Commit Graph

10 Commits

Author SHA1 Message Date
Wang, Fei
d4044c5152
[CELEBORN-1682] Add java tools.jar into classpath for JVM quake
### What changes were proposed in this pull request?

Add java tools.jar into classpath for JVM quake.

### Why are the changes needed?
Meet below issue with `celeborn.worker.jvmQuake.enabled=true`, see https://github.com/apache/celeborn/pull/2061
```
24/11/03 15:51:08,453 ERROR [main] Worker: Initialize worker failed.
java.lang.NoClassDefFoundError: sun/jvmstat/monitor/HostIdentifier
    at org.apache.celeborn.service.deploy.worker.monitor.JVMQuake$.monitoredVm$lzycompute(JVMQuake.scala:180)
    at org.apache.celeborn.service.deploy.worker.monitor.JVMQuake$.monitoredVm(JVMQuake.scala:179)
    at org.apache.celeborn.service.deploy.worker.monitor.JVMQuake$.ygcExitTimeMonitor$lzycompute(JVMQuake.scala:185)
    at org.apache.celeborn.service.deploy.worker.monitor.JVMQuake$.ygcExitTimeMonitor(JVMQuake.scala:184)
    at org.apache.celeborn.service.deploy.worker.monitor.JVMQuake$.org$apache$celeborn$service$deploy$worker$monitor$JVMQuake$$getLastExitTime(JVMQuake.scala:192)
    at org.apache.celeborn.service.deploy.worker.monitor.JVMQuake.start(JVMQuake.scala:66)
    at org.apache.celeborn.service.deploy.worker.Worker.<init>(Worker.scala:360)
    at org.apache.celeborn.service.deploy.worker.Worker$.main(Worker.scala:1041)
    at org.apache.celeborn.service.deploy.worker.Worker.main(Worker.scala)
Caused by: java.lang.ClassNotFoundException: sun.jvmstat.monitor.HostIdentifier
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 9 more
```

Related code:
c12e8881ab/project/JDKTools.scala (L58-L75)

Similar issue: https://github.com/vladimirvivien/jmx-cli/issues/4

After copy the `tools.jar` into worker-jars, the issue got resolved.

It is better that to involve the `tools.jar` automatically without copy.

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
<img width="1202" alt="image" src="https://github.com/user-attachments/assets/af8f6c0d-9123-4a73-93b5-69836c5f826d">

Closes #2874 from turboFei/jdk_tools.

Authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: SteNicholas <programgeek@163.com>
2024-11-04 11:05:10 +08:00
Aravind Patnam
cc26131f88 [CELEBORN-1572] Celeborn CLI initial REST API support
### What changes were proposed in this pull request?
Introducing the Celeborn CLI (based on this [CPIP](https://cwiki.apache.org/confluence/display/CELEBORN/CIP-7+Celeborn+CLI)). For the first iteration, adding support for querying the existing REST api endpoints.
After this will add a layer for external cluster manager support. Further improvements are needed such as pretty print, which can be added in subsequent PRs.

### Why are the changes needed?
see [CPIP](https://cwiki.apache.org/confluence/display/CELEBORN/CIP-7+Celeborn+CLI)

### Does this PR introduce _any_ user-facing change?
yes, new CLI tool.

### How was this patch tested?
added UTs and also tested internally.

Closes #2699 from akpatnam25/cli-CELEBORN-1572.

Lead-authored-by: Aravind Patnam <apatnam@linkedin.com>
Co-authored-by: Aravind Patnam <akpatnam25@gmail.com>
Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com>
2024-09-05 11:15:16 -05:00
Cheng Pan
de0fd8cc44 [CELEBORN-762] Always set JVM opts -XX:+IgnoreUnrecognizedVMOptions
### What changes were proposed in this pull request?

Always set JVM opts `-XX:+IgnoreUnrecognizedVMOptions`

### Why are the changes needed?

By default, JVM failed to start when unknown opts are set, it's not friendly for users who want to use different versions of JDK.

### Does this PR introduce _any_ user-facing change?

Yes, users can success start celeborn even if they provide unknown JVM opts.

### How was this patch tested?

Review.

Closes #1676 from pan3793/CELEBORN-762.

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-07-04 21:37:19 +08:00
Angerszhuuuu
e805c74aad
[CELEBORN-360] Export necessary env in load-celeborn-env.sh (#1308) 2023-03-03 21:09:11 +08:00
Angerszhuuuu
734f14991a
[CELEBORN-360] Add celeborn ratis shell command line (#1294)
* [CELEBORN-360] Add celeborn ratis shell command line
2023-03-02 16:30:45 +08:00
Binjie Yang
a31dcc8194
[IMPROVEMENT] Improve celeborn script logic (#1020) 2022-11-30 20:03:56 +08:00
Ethan Feng
59474c2f11
[INFRA]Update scripts and templates for new name. (#724) 2022-10-09 14:56:06 +08:00
Keyong Zhou
fe3b5988f2
[REFACTOR] Change package name to org.apache.celeborn (#710) 2022-10-02 18:10:29 +08:00
Cheng Pan
3dddb65f31
Enable Apache Rat and fix license header (#492) 2022-08-31 23:53:33 +08:00
zky.zhoukeyong
ba5920acde Initial Commit for RSS 2021-12-28 20:57:35 +08:00