<!--
Thanks for sending a pull request! Here are some tips for you:
- Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX] Your PR title ...'.
- Be sure to keep the PR description updated to reflect all changes.
- Please write your PR title to summarize what this PR proposes.
- If possible, provide a concise example to reproduce the issue for a faster review.
-->
CELEBORN-1901
### What changes were proposed in this pull request?
Update base docker image to ubuntu 24.04 (noble)
### Why are the changes needed?
Current used ubuntu 20.04 is close to end-of-life
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
Tested on our celeborn installation
Closes#3143 from jesusch/CELEBORN-1901-update-docker-base-image.
Authored-by: Björn Boschman <bjoern.boschman@innovid.com>
Signed-off-by: mingji <fengmingxiao.fmx@alibaba-inc.com>
### What changes were proposed in this pull request?
Dockerfile should support copying CLI jars.
### Why are the changes needed?
CLI jars are generated from `make-distribution.sh`. Therefore, Dockerfile could copy CLI jars to `/opt/celeborn/` directory.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
No.
Closes#2823 from SteNicholas/CELEBORN-1659.
Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: mingji <fengmingxiao.fmx@alibaba-inc.com>
### What changes were proposed in this pull request?
Delete `DEPLOY_ON_K8S.md`.
### Why are the changes needed?
`DEPLOY_ON_K8S.md` is replaced with `deploy_on_k8s.md`, which could be deleted.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
No.
Closes#2752 from SteNicholas/delete-deploy-on-k8s.
Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: mingji <fengmingxiao.fmx@alibaba-inc.com>
### What changes were proposed in this pull request?
Improve Celeborn document to fix typos, formats, unvalid link and unsynced default value of document. Meanwhile, the public interfaces of `shuffleclient.md` keep the consistent with `ShuffleClient`.
### Why are the changes needed?
There are some typos, formats, unvalid link and unsynced default value fixes in Celeborn document at present. Meanwhile, the public interfaces of `shuffleclient.md` is inconsistent with `ShuffleClient`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
No.
Closes#2410 from SteNicholas/CELEBORN-1341.
Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: mingji <fengmingxiao.fmx@alibaba-inc.com>
### What changes were proposed in this pull request?
Introduce env `WORKDIR` into celeborn `docker/Dockerfile`.
### Why are the changes needed?
We should add `WORKDIR` in Dockerfile, this will lead us to `/opt/celeborn` when we get into Celeborn Containers.
According to https://docs.docker.com/engine/reference/builder/{}
> The WORKDIR instruction sets the working directory for any RUN, CMD, ENTRYPOINT, COPY and ADD instructions that follow it in the Dockerfile. If the WORKDIR doesn't exist, it will be created even if it's not used in any subsequent Dockerfile instruction.
And also we can find same `WORKDIR` in spark project
3d119a5280/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile (L57)
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Local test
```log
hadoopXXXX:~/yangbinjie/XXXXe$ docker run cd3d2a0ccab5e88c202ad56c98d4db6ca5d36b2f7d44b5aa2a9166f075d5f950 ls -l
total 269
drwxrwxr-x 2 celeborn celeborn 4 Sep 11 05:37 bin
drwxrwxr-x 2 celeborn celeborn 9 Sep 11 05:37 conf
drwxrwxr-x 2 celeborn celeborn 78 Sep 11 05:37 jars
drwxrwxr-x 2 celeborn celeborn 79 Sep 11 05:37 master-jars
-rw-rw-r-- 1 celeborn celeborn 138 Sep 11 03:33 RELEASE
drwxrwxr-x 2 celeborn celeborn 11 Sep 11 05:37 sbin
drwxrwxr-x 2 celeborn celeborn 66 Sep 11 05:37 worker-jars
```
Closes#1896 from zwangsheng/CELEBORN-963.
Authored-by: zwangsheng <binjieyang@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### What changes were proposed in this pull request?
1. Provide `CELEBORN_PREFER_JEMALLOC` configuration to determine whether to enable jemalloc
2. Provide `CELEBORN_JEMALLOC_PATH` to configure the jemalloc path, for example, Centos is `/usr/lib64/libjemalloc.so`
3. Enable jemalloc by default in the docker environment
### Why are the changes needed?
Prevent unnecessary WARNING.
https://github.com/apache/incubator-celeborn/pull/1824#discussion_r1319909938
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
local test
Closes#1895 from cxzl25/CELEBORN-900_diable.
Lead-authored-by: sychen <sychen@ctrip.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: zky.zhoukeyong <zky.zhoukeyong@alibaba-inc.com>
### What changes were proposed in this pull request?
Only the Dockfile needs to change in this pr.
### Why are the changes needed?
When deploying celeborn for flink on kubernetes, Introducing jemalloc can improve pod memory usage.
### Does this PR introduce _any_ user-facing change?
None
### How was this patch tested?
Maybe starting a production job to test the memory usage improvement is needed.
Closes#1824 from mddxhj/feature/introduce_jemalloc.
Authored-by: Jun He <xuehaijuxian@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
