Commit Graph

13 Commits

Author SHA1 Message Date
zwangsheng
50c70265b5
[CELEBORN-963] Add WORKDIR in celeborn Dockerfile
### What changes were proposed in this pull request?
Introduce env `WORKDIR` into celeborn `docker/Dockerfile`.

### Why are the changes needed?
We should add `WORKDIR` in Dockerfile, this will lead us to `/opt/celeborn` when we get into Celeborn Containers.

According to https://docs.docker.com/engine/reference/builder/{}
> The WORKDIR instruction sets the working directory for any RUN, CMD, ENTRYPOINT, COPY and ADD instructions that follow it in the Dockerfile. If the WORKDIR doesn't exist, it will be created even if it's not used in any subsequent Dockerfile instruction.

And also we can find same `WORKDIR` in spark project
3d119a5280/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile (L57)
### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Local test

```log
hadoopXXXX:~/yangbinjie/XXXXe$ docker run cd3d2a0ccab5e88c202ad56c98d4db6ca5d36b2f7d44b5aa2a9166f075d5f950 ls -l
total 269
drwxrwxr-x 2 celeborn celeborn   4 Sep 11 05:37 bin
drwxrwxr-x 2 celeborn celeborn   9 Sep 11 05:37 conf
drwxrwxr-x 2 celeborn celeborn  78 Sep 11 05:37 jars
drwxrwxr-x 2 celeborn celeborn  79 Sep 11 05:37 master-jars
-rw-rw-r-- 1 celeborn celeborn 138 Sep 11 03:33 RELEASE
drwxrwxr-x 2 celeborn celeborn  11 Sep 11 05:37 sbin
drwxrwxr-x 2 celeborn celeborn  66 Sep 11 05:37 worker-jars
```

Closes #1896 from zwangsheng/CELEBORN-963.

Authored-by: zwangsheng <binjieyang@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-09-11 15:10:26 +08:00
sychen
8b7989ad0c [CELEBORN-900][FOLLOWUP] Disable jemalloc in non-docker environment
### What changes were proposed in this pull request?
1. Provide `CELEBORN_PREFER_JEMALLOC` configuration to determine whether to enable jemalloc
2. Provide `CELEBORN_JEMALLOC_PATH` to configure the jemalloc path, for example, Centos is `/usr/lib64/libjemalloc.so`
3. Enable jemalloc by default in the docker environment

### Why are the changes needed?
Prevent unnecessary WARNING.

https://github.com/apache/incubator-celeborn/pull/1824#discussion_r1319909938

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
local test

Closes #1895 from cxzl25/CELEBORN-900_diable.

Lead-authored-by: sychen <sychen@ctrip.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: zky.zhoukeyong <zky.zhoukeyong@alibaba-inc.com>
2023-09-11 14:55:10 +08:00
Jun He
ada12a2c0e
[CELEBORN-900] Prefer to use jemalloc for memory allocation
### What changes were proposed in this pull request?

Only the Dockfile needs to change in this pr.

### Why are the changes needed?

When deploying celeborn for flink on kubernetes, Introducing jemalloc can improve pod memory usage.

### Does this PR introduce _any_ user-facing change?
None

### How was this patch tested?
Maybe starting a production job to test the memory usage improvement is needed.

Closes #1824 from mddxhj/feature/introduce_jemalloc.

Authored-by: Jun He <xuehaijuxian@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-09-08 19:49:24 +08:00
Cheng Pan
1822896156
[CELEBORN-151][K8S] Celeborn should run as celeborn instead of root in container (#895) 2022-12-20 15:57:14 +08:00
Cheng Pan
dc66369973
[CELEBORN-150][BUILD] Reduce binary tarball size by sharing jars (#1095)
* [CELEBORN-150][BUILD] Reduce binary tarball size by sharing jars

* nit

* nit

* docker

* nit

* cp -R
2022-12-16 14:30:17 +08:00
Binjie Yang
9ef4751d22
[ISSUE-882][REFACTOR][K8S] Refactor dockerfile to build docker image from binary instead of ADD tgz (#884)
* init

* fix
2022-10-31 10:45:00 +08:00
Cheng Pan
873eeeb1ed
[BUILD] Add apache- prefix in release tarball name (#854) 2022-10-25 22:39:48 +08:00
Cheng Pan
ab16b4f101
[INFRA] Rename modules w/ celeborn prefix (#723) 2022-10-08 08:05:57 +08:00
Cheng Pan
89a2056a07
Fix binary tarball name in Dockerfile (#596) 2022-09-13 11:54:56 +08:00
Binjie Yang
9152906bbd
[REFACTOR] Switch docker base image to Eclipse Temurin (#509) 2022-09-07 21:22:58 +08:00
Cheng Pan
3dddb65f31
Enable Apache Rat and fix license header (#492) 2022-08-31 23:53:33 +08:00
Binjie Yang
2137e84ab7
add rss version (#316) 2022-08-10 21:03:46 +08:00
Fan Yilun
7a858bd70b
[DOC] Add doc and helm chart to describe how to deploy on k8s (#309) 2022-08-09 21:15:15 +08:00