Commit Graph

35 Commits

Author SHA1 Message Date
zwangsheng
50c70265b5
[CELEBORN-963] Add WORKDIR in celeborn Dockerfile
### What changes were proposed in this pull request?
Introduce env `WORKDIR` into celeborn `docker/Dockerfile`.

### Why are the changes needed?
We should add `WORKDIR` in Dockerfile, this will lead us to `/opt/celeborn` when we get into Celeborn Containers.

According to https://docs.docker.com/engine/reference/builder/{}
> The WORKDIR instruction sets the working directory for any RUN, CMD, ENTRYPOINT, COPY and ADD instructions that follow it in the Dockerfile. If the WORKDIR doesn't exist, it will be created even if it's not used in any subsequent Dockerfile instruction.

And also we can find same `WORKDIR` in spark project
3d119a5280/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile (L57)
### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Local test

```log
hadoopXXXX:~/yangbinjie/XXXXe$ docker run cd3d2a0ccab5e88c202ad56c98d4db6ca5d36b2f7d44b5aa2a9166f075d5f950 ls -l
total 269
drwxrwxr-x 2 celeborn celeborn   4 Sep 11 05:37 bin
drwxrwxr-x 2 celeborn celeborn   9 Sep 11 05:37 conf
drwxrwxr-x 2 celeborn celeborn  78 Sep 11 05:37 jars
drwxrwxr-x 2 celeborn celeborn  79 Sep 11 05:37 master-jars
-rw-rw-r-- 1 celeborn celeborn 138 Sep 11 03:33 RELEASE
drwxrwxr-x 2 celeborn celeborn  11 Sep 11 05:37 sbin
drwxrwxr-x 2 celeborn celeborn  66 Sep 11 05:37 worker-jars
```

Closes #1896 from zwangsheng/CELEBORN-963.

Authored-by: zwangsheng <binjieyang@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-09-11 15:10:26 +08:00
sychen
8b7989ad0c [CELEBORN-900][FOLLOWUP] Disable jemalloc in non-docker environment
### What changes were proposed in this pull request?
1. Provide `CELEBORN_PREFER_JEMALLOC` configuration to determine whether to enable jemalloc
2. Provide `CELEBORN_JEMALLOC_PATH` to configure the jemalloc path, for example, Centos is `/usr/lib64/libjemalloc.so`
3. Enable jemalloc by default in the docker environment

### Why are the changes needed?
Prevent unnecessary WARNING.

https://github.com/apache/incubator-celeborn/pull/1824#discussion_r1319909938

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
local test

Closes #1895 from cxzl25/CELEBORN-900_diable.

Lead-authored-by: sychen <sychen@ctrip.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: zky.zhoukeyong <zky.zhoukeyong@alibaba-inc.com>
2023-09-11 14:55:10 +08:00
Jun He
ada12a2c0e
[CELEBORN-900] Prefer to use jemalloc for memory allocation
### What changes were proposed in this pull request?

Only the Dockfile needs to change in this pr.

### Why are the changes needed?

When deploying celeborn for flink on kubernetes, Introducing jemalloc can improve pod memory usage.

### Does this PR introduce _any_ user-facing change?
None

### How was this patch tested?
Maybe starting a production job to test the memory usage improvement is needed.

Closes #1824 from mddxhj/feature/introduce_jemalloc.

Authored-by: Jun He <xuehaijuxian@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-09-08 19:49:24 +08:00
Binjie Yang
2bb8e7a2a2
[CELEBORN-218][HELM] Move helm chart to dedicated directory (#1160) 2023-01-11 16:11:27 +08:00
Binjie Yang
fb91544843
[CELEBORN-210][HELM] Add recommended labels in celeborn chart (#1156)
* [HELM] Add recommended labels in celeborn chart

* fix master and worker affinity
2023-01-10 19:18:14 +08:00
Cheng Pan
1822896156
[CELEBORN-151][K8S] Celeborn should run as celeborn instead of root in container (#895) 2022-12-20 15:57:14 +08:00
Cheng Pan
dc66369973
[CELEBORN-150][BUILD] Reduce binary tarball size by sharing jars (#1095)
* [CELEBORN-150][BUILD] Reduce binary tarball size by sharing jars

* nit

* nit

* docker

* nit

* cp -R
2022-12-16 14:30:17 +08:00
Binjie Yang
f236b0bb14
[CELEBORN-143][HELM] Exposing pods details to users (#1084)
* Exposing details to users

* fix
2022-12-15 10:12:01 +08:00
Binjie Yang
f8970a120a
[CELEBORN-138] Respect users' configurations which are set in values.yaml 2022-12-14 16:24:21 +08:00
Gabriel
5ecb09d62a
[ISSUE-911] Decrease numConnectionsPerPeer to achieve better performance (#983) 2022-11-20 11:46:17 +08:00
leesf
3699683a3b
Fix and migrate some configs (#927) 2022-11-07 09:41:38 +08:00
Binjie Yang
25a8d78634
remove dup affinity (#923) 2022-11-04 20:46:05 +08:00
Binjie Yang
9ef4751d22
[ISSUE-882][REFACTOR][K8S] Refactor dockerfile to build docker image from binary instead of ADD tgz (#884)
* init

* fix
2022-10-31 10:45:00 +08:00
Cheng Pan
873eeeb1ed
[BUILD] Add apache- prefix in release tarball name (#854) 2022-10-25 22:39:48 +08:00
AngersZhuuuu
0fdb19065a
[ISSUE-841][REFACTOR] Migrate shuffle client side conf to Celeborn Configuration System (#842) 2022-10-24 20:48:48 +08:00
Cheng Pan
8d7d397e71
Fix Configuration page and polish naming (#838)
* Fix Configuration page and polish naming

* nit

* nit

* comment
2022-10-24 12:46:25 +08:00
Cheng Pan
efad4abb5d
Migrate a bunch of configurations (#786) 2022-10-18 10:44:01 +08:00
Cheng Pan
ea67f4e060
Introduce categories to ConfigEntry and migrate configurations (#775) 2022-10-17 16:56:54 +08:00
Cheng Pan
5829bda21a
Rework and migrate HA configuration system (#763) 2022-10-13 22:35:01 +08:00
Binjie Yang
1ab0ab2dcc
[K8S] Should move common env to values.yml instead of set in each yaml #762 2022-10-12 15:50:20 +08:00
Cheng Pan
f01a696313
Migrate and refactor configuration for master endpoints (#752) 2022-10-11 21:33:21 +08:00
Binjie Yang
7f9e263113
Rename helm template and values 2022-10-08 20:14:37 +08:00
Cheng Pan
ab16b4f101
[INFRA] Rename modules w/ celeborn prefix (#723) 2022-10-08 08:05:57 +08:00
Keyong Zhou
a2d2379153
[DOC] Replace RSS with Celeborn in docs (#715) 2022-10-06 10:37:46 +08:00
Keyong Zhou
fe3b5988f2
[REFACTOR] Change package name to org.apache.celeborn (#710) 2022-10-02 18:10:29 +08:00
Binjie Yang
1d6939bdf4
split (#657) 2022-09-22 17:53:42 +08:00
Cheng Pan
89a2056a07
Fix binary tarball name in Dockerfile (#596) 2022-09-13 11:54:56 +08:00
Binjie Yang
9152906bbd
[REFACTOR] Switch docker base image to Eclipse Temurin (#509) 2022-09-07 21:22:58 +08:00
Binjie Yang
8da0dfa948
[BUG] Fix start up and defaults config map using rss-svc (#558) 2022-09-06 14:59:46 +08:00
Cheng Pan
4b42219595
Remove log4j1 (#501) 2022-09-05 19:30:15 +08:00
Cheng Pan
3dddb65f31
Enable Apache Rat and fix license header (#492) 2022-08-31 23:53:33 +08:00
Binjie Yang
44f01a39b6
add worker svc (#455) 2022-08-24 21:56:25 +08:00
Binjie Yang
b8533b570d
fix (#377) 2022-08-17 22:45:57 +08:00
Binjie Yang
2137e84ab7
add rss version (#316) 2022-08-10 21:03:46 +08:00
Fan Yilun
7a858bd70b
[DOC] Add doc and helm chart to describe how to deploy on k8s (#309) 2022-08-09 21:15:15 +08:00