apache/celeborn
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
41df4ebbea
celeborn/docs/developers
History
Fu Chen 41df4ebbea [CELEBORN-1156][BUILD] SBT publish support 
### What changes were proposed in this pull request?

As title

### Why are the changes needed?

As title

### Does this PR introduce _any_ user-facing change?

Yes, the user can publish shade clients via SBT

### How was this patch tested?

```shell
docker run -d -p 8081:8081 sonatype/nexus3
```

```shell
export SONATYPE_SNAPSHOTS_URL=http://192.168.3.46:8081/repository/maven-snapshots/
export SONATYPE_RELEASES_URL=http://192.168.3.46:8081/repository/maven-releases/
export ASF_USERNAME=admin
export ASF_PASSWORD=123456
```

- Publish the shade client for Spark 3.5:
```shell
./build/sbt -Pspark-3.4 celeborn-client-spark-3-shaded/publish
```

<img width="1673" alt="截屏2023-12-08 下午10 22 07" src="https://github.com/apache/incubator-celeborn/assets/8537877/1e87e7e2-cf3b-4bc0-8272-0f5b03ee65bf">

- Publish the shade client for Flink 1.18:

```shell
$ ./build/sbt -Pflink-1.18 celeborn-client-flink-1_18-shaded/publish
```
<img width="1676" alt="截屏2023-12-08 下午10 25 28" src="https://github.com/apache/incubator-celeborn/assets/8537877/62d0c3c4-e105-4e8a-8d8d-e78650a2eb09">

- Publish the shade client for MapReduce:
```shell
$ ./build/sbt -Pmr celeborn-client-mr-shaded/publish
```
<img width="1672" alt="截屏2023-12-08 下午10 25 47" src="https://github.com/apache/incubator-celeborn/assets/8537877/563d5ad5-fa6d-46fc-9465-8279ef96385a">

Closes #2129 from cfmcgrady/sbt-publish.

Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Fu Chen <cfmcgrady@gmail.com>
2023-12-15 11:22:35 +08:00
..
client.md [CELEBORN-853][DOC] Document on LifecycleManager 2023-07-31 17:36:42 +08:00
faulttolerant.md [CELEBORN-860][DOC] Document on ShuffleClient 2023-07-31 20:07:20 +08:00
flink.md [CELEBORN-869][DOC] Document on Integrating Celeborn 2023-08-02 17:22:41 +08:00
glutensupport.md [CELEBORN-1010] Update docs about spark.shuffle.service.enabled 2023-10-08 09:15:42 +08:00
integrate.md [CELEBORN-869][FOLLOWUP][DOC] Document on Integrating Celeborn 2023-08-02 18:17:17 +08:00
lifecyclemanager.md [MINOR] Fix some typos 2023-10-12 20:34:07 +08:00
master.md [CELEBORN-853][DOC] Document on LifecycleManager 2023-07-31 17:36:42 +08:00
overview.md [CELEBORN-997][DOC] Fix Rolling upgrade broken link 2023-09-20 16:44:42 +08:00
readdata.md [CELEBORN-823][DOC] Add Celeborn architecture document 2023-07-22 23:57:22 +08:00
sbt.md [CELEBORN-1156][BUILD] SBT publish support 2023-12-15 11:22:35 +08:00
shuffleclient.md [CELEBORN-860][DOC] Document on ShuffleClient 2023-07-31 20:07:20 +08:00
slotsallocation.md [CELEBORN-1081][FOLLOWUP] Remove UNKNOWN_DISK and allocate all slots to disk 2023-11-28 11:26:00 +08:00
spark.md [CELEBORN-869][DOC] Document on Integrating Celeborn 2023-08-02 17:22:41 +08:00
storage.md [CELEBORN-860][DOC] Document on ShuffleClient 2023-07-31 20:07:20 +08:00
trafficcontrol.md [MINOR] Fix incorrect default resume ratio in trafficcontrol doc 2023-09-21 11:18:48 +08:00
worker.md [CELEBORN-877][DOC] Document on SBT 2023-08-11 12:17:55 +08:00
workerexclusion.md [CELEBORN-869][DOC] Document on Integrating Celeborn 2023-08-02 17:22:41 +08:00