cfb490c938
### What changes were proposed in this pull request? This PR adds support for lz4 decompression in CppClient. ### Why are the changes needed? To support reading from Celeborn with CppClient. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? By compilation and UTs. Closes #3402 from Jraaay/feat/cpp_client_lz4_decompression. Authored-by: Jray <1075860716@qq.com> Signed-off-by: SteNicholas <programgeek@163.com>
59 lines
1.5 KiB
Markdown
59 lines
1.5 KiB
Markdown
# [WIP] Celeborn Cpp Support
|
|
|
|
## Environment Setup
|
|
We provide several methods to setup dev environment for CelebornCpp.
|
|
Note that currently the scripts only take care of the cpp-related dependencies,
|
|
and java dependencies are not included.
|
|
|
|
### Clone the project
|
|
We should clone the project as:
|
|
```
|
|
# the option `--config core.symlinks=true` is to make sure that the softlink works
|
|
git clone git@github.com:apache/celeborn.git --config core.symlinks=true
|
|
```
|
|
|
|
### Use container with prebuilt image
|
|
We provide a pre-built image ready to be pulled and used so you could launch a container directly:
|
|
```
|
|
export PROJECT_DIR=/your/path/to/celeborn/dir
|
|
docker run \
|
|
-v ${PROJECT_DIR}:/celeborn \
|
|
-w /celeborn \
|
|
-it --rm \
|
|
--name celeborn-cpp-dev-container \
|
|
jraaaay/celeborn-cpp-dev:0.4 \
|
|
/bin/bash
|
|
```
|
|
|
|
### Build image and use container
|
|
We provide the dev image building scripts so you could build the image and launch a container as follows:
|
|
```
|
|
cd scripts
|
|
|
|
# build image
|
|
bash ./build-docker-image.sh
|
|
|
|
# launch container with image above
|
|
bash ./launch-docker-container.sh
|
|
```
|
|
|
|
### Build on local machine (Ubuntu-Only)
|
|
Currently, we only provide the dev-environment setup script for Ubuntu:
|
|
```
|
|
cd scripts
|
|
bash setup-ubuntu.sh
|
|
```
|
|
Other platforms are not supported yet, and you could use the container above as your dev environment.
|
|
|
|
## Compile and test
|
|
Currently, the modules are under development.
|
|
You could compile the code and run the tests within the dev container by
|
|
```
|
|
cd celeborn/cpp
|
|
mkdir -p build && cd build
|
|
cmake ..
|
|
make
|
|
ctest
|
|
```
|
|
|