Commit Graph

7 Commits

Author SHA1 Message Date
sychen
efa22a4936 [CELEBORN-1105][FLINK] Support Flink 1.18
### What changes were proposed in this pull request?

### Why are the changes needed?

```bash
flink-1.18.0
./bin/start-cluster.sh
./bin/flink run examples/streaming/WordCount.jar --execution-mode BATCH
```

```java
Caused by: java.lang.NoSuchMethodError: org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.<init>(Ljava/lang/String;ILorg/apache/flink/runtime/jobgraph/IntermediateDataSetID;Lorg/apache/flink/runtime/io/network/partition/ResultPartitionType;Lorg/apache/flink/runtime/executiongraph/IndexRange;ILorg/apache/flink/runtime/io/network/partition/PartitionProducerStateProvider;Lorg/apache/flink/util/function/SupplierWithException;Lorg/apache/flink/runtime/io/network/buffer/BufferDecompressor;Lorg/apache/flink/core/memory/MemorySegmentProvider;ILorg/apache/flink/runtime/throughput/ThroughputCalculator;Lorg/apache/flink/runtime/throughput/BufferDebloater;)V
	at org.apache.celeborn.plugin.flink.RemoteShuffleInputGate$FakedRemoteInputChannel.<init>(RemoteShuffleInputGate.java:225)
	at org.apache.celeborn.plugin.flink.RemoteShuffleInputGate.getChannel(RemoteShuffleInputGate.java:179)
	at org.apache.flink.runtime.io.network.partition.consumer.InputGate.setChannelStateWriter(InputGate.java:90)
	at org.apache.flink.runtime.taskmanager.InputGateWithMetrics.setChannelStateWriter(InputGateWithMetrics.java:120)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.injectChannelStateWriterIntoChannels(StreamTask.java:524)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.<init>(StreamTask.java:496)
```

Flink 1.18.0 release
https://flink.apache.org/2023/10/24/announcing-the-release-of-apache-flink-1.18/

Interface `org.apache.flink.runtime.io.network.buffer.Buffer` adds `setRecycler` method.
[[FLINK-32549](https://issues.apache.org/jira/browse/FLINK-32549)][network] Tiered storage memory manager supports ownership transfer for buffers

`org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate` constructor adds parameters.
[[FLINK-31638](https://issues.apache.org/jira/browse/FLINK-31638)][network] Introduce the TieredStorageConsumerClient to SingleInputGate
[[FLINK-31642](https://issues.apache.org/jira/browse/FLINK-31642)][network] Introduce the MemoryTierConsumerAgent to TieredStorageConsumerClient

### Does this PR introduce _any_ user-facing change?

### How was this patch tested?
```bash
flink-1.18.0 ./bin/flink run examples/streaming/WordCount.jar --execution-mode BATCH
Executing example with default input data.
Use --input to specify file input.
Printing result to stdout. Use --output to specify output path.
Job has been submitted with JobID d7fc5f0ca018a54e9453c4d35f7c598a
Program execution finished
Job with JobID d7fc5f0ca018a54e9453c4d35f7c598a has finished.
Job Runtime: 1635 ms
```

<img width="1297" alt="image" src="https://github.com/apache/incubator-celeborn/assets/3898450/6a5266bf-2386-4386-b98b-a60d2570fa99">

Closes #2063 from cxzl25/CELEBORN-1105.

Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Shuang <lvshuang.tb@gmail.com>
2023-11-06 15:53:39 +08:00
Cheng Pan
7da6942367
[CELEBORN-942] Release script supports uploading Nexus
### What changes were proposed in this pull request?

`build/release/release.sh` supports uploading client jars to Nexus.

### Why are the changes needed?

We should deploy the client jars to maven central.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Manually verification.

Closes #1875 from pan3793/CELEBORN-942.

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-08-31 23:39:30 +08:00
Cheng Pan
d8b6c7e24d
[CELEBORN-180][FOLLOWUP] Declare PROJECT_DIR before using 2023-01-03 16:44:07 +08:00
Cheng Pan
c9dcf312f8
[CELEBORN-186][BUILD] Auto upload/move release artifacts to svn (#1130) 2023-01-03 15:04:40 +08:00
Cheng Pan
a16ba0e807
[CELEBORN-180][BUILD] Script for creating binary release artifact (#1129) 2023-01-03 12:58:42 +08:00
Cheng Pan
932a6c2703
[CELEBORN-177] Script for creating source release artifact (#1122) 2022-12-29 18:04:54 +08:00
Ethan Feng
489ac422c8
[CELEBORN-166] Add LICENSE and NOTICE for binary release artifacts (#1114)
* [CELEBORN-166] Automate release build process.

* Update .gitignore

Co-authored-by: Cheng Pan <chengpan@apache.org>

* sort dependencies.

* remove dependency list.

* 1.add git ignore.
2.add notice binary.

* update.

* update.

* style.

* style.

* update.

* Revert "style."

This reverts commit b90eb94196a9080d04e732c6cfa5ddf8c570ce01.

* Revert "style."

This reverts commit ef7e70d33f4251f69631c9c4a3c571a714dd902c.

* partial revert 63ae9f6

* Revert "update."

This reverts commit 9fcb3580a229ac53b3b0b2a8e52a6a7f0ef7daad.

* move to release folder

Co-authored-by: Cheng Pan <chengpan@apache.org>
2022-12-29 11:07:30 +08:00