celeborn/docs/developers
SteNicholas adaa96fc60 [CELEBORN-1310][FLINK] Support Flink 1.19
### What changes were proposed in this pull request?

Support Flink 1.19.

### Why are the changes needed?

Flink 1.19.0 is announced to release: [Announcing the Release of Apache Flink 1.19] (https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19).

The main changes includes:

- `org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel` constructor change parameters:
   - `consumedSubpartitionIndex` changes to `consumedSubpartitionIndexSet`: [[FLINK-33743][runtime] Support consuming multiple subpartitions on a single channel](https://github.com/apache/flink/pull/23927).
   - adds `partitionRequestListenerTimeout`: [[FLINK-25055][network] Support listen and notify mechanism for partition request](https://github.com/apache/flink/pull/23565).
- `org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate` constructor removes parameters `subpartitionIndexRange`, `tieredStorageConsumerClient`, `nettyService` and `tieredStorageConsumerSpecs`: [[FLINK-33743][runtime] Support consuming multiple subpartitions on a single channel](https://github.com/apache/flink/pull/23927).
- Change the default config file to `config.yaml` in `flink-dist`: [[FLINK-33577][dist] Change the default config file to config.yaml in flink-dist](https://github.com/apache/flink/pull/24177).
- `org.apache.flink.configuration.RestartStrategyOptions` uses `org.apache.commons.compress.utils.Sets` of `commons-compress` dependency: [[FLINK-33865][runtime] Adding an ITCase to ensure exponential-delay.attempts-before-reset-backoff works well](https://github.com/apache/flink/pull/23942).

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Local test:

- Flink batch job submission

```
$ ./bin/flink run examples/streaming/WordCount.jar --execution-mode BATCH
Executing example with default input data.
Use --input to specify file input.
Printing result to stdout. Use --output to specify output path.
Job has been submitted with JobID 2e9fb659991a9c29d376151783bdf6de
Program execution finished
Job with JobID 2e9fb659991a9c29d376151783bdf6de has finished.
Job Runtime: 1912 ms
```

- Flink batch job execution

![image](https://github.com/apache/incubator-celeborn/assets/10048174/18b60861-cafc-4df3-b94d-93307e728be2)

- Celeborn master log
```

24/03/18 20:52:47,513 INFO [celeborn-dispatcher-42] Master: Offer slots successfully for 1 reducers of 1710766312631-2e9fb659991a9c29d376151783bdf6de-0 on 1 workers.
```

- Celeborn worker log
```
24/03/18 20:52:47,704 INFO [celeborn-dispatcher-1] StorageManager: created file at /Users/nicholas/Software/Celeborn/apache-celeborn-0.5.0-SNAPSHOT/shuffle/celeborn-worker/shuffle_data/1710766312631-2e9fb659991a9c29d376151783bdf6de/0/0-0-0
24/03/18 20:52:47,707 INFO [celeborn-dispatcher-1] Controller: Reserved 1 primary location and 0 replica location for 1710766312631-2e9fb659991a9c29d376151783bdf6de-0
24/03/18 20:52:47,874 INFO [celeborn-dispatcher-2] Controller: Start commitFiles for 1710766312631-2e9fb659991a9c29d376151783bdf6de-0
24/03/18 20:52:47,890 INFO [worker-rpc-async-replier] Controller: CommitFiles for 1710766312631-2e9fb659991a9c29d376151783bdf6de-0 success with 1 committed primary partitions, 0 empty primary partitions, 0 failed primary partitions, 0 committed replica partitions, 0 empty replica partitions, 0 failed replica partitions.
```

Closes #2399 from SteNicholas/CELEBORN-1310.

Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Shuang <lvshuang.xjs@alibaba-inc.com>
2024-03-20 11:51:23 +08:00
..
client.md [CELEBORN-853][DOC] Document on LifecycleManager 2023-07-31 17:36:42 +08:00
configuration.md [CELEBORN-1286] Introduce configuration.md to document dynamic config and config service 2024-02-28 11:49:28 +08:00
faulttolerant.md [CELEBORN-860][DOC] Document on ShuffleClient 2023-07-31 20:07:20 +08:00
flink.md [CELEBORN-869][DOC] Document on Integrating Celeborn 2023-08-02 17:22:41 +08:00
glutensupport.md [MINOR] Fix style and Gluten link in Developers Doc 2024-03-11 12:07:01 +08:00
integrate.md [CELEBORN-869][FOLLOWUP][DOC] Document on Integrating Celeborn 2023-08-02 18:17:17 +08:00
lifecyclemanager.md [MINOR] Fix some typos 2023-10-12 20:34:07 +08:00
master.md [CELEBORN-853][DOC] Document on LifecycleManager 2023-07-31 17:36:42 +08:00
overview.md [MINOR] Fix typo in developer docs - overview 2024-03-17 16:15:16 +08:00
readdata.md
sbt.md [CELEBORN-1310][FLINK] Support Flink 1.19 2024-03-20 11:51:23 +08:00
shuffleclient.md [CELEBORN-860][DOC] Document on ShuffleClient 2023-07-31 20:07:20 +08:00
slotsallocation.md [MINOR] Fix style and Gluten link in Developers Doc 2024-03-11 12:07:01 +08:00
spark.md [CELEBORN-869][DOC] Document on Integrating Celeborn 2023-08-02 17:22:41 +08:00
storage.md [CELEBORN-860][DOC] Document on ShuffleClient 2023-07-31 20:07:20 +08:00
trafficcontrol.md [MINOR] Fix incorrect default resume ratio in trafficcontrol doc 2023-09-21 11:18:48 +08:00
worker.md [CELEBORN-877][DOC] Document on SBT 2023-08-11 12:17:55 +08:00
workerexclusion.md [CELEBORN-869][DOC] Document on Integrating Celeborn 2023-08-02 17:22:41 +08:00