celeborn/.github/workflows
SteNicholas adaa96fc60 [CELEBORN-1310][FLINK] Support Flink 1.19
### What changes were proposed in this pull request?

Support Flink 1.19.

### Why are the changes needed?

Flink 1.19.0 is announced to release: [Announcing the Release of Apache Flink 1.19] (https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19).

The main changes includes:

- `org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel` constructor change parameters:
   - `consumedSubpartitionIndex` changes to `consumedSubpartitionIndexSet`: [[FLINK-33743][runtime] Support consuming multiple subpartitions on a single channel](https://github.com/apache/flink/pull/23927).
   - adds `partitionRequestListenerTimeout`: [[FLINK-25055][network] Support listen and notify mechanism for partition request](https://github.com/apache/flink/pull/23565).
- `org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate` constructor removes parameters `subpartitionIndexRange`, `tieredStorageConsumerClient`, `nettyService` and `tieredStorageConsumerSpecs`: [[FLINK-33743][runtime] Support consuming multiple subpartitions on a single channel](https://github.com/apache/flink/pull/23927).
- Change the default config file to `config.yaml` in `flink-dist`: [[FLINK-33577][dist] Change the default config file to config.yaml in flink-dist](https://github.com/apache/flink/pull/24177).
- `org.apache.flink.configuration.RestartStrategyOptions` uses `org.apache.commons.compress.utils.Sets` of `commons-compress` dependency: [[FLINK-33865][runtime] Adding an ITCase to ensure exponential-delay.attempts-before-reset-backoff works well](https://github.com/apache/flink/pull/23942).

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Local test:

- Flink batch job submission

```
$ ./bin/flink run examples/streaming/WordCount.jar --execution-mode BATCH
Executing example with default input data.
Use --input to specify file input.
Printing result to stdout. Use --output to specify output path.
Job has been submitted with JobID 2e9fb659991a9c29d376151783bdf6de
Program execution finished
Job with JobID 2e9fb659991a9c29d376151783bdf6de has finished.
Job Runtime: 1912 ms
```

- Flink batch job execution

![image](https://github.com/apache/incubator-celeborn/assets/10048174/18b60861-cafc-4df3-b94d-93307e728be2)

- Celeborn master log
```

24/03/18 20:52:47,513 INFO [celeborn-dispatcher-42] Master: Offer slots successfully for 1 reducers of 1710766312631-2e9fb659991a9c29d376151783bdf6de-0 on 1 workers.
```

- Celeborn worker log
```
24/03/18 20:52:47,704 INFO [celeborn-dispatcher-1] StorageManager: created file at /Users/nicholas/Software/Celeborn/apache-celeborn-0.5.0-SNAPSHOT/shuffle/celeborn-worker/shuffle_data/1710766312631-2e9fb659991a9c29d376151783bdf6de/0/0-0-0
24/03/18 20:52:47,707 INFO [celeborn-dispatcher-1] Controller: Reserved 1 primary location and 0 replica location for 1710766312631-2e9fb659991a9c29d376151783bdf6de-0
24/03/18 20:52:47,874 INFO [celeborn-dispatcher-2] Controller: Start commitFiles for 1710766312631-2e9fb659991a9c29d376151783bdf6de-0
24/03/18 20:52:47,890 INFO [worker-rpc-async-replier] Controller: CommitFiles for 1710766312631-2e9fb659991a9c29d376151783bdf6de-0 success with 1 committed primary partitions, 0 empty primary partitions, 0 failed primary partitions, 0 committed replica partitions, 0 empty replica partitions, 0 failed replica partitions.
```

Closes #2399 from SteNicholas/CELEBORN-1310.

Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Shuang <lvshuang.xjs@alibaba-inc.com>
2024-03-20 11:51:23 +08:00
..
benchmark_manual.yml [CELEBORN-180][BUILD][FOLLOWUP] Update CI workflow and docs (#1134) 2023-01-03 17:58:51 +08:00
benchmark.yml [CELEBORN-180][BUILD][FOLLOWUP] Update CI workflow and docs (#1134) 2023-01-03 17:58:51 +08:00
check_stale_prs.yml [CELEBORN-135][INFRA] Remove unused GitHub slash workflow (#1080) 2022-12-14 10:13:13 +08:00
deps.yml [CELEBORN-1310][FLINK] Support Flink 1.19 2024-03-20 11:51:23 +08:00
integration.yml [CELEBORN-105][TEST] Kubernetes Integration Test 2023-06-05 12:11:29 +08:00
license.yml [CELEBORN-1108][FOLLOWUP] Use rat plugin check Flink 1.18 2023-11-08 12:54:23 +08:00
maven.yml [CELEBORN-1310][FLINK] Support Flink 1.19 2024-03-20 11:51:23 +08:00
regression_manual.yml [CELEBORN-180][BUILD][FOLLOWUP] Update CI workflow and docs (#1134) 2023-01-03 17:58:51 +08:00
regression.yml [CELEBORN-180][BUILD][FOLLOWUP] Update CI workflow and docs (#1134) 2023-01-03 17:58:51 +08:00
sbt.yml [CELEBORN-1310][FLINK] Support Flink 1.19 2024-03-20 11:51:23 +08:00
style.yml [CELEBORN-1310][FLINK] Support Flink 1.19 2024-03-20 11:51:23 +08:00
web_lint.yml [CELEBORN-1240][FOLLOWUP] Web lint check uses different groups 2024-02-26 20:28:19 +08:00