adaa96fc60
### What changes were proposed in this pull request? Support Flink 1.19. ### Why are the changes needed? Flink 1.19.0 is announced to release: [Announcing the Release of Apache Flink 1.19] (https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19). The main changes includes: - `org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel` constructor change parameters: - `consumedSubpartitionIndex` changes to `consumedSubpartitionIndexSet`: [[FLINK-33743][runtime] Support consuming multiple subpartitions on a single channel](https://github.com/apache/flink/pull/23927). - adds `partitionRequestListenerTimeout`: [[FLINK-25055][network] Support listen and notify mechanism for partition request](https://github.com/apache/flink/pull/23565). - `org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate` constructor removes parameters `subpartitionIndexRange`, `tieredStorageConsumerClient`, `nettyService` and `tieredStorageConsumerSpecs`: [[FLINK-33743][runtime] Support consuming multiple subpartitions on a single channel](https://github.com/apache/flink/pull/23927). - Change the default config file to `config.yaml` in `flink-dist`: [[FLINK-33577][dist] Change the default config file to config.yaml in flink-dist](https://github.com/apache/flink/pull/24177). - `org.apache.flink.configuration.RestartStrategyOptions` uses `org.apache.commons.compress.utils.Sets` of `commons-compress` dependency: [[FLINK-33865][runtime] Adding an ITCase to ensure exponential-delay.attempts-before-reset-backoff works well](https://github.com/apache/flink/pull/23942). ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Local test: - Flink batch job submission ``` $ ./bin/flink run examples/streaming/WordCount.jar --execution-mode BATCH Executing example with default input data. Use --input to specify file input. Printing result to stdout. Use --output to specify output path. Job has been submitted with JobID 2e9fb659991a9c29d376151783bdf6de Program execution finished Job with JobID 2e9fb659991a9c29d376151783bdf6de has finished. Job Runtime: 1912 ms ``` - Flink batch job execution  - Celeborn master log ``` 24/03/18 20:52:47,513 INFO [celeborn-dispatcher-42] Master: Offer slots successfully for 1 reducers of 1710766312631-2e9fb659991a9c29d376151783bdf6de-0 on 1 workers. ``` - Celeborn worker log ``` 24/03/18 20:52:47,704 INFO [celeborn-dispatcher-1] StorageManager: created file at /Users/nicholas/Software/Celeborn/apache-celeborn-0.5.0-SNAPSHOT/shuffle/celeborn-worker/shuffle_data/1710766312631-2e9fb659991a9c29d376151783bdf6de/0/0-0-0 24/03/18 20:52:47,707 INFO [celeborn-dispatcher-1] Controller: Reserved 1 primary location and 0 replica location for 1710766312631-2e9fb659991a9c29d376151783bdf6de-0 24/03/18 20:52:47,874 INFO [celeborn-dispatcher-2] Controller: Start commitFiles for 1710766312631-2e9fb659991a9c29d376151783bdf6de-0 24/03/18 20:52:47,890 INFO [worker-rpc-async-replier] Controller: CommitFiles for 1710766312631-2e9fb659991a9c29d376151783bdf6de-0 success with 1 committed primary partitions, 0 empty primary partitions, 0 failed primary partitions, 0 committed replica partitions, 0 empty replica partitions, 0 failed replica partitions. ``` Closes #2399 from SteNicholas/CELEBORN-1310. Authored-by: SteNicholas <programgeek@163.com> Signed-off-by: Shuang <lvshuang.xjs@alibaba-inc.com>
35 lines
1.3 KiB
Bash
Executable File
35 lines
1.3 KiB
Bash
Executable File
#!/bin/bash
|
|
#
|
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
|
# contributor license agreements. See the NOTICE file distributed with
|
|
# this work for additional information regarding copyright ownership.
|
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
|
# (the "License"); you may not use this file except in compliance with
|
|
# the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
#
|
|
|
|
set -x
|
|
|
|
PROJECT_DIR="$(cd "`dirname "$0"`/.."; pwd)"
|
|
|
|
if [ "$1" == "--web" ]; then
|
|
${PROJECT_DIR}/build/mvn clean install -pl web
|
|
else
|
|
${PROJECT_DIR}/build/mvn spotless:apply -Pflink-1.14
|
|
${PROJECT_DIR}/build/mvn spotless:apply -Pflink-1.15
|
|
${PROJECT_DIR}/build/mvn spotless:apply -Pflink-1.17
|
|
${PROJECT_DIR}/build/mvn spotless:apply -Pflink-1.18
|
|
${PROJECT_DIR}/build/mvn spotless:apply -Pflink-1.19
|
|
${PROJECT_DIR}/build/mvn spotless:apply -Pspark-2.4
|
|
${PROJECT_DIR}/build/mvn spotless:apply -Pspark-3.3
|
|
${PROJECT_DIR}/build/mvn spotless:apply -Pmr
|
|
fi
|