efa22a4936
### What changes were proposed in this pull request? ### Why are the changes needed? ```bash flink-1.18.0 ./bin/start-cluster.sh ./bin/flink run examples/streaming/WordCount.jar --execution-mode BATCH ``` ```java Caused by: java.lang.NoSuchMethodError: org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.<init>(Ljava/lang/String;ILorg/apache/flink/runtime/jobgraph/IntermediateDataSetID;Lorg/apache/flink/runtime/io/network/partition/ResultPartitionType;Lorg/apache/flink/runtime/executiongraph/IndexRange;ILorg/apache/flink/runtime/io/network/partition/PartitionProducerStateProvider;Lorg/apache/flink/util/function/SupplierWithException;Lorg/apache/flink/runtime/io/network/buffer/BufferDecompressor;Lorg/apache/flink/core/memory/MemorySegmentProvider;ILorg/apache/flink/runtime/throughput/ThroughputCalculator;Lorg/apache/flink/runtime/throughput/BufferDebloater;)V at org.apache.celeborn.plugin.flink.RemoteShuffleInputGate$FakedRemoteInputChannel.<init>(RemoteShuffleInputGate.java:225) at org.apache.celeborn.plugin.flink.RemoteShuffleInputGate.getChannel(RemoteShuffleInputGate.java:179) at org.apache.flink.runtime.io.network.partition.consumer.InputGate.setChannelStateWriter(InputGate.java:90) at org.apache.flink.runtime.taskmanager.InputGateWithMetrics.setChannelStateWriter(InputGateWithMetrics.java:120) at org.apache.flink.streaming.runtime.tasks.StreamTask.injectChannelStateWriterIntoChannels(StreamTask.java:524) at org.apache.flink.streaming.runtime.tasks.StreamTask.<init>(StreamTask.java:496) ``` Flink 1.18.0 release https://flink.apache.org/2023/10/24/announcing-the-release-of-apache-flink-1.18/ Interface `org.apache.flink.runtime.io.network.buffer.Buffer` adds `setRecycler` method. [[FLINK-32549](https://issues.apache.org/jira/browse/FLINK-32549)][network] Tiered storage memory manager supports ownership transfer for buffers `org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate` constructor adds parameters. [[FLINK-31638](https://issues.apache.org/jira/browse/FLINK-31638)][network] Introduce the TieredStorageConsumerClient to SingleInputGate [[FLINK-31642](https://issues.apache.org/jira/browse/FLINK-31642)][network] Introduce the MemoryTierConsumerAgent to TieredStorageConsumerClient ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? ```bash flink-1.18.0 ./bin/flink run examples/streaming/WordCount.jar --execution-mode BATCH Executing example with default input data. Use --input to specify file input. Printing result to stdout. Use --output to specify output path. Job has been submitted with JobID d7fc5f0ca018a54e9453c4d35f7c598a Program execution finished Job with JobID d7fc5f0ca018a54e9453c4d35f7c598a has finished. Job Runtime: 1635 ms ``` <img width="1297" alt="image" src="https://github.com/apache/incubator-celeborn/assets/3898450/6a5266bf-2386-4386-b98b-a60d2570fa99"> Closes #2063 from cxzl25/CELEBORN-1105. Authored-by: sychen <sychen@ctrip.com> Signed-off-by: Shuang <lvshuang.tb@gmail.com>
29 lines
1.2 KiB
Bash
Executable File
29 lines
1.2 KiB
Bash
Executable File
#!/bin/bash
|
|
#
|
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
|
# contributor license agreements. See the NOTICE file distributed with
|
|
# this work for additional information regarding copyright ownership.
|
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
|
# (the "License"); you may not use this file except in compliance with
|
|
# the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
#
|
|
|
|
set -x
|
|
|
|
PROJECT_DIR="$(cd "`dirname "$0"`/.."; pwd)"
|
|
${PROJECT_DIR}/build/mvn spotless:apply -Pflink-1.14
|
|
${PROJECT_DIR}/build/mvn spotless:apply -Pflink-1.15
|
|
${PROJECT_DIR}/build/mvn spotless:apply -Pflink-1.17
|
|
${PROJECT_DIR}/build/mvn spotless:apply -Pflink-1.18
|
|
${PROJECT_DIR}/build/mvn spotless:apply -Pspark-2.4
|
|
${PROJECT_DIR}/build/mvn spotless:apply -Pspark-3.3
|
|
${PROJECT_DIR}/build/mvn spotless:apply -Pmr
|