efa22a4936
### What changes were proposed in this pull request? ### Why are the changes needed? ```bash flink-1.18.0 ./bin/start-cluster.sh ./bin/flink run examples/streaming/WordCount.jar --execution-mode BATCH ``` ```java Caused by: java.lang.NoSuchMethodError: org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.<init>(Ljava/lang/String;ILorg/apache/flink/runtime/jobgraph/IntermediateDataSetID;Lorg/apache/flink/runtime/io/network/partition/ResultPartitionType;Lorg/apache/flink/runtime/executiongraph/IndexRange;ILorg/apache/flink/runtime/io/network/partition/PartitionProducerStateProvider;Lorg/apache/flink/util/function/SupplierWithException;Lorg/apache/flink/runtime/io/network/buffer/BufferDecompressor;Lorg/apache/flink/core/memory/MemorySegmentProvider;ILorg/apache/flink/runtime/throughput/ThroughputCalculator;Lorg/apache/flink/runtime/throughput/BufferDebloater;)V at org.apache.celeborn.plugin.flink.RemoteShuffleInputGate$FakedRemoteInputChannel.<init>(RemoteShuffleInputGate.java:225) at org.apache.celeborn.plugin.flink.RemoteShuffleInputGate.getChannel(RemoteShuffleInputGate.java:179) at org.apache.flink.runtime.io.network.partition.consumer.InputGate.setChannelStateWriter(InputGate.java:90) at org.apache.flink.runtime.taskmanager.InputGateWithMetrics.setChannelStateWriter(InputGateWithMetrics.java:120) at org.apache.flink.streaming.runtime.tasks.StreamTask.injectChannelStateWriterIntoChannels(StreamTask.java:524) at org.apache.flink.streaming.runtime.tasks.StreamTask.<init>(StreamTask.java:496) ``` Flink 1.18.0 release https://flink.apache.org/2023/10/24/announcing-the-release-of-apache-flink-1.18/ Interface `org.apache.flink.runtime.io.network.buffer.Buffer` adds `setRecycler` method. [[FLINK-32549](https://issues.apache.org/jira/browse/FLINK-32549)][network] Tiered storage memory manager supports ownership transfer for buffers `org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate` constructor adds parameters. [[FLINK-31638](https://issues.apache.org/jira/browse/FLINK-31638)][network] Introduce the TieredStorageConsumerClient to SingleInputGate [[FLINK-31642](https://issues.apache.org/jira/browse/FLINK-31642)][network] Introduce the MemoryTierConsumerAgent to TieredStorageConsumerClient ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? ```bash flink-1.18.0 ./bin/flink run examples/streaming/WordCount.jar --execution-mode BATCH Executing example with default input data. Use --input to specify file input. Printing result to stdout. Use --output to specify output path. Job has been submitted with JobID d7fc5f0ca018a54e9453c4d35f7c598a Program execution finished Job with JobID d7fc5f0ca018a54e9453c4d35f7c598a has finished. Job Runtime: 1635 ms ``` <img width="1297" alt="image" src="https://github.com/apache/incubator-celeborn/assets/3898450/6a5266bf-2386-4386-b98b-a60d2570fa99"> Closes #2063 from cxzl25/CELEBORN-1105. Authored-by: sychen <sychen@ctrip.com> Signed-off-by: Shuang <lvshuang.tb@gmail.com>
98 lines
2.5 KiB
YAML
98 lines
2.5 KiB
YAML
#
|
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
|
# contributor license agreements. See the NOTICE file distributed with
|
|
# this work for additional information regarding copyright ownership.
|
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
|
# (the "License"); you may not use this file except in compliance with
|
|
# the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
#
|
|
|
|
name: Dependency
|
|
|
|
on:
|
|
pull_request:
|
|
branches:
|
|
- main
|
|
- branch-*
|
|
paths:
|
|
- '**/pom.xml'
|
|
- 'project/**'
|
|
- '.github/workflows/deps.yml'
|
|
- 'dev/dependencies.sh'
|
|
|
|
concurrency:
|
|
group: deps-${{ github.head_ref || github.run_id }}
|
|
cancel-in-progress: true
|
|
|
|
jobs:
|
|
sbt:
|
|
runs-on: ubuntu-22.04
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
module:
|
|
- 'server'
|
|
- 'spark-2.4'
|
|
- 'spark-3.0'
|
|
- 'spark-3.1'
|
|
- 'spark-3.2'
|
|
- 'spark-3.3'
|
|
- 'spark-3.4'
|
|
- 'spark-3.5'
|
|
- 'flink-1.14'
|
|
- 'flink-1.15'
|
|
- 'flink-1.17'
|
|
- 'flink-1.18'
|
|
- 'mr'
|
|
steps:
|
|
- uses: actions/checkout@v2
|
|
- name: Setup JDK 8
|
|
uses: actions/setup-java@v2
|
|
with:
|
|
distribution: zulu
|
|
java-version: 8
|
|
check-latest: false
|
|
- name: SBT Check dependency list
|
|
run: |
|
|
./dev/dependencies.sh --sbt --module ${{ matrix.module }} --check
|
|
|
|
maven:
|
|
runs-on: ubuntu-22.04
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
module:
|
|
- 'server'
|
|
- 'spark-2.4'
|
|
- 'spark-3.0'
|
|
- 'spark-3.1'
|
|
- 'spark-3.2'
|
|
- 'spark-3.3'
|
|
- 'spark-3.4'
|
|
- 'spark-3.5'
|
|
- 'flink-1.14'
|
|
- 'flink-1.15'
|
|
- 'flink-1.17'
|
|
- 'flink-1.18'
|
|
- 'mr'
|
|
steps:
|
|
- uses: actions/checkout@v2
|
|
- name: Setup JDK 8
|
|
uses: actions/setup-java@v2
|
|
with:
|
|
distribution: zulu
|
|
java-version: 8
|
|
cache: maven
|
|
check-latest: false
|
|
- name: Maven Check dependency list
|
|
run: |
|
|
./dev/dependencies.sh --module ${{ matrix.module }} --check
|