[CELEBORN-1250][FOLLOWUP] Fix license issues

### What changes were proposed in this pull request?

As title

### Why are the changes needed?

Fix license issues for the main branch

cherry-pick https://github.com/apache/incubator-celeborn/pull/2259 and https://github.com/apache/incubator-celeborn/pull/2268 into the main branch.

### Does this PR introduce _any_ user-facing change?

No

### How was this patch tested?

Pass GA

Closes #2271 from cfmcgrady/license-main.

Authored-by: mingji <fengmingxiao.fmx@alibaba-inc.com>
Signed-off-by: mingji <fengmingxiao.fmx@alibaba-inc.com>

LICENSE

@@ -202,9 +202,8 @@
 
 
 ------------------------------------------------------------------------------------
-This product bundles various third-party components under other open source licenses.
+This product bundles various third-party components under the same license.
-This section summarizes those components and their licenses. See licenses/
+This section summarizes those components.
-for text of these licenses.
 
 
 Apache License 2.0

LICENSE-binary

@@ -203,8 +203,14 @@
 ------------------------------------------------------------------------------------
 This project bundles the following dependencies under the Apache License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt):
 
+com.fasterxml.jackson.core:jackson-annotations
+com.fasterxml.jackson.core:jackson-core
+com.fasterxml.jackson.core:jackson-databind
+com.fasterxml.jackson.module:jackson-module-scala
 com.google.code.findbugs:jsr305
+com.google.guava:failureaccess
 com.google.guava:guava
+com.thoughtworks.paranamer:paranamer
 commons-cli:commons-cli
 commons-io:commons-io
 commons-logging:commons-logging
@@ -243,20 +249,8 @@ io.netty:netty-transport-sctp
 io.netty:netty-transport-udt
 org.apache.commons:commons-crypto
 org.apache.commons:commons-lang3
-org.apache.hadoop:hadoop-auth
-org.apache.hadoop:hadoop-client
 org.apache.hadoop:hadoop-client-api
 org.apache.hadoop:hadoop-client-runtime
-org.apache.hadoop:hadoop-common
-org.apache.hadoop:hadoop-hdfs-client
-org.apache.hadoop:hadoop-mapreduce-client-app
-org.apache.hadoop:hadoop-mapreduce-client-common
-org.apache.hadoop:hadoop-mapreduce-client-core
-org.apache.hadoop:hadoop-mapreduce-client-jobclient
-org.apache.hadoop:hadoop-mapreduce-client-shuffle
-org.apache.hadoop:hadoop-yarn-api
-org.apache.hadoop:hadoop-yarn-common
-org.apache.htrace:htrace-core4
 org.apache.logging.log4j:log4j-1.2-api
 org.apache.logging.log4j:log4j-api
 org.apache.logging.log4j:log4j-core
@@ -275,17 +269,12 @@ org.javassist:javassist
 org.reflections:reflections
 org.roaringbitmap:RoaringBitmap
 org.roaringbitmap:shims
+org.rocksdb:rocksdbjni
 org.scala-lang:scala-library
 org.scala-lang:scala-reflect
 org.slf4j:jcl-over-slf4j
+org.xerial.snappy:snappy-java
 org.yaml:snakeyaml
-org.rocksdb:rocksdbjni
-com.fasterxml.jackson.module:jackson-module-scala
-com.fasterxml.jackson.core:jackson-databind
-com.fasterxml.jackson.core:jackson-annotations
-com.fasterxml.jackson.core:jackson-core
-com.thoughtworks.paranamer:paranamer
-
 
 ------------------------------------------------------------------------------------
 This product bundles various third-party components under other open source licenses.
@@ -295,25 +284,25 @@ for text of these licenses.
 
 BSD 3-clause
 ------------
-See license/LICENSE-protobuf.txt for details.
+See licenses/LICENSE-protobuf.txt for details.
 com.google.protobuf:protobuf-java
-See license/LICENSE-leveldbjni.txt for details.
+See licenses/LICENSE-leveldbjni.txt for details.
 org.fusesource.leveldbjni:leveldbjni-all
 
 
 Common Development and Distribution License (CDDL) 1.0
 ------------
-See license/LICENSE-CDDL.txt for details.
+See licenses/LICENSE-CDDL.txt for details.
 javax.servlet:javax.servlet-api
 
 
 MIT License
 ------------
-See license/LISENCE-jdktools.txt for details.
+See licenses/LICENSE-jdktools.txt for details.
 com.github.olivergondza:maven-jdk-tools-wrapper
-See license/LICENSE-slf4j.txt for details.
+See licenses/LICENSE-slf4j.txt for details.
 org.slf4j:jul-to-slf4j
 org.slf4j:slf4j-api
 ------------
-See license/LICENSE-javassist.txt for detail.
+See licenses/LICENSE-javassist.txt for detail.
 org.javassist:javassist

NOTICE-binary

@@ -47,6 +47,9 @@ This product includes software developed at The Alibaba Group Holding Ltd
 Apache Commons CLI
 Copyright 2002-2021 The Apache Software Foundation
 
+Apache Commons Crypto
+Copyright 2016-2024 The Apache Software Foundation
+
 Apache Commons IO
 Copyright 2002-2023 The Apache Software Foundation
 
@@ -56,9 +59,6 @@ Copyright 2001-2021 The Apache Software Foundation
 Apache Commons Logging
 Copyright 2003-2013 The Apache Software Foundation
 
-htrace-core4
-Copyright 2016 The Apache Software Foundation
-
 Apache Log4j 1.x Compatibility API
 Copyright 1999-2022 The Apache Software Foundation
 
@@ -120,3 +120,6 @@ See the License for the specific language governing permissions and
 limitations under the License.
 
 This software includes projects with other licenses -- see `doc/LICENSE.md`.
+
+This product includes software developed by Google
+ Snappy: http://code.google.com/p/snappy/ (New BSD License)

client-flink/flink-1.14-shaded/src/main/resources/META-INF/LICENSE

@@ -208,7 +208,8 @@ This project bundles the following dependencies under the Apache License 2.0 (ht
 Apache License 2.0
 --------------------------------------
 
-org.apache.commons:commons-lang3
+com.google.guava:failureaccess
+com.google.guava:guava
 io.netty:netty-all
 io.netty:netty-buffer
 io.netty:netty-codec
@@ -225,23 +226,23 @@ io.netty:netty-codec-stomp
 io.netty:netty-codec-xml
 io.netty:netty-common
 io.netty:netty-handler
-io.netty:netty-transport-native-unix-common
 io.netty:netty-handler-proxy
 io.netty:netty-resolver
 io.netty:netty-resolver-dns
 io.netty:netty-transport
-io.netty:netty-transport-rxtx
-io.netty:netty-transport-sctp
-io.netty:netty-transport-udt
 io.netty:netty-transport-classes-epoll
 io.netty:netty-transport-classes-kqueue
 io.netty:netty-transport-native-epoll
 io.netty:netty-transport-native-kqueue
-com.google.guava:guava
+io.netty:netty-transport-native-unix-common
+io.netty:netty-transport-rxtx
+io.netty:netty-transport-sctp
+io.netty:netty-transport-udt
+org.apache.commons:commons-lang3
 org.roaringbitmap:RoaringBitmap
 
 
 BSD 3-clause
 ------------
-See license/LICENSE-protobuf.txt for details.
+See licenses/LICENSE-protobuf.txt for details.
 com.google.protobuf:protobuf-java

client-flink/flink-1.15-shaded/src/main/resources/META-INF/LICENSE

@@ -208,7 +208,8 @@ This project bundles the following dependencies under the Apache License 2.0 (ht
 Apache License 2.0
 --------------------------------------
 
-org.apache.commons:commons-lang3
+com.google.guava:failureaccess
+com.google.guava:guava
 io.netty:netty-all
 io.netty:netty-buffer
 io.netty:netty-codec
@@ -225,23 +226,23 @@ io.netty:netty-codec-stomp
 io.netty:netty-codec-xml
 io.netty:netty-common
 io.netty:netty-handler
-io.netty:netty-transport-native-unix-common
 io.netty:netty-handler-proxy
 io.netty:netty-resolver
 io.netty:netty-resolver-dns
 io.netty:netty-transport
-io.netty:netty-transport-rxtx
-io.netty:netty-transport-sctp
-io.netty:netty-transport-udt
 io.netty:netty-transport-classes-epoll
 io.netty:netty-transport-classes-kqueue
 io.netty:netty-transport-native-epoll
 io.netty:netty-transport-native-kqueue
-com.google.guava:guava
+io.netty:netty-transport-native-unix-common
+io.netty:netty-transport-rxtx
+io.netty:netty-transport-sctp
+io.netty:netty-transport-udt
+org.apache.commons:commons-lang3
 org.roaringbitmap:RoaringBitmap
 
 
 BSD 3-clause
 ------------
-See license/LICENSE-protobuf.txt for details.
+See licenses/LICENSE-protobuf.txt for details.
 com.google.protobuf:protobuf-java

client-flink/flink-1.17-shaded/src/main/resources/META-INF/LICENSE

@@ -208,7 +208,8 @@ This project bundles the following dependencies under the Apache License 2.0 (ht
 Apache License 2.0
 --------------------------------------
 
-org.apache.commons:commons-lang3
+com.google.guava:failureaccess
+com.google.guava:guava
 io.netty:netty-all
 io.netty:netty-buffer
 io.netty:netty-codec
@@ -225,23 +226,23 @@ io.netty:netty-codec-stomp
 io.netty:netty-codec-xml
 io.netty:netty-common
 io.netty:netty-handler
-io.netty:netty-transport-native-unix-common
 io.netty:netty-handler-proxy
 io.netty:netty-resolver
 io.netty:netty-resolver-dns
 io.netty:netty-transport
-io.netty:netty-transport-rxtx
-io.netty:netty-transport-sctp
-io.netty:netty-transport-udt
 io.netty:netty-transport-classes-epoll
 io.netty:netty-transport-classes-kqueue
 io.netty:netty-transport-native-epoll
 io.netty:netty-transport-native-kqueue
-com.google.guava:guava
+io.netty:netty-transport-native-unix-common
+io.netty:netty-transport-rxtx
+io.netty:netty-transport-sctp
+io.netty:netty-transport-udt
+org.apache.commons:commons-lang3
 org.roaringbitmap:RoaringBitmap
 
 
 BSD 3-clause
 ------------
-See license/LICENSE-protobuf.txt for details.
+See licenses/LICENSE-protobuf.txt for details.
 com.google.protobuf:protobuf-java

client-flink/flink-1.18-shaded/src/main/resources/META-INF/LICENSE

@@ -208,7 +208,8 @@ This project bundles the following dependencies under the Apache License 2.0 (ht
 Apache License 2.0
 --------------------------------------
 
-org.apache.commons:commons-lang3
+com.google.guava:failureaccess
+com.google.guava:guava
 io.netty:netty-all
 io.netty:netty-buffer
 io.netty:netty-codec
@@ -225,23 +226,23 @@ io.netty:netty-codec-stomp
 io.netty:netty-codec-xml
 io.netty:netty-common
 io.netty:netty-handler
-io.netty:netty-transport-native-unix-common
 io.netty:netty-handler-proxy
 io.netty:netty-resolver
 io.netty:netty-resolver-dns
 io.netty:netty-transport
-io.netty:netty-transport-rxtx
-io.netty:netty-transport-sctp
-io.netty:netty-transport-udt
 io.netty:netty-transport-classes-epoll
 io.netty:netty-transport-classes-kqueue
 io.netty:netty-transport-native-epoll
 io.netty:netty-transport-native-kqueue
-com.google.guava:guava
+io.netty:netty-transport-native-unix-common
+io.netty:netty-transport-rxtx
+io.netty:netty-transport-sctp
+io.netty:netty-transport-udt
+org.apache.commons:commons-lang3
 org.roaringbitmap:RoaringBitmap
 
 
 BSD 3-clause
 ------------
-See license/LICENSE-protobuf.txt for details.
+See licenses/LICENSE-protobuf.txt for details.
 com.google.protobuf:protobuf-java

client-mr/mr-shaded/src/main/resources/META-INF/LICENSE

@@ -208,7 +208,9 @@ This project bundles the following dependencies under the Apache License 2.0 (ht
 Apache License 2.0
 --------------------------------------
 
-org.apache.commons:commons-lang3
+com.google.guava:failureaccess
+com.google.guava:guava
+io.netty:netty
 io.netty:netty-all
 io.netty:netty-buffer
 io.netty:netty-codec
@@ -225,26 +227,27 @@ io.netty:netty-codec-stomp
 io.netty:netty-codec-xml
 io.netty:netty-common
 io.netty:netty-handler
-io.netty:netty-transport-native-unix-common
 io.netty:netty-handler-proxy
 io.netty:netty-resolver
 io.netty:netty-resolver-dns
 io.netty:netty-transport
-io.netty:netty-transport-rxtx
-io.netty:netty-transport-sctp
-io.netty:netty-transport-udt
 io.netty:netty-transport-classes-epoll
 io.netty:netty-transport-classes-kqueue
 io.netty:netty-transport-native-epoll
 io.netty:netty-transport-native-kqueue
-com.google.guava:guava
+io.netty:netty-transport-native-unix-common
-org.roaringbitmap:RoaringBitmap
+io.netty:netty-transport-rxtx
+io.netty:netty-transport-sctp
+io.netty:netty-transport-udt
+org.apache.commons:commons-lang3
 org.lz4:lz4-java
-io.netty:netty
+org.roaringbitmap:RoaringBitmap
 org.scala-lang:scala-library
 
 
-BSD 3-clause
+BSD
 ------------
-See license/LICENSE-protobuf.txt for details.
+See licenses/LICENSE-protobuf.txt for details.
 com.google.protobuf:protobuf-java
+See licenses/LICENSE-zstd-jni.txt for details.
+com.github.luben:zstd-jni

client-mr/mr-shaded/src/main/resources/META-INF/licenses/LICENSE-zstd-jni.txt

@@ -0,0 +1,26 @@
+Zstd-jni: JNI bindings to Zstd Library
+
+Copyright (c) 2015-present, Luben Karavelov/ All rights reserved.
+
+BSD License
+
+Redistribution and use in source and binary forms, with or without modification,
+are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+  list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice, this
+  list of conditions and the following disclaimer in the documentation and/or
+  other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
+ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
+ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

client-spark/spark-2-shaded/src/main/resources/META-INF/LICENSE

@@ -208,7 +208,8 @@ This project bundles the following dependencies under the Apache License 2.0 (ht
 Apache License 2.0
 --------------------------------------
 
-org.apache.commons:commons-lang3
+com.google.guava:failureaccess
+com.google.guava:guava
 io.netty:netty-all
 io.netty:netty-buffer
 io.netty:netty-codec
@@ -225,23 +226,23 @@ io.netty:netty-codec-stomp
 io.netty:netty-codec-xml
 io.netty:netty-common
 io.netty:netty-handler
-io.netty:netty-transport-native-unix-common
 io.netty:netty-handler-proxy
 io.netty:netty-resolver
 io.netty:netty-resolver-dns
 io.netty:netty-transport
-io.netty:netty-transport-rxtx
-io.netty:netty-transport-sctp
-io.netty:netty-transport-udt
 io.netty:netty-transport-classes-epoll
 io.netty:netty-transport-classes-kqueue
 io.netty:netty-transport-native-epoll
 io.netty:netty-transport-native-kqueue
-com.google.guava:guava
+io.netty:netty-transport-native-unix-common
+io.netty:netty-transport-rxtx
+io.netty:netty-transport-sctp
+io.netty:netty-transport-udt
+org.apache.commons:commons-lang3
 org.roaringbitmap:RoaringBitmap
 
 
 BSD 3-clause
 ------------
-See license/LICENSE-protobuf.txt for details.
+See licenses/LICENSE-protobuf.txt for details.
 com.google.protobuf:protobuf-java

client-spark/spark-3-shaded/src/main/resources/META-INF/LICENSE

@@ -208,7 +208,8 @@ This project bundles the following dependencies under the Apache License 2.0 (ht
 Apache License 2.0
 --------------------------------------
 
-org.apache.commons:commons-lang3
+com.google.guava:failureaccess
+com.google.guava:guava
 io.netty:netty-all
 io.netty:netty-buffer
 io.netty:netty-codec
@@ -225,23 +226,23 @@ io.netty:netty-codec-stomp
 io.netty:netty-codec-xml
 io.netty:netty-common
 io.netty:netty-handler
-io.netty:netty-transport-native-unix-common
 io.netty:netty-handler-proxy
 io.netty:netty-resolver
 io.netty:netty-resolver-dns
 io.netty:netty-transport
-io.netty:netty-transport-rxtx
-io.netty:netty-transport-sctp
-io.netty:netty-transport-udt
 io.netty:netty-transport-classes-epoll
 io.netty:netty-transport-classes-kqueue
 io.netty:netty-transport-native-epoll
 io.netty:netty-transport-native-kqueue
-com.google.guava:guava
+io.netty:netty-transport-native-unix-common
+io.netty:netty-transport-rxtx
+io.netty:netty-transport-sctp
+io.netty:netty-transport-udt
+org.apache.commons:commons-lang3
 org.roaringbitmap:RoaringBitmap
 
 
 BSD 3-clause
 ------------
-See license/LICENSE-protobuf.txt for details.
+See licenses/LICENSE-protobuf.txt for details.
 com.google.protobuf:protobuf-java