apache/celeborn
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

[CELEBORN-2025] RpcFailure Scala 2.13 serialization is incompatible

Browse Source 
### What changes were proposed in this pull request?

### Why are the changes needed?
Spark4 uses scala 2.13, but it cannot be connected to the celeborn master compiled with scala 2.12.

For example, the first node of `celeborn.master.endpoints` configured by client is not a leader, and RpcFailure will be returned at this time.

```
https://github.com/scala/bug/issues/11207

https://users.scala-lang.org/t/serialversionuid-change-between-scala-2-12-6-and-2-12-7/3478
```

```java
java.io.InvalidClassException: org.apache.celeborn.common.rpc.netty.RpcFailure; local class incompatible: stream classdesc serialVersionUID = 2793139166962436434, local class serialVersionUID = -1724324816907181707
	at java.base/java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:598) ~[?:?]
```

```bash
scala  -cp /tmp/celeborn-client-spark-3-shaded_2.12-0.5.4.jar
```

```scala

:paste -raw

  package org.apache.celeborn {
    class Y {
        def printId = {
            val clazz = classOf[org.apache.celeborn.common.rpc.netty.RpcFailure]
            val uid = java.io.ObjectStreamClass.lookup(clazz).getSerialVersionUID
            println(s"Scala version: ${scala.util.Properties.versionNumberString}")
            println(s"serialVersionUID: $uid")
      }
    }
  }

new org.apache.celeborn.Y().printId
```

2.11
```
Scala version: 2.11.12
serialVersionUID: 2793139166962436434
```
2.12
```
Scala version: 2.12.19
serialVersionUID: 2793139166962436434
```
2.13
```
Scala version: 2.13.16
serialVersionUID: -1724324816907181707
```

### Does this PR introduce _any_ user-facing change?
If we used the master compiled with 2.13 before, it may be incompatible.

### How was this patch tested?
local test

```
Scala version: 2.13.16
serialVersionUID: 2793139166962436434
```

Closes #3309 from cxzl25/CELEBORN-2025.

Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Shuang <lvshuang.xjs@alibaba-inc.com>
This commit is contained in:
sychen 2025-06-05 22:05:56 +08:00 committed by Shuang
parent a24164c0fa
commit 211046d0ec
1 changed files with 1 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
common/src/main/scala/org/apache/celeborn/common/rpc/netty/NettyRpcEnv.scala
View File

@ -576,6 +576,7 @@ private[celeborn] object RequestMessage {
/**
* A response that indicates some failure happens in the receiver side.
*/
@SerialVersionUID(2793139166962436434L)
private[celeborn] case class RpcFailure(e: Throwable)
/**