apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

[KYUUBI #6998] [TEST] Harness SparkProcessBuilderSuite

Browse Source 
### Why are the changes needed?

Fix the missing `assert` in `SparkProcessBuilderSuite - spark process builder`.

Fix the flaky test `SparkProcessBuilderSuite - capture error from spark process builder` by increasing `kyuubi.session.engine.startup.maxLogLines` from 10 to 4096, this is easy to fail, especially in Spark 4.0 due to increased error stack trace. for example, https://github.com/apache/kyuubi/actions/runs/13974413470/job/39290129824

```
SparkProcessBuilderSuite:
- spark process builder
- capture error from spark process builder *** FAILED ***
  The code passed to eventually never returned normally. Attempted 167 times over 1.5007926256666668 minutes. Last failure message: "org.apache.kyuubi.KyuubiSQLException: 	Suppressed: org.apache.spark.util.Utils$OriginalTryStackTraceException: Full stacktrace of original doTryWithCallerStacktrace caller
   See more: /home/runner/work/kyuubi/kyuubi/kyuubi-server/target/work/kentyao/kyuubi-spark-sql-engine.log.2
  	at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69)
  	at org.apache.kyuubi.engine.ProcBuilder.$anonfun$start$1(ProcBuilder.scala:239)
  	at java.base/java.lang.Thread.run(Thread.java:1583)
  .
  FYI: The last 10 line(s) of log are:
  25/03/24 12:53:39 INFO MemoryStore: MemoryStore started with capacity 434.4 MiB
  25/03/24 12:53:39 INFO MemoryStore: MemoryStore cleared
  25/03/24 12:53:39 INFO BlockManager: BlockManager stopped
  25/03/24 12:53:39 INFO BlockManagerMaster: BlockManagerMaster stopped
  25/03/24 12:53:39 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
  25/03/24 12:53:39 INFO SparkContext: Successfully stopped SparkContext
  25/03/24 12:53:39 INFO ShutdownHookManager: Shutdown hook called
  25/03/24 12:53:39 INFO ShutdownHookManager: Deleting directory /tmp/spark-18455622-344e-48ac-92eb-4b368c35e697
  25/03/24 12:53:39 INFO ShutdownHookManager: Deleting directory /home/runner/work/kyuubi/kyuubi/kyuubi-server/target/work/kentyao/artifacts/spark-7479249b-44a2-4fe5-aa0f-544074f9c356
  25/03/24 12:53:39 INFO ShutdownHookManager: Deleting directory /tmp/spark-5ba8250f-1ff2-4e0d-a365-27d7518308e1" did not contain "org.apache.hadoop.hive.ql.metadata.HiveException:". (SparkProcessBuilderSuite.scala:77)
```

### How was this patch tested?

Pass GHA, and verified locally with Spark 4.0.0 RC3 by running tests 10 times with constant success.

### Was this patch authored or co-authored using generative AI tooling?

No

Closes #6998 from pan3793/spark-pb-ut.

Closes #6998

a4290b413 [Cheng Pan] harness SparkProcessBuilderSuite

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
This commit is contained in:
Cheng Pan 2025-03-25 17:28:58 +08:00
parent 196b47e32a
commit 6459680d89
No known key found for this signature in database
GPG Key ID: 8001952629BCC75D
1 changed files with 6 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
kyuubi-server/src/test/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilderSuite.scala
View File

@ -29,7 +29,7 @@ import org.scalatestplus.mockito.MockitoSugar
import org.apache.kyuubi._
import org.apache.kyuubi.config.KyuubiConf
import org.apache.kyuubi.config.KyuubiConf.{ENGINE_LOG_TIMEOUT, ENGINE_SPARK_MAIN_RESOURCE, KUBERNETES_FORCIBLY_REWRITE_DRIVER_POD_NAME, KUBERNETES_FORCIBLY_REWRITE_EXEC_POD_NAME_PREFIX}
import org.apache.kyuubi.config.KyuubiConf._
import org.apache.kyuubi.engine.ProcBuilder.KYUUBI_ENGINE_LOG_PATH_KEY
import org.apache.kyuubi.engine.spark.SparkProcessBuilder._
import org.apache.kyuubi.ha.HighAvailabilityConf
@ -39,15 +39,17 @@ import org.apache.kyuubi.util.AssertionUtils._
import org.apache.kyuubi.util.command.CommandLineUtils._
class SparkProcessBuilderSuite extends KerberizedTestHelper with MockitoSugar {
private def conf = KyuubiConf().set("kyuubi.on", "off")
private def conf = KyuubiConf()
.set(SESSION_ENGINE_STARTUP_MAX_LOG_LINES, 4096)
.set("kyuubi.on", "off")
test("spark process builder") {
val builder = new SparkProcessBuilder("kentyao", true, conf)
val commands = builder.toString.split(' ')
assert(commands(2) === "org.apache.kyuubi.engine.spark.SparkSQLEngine")
assert(commands.contains("spark.kyuubi.on=off"))
builder.toString.contains("\\\n\t--class")
builder.toString.contains("\\\n\t--conf spark.kyuubi.on=off")
assert(builder.toString.contains("\\\n\t--class"))
assert(builder.toString.contains("\\\n\t--conf spark.kyuubi.on=off"))
val pb = new ProcessBuilder(commands.head, "--help")
assert(pb.start().waitFor() === 0)
assert(Files.exists(Paths.get(commands.last)))