kyuubi/integration-tests
Bowen Liang e33df9ce66 [KYUUBI #5196] Enable CI Test on Scala 2.13 and support custom or spark-core extracted Scala version for Spark's engine
### _Why are the changes needed?_

- enable CI test on Scala-2.13 for all modules except Flink SQL engine
- For testing, choose available Spark engine home in `download` module by `SCALA_COMPILE_VERSION` of Kyuubi server
- Choose the Scala version of Spark engine main resource Jar  in the following order:
  1. `SPARK_SCALA_VERSION` system env
  2. Extract Scala version from Spark home's `spark-core` jar filename
- Fixed 1 assertion error of kyuubi-spark-lineage module, as Spark on Scala 2.12 and 2.13 show different order of column linage output in `MergeIntoTable` ut
```
SparkSQLLineageParserHelperSuite:
- columns lineage extract - MergeIntoTable *** FAILED ***
  inputTables(List(v2_catalog.db.source_t))
  outputTables(List(v2_catalog.db.target_t))
  columnLineage(List(ColumnLineage(v2_catalog.db.target_t.name,Set(v2_catalog.db.source_t.name)), ColumnLineage(v2_catalog.db.target_t.price,Set(v2_catalog.db.source_t.price)), ColumnLineage(v2_catalog.db.target_t.id,Set(v2_catalog.db.source_t.id)))) did not equal inputTables(List(v2_catalog.db.source_t))
  outputTables(List(v2_catalog.db.target_t))
  columnLineage(List(ColumnLineage(v2_catalog.db.target_t.id,Set(v2_catalog.db.source_t.id)), ColumnLineage(v2_catalog.db.target_t.name,Set(v2_catalog.db.source_t.name)), ColumnLineage(v2_catalog.db.target_t.price,Set(v2_catalog.db.source_t.price)))) (SparkSQLLineageParserHelperSuite.scala:182)
```
- Fixed other tests relying on Scala scripting results

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

Closes #5196 from bowenliang123/scala213-test.

Closes #5196

97fafacd3 [liangbowen] prevent repeated compilation for regrex pattern
76b99d423 [Bowen Liang] test on scala-2.13

Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2023-10-10 08:42:40 +08:00
..
kyuubi-flink-it [KYUUBI #5196] Enable CI Test on Scala 2.13 and support custom or spark-core extracted Scala version for Spark's engine 2023-10-10 08:42:40 +08:00
kyuubi-hive-it [KYUUBI #5196] Enable CI Test on Scala 2.13 and support custom or spark-core extracted Scala version for Spark's engine 2023-10-10 08:42:40 +08:00
kyuubi-jdbc-it [KYUUBI #5196] Enable CI Test on Scala 2.13 and support custom or spark-core extracted Scala version for Spark's engine 2023-10-10 08:42:40 +08:00
kyuubi-kubernetes-it [KYUUBI #5339] [MINOR] Avoid using kyuubi.session.engine.initialize.timeout string 2023-09-28 05:05:40 +00:00
kyuubi-trino-it [KYUUBI #5196] Enable CI Test on Scala 2.13 and support custom or spark-core extracted Scala version for Spark's engine 2023-10-10 08:42:40 +08:00
kyuubi-zookeeper-it Bump 1.9.0-SNAPSHOT 2023-09-04 14:23:12 +08:00
pom.xml Bump 1.9.0-SNAPSHOT 2023-09-04 14:23:12 +08:00