e33df9ce66
### _Why are the changes needed?_ - enable CI test on Scala-2.13 for all modules except Flink SQL engine - For testing, choose available Spark engine home in `download` module by `SCALA_COMPILE_VERSION` of Kyuubi server - Choose the Scala version of Spark engine main resource Jar in the following order: 1. `SPARK_SCALA_VERSION` system env 2. Extract Scala version from Spark home's `spark-core` jar filename - Fixed 1 assertion error of kyuubi-spark-lineage module, as Spark on Scala 2.12 and 2.13 show different order of column linage output in `MergeIntoTable` ut ``` SparkSQLLineageParserHelperSuite: - columns lineage extract - MergeIntoTable *** FAILED *** inputTables(List(v2_catalog.db.source_t)) outputTables(List(v2_catalog.db.target_t)) columnLineage(List(ColumnLineage(v2_catalog.db.target_t.name,Set(v2_catalog.db.source_t.name)), ColumnLineage(v2_catalog.db.target_t.price,Set(v2_catalog.db.source_t.price)), ColumnLineage(v2_catalog.db.target_t.id,Set(v2_catalog.db.source_t.id)))) did not equal inputTables(List(v2_catalog.db.source_t)) outputTables(List(v2_catalog.db.target_t)) columnLineage(List(ColumnLineage(v2_catalog.db.target_t.id,Set(v2_catalog.db.source_t.id)), ColumnLineage(v2_catalog.db.target_t.name,Set(v2_catalog.db.source_t.name)), ColumnLineage(v2_catalog.db.target_t.price,Set(v2_catalog.db.source_t.price)))) (SparkSQLLineageParserHelperSuite.scala:182) ``` - Fixed other tests relying on Scala scripting results ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ Closes #5196 from bowenliang123/scala213-test. Closes #5196 97fafacd3 [liangbowen] prevent repeated compilation for regrex pattern 76b99d423 [Bowen Liang] test on scala-2.13 Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn> Co-authored-by: liangbowen <liangbowen@gf.com.cn> Signed-off-by: Bowen Liang <liangbowen@gf.com.cn> |
||
|---|---|---|
| .. | ||
| server/kyuubi-server-plugin | ||
| spark | ||
| README.md | ||
For developers
This folder contains plugins/extension for kyuubi server and different engine types.
- ext
- kyuubi-server
- spark
- flink
- trino
- hive
- others
- ...