Commit Graph

9 Commits

Author SHA1 Message Date
Bowen Liang
e33df9ce66 [KYUUBI #5196] Enable CI Test on Scala 2.13 and support custom or spark-core extracted Scala version for Spark's engine
### _Why are the changes needed?_

- enable CI test on Scala-2.13 for all modules except Flink SQL engine
- For testing, choose available Spark engine home in `download` module by `SCALA_COMPILE_VERSION` of Kyuubi server
- Choose the Scala version of Spark engine main resource Jar  in the following order:
  1. `SPARK_SCALA_VERSION` system env
  2. Extract Scala version from Spark home's `spark-core` jar filename
- Fixed 1 assertion error of kyuubi-spark-lineage module, as Spark on Scala 2.12 and 2.13 show different order of column linage output in `MergeIntoTable` ut
```
SparkSQLLineageParserHelperSuite:
- columns lineage extract - MergeIntoTable *** FAILED ***
  inputTables(List(v2_catalog.db.source_t))
  outputTables(List(v2_catalog.db.target_t))
  columnLineage(List(ColumnLineage(v2_catalog.db.target_t.name,Set(v2_catalog.db.source_t.name)), ColumnLineage(v2_catalog.db.target_t.price,Set(v2_catalog.db.source_t.price)), ColumnLineage(v2_catalog.db.target_t.id,Set(v2_catalog.db.source_t.id)))) did not equal inputTables(List(v2_catalog.db.source_t))
  outputTables(List(v2_catalog.db.target_t))
  columnLineage(List(ColumnLineage(v2_catalog.db.target_t.id,Set(v2_catalog.db.source_t.id)), ColumnLineage(v2_catalog.db.target_t.name,Set(v2_catalog.db.source_t.name)), ColumnLineage(v2_catalog.db.target_t.price,Set(v2_catalog.db.source_t.price)))) (SparkSQLLineageParserHelperSuite.scala:182)
```
- Fixed other tests relying on Scala scripting results

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

Closes #5196 from bowenliang123/scala213-test.

Closes #5196

97fafacd3 [liangbowen] prevent repeated compilation for regrex pattern
76b99d423 [Bowen Liang] test on scala-2.13

Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2023-10-10 08:42:40 +08:00
Cheng Pan
6061a05f24
Bump 1.9.0-SNAPSHOT 2023-09-04 14:23:12 +08:00
liangbowen
4213e20945 [KYUUBI #5177] Use Scala binary version placeholder in Maven module's artifactId suffix
### _Why are the changes needed?_

- Change hardcoded Scala's version 2.12 in Maven module's `artifactId` to placeholder `scala.binary.version` which is defined in project parent pom as 2.12
- Preparation for Scala 2.13/3.x support in the future
- No impact on using or building Maven modules
- Some ignorable warning messages for unstable artifactId will be thrown by Maven.
```
Warning:  Some problems were encountered while building the effective model for org.apache.kyuubi:kyuubi-server_2.12🫙1.8.0-SNAPSHOT
Warning:  'artifactId' contains an expression but should be a constant
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No.

Closes #5175 from bowenliang123/artifactId-scala.

Closes #5177

2eba29cfa [liangbowen] use placeholder of scala binary version for artifactId

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-08-20 16:03:23 +00:00
Cheng Pan
4e226ac3cc
Bump 1.8.0-SNAPSHOT 2023-02-10 15:25:49 +08:00
sychen
e49f775431
[KYUUBI #3847] Add jdbc-shaded profile to support IDE debug
### _Why are the changes needed?_
1. `mvn antrun:runbuild-info -pl kyuubi-common`
2. debug engine suite

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3847 from cxzl25/profile_jdbc_debug.

Closes #3847

e1c32067 [Cheng Pan] Merge branch 'master' into profile_jdbc_debug
f41d3cd9 [sychen] use jdbc-shaded
25b54b18 [sychen] use jdbc-shaded
dd23f864 [sychen] use kyuubi-hive-jdbc as default
79b0ffcb [sychen] add jdbc profile

Lead-authored-by: sychen <sychen@ctrip.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-11-30 22:34:48 +08:00
liangbowen
2ac10f91d5
[KYUUBI #3842] [Improvement] Support maven pom.xml code style check with spotless plugin
### _Why are the changes needed?_

Introduce code style check support for Maven's pom.xml with sortPom in spotless maven plugin.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3843 from bowenliang123/spotless-pom.

Closes #3842

3c654597 [liangbowen] apply to pom.xml
fd1536f7 [liangbowen] set expandEmptyElements to true
e498423f [liangbowen] apply spotless:apply to all pom.xml
e46bcfec [liangbowen] add pom style check support in spotless

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-11-23 22:08:00 +08:00
SteNicholas
77b036f3a8
[KYUUBI #3264] [RELEASE] Bump 1.7.0-SNAPSHOT
### _Why are the changes needed?_

Preparing v1.7.0-SNAPSHOT with branch-1.6 cut

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3264 from SteNicholas/prepare-1.7.0-snapshot.

Closes #3264

374d56bf [SteNicholas] preparing v1.7.0-SNAPSHOT with branch-1.6 cut

Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-08-18 11:23:54 +08:00
Cheng Pan
c5d29260a3
[KYUUBI #3037] Handles configuring the JUL -> SLF4J bridge
### _Why are the changes needed?_

Handles configuring the JUL -> SLF4J bridge, otherwise, the components which use JUL like Jersey will be not controlled by log4j2 configurations

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3037 from pan3793/jul.

Closes #3037

fda05ff6 [Cheng Pan] update license
31f579fd [Cheng Pan] nit
7da0f9fb [Cheng Pan] Fix dependencyList
3179e6bd [Cheng Pan] Handles configuring the JUL -> SLF4J bridge

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-07-11 17:54:39 +08:00
Cheng Pan
2fd4e3a858
[KYUUBI #2827] [BUILD][TEST] Decouple integration tests from kyuubi-server
### _Why are the changes needed?_

This PR moves engine integration tests except to spark engine from `kyuubi-server` module to dedicated IT modules e.g. `kyuubi-hive-it`, then we can spilt different engine test into jobs to speed up CI tes

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

before
<img width="836" alt="image" src="https://user-images.githubusercontent.com/26535726/173265839-5d0f1ce6-e133-4d60-9a95-c95aa16f817f.png">

after
<img width="825" alt="image" src="https://user-images.githubusercontent.com/26535726/173265903-fb01e910-8f78-4091-91da-dca25be4357d.png">

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2827 from pan3793/hive-it.

Closes #2827

df3798f1 [Cheng Pan] Remove Java 11 test for Hive
e1645338 [Cheng Pan] nit
66524226 [Cheng Pan] nit
3aef2cef [Cheng Pan] review
500c9cff [Cheng Pan] MVN_OPT
3b4ad142 [Cheng Pan] nit
daa6e0ab [Cheng Pan] review
3383d4c1 [Cheng Pan] [BUILD][TEST] Decouple integration tests from kyuubi-server

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-06-18 00:01:34 +08:00