Commit Graph

12 Commits

Author SHA1 Message Date
Cheng Pan
6bdf2bdaf8
[KYUUBI #6392] Support javax.servlet and jakarta.servlet co-exist
# 🔍 Description

This PR makes `javax.servlet` and `jakarta.servlet` co-exist, by introducing `javax.servlet-api-4.0.1` and upgrade `jakarta.servlet-api` to 5.0.0. (6.0.0 requires JDK 11)

Spark 4.0 migrated from `javax.servlet` to `jakarta.servlet` in SPARK-47118 while Kyuubi still uses `javax.servlet` in other modules, we should allow them to co-exist for a while.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Pass GHA.

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6392 from pan3793/servlet.

Closes #6392

27d412599 [Cheng Pan] fix
9f1e72272 [Cheng Pan] other spark modules
f4545dc76 [Cheng Pan] fix
313826fa7 [Cheng Pan] exclude
7d5028154 [Cheng Pan] Support javax.servlet and jakarta.servlet co-exist

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-05-20 21:09:30 +08:00
Binjie Yang
eb278c562d
[RELEASE] Bump 1.10.0-SNAPSHOT 2024-03-13 14:24:49 +08:00
Cheng Pan
6061a05f24
Bump 1.9.0-SNAPSHOT 2023-09-04 14:23:12 +08:00
liangbowen
4213e20945 [KYUUBI #5177] Use Scala binary version placeholder in Maven module's artifactId suffix
### _Why are the changes needed?_

- Change hardcoded Scala's version 2.12 in Maven module's `artifactId` to placeholder `scala.binary.version` which is defined in project parent pom as 2.12
- Preparation for Scala 2.13/3.x support in the future
- No impact on using or building Maven modules
- Some ignorable warning messages for unstable artifactId will be thrown by Maven.
```
Warning:  Some problems were encountered while building the effective model for org.apache.kyuubi:kyuubi-server_2.12🫙1.8.0-SNAPSHOT
Warning:  'artifactId' contains an expression but should be a constant
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No.

Closes #5175 from bowenliang123/artifactId-scala.

Closes #5177

2eba29cfa [liangbowen] use placeholder of scala binary version for artifactId

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-08-20 16:03:23 +00:00
wforget
408862af72
[KYUUBI #4814] Introduce Apache Atlas hook support in lineage plugin
### _Why are the changes needed?_

Implements AtlasLineageDispatcher to send lineage to Apache Atlas.

close #4814

Atlas Spark Model Definition: https://github.com/apache/atlas/blob/master/addons/models/1000-Hadoop/1100-spark_model.json

spark process:

![1](https://github.com/apache/kyuubi/assets/17894939/28e2c68c-0ffd-4f1d-b805-a7e964f85aab)

table lineage:

![2](https://github.com/apache/kyuubi/assets/17894939/76b3db6d-ed50-42e3-97cf-2f96d4e403df)

column lineage:

![3](https://github.com/apache/kyuubi/assets/17894939/41ae6ef8-acbf-43b9-ad05-42d669c5e950)

### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [X] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4815 from wForget/KYUUBI-4814.

Closes #4814

3df8a7ec9 [wforget] comments
c58eae7c5 [wforget] comments
926bcf211 [wforget] comment
e0b4067c3 [wforget] comment
e4cc3e3f8 [wforget] comments
adc72b96f [Bowen Liang] Update extensions/spark/kyuubi-spark-lineage/src/main/scala/org/apache/kyuubi/plugin/lineage/dispatcher/atlas/AtlasEntityHelper.scala
e3bdd1c65 [Bowen Liang] Update extensions/spark/kyuubi-spark-lineage/src/main/scala/org/apache/kyuubi/plugin/lineage/dispatcher/atlas/AtlasEntityHelper.scala
baf1711ac [Bowen Liang] Update extensions/spark/kyuubi-spark-lineage/src/test/scala/org/apache/kyuubi/plugin/lineage/dispatcher/atlas/AtlasLineageDispatcherSuite.scala
61e79f3d5 [Bowen Liang] Update extensions/spark/kyuubi-spark-lineage/src/test/scala/org/apache/kyuubi/plugin/lineage/dispatcher/atlas/AtlasLineageDispatcherSuite.scala
541df3780 [Bowen Liang] Update extensions/spark/kyuubi-spark-lineage/src/test/scala/org/apache/kyuubi/plugin/lineage/dispatcher/atlas/AtlasLineageDispatcherSuite.scala
5dd310657 [wforget] fix
cea1e137d [wforget] fix
f028d4b09 [wforget] fix
0c9b4516b [wforget] fix
6f8113032 [wforget] add close atlas client shutdown hook
3f4d2a7db [wforget] add remote user
a0db58afc [wforget] comments
6dd3c66df [wforget] comments
f2b2a30dc [wforget] style
83eb1e481 [wforget] add atlas.column.lineage.enable configuration
0719a2b65 [wforget] doc
05f936005 [wforget] fix
d169b661d [wforget] fix
6da80d742 [wforget] fix
820ae5c5f [wforget] column lineages
dabe8173e [wforget] license
f22e044d2 [wforget] test
b948bce90 [wforget] fix and add test
0aef1be6b [wforget] fix
368b5ab3f [wforget] [KYUUBI-4814] Implements AtlasLineageDispatcher to send lineage to Apache Atlas

Lead-authored-by: wforget <643348094@qq.com>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-06-06 17:47:19 +08:00
Cheng Pan
01d80eb272
[KYUUBI #4870] Add kyuubi-util and kyuubi-util-scala modules
### _Why are the changes needed?_

Close #4870

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4872 from pan3793/util.

Closes #4870

0b9fe3cba [Cheng Pan] nit
ecc5ee4f2 [Cheng Pan] fix
63be7a20c [Cheng Pan] test
85363c187 [Cheng Pan] style
2227247dd [Cheng Pan] fix package
11d10a081 [Cheng Pan] Add kyuubi-util and kyuubi-util-scala modules

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-05-22 22:13:56 +08:00
Cheng Pan
ab1f67cb31
[KYUUBI #4741] Kyuubi Spark Engine/TPC connectors support Spark 3.4
### _Why are the changes needed?_

- Add CI for Spark 3.4
- Kyuubi Spark TPC-DS/H connectors support Spark 3.4

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4741 from pan3793/spark-3.4.

Closes #4741

84a2d6ad7 [Cheng Pan] log
b9b2ec1fb [Cheng Pan] Add spark-3.4 profile

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-04-23 20:17:20 +08:00
wForget
c4f3195bd6 [KYUUBI #3929] Refactor lineage plugin to add LineageDispatcher
### _Why are the changes needed?_

Refactor lineage plugin to add LineageDispatcher.

close #3929

### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3919 from wForget/dev-lineage-dispatcher.

Closes #3929

5df2aa2f [wforget] add doc
98683ebc [wforget] fix
7b97b2e0 [wForget] rebase
4b046868 [wForget] separate LineageDispatcherType class file
e14cf838 [wForget] Refactor lineage plugin to add LineageDispatcher

Lead-authored-by: wForget <643348094@qq.com>
Co-authored-by: wforget <643348094@qq.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
2023-03-22 09:46:14 +08:00
Cheng Pan
4e226ac3cc
Bump 1.8.0-SNAPSHOT 2023-02-10 15:25:49 +08:00
liangbowen
2ac10f91d5
[KYUUBI #3842] [Improvement] Support maven pom.xml code style check with spotless plugin
### _Why are the changes needed?_

Introduce code style check support for Maven's pom.xml with sortPom in spotless maven plugin.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3843 from bowenliang123/spotless-pom.

Closes #3842

3c654597 [liangbowen] apply to pom.xml
fd1536f7 [liangbowen] set expandEmptyElements to true
e498423f [liangbowen] apply spotless:apply to all pom.xml
e46bcfec [liangbowen] add pom style check support in spotless

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-11-23 22:08:00 +08:00
Wang Zhen
7f83c454b6 [KYUUBI #3537] Remove kyuubi dependency of the spark lineage plugin
### _Why are the changes needed?_

Remove kyuubi dependency of the spark lineage plugin.

### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [X] Add screenshots for manual tests if appropriate
![image](https://user-images.githubusercontent.com/17894939/191468933-a7b76b2c-a888-4a45-b294-794194c8ef03.png)

- [X] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3537 from wForget/dev-lineage.

Closes #3537

1cd2f549 [Wang Zhen] comment
71b11506 [Wang Zhen] fix and add test
d7465799 [Wang Zhen] fix test
9e44840f [Wang Zhen] Remove kyuubi dependency of the spark lineage plugin

Authored-by: Wang Zhen <wangzhen07@qiyi.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-11-03 15:46:48 +00:00
odone
9716548380
[KYUUBI #2282] Add lineage records for sql statement execution in Kyuubi engine logs
### _Why are the changes needed?_

Lineage information:
```
col0 -> (table.a, table.b)
col1 -> (table.c, table.a)
```

SQL lineage logger JSON format example.
**SQL:**
```
select a as col0, b as col1 from test_table0
```
**Lineage:**
```
{
   "inputTables": ["default.test_table0"],
   "outputTables": [],
   "columnLineage": [{
      "column": "col0",
      "originalColumns": ["default.test_table0.a"]
   }, {
      "column": "col1",
      "originalColumns": ["default.test_table0.b"]
   }]
}
```

Currently supported column lineage for spark `Command` and `Query` `TreeNode`:

### Query
- `Select`

### Command
- `CreateDataSourceTableAsSelectCommand`
- `CreateHiveTableAsSelectCommand`
- `OptimizedCreateHiveTableAsSelectCommand`
- `CreateTableAsSelect`
- `ReplaceTableAsSelect`
- `InsertIntoDataSourceCommand`
- `InsertIntoHadoopFsRelationCommand`
- `InsertIntoDataSourceDirCommand`
- `InsertIntoHiveDirCommand`
- `InsertIntoHiveTable`

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3185 from iodone/kyuubi-2282.

Closes #2282

002c6d61 [odone] delete spark-sql-engine test for lineage
e1728a79 [odone] update lineage entity schema
de2a3e9a [odone] change kyuubi-spark-listener module to kyuubi-spark-lineage module
9258125e [odone] optimize lineage output
834669ed [odone] delete engine lineage parse
d9c7a3dc [odone] add spark listener to support lineage
4bae8c2f [odone] update for code cleaning
32b3392b [odone] update for review
fe09e478 [odone] add some test

Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-08-24 14:11:35 +08:00