70a2451951
56 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Cheng Pan
|
14e12e9aa4
|
[KYUUBI #6861] Configuration guide of structured logging for Kyuubi server
### Why are the changes needed? It's a common use case that the user may want to send the service logs in a structured format to Kafka and then collect them into centralized log services for further analysis, fortunately, the Kyuubi used logging frameworks Log4j2 has built-in [KafkaAppender](https://logging.apache.org/log4j/2.x/manual/appenders/message-queue.html#KafkaAppender) and [JSON Template Layout](https://logging.apache.org/log4j/2.x/manual/json-template-layout.html), thus the goal could be achieved by just a few configurations. To simplify the user setup steps, this PR adds `log4j-layout-template-json-<version>.jar` into Kyuubi binary tarball. PS: I also plan to support sending engine bootstrap process(e.g. `spark-submit`) logs into Kafka with specific labels in the follow-up PRs. ### How was this patch tested? Manually test. Configuration in `$KYUUBI_HOME/conf/log4j2.xml` ```xml <Configuration status="INFO"> <Appenders> <Kafka name="kafka" topic="ecs-json-logs" syncSend="false"> <JsonTemplateLayout> <EventTemplateAdditionalField key="app" value="kyuubi"/> <EventTemplateAdditionalField key="cluster" value="hadoop-testing"/> <EventTemplateAdditionalField key="host" value="${hostName}"/> </JsonTemplateLayout> <Property name="bootstrap.servers" value="kafka-1:9092,kafka-2:9092,kafka-3:9092"/> <Property name="compression.type" value="gzip"/> </Kafka> </Appenders> <Loggers> <Root level="INFO"> <AppenderRef ref="kafka"/> </Root> </Loggers> </Configuration> ``` Check that Kafka receives the expected structured logging message in the Elastic Common Schema(ECS) layout.  ### Was this patch authored or co-authored using generative AI tooling? No Closes #6861 from pan3793/structured-logging. Closes #6861 9556da2a7 [Cheng Pan] Structured Logs 7dc6dda86 [Cheng Pan] Add log4j-layout-template-json Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
f2cacd3f18
|
[KYUUBI #6836] Ship kafka-clients in binary distribution tarball without compression libs
### Why are the changes needed? I'd like to include `kafka-clients` in the Kyuubi binary distribution tarball to enable the out-of-box support for sinking Kyuubi events to Kafka. - Kafka is an important component in modern data platforms, and is a defacto message queue implementation, especially in the big data domain - `kafka-clients` is released under Apache License V2, has no legal issue - `kafka-clients` is quite a light lib, has no third-party deps except for `slf4j-api` and a few optional compression libs - `kafka-clients` uses "none" compression as default, in practice, "gzip"(delegate to JDK gzip algorithm, no additional libs are required) already performs well for non-extreme cases Additionally, LOG4J2 has a built-in `KafkaAppender` that supports sinking logging to Kafka, which also requires `kafka-clients` in the classpath, I have some initial ideas to forward both Kyuubi server's and engine bootstrap processes log to Kafka in a structured format, will send another PR to add docs to guide users in configuring that. ### How was this patch tested? Review ### Was this patch authored or co-authored using generative AI tooling? No. Closes #6836 from pan3793/kafka-lib. Closes #6836 b069eb199 [Cheng Pan] Ship kafka-clients in binary distribution tarball without compression libs Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
017d8ccd7e
|
[KYUUBI #6458] Remove commons-logging from binary release
# 🔍 Description [`jcl-over-slf4j`](https://www.slf4j.org/legacy.html#jcl-over-slf4j) is a drop-in replacement of `commons-logging`, the latter one should not be present in the final classpath, otherwise, there are potential class conflict issues. The current dep check is problematic, this PR also changes it to always perform "install" to fix the false negative report. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Simply delete `commons-logging-1.1.3.jar` from `apache-kyuubi-1.9.1-bin.tgz` and everything goes well. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6458 from pan3793/commons-logging. Closes #6458 114ec766a [Cheng Pan] fix 79d4121a1 [Cheng Pan] fix 6633e83ee [Cheng Pan] fix 21127ed0b [Cheng Pan] always perform install on dep check 98b13dfcf [Cheng Pan] Remove commons-logging from binary release Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
6bdf2bdaf8
|
[KYUUBI #6392] Support javax.servlet and jakarta.servlet co-exist
# 🔍 Description This PR makes `javax.servlet` and `jakarta.servlet` co-exist, by introducing `javax.servlet-api-4.0.1` and upgrade `jakarta.servlet-api` to 5.0.0. (6.0.0 requires JDK 11) Spark 4.0 migrated from `javax.servlet` to `jakarta.servlet` in SPARK-47118 while Kyuubi still uses `javax.servlet` in other modules, we should allow them to co-exist for a while. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Pass GHA. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6392 from pan3793/servlet. Closes #6392 27d412599 [Cheng Pan] fix 9f1e72272 [Cheng Pan] other spark modules f4545dc76 [Cheng Pan] fix 313826fa7 [Cheng Pan] exclude 7d5028154 [Cheng Pan] Support javax.servlet and jakarta.servlet co-exist Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
4ee9481674
|
[KYUUBI #6391] Bump Arrow from 15.0.2 to 16.0.0
# 🔍 Description A regular dependency upgrading, additionally, Arrow 15 introduced the eclipse-collections dependencies but removed in Arrow 16. https://github.com/apache/arrow/issues/40896 Note: This PR upgrades Arrow to 16.0.0 instead of 16.1.0 due to https://github.com/apache/arrow/issues/41717 ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Pass GHA --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6391 from pan3793/arrow-16. Closes #6391 d8ea702a3 [Cheng Pan] 16.0.0 8a8bc46ab [Cheng Pan] Bump Arrow from 15.0.2 to 16.1.0 Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
dupeng
|
3133b5911c
|
[KYUUBI #6293] Upgrade Arrow from 12.0.0 to 15.0.2
# 🔍 Description ## Issue References 🔗 This pull request fixes #6293 ## Describe Your Solution 🔧 Here are my main modifications: 1. In the project POM file, I have updated the Apache Arrow version from 12.0.0 to 15.0.2. 2. After executing `build/dependency.sh --replace`, the `dev/dependencyList` file was automatically updated. 3. I noticed the addition of the "Eclipse Collections" dependency and checked its open-source license. Following the existing format, I have accordingly modified the `NOTICE-binary` and `LICENSE-binary` files. However, I remain uncertain whether my changes to the LICENSE/NOTICE sections adhere to the community's standards. I kindly request guidance from the community regarding this matter. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6304 from dupen01/issue-arrow. Closes #6293 e858a1c12 [dupeng] Merge remote-tracking branch 'origin/issue-arrow' into issue-arrow 205d905ef [Perl Du] Merge branch 'apache:master' into issue-arrow 238ef7ac6 [dupeng] Merge branch 'master' into issue-arrow 14a53f293 [dupeng] update kyuubi-hive-jdbc-shaded pom 62968d2e2 [dupeng] Update the license information for Eclipse Collections. 323136ec4 [dupeng] Upgrade Arrow to 15.0.2 and update LICENSE/NOTICE Lead-authored-by: dupeng <dunett@163.com> Co-authored-by: Perl Du <34719039+dupen01@users.noreply.github.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
zwangsheng
|
32660d7e8a
|
[KYUUBI #4453][FOLLOW] Delete LICENSE for removed kubernetes client dependencies
# 🔍 Description ## Issue References 🔗 This pull request fixes #4453 Those dependencies was removed in #4453 without update `LICENSE-binary` ## Describe Your Solution 🔧 Delete those removed dependencies ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6159 from zwangsheng/dep/clean_up_dep. Closes #4453 150e74dab [zwangsheng] [KYUUBI #4453][FOLLOW] Delete removed kubernetes client dependencies Authored-by: zwangsheng <binjieyang@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
zhouyifan279
|
6fc7552ef0
|
[KYUUBI #6118] Migrate to shaded HMS client for getting delegation token on server
# 🔍 Description ## Issue References 🔗 ## Describe Your Solution 🔧 Kyuubi Shaded 0.3 introduces a light kyuubi-relocated-hive-metastore-client, for refreshing delegation token, this PR aims to migrate from the vanilla HMS client to this light shaded HMS client, then we can get rid of Hive dependencies, especially the vulnerable thrift 0.9, from the Kyuubi server. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Pass GA. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6118 from zhouyifan279/relocated-hms-client. Closes #6118 4d7e50915 [Cheng Pan] comment 845e39f87 [Cheng Pan] notice b4a58a614 [Cheng Pan] Update NOTICE f4bfa9310 [zhouyifan279] Use kyuubi-relocated-hive-metastore-client af17be1fe [zhouyifan279] Use kyuubi-relocated-hive-metastore-client Lead-authored-by: zhouyifan279 <zhouyifan279@gmail.com> Co-authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
d007d7af10
|
[KYUUBI #6099] Remove support of Derby for Kyuubi metastore
# 🔍 Description ## Issue References 🔗 This pull request removes support of Derby for Kyuubi metastore. ## Describe Your Solution 🔧 Previously, we migrated the embedded DB of Kyuubi metastore from Derby to SQLite, and also marked Derby as deprecated (#4950), now, I propose to remove support of Derby for Kyuubi metastore. Note, that both Derby and SQLite are mainly for testing purposes, and they're not supposed to be used in production. Users should not be surprised by this removal. Also, the Derby we used suffers CVE-2022-46337 > Mitigation: > > Users should upgrade to Java 21 and Derby 10.17.1.0. > > Alternatively, users who wish to remain on older Java versions should build their own Derby distribution from one of the release families to which the fix was backported: 10.16, 10.15, and 10.14. Those are the releases which correspond, respectively, with Java LTS versions 17, 11, and 8. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [x] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Some tests are modified accordingly. Pass GA. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6099 from pan3793/remove-derby. Closes #6099 7e9dfd692 [Cheng Pan] Remove support of Derby for Kyuubi metastore Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
10230b3f94
|
[KYUUBI #5674][LICENSE][FOLLOWUP] Update license files
# 🔍 Description ## Issue References 🔗 This pull request is a follow-up of https://github.com/apache/kyuubi/pull/6027, to update the license files as new dependencies are involved. ## Describe Your Solution 🔧 Follow the https://infra.apache.org/licensing-howto.html to update the LICENSE and NOTICE files. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 It's not a code change, review is required. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6029 from pan3793/5674-license. Closes #5674 216e34802 [Cheng Pan] [KYUUBI #5674][LICENSE][FOLLOWUP] Update license files Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
87ed400562
|
[KYUUBI #5975] Bump trino-client 411
# 🔍 Description ## Issue References 🔗 This is a regular dependency upgrading, ## Describe Your Solution 🔧 Upgrade `trino-client` from 363 to 411. 411 is the latest version which uses okhttp 3.x, hence it does not have kotlin runtime dependencies. This PR also updates the docs, especially the Trino cluster version requirement. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Pass GA. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #5975 from pan3793/trino-411. Closes #5975 2b57df34d [Cheng Pan] fix c498a5bb3 [Cheng Pan] fix 21948ca4f [Cheng Pan] Fix compile e4f1397cc [Cheng Pan] license 66583ca16 [Cheng Pan] Bump trino-client 411 Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
35e988d182
|
[KYUUBI #5974] Bump jetcd 0.7.7
# 🔍 Description ## Issue References 🔗 It's regular dependency upgrading, and jetcd 0.7.7 may be the latest version that supports Java 8. ## Describe Your Solution 🔧 Upgrading jetcd to 0.7.7, and this upgrading involves the transitive dep `failsafe` major upgrades, the group is changed from `net.jodah` to `dev.failsafe` ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Pass GA. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #5974 from pan3793/jetcd-version. Closes #5974 7c1d815a4 [Cheng Pan] relocation 5eca6cdb4 [Cheng Pan] Bump jetcd 0.7.7 Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
f35761b50f
|
[KYUUBI #5973] Bump gRPC from 1.53.0 to 1.60.1
# 🔍 Description ## Issue References 🔗 It's a regular dependency upgrading task. ## Describe Your Solution 🔧 Bump gRPC from 1.53.0 to 1.60.1, a new module [grpc-util](https://mvnrepository.com/artifact/io.grpc/grpc-util) was added in 1.58.0, LICENSE files updated accordingly. https://github.com/grpc/grpc-java/releases ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Pass GA. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #5973 from pan3793/grpc. Closes #5973 b47c7f295 [Cheng Pan] nit dd0e966f0 [Cheng Pan] Bump gRPC from 1.53.0 to 1.60.1 Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
c0f634948a |
[KYUUBI #5671] Bump axios from 0.27.2 to 1.6.0 in /kyuubi-server/web-ui
Bumps [axios](https://github.com/axios/axios) from 0.27.2 to 1.6.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/axios/axios/releases">axios's releases</a>.</em></p> <blockquote> <h2>Release v1.6.0</h2> <h2>Release notes:</h2> <h3>Bug Fixes</h3> <ul> <li><strong>CSRF:</strong> fixed CSRF vulnerability CVE-2023-45857 (<a href="https://redirect.github.com/axios/axios/issues/6028">#6028</a>) (<a href=" |
||
|
labbomb
|
2b20f7b626 |
[KYUUBI #5517] [UI] Initial implement the SQL Lab page
### _Why are the changes needed?_ 1. New SQL Lab Page 2. New core editor functionality <img width="1674" alt="image" src="https://github.com/apache/kyuubi/assets/15062456/00cc9194-f43b-422f-a3dd-7b4456d9eb5b"> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No. Closes #5517 from labbomb/dev. Closes #5517 7b9f6dcea [labbomb] fix: adjust keyword case edd71d232 [labbomb] fix: delete some language configs 45d999b88 [labbomb] fix: update license-binary 70edf9f47 [labbomb] fix: fix some errors becb4e3a6 [labbomb] fix: fix some errors d47e7f5c7 [labbomb] feat: add editor component Authored-by: labbomb <739955946@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
75428bb979
|
[KYUUBI #5484] Remove legacy Web UI
### _Why are the changes needed?_ Close https://github.com/apache/kyuubi/issues/5484 Kyuubi provides a basic new Web UI which is built on top of Vue3, we can remove the legacy dummy Web UI in 1.8. The new UI hosts at `http://<host>:<port>/ui/` and the legacy UI hosts at `http://<host>:<port>/`, we should 1. Remove the legacy UI routing from Jetty 2. Remove all files related to legacy UI 3. Redirect `http://<host>:<port>/` to `http://<host>:<port>/ui/` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate building with the command `build/dist --web-ui`, then `cd dist` and perform `bin/kyuubi run` access http://0.0.0.0:10099 could correctly redirect to http://0.0.0.0:10099/ui/ <img width="1428" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/1e8a67f6-e4db-415e-8a47-dd7c41b487cf"> swagger is render correctly too. <img width="1428" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/1cb4ba31-9965-4468-b7c3-b0319ba959e6"> - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No. Closes #5516 from pan3793/5484. Closes #5484 9d58ef758 [Cheng Pan] address comment and fix test 6d4c0981b [Cheng Pan] [KYUUBI #5484] Remove legacy Web UI Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
zwangsheng
|
60b7e8fb9a |
[KYUUBI #5259][UI] Web UI supports Swagger pages
### _Why are the changes needed?_ As title, add swagger ui into new kyuubi web ui. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes #5263 from zwangsheng/KYUUBI#5259. Closes #5259 34c87797c [zwangsheng] keep static swagger ui in kyuubi server 2efbae03f [zwangsheng] fix license 21c8cae6e [zwangsheng] remove dup dependence and add swagger ui into notive c1bedcbe1 [Cheng Pan] Update kyuubi-server/web-ui/src/views/swagger/index.vue 182042009 [Fu Chen] Update kyuubi-server/web-ui/src/views/swagger/index.vue 05c3b87c4 [zwangsheng] fix style fc05bb82f [zwangsheng] Add License header e56252036 [zwangsheng] Try 9c968005b [zwangsheng] Init swagger page Lead-authored-by: zwangsheng <binjieyang@apache.org> Co-authored-by: Cheng Pan <pan3793@gmail.com> Co-authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
06a915aff4 |
[KYUUBI #4950] Migrate Kyuubi embedded database from Derby to SQLite
### _Why are the changes needed?_ Apache Derby is no longer active. One major drawback of Derby is that it does not support multiple connections to single db files, making it hard to analyze the data on local development. SQLite may be the most popular embedded DBMS in the world. It lives almost in every smartphone (at least Android and iOS integrate SQLite), which means SQLite is quite stable and may be a good choice for standalone production deployment. SQLite provides a CLI command `sqlite3` which is easy to use to connect a data file and run queries. Multi connections to a single db file is allowed, which helps a lot to analyze the data when the Kyuubi server is running. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4950 from pan3793/sqlite. Closes #4950 738c39e6a [Cheng Pan] Update docs/deployment/migration-guide.md 5facdad9c [Cheng Pan] Update docs/deployment/migration-guide.md b9883489c [Cheng Pan] migrate default metadata store to sqlite c785e1a77 [Cheng Pan] migrate jdbc auth test to sqlite 23f63b932 [Cheng Pan] introduce sqlite deps Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
zwangsheng
|
4cd00a8777
|
[KYUUBI #3420][UI] Kyuubi Server Proxy Engine UI
### _Why are the changes needed?_ Kyuubi Server Proxy Engine UI ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate  - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4795 from zwangsheng/KYUUBI_3420. Closes #3420 079dc1c60 [zwangsheng] fix frontend unit test case 6e71b4518 [Cheng Pan] fix cf7ca5145 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/ApiRootResource.scala 9a91d62a0 [Cheng Pan] polish a5dcfae18 [zwangsheng] fix 5d4a8c239 [zwangsheng] Rebase 71d22fc9a [zwangsheng] fix 3b0152f33 [zwangsheng] [KYUUBI #3420][UI] Proxy Engnie UI Lead-authored-by: zwangsheng <2213335496@qq.com> Co-authored-by: Cheng Pan <chengpan@apache.org> Co-authored-by: Cheng Pan <pan3793@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
208ab3af62
|
[KYUUBI #4852] Switch to Kyuubi Shaded Zookeeper
### _Why are the changes needed?_ This PR aims to migrate the vanilla Zookeeper and Curator to the Kyuubi Shaded Zookeeper. It's the first step to adapting JDK 17. There is a known issue [ZOOKEEPER-3779](https://issues.apache.org/jira/browse/ZOOKEEPER-3779) that Zookeeper 3.4 client can not run on JDK 14 and above, in https://github.com/apache/kyuubi-shaded/pull/5, we fixed this issue by a surgical. With the above fixing, zk-3.4 and zk-3.6 clients both work well on JDK 17, we just randomly pick some cases to make sure zk-3.6 is tested zk-3.4 client supports zk-3.4+ server, but zk-3.6 client only supports zk-3.5+ server; in the meanwhile, zk-3.4 is adopted widely, (CDH 5/6, HDP, EMR created before 2023). We are sticky to zk-3.4 to ensure that Kyuubi can be out-of-box in the most existing Hadoop cluster but also provide zk-3.6 as an alternative(simply replace the kyuubi-shaded-zk-3.4 jar w/ kyuubi-shaded-zk-3.6, or build w/ -Pzookeeper-3.6) for users who concerns that zk-3.4 is EOL. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4852 from pan3793/shaded-zk. Closes #4852 d960cc945 [Cheng Pan] remove staging repo 1b3622080 [Cheng Pan] Switch to Kyuubi Shaded Zookeeper Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
liangbowen
|
d73ec64b37 |
[KYUUBI #4733] Introduce Kafka event logger for server events
### _Why are the changes needed?_ - introduce new event logger type `KAFKA` - send server events to the Kafka topic with initializing and closing Kafka producer properly with server's lifecyle - use Kafka 3.4.0 as the client version, and tested with Kakfa servers of 2.8.x and 3.4.x ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4733 from bowenliang123/kafka-logger. Closes #4733 b5220d234 [liangbowen] introduce kafka server event logger Authored-by: liangbowen <liangbowen@gf.com.cn> Signed-off-by: liangbowen <liangbowen@gf.com.cn> |
||
|
He Zhao
|
0d5eaa2d0b
|
[KYUUBI #3646][UI] Init Session Statistic Page
### _Why are the changes needed?_ Init Session Statistic Page Close #3646 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request  Closes #4564 from zwangsheng/KYUUBI_3646. Closes #3646 cbe0842ba [zwangsheng] [KYUUBI #3646] fix style 930cbb12a [zwangsheng] [KYUUBI #3646] bracket same line d2ab1d5fd [zwangsheng] [KYUUBI #3646] Fix i18n about status 7f059e2df [zwangsheng] [KYUUBI #3646] Fix i18n about status 452c3ee5a [zwangsheng] [KYUUBI #3646] Remove unused style class 8967652bc [zwangsheng] [KYUUBI #3646] Add date fns license 70f2472c2 [zwangsheng] [KYUUBI #3646] install date-fns 8a3e845ff [zwangsheng] [KYUUBI #3646][UI] Init Session Statistic Page Lead-authored-by: He Zhao <hezhao2@cisco.com> Co-authored-by: zwangsheng <2213335496@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
liangbowen
|
24e87ef21a |
[KYUUBI #4556] [CHAT] Refactor ChatGPTProvider to use openai-java client
### _Why are the changes needed?_ - use Java SDK `openai-java` for ChatGPT which is popular and listed in official website, https://github.com/TheoKanning/openai-java - Focus on lifecycle in ChatGPTProvider, and prevent handling lower-level concepts in details, like POJO mapping, HTTP request handling. - follow the changes from upstream changes from OpenAI ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4556 from bowenliang123/chatgpt-third. Closes #4556 ecf1e2cf6 [liangbowen] manually add `openai-gpt3-java:*` and its dependency to LICENSE-binary 53b8375a5 [liangbowen] refactor ChatGPTProvider to use `openai-java` SDK Authored-by: liangbowen <liangbowen@gf.com.cn> Signed-off-by: liangbowen <liangbowen@gf.com.cn> |
||
|
Cheng Pan
|
eb1b11cd17 |
[KYUUBI #4152] Enhance LDAP authentication
### _Why are the changes needed?_ This PR proposes to enhance the LDAP support, which mainly referring the code introduced in HIVE-14713. Currently, Kyuubi has very limited LDAP support, and the implementation is from the early Hive codebase. Hive enhanced the LDAP support in later versions, considering the Hive ecosystem is quite mature, I think we'd better to porting this functionality and keep the same behavior w/ Hive first, and we can improve it if meet certain requirements/issues in the future. Basically, this PR introduces the following configurations ``` kyuubi.authentication.ldap.url (since 1.0.0) kyuubi.authentication.ldap.domain (since 1.0.0) kyuubi.authentication.ldap.guidKey (since 1.2.0) kyuubi.authentication.ldap.base.dn (since 1.0.0 deprecated) kyuubi.authentication.ldap.baseDN kyuubi.authentication.ldap.groupMembershipKey kyuubi.authentication.ldap.userMembershipKey kyuubi.authentication.ldap.groupClassKey kyuubi.authentication.ldap.groupDNPattern kyuubi.authentication.ldap.userDNPattern kyuubi.authentication.ldap.groupFilter kyuubi.authentication.ldap.userFilter kyuubi.authentication.ldap.customLDAPQuery kyuubi.authentication.ldap.binddn kyuubi.authentication.ldap.bindpw ``` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible This PR ports all LDAP-related UT&IT from Hive codebase - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4152 from pan3793/ldap. Closes #4152 d251c959 [Cheng Pan] nit 6d14f44b [Cheng Pan] nit 6b3d116c [Cheng Pan] nit ab47d822 [Cheng Pan] nit a56e8702 [Cheng Pan] nit 4624619a [Cheng Pan] nit b82c0c05 [Cheng Pan] LDAP test password uses alphanumeric 86a01cca [Cheng Pan] Enhance LDAP authentication Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Yikf
|
246be77632 |
[KYUUBI #3926][FOLLOWUP] Supplement LICENSE-binary
### _Why are the changes needed?_ This pr is followup of the https://github.com/apache/kyuubi/pull/3944, it supplement LICENSE-binary ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4163 from Yikf/license. Closes #3926 92d907dc [Yikf] [KYUUBI #3926][FOLLOWUP] Supplement LICENSE-binary Authored-by: Yikf <yikaifei1@gmail.com> Signed-off-by: Yikf <yikaifei@apache.org> |
||
|
Cheng Pan
|
b2831d7656
|
[KYUUBI #3971] [LICENSE] Update LICENSE-binary to include web-ui
### _Why are the changes needed?_ This leverages the feature `pnpm licenses list --prod` recently added in pnpm. https://github.com/pnpm/pnpm/releases/tag/v7.17.0 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #3971 from pan3793/fe-license. Closes #3971 c3af4546 [Cheng Pan] [LICENSE] Update LICENSE-binary to include web-ui Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
yuqi
|
c537557fcf
|
[KYUUBI #3813] [BUG] Caused by: java.lang.ClassNotFoundException: com.google.common.util.concurrent.internal.InternalFutureFailureAccess
### _Why are the changes needed?_ **Fix the following bug:** ``` 2022-11-16 13:08:45.540 ERROR org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Error opening session: org.apache.kyuubi.KyuubiSQLException: Error opening session for test_user client ip 127.0.0.1, due to com/google/common/util/concurrent/internal/InternalFutureFailureAccess at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) ~[kyuubi-common_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT] at org.apache.kyuubi.session.KyuubiSessionManager.openSession(KyuubiSessionManager.scala:102) ~[kyuubi-server_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT] at org.apache.kyuubi.service.AbstractBackendService.openSession(AbstractBackendService.scala:45) ~[kyuubi-common_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT] at org.apache.kyuubi.server.KyuubiServer$$anon$1.org$apache$kyuubi$server$BackendServiceMetric$$super$openSession(KyuubiServer.scala:111) ~[kyuubi-server_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT] at org.apache.kyuubi.server.BackendServiceMetric.$anonfun$openSession$1(BackendServiceMetric.scala:37) ~[kyuubi-server_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT] at org.apache.kyuubi.metrics.MetricsSystem$.timerTracing(MetricsSystem.scala:111) ~[kyuubi-metrics_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT] at org.apache.kyuubi.server.BackendServiceMetric.openSession(BackendServiceMetric.scala:37) ~[kyuubi-server_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT] at org.apache.kyuubi.server.BackendServiceMetric.openSession$(BackendServiceMetric.scala:30) ~[kyuubi-server_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT] at org.apache.kyuubi.server.KyuubiServer$$anon$1.openSession(KyuubiServer.scala:111) ~[kyuubi-server_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT] at org.apache.kyuubi.service.TFrontendService.getSessionHandle(TFrontendService.scala:181) ~[kyuubi-common_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT] at org.apache.kyuubi.server.KyuubiTBinaryFrontendService.OpenSession(KyuubiTBinaryFrontendService.scala:81) ~[kyuubi-server_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT] at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1497) ~[hive-service-rpc-3.1.3.jar:3.1.3] at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1482) ~[hive-service-rpc-3.1.3.jar:3.1.3] at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) ~[libthrift-0.9.3.jar:0.9.3] at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) ~[kyuubi-common_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT] at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) ~[libthrift-0.9.3.jar:0.9.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_222] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_222] Caused by: java.lang.NoClassDefFoundError: com/google/common/util/concurrent/internal/InternalFutureFailureAccess at java.lang.ClassLoader.defineClass1(Native Method) ~[?:1.8.0_222] at java.lang.ClassLoader.defineClass(ClassLoader.java:763) ~[?:1.8.0_222] at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) ~[?:1.8.0_222] at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) ~[?:1.8.0_222] at java.net.URLClassLoader.access$100(URLClassLoader.java:74) ~[?:1.8.0_222] at java.net.URLClassLoader$1.run(URLClassLoader.java:369) ~[?:1.8.0_222] at java.net.URLClassLoader$1.run(URLClassLoader.java:363) ~[?:1.8.0_222] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_222] at java.net.URLClassLoader.findClass(URLClassLoader.java:362) ~[?:1.8.0_222] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_222] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) ~[?:1.8.0_222] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_222] at java.lang.ClassLoader.defineClass1(Native Method) ~[?:1.8.0_222] at java.lang.ClassLoader.defineClass(ClassLoader.java:763) ~[?:1.8.0_222] at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) ~[?:1.8.0_222] at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) ~[?:1.8.0_222] at java.net.URLClassLoader.access$100(URLClassLoader.java:74) ~[?:1.8.0_222] at java.net.URLClassLoader$1.run(URLClassLoader.java:369) ~[?:1.8.0_222] at java.net.URLClassLoader$1.run(URLClassLoader.java:363) ~[?:1.8.0_222] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_222] at java.net.URLClassLoader.findClass(URLClassLoader.java:362) ~[?:1.8.0_222] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_222] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) ~[?:1.8.0_222] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_222] at java.lang.ClassLoader.defineClass1(Native Method) ~[?:1.8.0_222] at java.lang.ClassLoader.defineClass(ClassLoader.java:763) ~[?:1.8.0_222] at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) ~[?:1.8.0_222] at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) ~[?:1.8.0_222] at java.net.URLClassLoader.access$100(URLClassLoader.java:74) ~[?:1.8.0_222] at java.net.URLClassLoader$1.run(URLClassLoader.java:369) ~[?:1.8.0_222] at java.net.URLClassLoader$1.run(URLClassLoader.java:363) ~[?:1.8.0_222] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_222] at java.net.URLClassLoader.findClass(URLClassLoader.java:362) ~[?:1.8.0_222] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_222] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) ~[?:1.8.0_222] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_222] at com.google.common.cache.LocalCache$LoadingValueReference.<init>(LocalCache.java:3472) ~[guava-30.1-jre.jar:?] at com.google.common.cache.LocalCache$LoadingValueReference.<init>(LocalCache.java:3476) ~[guava-30.1-jre.jar:?] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2134) ~[guava-30.1-jre.jar:?] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045) ~[guava-30.1-jre.jar:?] at com.google.common.cache.LocalCache.get(LocalCache.java:3951) ~[guava-30.1-jre.jar:?] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) ~[guava-30.1-jre.jar:?] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4935) ~[guava-30.1-jre.jar:?] at org.apache.kyuubi.session.FileSessionConfAdvisor.getConfOverlay(FileSessionConfAdvisor.scala:41) ~[kyuubi-server_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT] at org.apache.kyuubi.session.KyuubiSessionImpl.<init>(KyuubiSessionImpl.scala:55) ~[kyuubi-server_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT] at org.apache.kyuubi.session.KyuubiSessionManager.createSession(KyuubiSessionManager.scala:83) ~[kyuubi-server_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT] at org.apache.kyuubi.session.SessionManager.openSession(SessionManager.scala:98) ~[kyuubi-common_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT] at org.apache.kyuubi.session.KyuubiSessionManager.openSession(KyuubiSessionManager.scala:95) ~[kyuubi-server_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT] ... 18 more Caused by: java.lang.ClassNotFoundException: com.google.common.util.concurrent.internal.InternalFutureFailureAccess at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_222] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_222] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) ~[?:1.8.0_222] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_222] at java.lang.ClassLoader.defineClass1(Native Method) ~[?:1.8.0_222] at java.lang.ClassLoader.defineClass(ClassLoader.java:763) ~[?:1.8.0_222] at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) ~[?:1.8.0_222] at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) ~[?:1.8.0_222] at java.net.URLClassLoader.access$100(URLClassLoader.java:74) ~[?:1.8.0_222] at java.net.URLClassLoader$1.run(URLClassLoader.java:369) ~[?:1.8.0_222] at java.net.URLClassLoader$1.run(URLClassLoader.java:363) ~[?:1.8.0_222] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_222] at java.net.URLClassLoader.findClass(URLClassLoader.java:362) ~[?:1.8.0_222] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_222] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) ~[?:1.8.0_222] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_222] at java.lang.ClassLoader.defineClass1(Native Method) ~[?:1.8.0_222] at java.lang.ClassLoader.defineClass(ClassLoader.java:763) ~[?:1.8.0_222] at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) ~[?:1.8.0_222] at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) ~[?:1.8.0_222] at java.net.URLClassLoader.access$100(URLClassLoader.java:74) ~[?:1.8.0_222] at java.net.URLClassLoader$1.run(URLClassLoader.java:369) ~[?:1.8.0_222] at java.net.URLClassLoader$1.run(URLClassLoader.java:363) ~[?:1.8.0_222] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_222] at java.net.URLClassLoader.findClass(URLClassLoader.java:362) ~[?:1.8.0_222] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_222] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) ~[?:1.8.0_222] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_222] at java.lang.ClassLoader.defineClass1(Native Method) ~[?:1.8.0_222] at java.lang.ClassLoader.defineClass(ClassLoader.java:763) ~[?:1.8.0_222] at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) ~[?:1.8.0_222] at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) ~[?:1.8.0_222] at java.net.URLClassLoader.access$100(URLClassLoader.java:74) ~[?:1.8.0_222] at java.net.URLClassLoader$1.run(URLClassLoader.java:369) ~[?:1.8.0_222] at java.net.URLClassLoader$1.run(URLClassLoader.java:363) ~[?:1.8.0_222] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_222] at java.net.URLClassLoader.findClass(URLClassLoader.java:362) ~[?:1.8.0_222] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_222] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) ~[?:1.8.0_222] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_222] at com.google.common.cache.LocalCache$LoadingValueReference.<init>(LocalCache.java:3472) ~[guava-30.1-jre.jar:?] at com.google.common.cache.LocalCache$LoadingValueReference.<init>(LocalCache.java:3476) ~[guava-30.1-jre.jar:?] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2134) ~[guava-30.1-jre.jar:?] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045) ~[guava-30.1-jre.jar:?] at com.google.common.cache.LocalCache.get(LocalCache.java:3951) ~[guava-30.1-jre.jar:?] at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) ~[guava-30.1-jre.jar:?] at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4935) ~[guava-30.1-jre.jar:?] at org.apache.kyuubi.session.FileSessionConfAdvisor.getConfOverlay(FileSessionConfAdvisor.scala:41) ~[kyuubi-server_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT] at org.apache.kyuubi.session.KyuubiSessionImpl.<init>(KyuubiSessionImpl.scala:55) ~[kyuubi-server_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT] at org.apache.kyuubi.session.KyuubiSessionManager.createSession(KyuubiSessionManager.scala:83) ~[kyuubi-server_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT] at org.apache.kyuubi.session.SessionManager.openSession(SessionManager.scala:98) ~[kyuubi-common_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT] at org.apache.kyuubi.session.KyuubiSessionManager.openSession(KyuubiSessionManager.scala:95) ~[kyuubi-server_2.12-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT] ... 18 more ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #3813 from ychris78/yuqi_1116. Closes #3813 7fbfdff8 [yuqi] update dev/dependencyList and LICENSE-binary f664907d [yuqi] fix bug: Caused by: java.lang.ClassNotFoundException: com.google.common.util.concurrent.internal.InternalFutureFailureAccess Authored-by: yuqi <1450636878@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
72c1f53dd0
|
[KYUUBI #3776] [BUILD] Revise Kyuubi Spark engine shaded
### _Why are the changes needed?_
This PR revises the shaded rule of spark engine module, especially to make sure that netty native libs is shaded properly.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
```
(kyuubi) ➜ apache-kyuubi git:(spark-engine-shade) tree externals/kyuubi-spark-sql-engine/target/unpacked -L 5 | grep -v class
externals/kyuubi-spark-sql-engine/target/unpacked
├── META-INF
│ ├── LICENSE
│ ├── MANIFEST.MF
│ ├── NOTICE
│ ├── io.netty.versions.properties
│ ├── native
│ │ ├── liborg_apache_kyuubi_shade_netty_transport_native_epoll_aarch_64.so
│ │ └── liborg_apache_kyuubi_shade_netty_transport_native_epoll_x86_64.so
│ ├── services
│ │ ├── org.apache.kyuubi.shade.io.grpc.LoadBalancerProvider
│ │ ├── org.apache.kyuubi.shade.io.grpc.ManagedChannelProvider
│ │ ├── org.apache.kyuubi.shade.io.grpc.NameResolverProvider
│ │ ├── org.apache.kyuubi.shade.io.grpc.ServerProvider
│ │ ├── org.apache.kyuubi.shade.io.vertx.core.spi.launcher.CommandFactory
│ │ ├── org.apache.spark.status.AppHistoryServerPlugin
│ │ └── reactor.blockhound.integration.BlockHoundIntegration
│ ├── versions
│ │ └── 11
│ │ └── io
│ │ └── vertx
│ └── vertx
│ └── vertx-version.txt
├── kyuubi-version-info.properties
├── log4j2-defaults.xml
├── org
│ └── apache
│ ├── kyuubi
│ │ ├── cli
│ │ ├── config
│ │ │ └── internal
│ │ ├── engine
│ │ │ └── spark
│ │ ├── events
│ │ │ └── handler
│ │ ├── ha
│ │ │ └── client
│ │ ├── operation
│ │ │ ├── log
│ │ │ └── meta
│ │ ├── reflection
│ │ ├── service
│ │ │ └── authentication
│ │ ├── session
│ │ ├── shade
│ │ │ ├── android
│ │ │ ├── com
│ │ │ ├── io
│ │ │ ├── net
│ │ │ └── org
│ │ └── util
│ └── spark
│ ├── api
│ │ └── python
│ ├── kyuubi
│ └── ui
└── python
├── execute_python.py
└── kyuubi_util.py
40 directories, 211 files
```
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes #3776 from pan3793/spark-engine-shade.
Closes #3776
6a6e2a64 [Cheng Pan] nit
e247923a [Cheng Pan] 1
a53b7c06 [Cheng Pan] nit
19382ef3 [Cheng Pan] [BUILD] Revisit Kyuubi Spark engine shaded
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
|
||
|
Cheng Pan
|
de0a92529a
|
[KYUUBI #3796] Bump Netty from 4.1.73.Final to 4.1.84.Final
### _Why are the changes needed?_ Bump Netty to latest version. https://github.com/netty/netty/pull/12146 explained that netty-tcnative-classes should be optional, otherwise will cause https://github.com/netty/netty/issues/12101 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #3796 from pan3793/netty. Closes #3796 2b299233 [Cheng Pan] license 447a8d08 [Cheng Pan] nit 62abe481 [Cheng Pan] nit 506f76bf [Cheng Pan] Bump Netty from 4.1.73.Final to 4.1.84.Final Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Fu Chen
|
bea8e4f1cb |
[KYUUBI #3710] Delete the swagger UI related resource files from the source folder
### _Why are the changes needed?_ This pr aims to reduce the source file size. This PR removes the swagger UI resource file in the folder `kyuubi-server/src/main/resources/org/apache/kyuubi/ui/swagger` which was introduced by [PR-2061](https://github.com/apache/incubator-kyuubi/pull/2061) , and now we rely on Maven dependency `org.webjars::swagger-ui::4.9.1` to package these resource files, we also add a new REST endpoint `/swagger-static/` for downloading these resource files ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #3710 from cfmcgrady/reduce-src-size. Closes #3710 c83bd313 [Fu Chen] update comment 441129b5 [Fu Chen] update LICENSE-binary 14eac59b [Fu Chen] update dev/dependencyList 7971c388 [Fu Chen] bump swagger-ui to 4.9.1 455aa206 [Fu Chen] reduce source file size Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
08f6f4e13d
|
Revert "[KYUUBI #2041] Bump Zookeeper from 3.4.14 to 3.6.3"
This reverts commit
|
||
|
Cheng Pan
|
812ea2bdfe
|
[KYUUBI #2041] Bump Zookeeper from 3.4.14 to 3.6.3
### _Why are the changes needed?_ #2040 adds the test to verify the compatibility for Zookeeper Server 3.4, I will rebase this PR after #2040 get merged. Close #1941 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2041 from pan3793/zk. Closes #2041 d6722f11 [Cheng Pan] license eb7101a0 [Cheng Pan] fix d9ce7c29 [Cheng Pan] Bump Zookeeper 3.6.3 Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
hongdongdong
|
b01022d8f3
|
[KYUUBI #3254] Supplement the licenses of support etcd discovery
### _Why are the changes needed?_ Supplement the licenses. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #3254 from hddong/supplement-license. Closes #3254 b9dfef64 [hongdongdong] remove netty-transport-native-kqueue from license-binary 2185c964 [hongdongdong] Fix adb8f479 [hongdongdong] Fix 0eeeb0f5 [hongdongdong] Update notice c296f78e [hongdd] Update license and notice 88482efe [hongdongdong] Supplement the licenses of support etcd discovery Lead-authored-by: hongdongdong <hongdd@apache.org> Co-authored-by: hongdd <hongdd@hongdddeMac-mini.local> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
hongdongdong
|
32970ce685
|
[KYUUBI #2644] Add etcd discovery client for HA
### _Why are the changes needed?_ Add etcd discovery client for HA ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2767 from hddong/etcd-support. Closes #2644 6fa37578 [hongdongdong] Rebase and remove from rat-excludes fc23a2b4 [hongdongdong] upgrade grpc-1.47.0 61c77d27 [hongdongdong] Fix tests 28abd13e [hongdongdong] fix comments 9e81a6a3 [hongdongdong] Rename HA_ZK_ENGINE_REF_ID to HA_ENGINE_REF_ID aa2b4260 [hongdongdong] [KYUUBI #2644][WIP] Add etcd discovery client for HA Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
c5d29260a3
|
[KYUUBI #3037] Handles configuring the JUL -> SLF4J bridge
### _Why are the changes needed?_ Handles configuring the JUL -> SLF4J bridge, otherwise, the components which use JUL like Jersey will be not controlled by log4j2 configurations ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #3037 from pan3793/jul. Closes #3037 fda05ff6 [Cheng Pan] update license 31f579fd [Cheng Pan] nit 7da0f9fb [Cheng Pan] Fix dependencyList 3179e6bd [Cheng Pan] Handles configuring the JUL -> SLF4J bridge Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
zwangsheng
|
3bc299d01e
|
[KYUUBI #2868] [K8S] Add KubernetesApplicationOperation
### _Why are the changes needed?_ close #2868 Add Kubernetes Application Operation. Help Tag Spark Driver with Engine Ref Id. 1) Find taged Engine: `KyuubiApplicationManager => KubernetesApplicationOperation =(doesn't found taged pod)=> KubernetesApplicationOperation#JpsApplicationOperation` 2) `SparkBatchProcessBuilder` add logic to help tag Kubernetes Engine Tag. 3) Add Unit Test help to check `KubernetesApplicationOperation` find & kill Spark Engine On Kubernetes with both client mode and cluster mode. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2869 from zwangsheng/feature/k8s-application-operation. Closes #2868 04c97210 [zwangsheng] remove fc151603 [zwangsheng] fix kyuubi on kubernetes integration test 29a3da52 [zwangsheng] fix f43f0989 [zwangsheng] fix 2f2c9581 [zwangsheng] test 209dd65d [zwangsheng] fix jps e12c681c [zwangsheng] fix tag 80971f98 [zwangsheng] fix 0760b63d [zwangsheng] fix 087eadb1 [zwangsheng] fix d4989b09 [zwangsheng] try 93d83f81 [zwangsheng] add test 19378da4 [zwangsheng] fix style 63007261 [zwangsheng] fix test fd4bf714 [zwangsheng] fix 66fc902b [zwangsheng] Using jps for client 2a1b797f [zwangsheng] fix setting 4827c459 [zwangsheng] fix 9ee90da6 [zwangsheng] decoupling spark 6b5831da [zwangsheng] useless driver label f9ca898c [zwangsheng] fix 2add72fe [zwangsheng] fix setting aaa081f8 [zwangsheng] fix style & dependency list e102ac73 [zwangsheng] add kubernetesApplicationOperation Authored-by: zwangsheng <2213335496@qq.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Fei Wang
|
6f308c434c |
[KYUUBI #2964] [SUB-TASK][KPIP-4] Refine the batch response and render
### _Why are the changes needed?_ This is make some enhancements for batch response: - combine batch name and batch app name - flatten the batchInfo map to `appId`, `appUrl`, `appState` and`appDiagnostic` Using https://github.com/JakeWharton/flip-tables for table render, which support that the column value could be a multiple-line string. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate Batch List:  Batch Report:  - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2964 from turboFei/refine_batch_resp. Closes #2964 deaedcbc [Fei Wang] add into LICENSE-binary 1f9691e3 [Fei Wang] [SUB-TASK][KPIP-4] Refine the batch response and render Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: Fei Wang <fwang12@ebay.com> |
||
|
Fei Wang
|
73c6b1b14a |
[KYUUBI #2717] [SUB-TASK][KPIP-4] Introduce jdbc session state store for batch session multiple HA
### _Why are the changes needed?_ Close #2715 Close #2374 Introduce jdbc based session state store to share the batch state information across kyuubi instances. This pr is based on docs: https://docs.google.com/document/d/1_BMekMT88P2vCHCl4ZacWEeWslpIv4pxsJH32PAnWSI and refactor it during implementation. In this PR, I introduce `StateStore` interface and implement the default `JDBCStateStore`. The data structures: 1. BatchState - the batch state info for multiple-HA, including batch operation state and application information 2. BatchMeta - the meta data used for batch recovery, including sessionConf, batchRequest information For JDBCStateStore: - support to specify data base type: - 1. DERBY - it will use derby jdbc driver by default - 2. MYSQL - it will use mysql jdbc by default - 3. CUSTOM - the user need specify the jdbc driver - support to init schema By default, the jdbc url is a derby in-memory database url for single kyuubi instance. To support multiple HA in production env, we should config the production jdbc url to share batch info across kyuubi instances. The datasource is managed by HiKariCP. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2717 from turboFei/kyuubi_2715_hikaricp_v2. Closes #2717 367b944a3 [Fei Wang] comments 9a7963dbd [Fei Wang] post session type in event b1c130853 [Fei Wang] move session type to session package 66def0cb6 [Fei Wang] add JDBCStateStoreConf 839b72bf6 [Fei Wang] refactor 644cbcd28 [Fei Wang] filter by session type 58f1c7854 [Fei Wang] refactor 1c0035208 [Fei Wang] add session type 34a087ee5 [Fei Wang] refactor fba38e473 [Fei Wang] refactor fcbde25a0 [Fei Wang] refactor 332a67537 [Fei Wang] comments 135ba9b6d [Fei Wang] Revert "revert class utils change" 0e045a483 [Fei Wang] Revert "Revert "move"" 05048416e [Fei Wang] Revert "test again" 01a50c35b [Fei Wang] test again b89159d13 [Fei Wang] Revert "move" 3764a9051 [Fei Wang] move eef17822c [Fei Wang] revert 4166db261 [Fei Wang] adjust engine idle timeout 5fc95a824 [Fei Wang] revert class utils change 23f88d63d [Fei Wang] refactor a66cb811d [Fei Wang] refactor 89fa0edb4 [Fei Wang] fix e5228fefd [Fei Wang] refactor 581908d86 [Fei Wang] exclude sql 3446aaa06 [Fei Wang] UPPER ab980c677 [Fei Wang] [KYUUBI #2715] Introduce session state store for batch session multiple HA Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: Fei Wang <fwang12@ebay.com> |
||
|
gabrywu
|
a6e14ac38d |
[KYUUBI #2560] Upgrade kyuubi-hive-jdbc hive version to 3.1.3
### Why are the changes needed? upgrade hive to 3.1.3 in the kyuubi-hive-jdbc module ### change details file|comment --- | --- kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/HttpBasicAuthInterceptor.java | add customCookies as a construct parameter kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/HttpKerberosRequestInterceptor.java | replace assumeSubject field with `Subject loggedInSubject` ; add customCookies as a construct parameter kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/HttpRequestInterceptorBase.java | add customCookies as a construct parameter; add customCookies initialization codes kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/HttpTokenAuthInterceptor.java | add customCookies as a construct parameter kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/JdbcColumn.java | support TIMESTAMP_WITH_TIMEZONE & UNION_TYPE type kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiBaseResultSet.java | support TIMESTAMPLOCALTZ_TYPE kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiConnection.java | add loggedInSubject,wmPool & wmApp fields and their initialization codes; add getAllUrls function; use Utils.getCanonicalHostName to extract host from connParams; clear hive conf in embedded mode; use SessionUtils.getTokenStrForm to get token string; remove isZkDynamicDiscoveryMode kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiDatabaseMetaData.java | support getSQLKeywords; add Object class type parameter to KyuubiMetaDataResultSet kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiMetaDataResultSet.java | optimize initialization logic of data & columnTypes field; add normalizeColumnNames function kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiStatement.java | split closeClientOperation into 2 functions(closeStatementIfNeeded & closeClientOperation) & add reInitState function to replace initFlags & ad error messages to exceptions in CANCELED_STATE kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/Utils.java | add dummyAuthorityString, SERVICE_DISCOVERY_MODE_ZOOKEEPER_HA, ZOOKEEPER_ACTIVE_PASSIVE_HA_DEFAULT_NAMESPACE, WM_POOL, HTTP_COOKIE_PREFIX, APPLICATION fields; replace authorityList field with suppliedURLAuthority in JdbcConnectionParams;add a construct function to JdbcConnectionParams; replace getAuthorityList function with getSuppliedURLAuthority; split parseURL into 2 different functions(parseURL, extractURLComponents); add configureConnParamsFromZooKeeper, handleAllDeprecations, getAuthorityFromJdbcURL functions; remove configureConnParams;remove joinStringArray;add getCanonicalHostName function kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/ZooKeeperHiveClientHelper.java | add more functions kyuubi-server/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier | change package name of DelegationTokenIdentifier kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HiveDelegationTokenProvider.scala | change package name of DelegationTokenIdentifier kyuubi-server/src/test/scala/org/apache/kyuubi/credentials/HiveDelegationTokenProviderSuite.scala | add configs, METASTORE_CONNECTION_POOLING_TYPE = none,METASTORE_AUTO_CREATE_ALL = true, METASTORE_SCHEMA_VERIFICATION = false; add construct parameter clientConf to HadoopThriftAuthBridgeWithServerContextClassLoader kyuubi-common/src/test/scala/org/apache/kyuubi/operation/SparkMetadataTests.scala | remove getSQLKeywords from unsupported list externals/kyuubi-hive-sql-engine/pom.xml | add HikariCP to test scope pom.xml | update hive version to 3.1.3 kyuubi-hive-jdbc-shaded/pom.xml | add hive-standalone-metastore dependency; add hive-llap-client dependency kyuubi-hive-jdbc/pom.xml | add hive-standalone-metastore dependency;add hive-llap-client dependency kyuubi-server/pom.xml | add hive-standalone-metastore dependency dev/dependencyList | add hive-standalone-metastore kyuubi-hive-jdbc-shaded/src/main/resources/META-INF/LICENSE | update content because of hive-standalone-metastore & hive-llap-client kyuubi-hive-jdbc-shaded/src/main/resources/META-INF/NOTICE | update content because of hive-standalone-metastore & hive-llap-client ### How was this patch tested? use existing UTs to test this patch Closes #2560 from gabrywu/jdbc-3.1. Closes #2560 9c5e3eda [gabry.wu] Merge branch 'master' into jdbc-3.1 c23cefc5 [gabrywu] update NOTICE rollback unnecessary change b9394144 [gabrywu] change hikaricp.version position to satisfy alphabetical order 59ebed37 [gabrywu] add hive-storage-api explicitly d22dc585 [gabrywu] rollback to minimize diff of this PR 33c13f0f [gabrywu] update LICENSE, NOTICE, LICENSE add hive-llap-client to kyuubi-hive-jdbc-shaded module d1ce2c6c [gabrywu] manage hikaricp.version under dependencyManagement 1f4b7c72 [gabrywu] Merge remote-tracking branch 'origin/jdbc-3.1' into jdbc-3.1 fd309054 [gabrywu] manage hikaricp.version under dependencyManagement 39aa7ae6 [gabrywu] manage hikaricp.version under dependencyManagement 85dc983a [gabrywu] manage hikaricp.version under dependencyManagement 545578c5 [gabrywu] update dependencyList file f479f642 [gabrywu] remove optional flag of hive-standalone-metastore in kyuubi-server 5ee8f89b [gabrywu] 1. add hive-storage-api 2. enable METASTORE_AUTO_CREATE_ALL 3. disable METASTORE_SCHEMA_VERIFICATION f70b0354 [gabrywu] add HikariCP to test scope c93ab167 [gabrywu] set METASTORE_CONNECTION_POOLING_TYPE to none feb53d6e [gabrywu] support getSQLKeywords func 4601b9b9 [gabrywu] exclude package.jdo 4f4a19bf [gabrywu] upgrade hive version to 3.3.1 in dependencyList 0b88a77a [gabrywu] upgrade hive to 3.1.3 Lead-authored-by: gabrywu <gabrywu@apache.org> Co-authored-by: gabry.wu <gabrywu@apache.org> Signed-off-by: Fei Wang <fwang12@ebay.com> |
||
|
Kent Yao
|
03b72268e8
|
[KYUUBI #2280] [INFRA] Replace BSD 3-clause with ASF License v2 for scala binaries
### _Why are the changes needed?_ Scala distribution is released under the ASF License v2 since 2.12.x, while 2.11.x is BSD 3-clause https://www.scala-lang.org/download/2.11.12.html https://www.scala-lang.org/download/2.13.8.html https://www.scala-lang.org/download/2.12.15.html ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2280 from yaooqinn/scalal. Closes #2280 9e0ff440 [Kent Yao] rm licenses-binary/LICENSE-scala.txt 720f5966 [Kent Yao] [INFRA] Replace BSD 3-clause with ASF License v2 for scala binaries Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
zwangsheng
|
0fd4738146
|
[KYUUBI #2246] [BUILD] Pick Commons-Logging dependence out of Hudi-Common
### _Why are the changes needed?_
```
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client-minicluster</artifactId>
<scope>test</scope>
</dependency>
```
`hadoop-client-minicluster` need `commons-logging` dependence. Now it works because `hudi-spark-common` include this dependency.
We extract it to better manage dependencies.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes #2246 from zwangsheng/fix/pick_commons_logging_dependice.
Closes #2246
5e75b964 [zwangsheng] update LICENSE
76913f15 [zwangsheng] recover unnecessary change
5fb2f3a6 [zwangsheng] rebuild dependency list
2979a3cd [zwangsheng] fix
42ae31c5 [zwangsheng] remove common-logging
d0d6401e [zwangsheng] replace
ff306b3f [zwangsheng] Pick Commons-Logging Dependices Out of Hudi
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
|
||
|
Cheng Pan
|
33424a1b47
|
[KYUUBI #2115] Update license and enhance collect_licenses script
### _Why are the changes needed?_ This PR updates the LICENSE and NOTICE files for both source release and binary release, and enhances the `collect_licenses.sh` script to support 1)collect `NOTICE*`(previous only `NOTICE`) files, 2) remove redundant or duplicated text from final NOTICE This PR also fix the LICENSE and NOTICE files for shaded modules ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2115 from pan3793/license. Closes #2115 e515eaaf [Cheng Pan] nit 53052ee6 [Cheng Pan] License for static resource b14837fe [Cheng Pan] Remove CCO 2fe01d58 [Cheng Pan] nit cb617e15 [Cheng Pan] log conf 7aaf2dbf [Cheng Pan] Update LICENSE 412a04f4 [Cheng Pan] Fix kyuubi-version-info.properties 0ed01fa7 [Cheng Pan] recover MANIFEST.MF f0a5d192 [Cheng Pan] Update LICENSE for shade modules 407b9ddb [Cheng Pan] python3 830c7a92 [Cheng Pan] Revert "Setup python3" 239890da [Cheng Pan] Revert "setup-python@v3" b0b27664 [Cheng Pan] setup-python@v3 3487cd45 [Cheng Pan] Setup python3 82ae8f01 [Cheng Pan] Revert "Update Dockerfile install python3" 4df10c7c [Cheng Pan] Update Dockerfile install python3 bbd87a43 [Cheng Pan] niy 152d2d03 [Cheng Pan] nit 85e89dcf [Cheng Pan] update license df891a63 [Cheng Pan] Revert "Clean LICENSE" 09dcba9f [Cheng Pan] enhance collect_license.sh acb624c9 [Cheng Pan] enhance collect-licenses.sh 7e0d5657 [Cheng Pan] Clean LICENSE 8eb5e356 [Cheng Pan] [LICENSE] Update LICENSE-binary Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
SteNicholas
|
463f851351
|
[KYUUBI #1850] Bump Prometheus 0.14.1
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ Bump Prometheus 0.14.1. There are some breaking changes, please go to https://github.com/prometheus/client_java/releases to learn details. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1857 from SteNicholas/KYUUBI-1850. Closes #1850 41c6f682 [SteNicholas] [KYUUBI #1850] Bump Prometheus 0.14.1 Authored-by: SteNicholas <programgeek@163.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Fei Wang
|
885ace06c8 |
[KYUUBI #1769][FOLLOWUP] Some cleanups for log4j properties
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This is a followup of #1769. Cleanup the log4j properties files and using log4j2.properties for UT. ### _How was this patch tested?_ Passed UT. Closes #1842 from turboFei/remove_log4j1. Closes #1769 b6757555 [Fei Wang] unused change 365fbec7 [Fei Wang] revert license 14c64ec9 [Fei Wang] exclude log4j 1.2.17 in slf4j-log4j12 034b04db [Fei Wang] recover jcl-over-slf4j and slf4j-api for k8s it 2beae75e [Fei Wang] remove log4j 1.2.17 b00f07b5 [Fei Wang] remove from license e55fd2ec [Fei Wang] remove unused dependencies ab86f023 [Fei Wang] [KYUUBI #1769][FOLLOWUP] Some cleanups for log4j properties Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: Fei Wang <fwang12@ebay.com> |
||
|
Kent Yao
|
d1de9ca529
|
[KYUUBI #1658] Revamp swagger ui dependencies
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This PR mainly removes the swagger-scala-module which is currently fine to remove. It also introduces transitive deps with `-jakarta` which could introduce jar conflicts like , ```scala Caused by: java.lang.NoClassDefFoundError: jakarta/xml/bind/annotation/XmlElement at io.swagger.v3.core.jackson.SwaggerAnnotationIntrospector.hasRequiredMarker(SwaggerAnnotationIntrospector.java:29) at com.fasterxml.jackson.databind.introspect.AnnotationIntrospectorPair.hasRequiredMarker(AnnotationIntrospectorPair.java:319) at com.fasterxml.jackson.databind.introspect.AnnotationIntrospectorPair.hasRequiredMarker(AnnotationIntrospectorPair.java:319) at com.fasterxml.jackson.databind.introspect.AnnotationIntrospectorPair.hasRequiredMarker(AnnotationIntrospectorPair.java:319) at com.fasterxml.jackson.databind.introspect.POJOPropertyBuilder.getMetadata(POJOPropertyBuilder.java:229) ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1658 from yaooqinn/dep. Closes #1658 d5ab0765 [Kent Yao] Merge branch 'master' into dep d1ddd5fd [Kent Yao] Revamp swagger ui dependencies ecfcc92a [Kent Yao] Revamp swagger ui dependencies 59196b94 [Kent Yao] Revamp swagger ui dependencies d7ca6a5f [Kent Yao] Revamp swagger ui dependencies 215b1b53 [Kent Yao] Revamp swagger ui dependencies Authored-by: Kent Yao <yao@apache.org> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
sunfangbin
|
fb8bf49dbc
|
[KYUUBI #1413] Improve usage of zookeeper cli
### _Why are the changes needed?_ This change enabled the JLine support so that the `kyuubi-zookeeper-shell` command can be easier to use with history, prompt and completion, etc. Note that the `jline` version `0.9.94` is dependent on `zookeeper` version `3.4.14` which is a lower version. BTW, users can consider upgrading the `zookeeper` version for the sake of performance improvements and new features. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1413 from murong00/branch-1413. Closes #1413 2d1ec084 [sunfangbin] Rename shell script to zk-cli d09f64dc [sunfangbin] Fixes failed ci ce88357a [sunfangbin] Improve usage of zookeeper cli Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Fu Chen
|
a5ef94accc
|
[KYUUBI #1354] Expose OpenAPI Specifications
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. add a new endpoint `/openapi.json` which describes the entire restful API of Kyuubi. 2. add a static web page `/api/v1/swagger-ui`. see [swagger-ui](https://github.com/swagger-api/swagger-ui) ``` curl http://localhost:10099/openapi.json | jq . ``` ```json { "openapi": "3.0.1", "info": { "title": "Apache Kyuubi RESTful Api Documentation", "description": "Apache Kyuubi RESTful Api Documentation", "contact": { "name": "Apache Kyuubi Community", "url": "https://kyuubi.apache.org/issue_tracking.html", "email": "devkyuubi.apache.org" }, "license": { "name": "Apache 2.0", "url": "https://www.apache.org/licenses/LICENSE-2.0.html" }, "version": "1.4.0" }, "tags": [ { "name": "Session" } ], "paths": { "/api/v1/ping": { "get": { "operationId": "ping", "responses": { "default": { "description": "default response", "content": { "text/plain": { "schema": { "type": "string" } } } } } } }, "/api/v1/sessions/{sessionHandle}": { "get": { "tags": [ "Session" ], "operationId": "sessionInfo", "parameters": [ { "name": "sessionHandle", "in": "path", "required": true, "schema": { "type": "string" } } ], "responses": { "200": { "description": "get a session via session handle identifier", "content": { "application/json": {} } } } }, "delete": { "tags": [ "Session" ], "operationId": "closeSession", "parameters": [ { "name": "sessionHandle", "in": "path", "required": true, "schema": { "type": "string" } } ], "responses": { "200": { "description": "close a Session", "content": { "application/json": {} } } } } }, "/api/v1/sessions/execPool/statistic": { "get": { "tags": [ "Session" ], "operationId": "execPoolStatistic", "responses": { "200": { "description": "get some statistic info of sessions", "content": { "application/json": {} } } } } }, "/api/v1/sessions/{sessionHandle}/info/{infoType}": { "get": { "tags": [ "Session" ], "operationId": "getInfo", "parameters": [ { "name": "sessionHandle", "in": "path", "required": true, "schema": { "type": "string" } }, { "name": "infoType", "in": "path", "required": true, "schema": { "type": "integer", "format": "int32" } } ], "responses": { "200": { "description": "get a information detail via session handle identifier and a specific information type", "content": { "application/json": {} } } } } }, "/api/v1/sessions": { "get": { "tags": [ "Session" ], "operationId": "sessionInfoList", "responses": { "200": { "description": "get all the session list hosted in SessionManager", "content": { "application/json": {} } } } }, "post": { "tags": [ "Session" ], "operationId": "openSession", "requestBody": { "content": { "application/json": { "schema": { "$ref": "#/components/schemas/SessionOpenRequest" } } } }, "responses": { "200": { "description": "Open(create) a Session", "content": { "application/json": {} } } } } }, "/api/v1/sessions/count": { "get": { "tags": [ "Session" ], "operationId": "sessionCount", "responses": { "200": { "description": "get open session count", "content": { "application/json": {} } } } } }, "/api/v1/exception": { "get": { "operationId": "test", "responses": { "default": { "description": "default response", "content": { "text/plain": {} } } } } }, "/application.wadl/{path}": { "get": { "operationId": "getExternalGrammar", "parameters": [ { "name": "path", "in": "path", "required": true, "schema": { "type": "string" } } ], "responses": { "default": { "description": "default response", "content": { "application/xml": {} } } } } }, "/application.wadl": { "get": { "operationId": "getWadl", "responses": { "default": { "description": "default response", "content": { "application/vnd.sun.wadl+xml": {}, "application/xml": {} } } } } } }, "components": { "schemas": { "ExecPoolStatistic": { "required": [ "execPoolSize" ], "type": "object", "properties": { "execPoolSize": { "type": "integer", "format": "int32" }, "execPoolActiveCount": { "type": "integer", "format": "int32" } } }, "InfoDetail": { "required": [ "infoType" ], "type": "object", "properties": { "infoType": { "type": "string" }, "infoValue": { "type": "string" } } }, "HandleIdentifier": { "required": [ "publicId" ], "type": "object", "properties": { "publicId": { "type": "string", "format": "uuid" }, "secretId": { "type": "string", "format": "uuid" } } }, "SessionHandle": { "required": [ "identifier", "protocol" ], "type": "object", "properties": { "identifier": { "$ref": "#/components/schemas/HandleIdentifier" }, "protocol": { "type": "string", "enum": [ "HIVE_CLI_SERVICE_PROTOCOL_V1", "HIVE_CLI_SERVICE_PROTOCOL_V2", "HIVE_CLI_SERVICE_PROTOCOL_V3", "HIVE_CLI_SERVICE_PROTOCOL_V4", "HIVE_CLI_SERVICE_PROTOCOL_V5", "HIVE_CLI_SERVICE_PROTOCOL_V6", "HIVE_CLI_SERVICE_PROTOCOL_V7", "HIVE_CLI_SERVICE_PROTOCOL_V8", "HIVE_CLI_SERVICE_PROTOCOL_V9", "HIVE_CLI_SERVICE_PROTOCOL_V10" ] } } }, "MapStringString": { "type": "object", "additionalProperties": { "type": "string" } }, "SessionOpenRequest": { "required": [ "configs", "protocolVersion", "user" ], "type": "object", "properties": { "protocolVersion": { "type": "integer", "format": "int32" }, "user": { "type": "string" }, "password": { "type": "string" }, "ipAddr": { "type": "string" }, "configs": { "type": "object", "additionalProperties": { "type": "string" } } } }, "SessionOpenCount": { "required": [ "openSessionCount" ], "type": "object", "properties": { "openSessionCount": { "type": "integer", "format": "int32" } } }, "SessionDetail": { "required": [ "configs", "createTime", "user" ], "type": "object", "properties": { "user": { "type": "string" }, "ipAddr": { "type": "string" }, "createTime": { "type": "integer", "format": "int64" }, "sessionHandle": { "$ref": "#/components/schemas/SessionHandle" }, "lastAccessTime": { "type": "integer", "format": "int64" }, "lastIdleTime": { "type": "integer", "format": "int64" }, "noOperationTime": { "type": "integer", "format": "int64" }, "configs": { "type": "object", "additionalProperties": { "type": "string" } } } }, "SeqSessionOverview": { "type": "array", "items": { "$ref": "#/components/schemas/SessionOverview" } }, "SessionList": { "required": [ "sessionList" ], "type": "object", "properties": { "sessionList": { "type": "array", "items": { "$ref": "#/components/schemas/SessionOverview" } } } }, "SessionOverview": { "required": [ "createTime", "user" ], "type": "object", "properties": { "user": { "type": "string" }, "ipAddr": { "type": "string" }, "createTime": { "type": "integer", "format": "int64" }, "sessionHandle": { "$ref": "#/components/schemas/SessionHandle" } } } } } } ``` http://localhost:10099/api/v1/swagger-ui > Note that this request will redirect to http://localhost:10099/swagger-ui-redirected/index.html?url=http://localhost:10099/openapi.json  ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1363 from cfmcgrady/kyuubi-1354. Closes #1354 8895d7fc [Fu Chen] typo 8f5dea9a [Fu Chen] addressed comment 4d325a45 [Fu Chen] constant MediaType.APPLICATION_JSON 0d96951e [Fu Chen] RESTful Api -> REST API be7d02dc [Fu Chen] add LIABILITY-edl.txt 4ff1053e [Fu Chen] update dev/dependencyList b126f7e6 [Fu Chen] swagger-ui 4.1.0 a35c2f29 [Fu Chen] addressed comment 7bde0534 [Fu Chen] update LICENSE-binary 38397dc2 [Fu Chen] align jackson version 923f3e7e [Fu Chen] update dev/dependencyList 7d5338c8 [Fu Chen] remove kyuubi-server static files 5cb66b33 [Fu Chen] update dev/dependencyList e8a8c421 [Fu Chen] fix style 5b13f69c [Fu Chen] add license and remove .map files c04d7a75 [Fu Chen] rebase master 2388dae8 [Fu Chen] fix style 8dcbd3db [Fu Chen] add static resource 4b0d6d0a [Fu Chen] expose OpenAPI Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
c122a23a0d
|
[KYUUBI #1348] [DEPS] Add netty and mysql jdbc driver to kyuubi-server module
### _Why are the changes needed?_ This PR is one part of #1334. Add netty to `kyuubi-server` module compile scope, and mysql jdbc driver to test scope, it's required for implementing MySQL frontend protocol ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1348 from pan3793/mysql-fe-deps. Closes #1348 288ad9c0 [Cheng Pan] [DEPS] Add netty to kyuubi-server module Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
3ebb50f1cb
|
[KYUUBI #978] [LICENSE] Fix the source license issue
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Fix the license issue mentioned by WillemJiang ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request The final `NOTICE` for binary release is not `NOTICE-binary` in root path but is generated by `build/release/collect_licenses.sh`, which will extract all NOTICE content from the bundled jars. ``` Apache Kyuubi Copyright 2021 and onwards The Apache Software Foundation. This product includes software developed at The Apache Software Foundation (https://www.apache.org/). Apache Spark Copyright 2014 and onwards The Apache Software Foundation. This product includes software developed at The Apache Software Foundation (http://www.apache.org/). Export Control Notice --------------------- This distribution includes cryptographic software. The country in which you currently reside may have restrictions on the import, possession, use, and/or re-export to another country, of encryption software. BEFORE using any encryption software, please check your country's laws, regulations and policies concerning the import, possession, or use, and re-export of encryption software, to see if this is permitted. See <http://www.wassenaar.org/> for more information. The U.S. Government Department of Commerce, Bureau of Industry and Security (BIS), has classified this software as Export Commodity Control Number (ECCN) 5D002.C.1, which includes information security software using or performing cryptographic functions with asymmetric algorithms. The form and manner of this Apache Software Foundation distribution makes it eligible for export under the License Exception ENC Technology Software Unrestricted (TSU) exception (see the BIS Export Administration Regulations, Section 740.13) for both object code and source code. The following provides more details on the included cryptographic software: This software uses Apache Commons Crypto (https://commons.apache.org/proper/commons-crypto/) to support authentication, and encryption and decryption of data sent across the network between services. Metrics Copyright 2010-2013 Coda Hale and Yammer, Inc. This product includes software developed by Coda Hale and Yammer, Inc. This product includes code derived from the JSR-166 project (ThreadLocalRandom, Striped64, LongAdder), which was released with the following comments: Written by Doug Lea with assistance from members of JCP JSR-166 Expert Group and released to the public domain, as explained at http://creativecommons.org/publicdomain/zero/1.0/ Curator Client Copyright 2011-2017 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/). Curator Framework Copyright 2011-2017 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/). Curator Recipes Copyright 2011-2017 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/). Hive Service RPC Copyright 2020 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/). htrace-core4 Copyright 2016 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/). # Jackson JSON processor Jackson is a high-performance, Free/Open Source JSON processing library. It was originally written by Tatu Saloranta (tatu.salorantaiki.fi), and has been in development since 2007. It is currently developed by a community of developers, as well as supported commercially by FasterXML.com. ## Licensing Jackson core and extension components may licensed under different licenses. To find the details that apply to this artifact see the accompanying LICENSE file. For more information, including possible other licensing options, contact FasterXML.com (http://fasterxml.com). ## Credits A list of contributors may be found from CREDITS file, which is included in some artifacts (usually source distributions); but is always available from the source code management (SCM) system project uses. # Jackson JSON processor Jackson is a high-performance, Free/Open Source JSON processing library. It was originally written by Tatu Saloranta (tatu.salorantaiki.fi), and has been in development since 2007. It is currently developed by a community of developers, as well as supported commercially by FasterXML.com. ## Licensing Jackson core and extension components may be licensed under different licenses. To find the details that apply to this artifact see the accompanying LICENSE file. For more information, including possible other licensing options, contact FasterXML.com (http://fasterxml.com). ## Credits A list of contributors may be found from CREDITS file, which is included in some artifacts (usually source distributions); but is always available from the source code management (SCM) system project uses. Apache log4j Copyright 2007 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/). ``` Closes #978 from pan3793/license. Closes #978 2a7eff44 [Cheng Pan] Update License d21825dd [Cheng Pan] [LICENSE] Fix the source license issue Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
timothy65535
|
5526898f7b |
[KYUUBI #894] Enhance kyuubi-zookeeper module
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### 1. Describe the feature Kyuubi use `TestingServer` as the EmbeddedZookeeper service in codebase, and require many args to init the service. These `electionPort`, `quorumPort`, `serverId` parameters only work in cluster mode, these ports are not bound to the host. From hadoop and bookkeeper project, they use `ZooKeeperServer` as a built-in zk service, especially the bookeeper project. - https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/server/services/MicroZookeeperService.java - https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/shims/zk/ZooKeeperServerShimImpl.java ### 2. Describe the solution Improve - Remove `electionPort`, `quorumPort`, `serverId` which are nerver be used, hadoop and bookkeer do the same - Replace `TestingServer` with `ZooKeeperServer` - Remove `curator-test` dependency ### 3. `deleteDataDirectoryOnClose` The name style for `deleteDataDirectoryOnClose` option, refer to hadoop `core-default.xml` https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/core-default.xml ``` hadoop.security.dns.log-slow-lookups.enabled hadoop.security.dns.log-slow-lookups.threshold.ms dfs.client.block.write.replace-datanode-on-failure.policy ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #907 from timothy65535/ky-894. Closes #894 7d7fde9f [timothy65535] remove useless config ce06a0fd [timothy65535] limit travis build only master branch ff825c3f [timothy65535] revert deleteDataDirectoryOnClose config a336544a [timothy65535] remove javassist 3da0bd55 [timothy65535] remove javassist 0abb44df [timothy65535] [KYUUBI #894] Enhance kyuubi-zookeeper module Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
