3a765079db
729 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Cheng Pan
|
4390cc6fa4
|
[KYUUBI #6058] Make Jetty server stop timeout configurable
# 🔍 Description ## Issue References 🔗 This pull request aims to make the Jetty server stop timeout configurable. The RESTful frontend protocol has been enabled by default since 1.8.0, while a side-effect is observed, the process shutdown is always blocked by the Jetty server until reaches the default 30s timeout. ## Describe Your Solution 🔧 Two configurations are introduced in this PR ``` kyuubi.frontend.trino.jetty.stopTimeout=5s kyuubi.frontend.rest.jetty.stopTimeout=5s ``` while the 5s is chosen as the default value according to https://git.eclipse.org/c/jetty/org.eclipse.jetty.project.git/tree/jetty-server/src/main/config/etc/jetty.xml?h=jetty-9.3.x#n124 ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 1. perform `bin/kyuubi run` 2. press `ctrl + c` to shutdown the Kyuubi process #### Behavior Without This Pull Request ⚰️ takes ~30s to complete the shutdown progress. #### Behavior With This Pull Request 🎉 takes ~5s to complete the shutdown progress. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6058 from pan3793/jetty-timeout. Closes #6058 47d15f9aa [Cheng Pan] fix cbac2b13e [Cheng Pan] Make Jetty server stop timeout configurable Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Fei Wang
|
8c3f471ae3 |
[KYUUBI #5952] Disconnect connections without running operations after engine maxlife time graceful period
# 🔍 Description ## Issue References 🔗 We found that, some kyuubi connections(maybe managed by jdbc connection pool likes hikari) always keep alive, and the engine can not be terminated after exceeds the max life time. So, In this pr, I introduce a graceful period after spark engine max life time, after the graceful period, the connections without running operations will be disconnected forcibly. Close #5952 ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6040 from turboFei/close_on_nooperation_rebase. Closes #5952 0b05ddc0c [Fei Wang] comments 4329a85cf [Fei Wang] async stop b39c9b3a0 [Fei Wang] use short sleep 7c123f612 [Fei Wang] save 45ad3489d [Fei Wang] check no running operation Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: Fei Wang <fwang12@ebay.com> |
||
|
Tigran Manasyan
|
35a910a43e
|
[KYUUBI #5674] Add PostgreSQL as backend database for kyuubi metadata
# 🔍 Description ## Issue References 🔗 This pull request fixes #5674 ## Describe Your Solution 🔧 Currently, Kyuubi supports the use of SQLite and MySQL to store metadata. Although PostgreSQL can be supported through the 'CUSTOM' method, there is a certain usage cost for users, therefore this MR adds support for PostgreSQL as a backend database for kyuubi metadata. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests `org.apache.kyuubi.server.metadata.jdbc.JDBCMetadataStoreSuite` --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6027 from tigrulya-exe/feature/5674-postgresql-metadata-backend. Closes #5674 111658002 [Tigran Manasyan] Fix metastore db type option comment 44a22bfcf [Tigran Manasyan] Fix dependency list and metastore db type option comment b638c8942 [Tigran Manasyan] Add PostgreSQL as backend database for kyuubi metadata Authored-by: Tigran Manasyan <t.manasyan@arenadata.io> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
3f993f405a
|
[KYUUBI #6003] Allow disabling user impersonation on launching engine
# 🔍 Description ## Issue References 🔗 HiveServer2 has a configuration `hive.server2.enable.doAs` to control the execution user between the session user and the server user, Kyuubi's CONNECTION and USER share levels always perform like doAs enabled do. In CDH 5/6, this is disabled by default, users who want to migrate from CDH to Kyuubi may encounter permission issues with the current implementation. ## Describe Your Solution 🔧 This pull request introduces a new configuration `kyuubi.engine.doAs.enabled` to allow enable/disable user impersonation on launching engine. For security purpose, it's not allowed to be overridden by session conf. The change in this PR has certain limitations: - only supports Spark engine - only supports interactive mode, specifically, it does not take effect on Spark batch mode now. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 The first step is passing all existing UTs when `kyuubi.engine.doAs.enabled=true`. Tested on internal Kerberized-environment, when `kyuubi.engine.share.level=CONNECTION` and `kyuubi.engine.doAs.enabled=false`, use user 'spark' to launch engine, and the engine submitted without `--proxy-user spark`, thus engine launched by server user `hive`, then run `select session_user(), current_user()` and returns ``` +-----------------+-----------------+ | session_user() | current_user() | +-----------------+-----------------+ | spark | hive | +-----------------+-----------------+ ``` And I checked the `spark.app.name` and registered path on Zookeeper also expected. ``` +-----------------+--------------------------------------------------------------------------+ | key | value | +-----------------+--------------------------------------------------------------------------+ | spark.app.name | kyuubi_USER_SPARK_SQL_spark_default_51a416e5-6023-4bac-a964-cd9605f17c61 | +-----------------+--------------------------------------------------------------------------+ ``` --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6003 from pan3793/doas. Closes #6003 c4002fef5 [Cheng Pan] grammar add20fd57 [Cheng Pan] nit 8711c2265 [Cheng Pan] address comment 033a32252 [Cheng Pan] 1.9.0 9273b9426 [Cheng Pan] fix a1563e1ca [Cheng Pan] HadoopCredentialsManager e982e2364 [Cheng Pan] Allow disable user impersonation on launching engine Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
d474768d97
|
[KYUUBI #6018] Speed up GetTables operation for Spark session catalog
# 🔍 Description ## Issue References 🔗 This pull request aims to speed up the GetTables operation for the Spark session catalog. As reported in https://github.com/apache/kyuubi/discussions/4956, https://github.com/apache/kyuubi/discussions/5949, the GetTables operation is quite slow in some cases, and in https://github.com/apache/kyuubi/pull/4444, `kyuubi.operation.getTables.ignoreTableProperties` was introduced to speed up the V2 catalog, but not covers session catalog. ## Describe Your Solution 🔧 Extend the scope of `kyuubi.operation.getTables.ignoreTableProperties` to cover the GetTables operation for the Spark session catalog. Currently, the basic step of GetTables in the Spark engine is ``` val catalog: String = getCatalog(spark, catalogName) val databases: Seq[String] = sessionCatalog.listDatabases(schemaPattern) val identifiers: Seq[TableIdentifier] = catalog.listTables(db, tablePattern, includeLocalTempViews = false) val tableObjects: Seq[CatalogTable] = catalog.getTablesByName(identifiers) ``` then filter `tableObjects` with `tableTypes: Set[String]`. The cost of `catalog.getTablesByName(identifiers)` is quite high when the table number is large, e.g. dozen thousand. For some cases, listing tables only for table name display, it is worth speeding up the operation while ignoring some properties(e.g. table comments) and query criteria(specifically in this case, when `kyuubi.operation.getTables.ignoreTableProperties=true`, criteria `tableTypes` will be ignored, and all tables and views will be treated as TABLE to return.) ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Pass GA --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6018 from pan3793/fast-get-table. Closes #6018 058001c6f [Cheng Pan] fix 405b12484 [Cheng Pan] fix 615b7470f [Cheng Pan] Speed up GetTables operation Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
yuqingyang
|
47a1091fa8
|
[KYUUBI #4279] Use new Apache 'closer.lua' syntax for kyuubi-download to obtain engine
…ne archives # 🔍 Description ## Issue References [4279](https://github.com/apache/kyuubi/issues/4279)🔗 This pull request fixes # ## Describe Your Solution 🔧 Use new Apache 'closer.lua' syntax for kyuubi-download to obtain engine ## Types of changes 🔖 - [x] New feature (non-breaking change which adds functionality) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6009 from sincewhen/master. Closes #4279 2152a9b92 [yuqingyang] Use new Apache 'closer.lua' syntax for kyuubi-download to obtain engine archives Authored-by: yuqingyang <yuqingyang@kingsoft.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
huangxiaoping
|
083f3c1e29
|
[KYUUBI #6010] Improve Gluten usage documentation
# 🔍 Description ## Issue References 🔗 This pull request fixes #6010 ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6011 from huangxiaopingRD/gluten-doc. Closes #6010 88bce5865 [huangxiaoping] [KYUUBI #6010] Improve Gluten usage documentation Authored-by: huangxiaoping <1754789345@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
senmiaoliu
|
777b784439
|
[KYUUBI #5968] Support set authentication user for Trino engine
# 🔍 Description ## Issue References 🔗 This pull request fixes #5968 ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #5970 from lsm1/branch-kyuubi-5968. Closes #5968 580eb45d2 [senmiaoliu] fix style a36380cde [senmiaoliu] add auth user conf for trino engine Authored-by: senmiaoliu <senmiaoliu@trip.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
swordy_zhao
|
750a9173c8
|
[KYUUBI #6000] Modify the incorrect configuration file in the trino-cli documentation
# 🔍 Description ## Issue References 🔗 This pull request fixes # ## Describe Your Solution 🔧 kyuubi conf file is kyuubi-defaults.conf, not kyuubi.conf ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6000 from SwordyZhao/trino-cli-doc-cheage. Closes #6000 95b839ada [swordy_zhao] kyuubi conf file is kyuubi-defaults.conf Authored-by: swordy_zhao <swordy_work@163.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
3b2e674364
|
[KYUUBI #5990] Always take the first declared SASL/PLAIN auth type
# 🔍 Description ## Issue References 🔗 https://github.com/apache/kyuubi/pull/5185 changed the type of `kyuubi.authentication` from `Seq`(ordered) to `Set`(unordered), which break the assumption of ``` Note that: for SASL authentication, KERBEROS and PLAIN auth types are supported at the same time, and only the first specified PLAIN auth type is valid. ``` ## Describe Your Solution 🔧 Restore the type to Seq ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 UT is updated --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #5990 from pan3793/auth-plain. Closes #5990 acae25f68 [Cheng Pan] fix doc cef7dba90 [Cheng Pan] fix 87b370ffe [Cheng Pan] Always take the first declared SASL/PLAIN auth type Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
87ed400562
|
[KYUUBI #5975] Bump trino-client 411
# 🔍 Description ## Issue References 🔗 This is a regular dependency upgrading, ## Describe Your Solution 🔧 Upgrade `trino-client` from 363 to 411. 411 is the latest version which uses okhttp 3.x, hence it does not have kotlin runtime dependencies. This PR also updates the docs, especially the Trino cluster version requirement. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Pass GA. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #5975 from pan3793/trino-411. Closes #5975 2b57df34d [Cheng Pan] fix c498a5bb3 [Cheng Pan] fix 21948ca4f [Cheng Pan] Fix compile e4f1397cc [Cheng Pan] license 66583ca16 [Cheng Pan] Bump trino-client 411 Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
yikaifei
|
518e918fa7
|
[KYUUBI #5867][FOLLOWUP] Remove unused config
# 🔍 Description ## Issue References 🔗 This pull request aims to remove unused config as #5867 followup. ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #5976 from Yikf/remove-INTERVAL. Closes #5867 e4183ffca [yikaifei] [KYUUBI #5867][FOLLOWUP] Remove unused config Authored-by: yikaifei <yikaifei@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
hezhao2
|
865a8b75c9
|
[KYUUBI #5951] [DOC] Improve doc for Session Conf Advisor
# 🔍 Description ## Issue References 🔗 This pull request is to improve doc for Session Conf Advisor ## Describe Your Solution 🔧 improve doc for Session Conf Advisor ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #5951 from zhaohehuhu/Improvement-0108. Closes #5951 9f2c66210 [hezhao2] Improve doc for Session Conf Advisor Authored-by: hezhao2 <hezhao2@cisco.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
zwangsheng
|
c6bba915b2
|
[KYUUBI #5941] Drop Kubernetes Block Cleaner Tool from Kyuubi
# 🔍 Description ## Issue References 🔗 This pull request fixes #5941 ## Describe Your Solution 🔧 Originally aims to support Spark On Kubernetes Shuffle Data Clean, limitations became more and more apparent over time, so let's drop this. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [x] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #5942 from zwangsheng/KYUUBI#5941. Closes #5941 23bf14f37 [Cheng Pan] Update docs/tools/spark_block_cleaner.md 1c3350186 [zwangsheng] fix comment 0bdbb1104 [zwangsheng] nit 0a5aa2bfa [zwangsheng] fix comments Lead-authored-by: zwangsheng <binjieyang@apache.org> Co-authored-by: Cheng Pan <pan3793@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
5d59cf1d6a
|
[KYUUBI #5867][FOLLOWUP] Fix generated configuration docs and typo
# 🔍 Description ## Issue References 🔗 This pull request is a follow-up of #5867, to fix generated configuration docs and typo to recover CI. ## Describe Your Solution 🔧 Regenerated docs by performing `dev/gen/gen_all_config_docs.sh` Address comment https://github.com/apache/kyuubi/pull/5868#discussion_r1438080698 ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Pass GA. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #5932 from pan3793/fix-docs. Closes #5867 93b8f97b0 [Cheng Pan] docs 0ad0d29f3 [Cheng Pan] typo Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
yikaifei
|
679aca5a6c
|
[KYUUBI #5867] HiveEngine support run on YARN mode
# 🔍 Description ## Issue References 🔗 This PR aims to support hive engine run on yarn mode, close https://github.com/apache/kyuubi/issues/5867 ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [ ] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [x] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5868 from Yikf/hive-on-yarn. Closes #5867 44f7287f5 [yikaifei] fix 3c17d2c4a [yikaifei] fix test 5474ebfba [yikaifei] parse classpath 6b97c4213 [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala 34a67b452 [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala 5e5045e66 [yikaifei] fix app type d1eb5aea7 [yikaifei] fix d89d09cfe [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala 1fa18ba1b [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/ApplicationMaster.scala 1b0b77f4d [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/ApplicationMaster.scala 2ed1d4492 [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala 98ff19ce6 [yikaifei] HiveEngine support run on YARN mode Lead-authored-by: yikaifei <yikaifei@apache.org> Co-authored-by: Cheng Pan <pan3793@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Kent Yao
|
c18dd61952
|
[KYUUBI #5916] Revert #5916
# 🔍 Description ## Issue References 🔗 This pull request fixes #5916 as a revert ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [x] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5928 from yaooqinn/revert. Closes #5916 5c43c0286 [Kent Yao] [KYUUBI #5916] Revert #5916 Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Bowen Liang
|
b1e726dcf5 |
[KYUUBI #5912] Cleanup unused config kyuubi.frontend.thrift.http.allow.user.substitution
# 🔍 Description ## Issue References 🔗 As decribed. ## Describe Your Solution 🔧 Drop the unused config of KyuubiConf, which are never used and linked in source code and test code: - FRONTEND_THRIFT_HTTP_ALLOW_USER_SUBSTITUTION: `kyuubi.frontend.thrift.http.allow.user.substitution` ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5912 from bowenliang123/unused-configs. Closes #5912 b938058b8 [Bowen Liang] update doc 9fd441dd0 [Bowen Liang] fix 9b0b31ee1 [Bowen Liang] cleanup unused config entries in KyuubiConf Authored-by: Bowen Liang <liangbowen@gf.com.cn> Signed-off-by: Bowen Liang <liangbowen@gf.com.cn> |
||
|
Bowen Liang
|
86bec4d914 |
[KYUUBI #5914] Update layer diagram on welcome page
# 🔍 Description ## Issue References 🔗 As described. ## Describe Your Solution 🔧 - replace `User Layer` with `User Scenarios` - realign the vertical margin of the logos in column - larger width for better displaying - correct StarRocks logo ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️  #### Behavior With This Pull Request 🎉  #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5914 from bowenliang123/layer-logo2. Closes #5914 238bc9d45 [Bowen Liang] update layer 77811b856 [Bowen Liang] update layer 7f613e536 [Bowen Liang] Update .gitignore a4b0b9514 [Bowen Liang] update 16ebe6e46 [Bowen Liang] update layer Authored-by: Bowen Liang <liangbowen@gf.com.cn> Signed-off-by: Bowen Liang <liangbowen@gf.com.cn> |
||
|
Bowen Liang
|
0f8edb2b2f |
[KYUUBI #5916] Use "(none)" placeholder for default value of un-preset configs on confing doc page
# 🔍 Description ## Issue References 🔗 As descirbed. ## Describe Your Solution 🔧 - Replace the placeholder for default value of un-preset configs on doc page, changed from `<undefined>` to `(none)` - The `(none)` placeholder style is also used in the docs of - Spark : https://spark.apache.org/docs/latest/configuration.html - Flink : https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/ - Trino (as `NONE`): https://trino.io/docs/current/admin/properties.html - Only affects the config doc page, no functional changes. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ No functional changes. Config doc page:  #### Behavior With This Pull Request 🎉 No functional changes. Config doc page:  #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5916 from bowenliang123/none-ph. Closes #5916 3b141dd61 [Bowen Liang] update test 04cf8cd2a [Bowen Liang] Use "(none)" placeholder for default value of unset configs on config page Authored-by: Bowen Liang <liangbowen@gf.com.cn> Signed-off-by: Bowen Liang <liangbowen@gf.com.cn> |
||
|
Cheng Pan
|
8f59355eb6
|
Revert "[KYUUBI #5908] [DOCS] Remove workaround for malformed table"
This reverts commit
|
||
|
Cheng Pan
|
97a0ba461c |
[KYUUBI #5908] [DOCS] Remove workaround for malformed table
# 🔍 Description ## Issue References 🔗 The `custom.css` was used to address the `sphinx-book-theme` bugs which generated malformed tables, see #1889 #2764 for details. With the recent upgrading of `sphinx-book-theme 1.1.0`, it's unnecessary. ## Describe Your Solution 🔧 Remove useless `custom.css` ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ <img width="1376" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/1ef116f8-6be1-49ee-8f6e-3b844eae284a"> #### Behavior With This Pull Request 🎉 <img width="1376" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/b12e461b-6294-4a4e-b034-fbd093b10ca4"> #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5908 from pan3793/remove-custom-css. Closes #5908 a958b774c [Cheng Pan] nit a3f592645 [Cheng Pan] [DOCS] Remove workaround for malformed table Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: liangbowen <liangbowen@gf.com.cn> |
||
|
Bowen Liang
|
5a751027f5 |
[KYUUBI #5905] [DOC] Workaround for long inlined code sample in config page
# 🔍 Description ## Issue References 🔗 ## Describe Your Solution 🔧 Workaround to fix the display issue of long inlined code sample on config page. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️  #### Behavior With This Pull Request 🎉  #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [ ] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5905 from bowenliang123/doc-block. Closes #5905 b4d28ed85 [Bowen Liang] update 7aa995b38 [Bowen Liang] update ef947ea15 [Bowen Liang] update 7b3263bee [Bowen Liang] update aefd206ae [Bowen Liang] update f02ec3acf [Bowen Liang] update d3302d55c [Bowen Liang] fix display by putting url demo in code block Authored-by: Bowen Liang <liangbowen@gf.com.cn> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Fei Wang
|
fd397d6277 |
[KYUUBI #5900] Support to deregister the engine on open failure
# 🔍 Description ## Issue References 🔗 This pull request fixes #5900 ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [ ] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5901 from turboFei/delete_engine. Closes #5900 57bcf1f24 [Fei Wang] docs 557714eed [Fei Wang] enum bc479e6e0 [Fei Wang] save 2b2a4952c [Fei Wang] refine f0ace3574 [Fei Wang] reset after close client cf07877fe [Fei Wang] re-version d93dd6add [Fei Wang] refactor Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: Fei Wang <fwang12@ebay.com> |
||
|
Fei Wang
|
58db6ca225 |
[KYUUBI #5877][FOLLOWUP] Add spark output mode to better support PySpark notebook
# 🔍 Description ## Issue References 🔗 This pull request fixes # ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [ ] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5898 from turboFei/notebook_mode. Closes #5877 7f1c607b9 [Fei Wang] PySpark 644d036bc [Fei Wang] docs 7c68b7742 [Fei Wang] add option to support notebook well Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: Fei Wang <fwang12@ebay.com> |
||
|
Bowen Liang
|
0640a00e3c |
[KYUUBI #5893] [DOC] Update logos in layer picture on welcome page
# 🔍 Description ## Issue References 🔗 This pull request fixes # ## Describe Your Solution 🔧 - update the logos of DBeaver, Mesos and DeltaLake with white background - add StarRocks logo to the engine layer - add Apache Paimon(Incubating) logo to the storage layer - update JDBC and Python logo on driver layer - add REST API text on driver layer - remove outdated SQuirrelSQLClient logo - fix typo in `etc,` into `etc.,` ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️   #### Behavior With This Pull Request 🎉  <img width="750" alt="image" src="https://github.com/apache/kyuubi/assets/1935105/6d6b058d-b295-478f-a561-0581107c8206"> #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5893 from bowenliang123/layer-logo. Closes #5893 8b13ec65f [Bowen Liang] mentioning paimon ea4686d71 [liangbowen] update f1a9c1b1c [Bowen Liang] update 87c8b94e5 [Bowen Liang] update logos in layer Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn> Co-authored-by: liangbowen <liangbowen@gf.com.cn> Signed-off-by: Bowen Liang <liangbowen@gf.com.cn> |
||
|
Bowen Liang
|
feddca4523 |
[KYUUBI #5902] Bump doc build dependencies
# 🔍 Description ## Issue References 🔗 As descirbed. ## Describe Your Solution 🔧 - Bump pip dependencies' version of doc building tools - sphinx 4.5.0 -> 7.2.6 - sphinx-book-theme 0.3.3-> 1.1.0 - markdown 3.4.1-> 3.5.1 - Upgraded `sphinx-book-theme` brings - dark mode support (light/dark/system) - new search bar style - less text hits shown in searched results ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️   #### Behavior With This Pull Request 🎉    #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [ ] Milestone correctly set? - [x] Test coverage is ok - [ ] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5902 from bowenliang123/docdep-bump. Closes #5902 db9d8671a [Bowen Liang] bump doc build dependencies Authored-by: Bowen Liang <liangbowen@gf.com.cn> Signed-off-by: Fei Wang <fwang12@ebay.com> |
||
|
Cheng Pan
|
2aa330c850 |
[KYUUBI #5892] Metadata store JDBC url supports <KYUUBI_HOME> variable
# 🔍 Description ## Issue References 🔗 Similar to https://github.com/apache/kyuubi/pull/5412, this PR aims to change the default SQLite file `kyuubi_state_store.db` for Metadata store is located under `$KYUUBI_HOME` instead of `$PWD`. ## Describe Your Solution 🔧 Make `kyuubi.metadata.store.jdbc.url` supports the variables substitution: `<KYUUBI_HOME>`, and change the default value to `jdbc:sqlite:<KYUUBI_HOME>/kyuubi_state_store.db` ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Verified locally. Make a binary distribution, and bootstrap the Kyuubi Server process from a non-KYUUBI_HOME dir, `kyuubi_state_store.db` is created at KYUUBI_HOME as expected. --- # Checklists ## 📝 Author Self Checklist - [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5892 from pan3793/sqlite-jdbc-url. Closes #5892 4341f4263 [Cheng Pan] docs 7062bcdf9 [Cheng Pan] migration guide 989de0e5b [Cheng Pan] Metadata store JDBC url supports <KYUUBI_HOME> variable Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Fei Wang <fwang12@ebay.com> |
||
|
Bowen Liang
|
4be20e1ca7 |
[KYUUBI #5404] JDBC Engine supports StarRocks
# 🔍 Description ## Issue References 🔗 This pull request fixes #5404 ## Describe Your Solution 🔧 - Introduce StarRocks support in JDBC Engine - Adding dialects and tests for StarRocks - Tested with StarRocks 3.x official open-sourced version docker image, with a mini cluster with FE and BE setup. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 Supported StarRocks in JDBC engine. #### Related Unit Tests Add tests of `StarRocksOperationSuite`, `StarRocksOperationWithEngineSuite`, `StarRocksSessionSuite` and `StarRocksStatementSuite`. --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5882 from bowenliang123/jdbc-sr. Closes #5404 54927d341 [Bowen Liang] update 5b1bbf71f [Bowen Liang] update doc f01da74ea [Bowen Liang] update StarRocksStatementSuite 1018bc95a [Bowen Liang] MySQL8ConnectionProvider 59cba957f [Bowen Liang] simplify StarRocksDialect 4bf55bba7 [Bowen Liang] simplify StarRocksTRowSetGenerator c85722481 [Bowen Liang] jdbc starrocks Authored-by: Bowen Liang <liangbowen@gf.com.cn> Signed-off-by: Bowen Liang <liangbowen@gf.com.cn> |
||
|
wforget
|
7b7e17b1ad
|
[KYUUBI #5244][FOLLOWUP] Corrected ENGINE_ALIVE_MAX_FAILURES conf version
# 🔍 Description ## Issue References 🔗 This pull request fixes #5244 ## Describe Your Solution 🔧 As discussed in https://github.com/apache/kyuubi/pull/5888#discussion_r1433511355, we need to correct the ENGINE_ALIVE_MAX_FAILURES conf version. ## Types of changes 🔖 - [X] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [x] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5889 from wForget/KYUUBI-5244-FOLLOWUP. Closes #5244 a7854496e [wforget] [KYUUBI #5244][FOLLOWUP] Corrected ENGINE_ALIVE_MAX_FAILURES conf version Authored-by: wforget <643348094@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
senmiaoliu
|
4c029f991f |
[KYUUBI #5377] Spark engine query result save to file
### _Why are the changes needed?_ close #5377 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ NO Closes #5591 from lsm1/branch-kyuubi-5377. Closes #5377 9d1a18c15 [senmiaoliu] ignore empty file 3c70a1e9e [LSM] fix doc 73d3c3abf [senmiaoliu] fix style and add some comment 80e1f0d70 [senmiaoliu] Close orc fetchOrcStatement and remove result save file when ExecuteStatement close 42634a1d0 [senmiaoliu] fix style 979125d9b [senmiaoliu] fix style 1dc07a553 [senmiaoliu] spark engine save into hdfs file Lead-authored-by: senmiaoliu <senmiaoliu@trip.com> Co-authored-by: LSM <senmiaoliu@trip.com> Signed-off-by: Fu Chen <cfmcgrady@gmail.com> |
||
|
hezhao2
|
8ab4763ce4 |
[KYUUBI #5828] [CHAT] Kyuubi chat engine supports ernie bot(文心一言)
# 🔍 Description ## Issue References 🔗 This pull request fixes #5386 ## Describe Your Solution 🔧 add a new backend(ernie bot) for the Chat engine. <img width="1672" alt="Screenshot 2023-12-07 at 16 20 56" src="https://github.com/apache/kyuubi/assets/32693629/9850916c-3c4a-433c-8278-3068e7b37314"> ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [ ] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [ ] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5828 from zhaohehuhu/dev-1207. Closes #5828 c7314d4a8 [Cheng Pan] Update externals/kyuubi-chat-engine/src/main/scala/org/apache/kyuubi/engine/chat/ernie/service/ErnieBotService.scala 0c4e01007 [hezhao2] update doc 78e51b3d1 [hezhao2] add ernie into doc 2f4a63845 [hezhao2] refactor 832bc0453 [hezhao2] delete enum model 67214c575 [hezhao2] get rid of some java code 567772679 [hezhao2] java bean to scale case class 4d5c5940d [hezhao2] refactor some params 7c44eb83f [hezhao2] refactor some params 56b9ad13a [hezhao2] refactor a8e3d6cf6 [hezhao2] refactor 7376d800d [hezhao2] Kyuubi chat engine supports ernie bot 4b72a09c0 [hezhao2] Kyuubi chat engine supports ernie bot Lead-authored-by: hezhao2 <hezhao2@cisco.com> Co-authored-by: Cheng Pan <pan3793@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
b594105f14
|
[KYUUBI #5830] Test Hive engine with CDH Hive 2.1.1-cdh6.3.2
# 🔍 Description ## Issue References 🔗 The CDH Hive 2.1.1-cdh6.3.2 is the latest free version of CDH, which is adopted widely, it would be great if the Kyuubi Hive engine could support this version. ## Describe Your Solution 🔧 Actually, the Kyuubi Hive engine could work with CDH Hive 2.1.1-cdh6.3.2, this pull request just adds an integration test and fixes one test case. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [x] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5830 from pan3793/hive-211-cdh6. Closes #5830 afa80bd82 [Cheng Pan] nit 89c9ae96e [Cheng Pan] nit e402e49cd [Cheng Pan] nit 3c126a2a6 [Cheng Pan] Test Hive engine with CDH Hive 2.1.1-cdh6.3.2 Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
yeatsliao
|
27ad1024b5
|
[KYUUBI #5795][K8S] Support to cleanup the spark driver pod periodically
# 🔍 Description ## Issue References 🔗 This pull request fixes #5795 ## Describe Your Solution 🔧 Create a single daemon thread to traverse cache map periodically, which will evict expired cache and trigger a pod clean up operation. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [ ] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [ ] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5806 from liaoyt/master. Closes #5795 75c2b68cc [yeatsliao] cleanup driver pod periodically Authored-by: yeatsliao <liaoyt66066@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
kandy01.wang
|
f3f643a309
|
[KYUUBI #5756] Introduce specified initialized SQL to every engine
# 🔍 Description ## Issue References 🔗 This pull request fixes #5756 ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [x] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5821 from hadoopkandy/KYUUBI-5756. Closes #5756 046fe2a58 [kandy01.wang] [KYUUBI #5756] Introduce specified initialized SQL to every engine Authored-by: kandy01.wang <kandy01.wang@vipshop.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
wangjunbo
|
5b6a729fa8
|
[KYUUBI #5800] [KYUUBI#5467] Integrate Intel Gluten with Spark engine
# 🔍 Description ## Issue References 🔗 This pull request fixes #5467 ## Describe Your Solution 🔧 1. Add Gluten UTs. 2. Setup CI for Gluten testing 3. Write docs to guide users in setting up Kyuubi with Spark plus Gluten. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests github action ci tests: [Gluten Test CI](https://github.com/Kwafoor/incubator-kyuubi/actions/runs/7111586978) --- # Checklists ## 📝 Author Self Checklist - [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5800 from Kwafoor/kyuubi_5467. Closes #5800 c6dd26f93 [wangjunbo] fix 7818ae0c5 [wangjunbo] fix Scala Test 296f08c8c [wangjunbo] remove spark-3.2 gluten test 5a704675d [wangjunbo] [KYUUBI#5467] Integrate Intel Gluten with Spark engine Authored-by: wangjunbo <wangjunbo@qiyi.com> Signed-off-by: ulyssesyou <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
52d25c7cf1
|
[KYUUBI #5819] [DOCS] Fix rst syntax of missing blank line after code-block
# 🔍 Description ## Issue References 🔗 This pull request fixes a rst syntax of missing blank line after code-block ## Describe Your Solution 🔧 Add a blank line. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ <img width="1316" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/a35725aa-03be-4402-a482-6cfa74430293"> #### Behavior With This Pull Request 🎉 <img width="1337" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/acc643c0-9b91-49c6-97ae-44cc83874ea5"> #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5819 from pan3793/hudi-doc. Closes #5819 45149c5ac [Cheng Pan] fix 3236c9692 [Cheng Pan] [DOCS] Fix rst syntax in Hudi connector page Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
liangbowen
|
3ff4593b09 |
[KYUUBI #5802] [DOC] Add scala-2.13 as supported Scala profile to the docs of building from source
# 🔍 Description ## Issue References 🔗 As described. ## Describe Your Solution 🔧 Add docs for spark-3.4 and spark-3.5 as supported Spark profiles. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ No related docs. #### Behavior With This Pull Request 🎉 <img width="766" alt="image" src="https://github.com/apache/kyuubi/assets/1935105/a217b939-f97a-4176-b320-a047eb142609"> #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5802 from bowenliang123/docs-scala-213. Closes #5802 d73420713 [liangbowen] add scala-2.13 profile to the docs of building from code Authored-by: liangbowen <liangbowen@gf.com.cn> Signed-off-by: wforget <643348094@qq.com> |
||
|
zml1206
|
762ccd8295 |
[KYUUBI #5786] Disable spark script transformation
# 🔍 Description ## Issue References 🔗 This pull request fixes #5786. ## Describe Your Solution 🔧 Add spark check rule. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests org.apache.kyuubi.plugin.spark.authz.rule.AuthzUnsupportedOperationsCheckSuite.test("disable script transformation") --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5788 from zml1206/KYUUBI-5786. Closes #5786 06c0098be [zml1206] fix e2c3fee22 [zml1206] fix 37744f4c3 [zml1206] move to spark extentions deb09fb30 [zml1206] add configuration cfea4845a [zml1206] Disable spark script transformation in Authz Authored-by: zml1206 <zhuml1206@gmail.com> Signed-off-by: wforget <643348094@qq.com> |
||
|
liangbowen
|
2a39d697c0 |
[KYUUBI #5801] [DOC] Add spark-3.4 and spark-3.5 as supported Spark profiles to the docs of building from source
# 🔍 Description ## Issue References 🔗 ## Describe Your Solution 🔧 Add docs for spark-3.4 and spark-3.5 as supported Spark profiles. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ <img width="774" alt="image" src="https://github.com/apache/kyuubi/assets/1935105/29b0690d-574f-4b3c-ac70-50d5fb67a75e"> #### Behavior With This Pull Request 🎉 <img width="766" alt="image" src="https://github.com/apache/kyuubi/assets/1935105/58af356b-0f9c-4fe9-8f2b-8769ccebcc3a"> #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5801 from bowenliang123/doc-spark-profiles. Closes #5801 35254df09 [liangbowen] style 2b55824fd [liangbowen] add spark-3.4 and spark-3.5 to the docs of building from code Authored-by: liangbowen <liangbowen@gf.com.cn> Signed-off-by: Bowen Liang <liangbowen@gf.com.cn> |
||
|
hezhao2
|
3677f3a26e
|
[KYUUBI #5762] [DOC] Introduce Kyuubi Web UI Engine Page
# 🔍 Description ## Issue References 🔗 This pull request fixes #5572 ## Describe Your Solution 🔧 Introduce Kyuubi Web UI Engine Page as the screenshot shown. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 <img width="835" alt="Screenshot 2023-11-27 at 10 39 55" src="https://github.com/apache/kyuubi/assets/32693629/363426e8-9808-4fd7-a711-2f0f80164147"> #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [x] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5762 from zhaohehuhu/dev-1123. Closes #5762 4a85af03e [hezhao2] compress picture 5aed43e63 [hezhao2] refactor 16d2d2f83 [hezhao2] reformat ad9c30bf1 [hezhao2] Introduce Kyuubi Web UI Engine Page Authored-by: hezhao2 <hezhao2@cisco.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
wangjunbo
|
5481bf54f2 |
[KYUUBI #5464] JDBC Engine supports MySQL
### _Why are the changes needed?_ To close https://github.com/apache/kyuubi/issues/5464. To support JDBC engine use MySQL Dialect (kyuubi.engine.jdbc.type=mysql). ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No. Closes #5588 from Kwafoor/kyuubi_5464. Closes #5464 1019a6118 [wangjunbo] [KYUUBI #5464]rename function name `getProviderClass` to `getDriverClass` 9901bbad4 [wangjunbo] [KYUUBI #5464]handle properly to keep compatiblity b33d79ed2 [wangjunbo] [KYUUBI #5464]handle properly to keep compatiblity 86e6ee2b3 [wangjunbo] [KYUUBI #5464]handle properly to keep compatiblity d76cb3275 [wangjunbo] [KYUUBI #5464]update the docs 4a1acffd1 [wangjunbo] [KYUUBI #5464]update the docs 1aff55ecd [wangjunbo] [KYUUBI #5464]update the docs of kyuubi.engine.type 84202ea0c [wangjunbo] [KYUUBI #5464] update the docs of kyuubi.engine.type e3c1e94db [wangjunbo] [KYUUBI #5464] fix check cdf820da0 [wangjunbo] [KYUUBI #5464] fix check ff0f30ad8 [wangjunbo] [KYUUBI #5464] fix check 01321dc44 [wangjunbo] [KYUUBI #5464] JDBC Engine supports MySQL 756f5303c [wangjunbo] [KYUUBI #5464] JDBC Engine supports MySQL Authored-by: wangjunbo <wangjunbo@qiyi.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Yuwei Zhan
|
931b5026e7 |
[KYUUBI #5737] [DOC] Correct JDBC URL demo in Flink docs
# 🔍 Description flink setting need additional prefix flink. to override settings ## Issue References 🔗 This pull request fixes #5737 ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [ ] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [ ] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5745 from dik111/update-doc. Closes #5737 27ac6c1db [yuwei zhan] Update AllKyuubiConfiguration about flink jdbc url a1b1b2cba [yuwei zhan] Update settings.md Lead-authored-by: Yuwei Zhan <195812208@qq.com> Co-authored-by: yuwei zhan <195812208@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
yeatsliao
|
dc03687248 |
[KYUUBI #5731][K8S] Support to cleanup the spark driver pod with specified cleanup strategy
# 🔍 Description ## Describe Your Solution 🔧 A new feature introduced from #5714 supports kyuubi to clean up spark driver pods automatically, but all pod would be clean up without considering app's terminated state. This PR make user can chose which pods should be delete by setting up a cleanup strategy. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Test locally. --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5728 from liaoyt/master. Closes #5731 d2cc8cb55 [yeatsliao] regenerate docs 4caf8b152 [yeatsliao] rename conf 'KUBERNETES_SPARK_DELETE_DRIVER_POD_ON_TERMINATION' to 'KUBERNETES_SPARK_CLEANUP_TERMINATED_DRIVER_POD' 4d970fa14 [yeatsliao] [K8S] Support to cleanup the spark driver pod with specified clean up strategy Authored-by: yeatsliao <liaoyt66066@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
fwang12
|
3478fc9dfb |
[KYUUBI #5717] Infer the proxy user automatically for delete batch operation
# 🔍 Description Infer the batch user from session or metadata, user do not need to specify the proxy user anymore. This pr also align the behavior of BatchesResource with that of SessionsResource and OperationsResource(no proxy user parameter). For Kyuubi Batch, Session and Operation, these resources have the user attiribute. So we only need to check whether the authentication user has the permission to access the resource. ## Issue References 🔗 This pull request fixes # ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [ ] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5717 from turboFei/hive_server2_proxy_user. Closes #5717 70ad7e76d [fwang12] comment c721a751a [fwang12] ignore da92bd5a1 [fwang12] fix ut 9a197d005 [fwang12] doc c8ed5f9cf [fwang12] ut cef9e329c [fwang12] do not use proxy user Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
fwang12
|
765cdaa255 |
[KYUUBI #5711] [K8S] Support to build application state and error from pod container
# 🔍 Description This pr is to support to build the application state from pod container instead of pod. The use case is that: For spark on k8s, the pod contains two containers. - one is the spark driver main container - the another one is a sidecar container If the spark driver main container completed but the sidecar container not, the batch application state is always RUNNING as the pod state is `NotReady`. It is not expected. So in this pr, I want to introduce a new option config to support to build application state from the pod container. ## Issue References 🔗 This pull request fixes # ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5711 from turboFei/k8s_pod_state. Closes #5711 9e0575b5f [fwang12] save eaec5d2bb [fwang12] comment 61ae5f723 [fwang12] comment 1eac84f2d [fwang12] save bca19be3a [fwang12] nit 38115b752 [fwang12] [K8S] Support to build application state and error from pod container Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
fwang12
|
88fae49d7f |
[KYUUBI #5714] [K8S] Support to cleanup the spark driver pod after application terminates for retain period
# 🔍 Description ## Describe Your Solution 🔧 As title, support to cleanup the application pod after application terminates for retain period ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Test locally. --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [ ] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5714 from turboFei/kill_k8s_pod. Closes #5714 1e0787868 [fwang12] doc 0c9ff1a44 [fwang12] cleanup pod ab95d4c03 [fwang12] save Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
Kent Yao
|
10d0aecde1
|
[KYUUBI #5701] Enhance the PR Template to make PR desc informative
# 🔍 Description ## Issue References 🔗 This pull request fixes #5701 ## Describe Your Solution 🔧 This PR updates the PR template to ensure that PR authors provide informative PR descriptions for reviewers to speed up the review process. It's also suitable for improving our code quality. Besides, `docs/contributing/code/style.rst` is also changed to add a link to the doc style so that we can keep the PR template neat. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ See the screenshot below.  #### Behavior With This Pull Request 🎉 It's able to check it from the description of this PR directly. #### Related Unit Tests No --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] This patch was authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [x] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5702 from yaooqinn/pr. Closes #5701 19bdea9a4 [Kent Yao] not d7d9d8cfa [Kent Yao] Enhance PR Template to make PR desc informative be026e4a6 [Kent Yao] Enhance PR Template to make PR desc informative Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
senmiaoliu
|
e498bdba00
|
[KYUUBI #5582] JDBC Engine supports configurable default fetchSize
### _Why are the changes needed?_ close #5582 JDBC Engine supports configurable default fetchSize ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ NO Closes #5614 from lsm1/branch-kyuubi-5582. Closes #5582 dd58a6d51 [senmiaoliu] fix style d0eb7c5dc [senmiaoliu] set jdbc fetch size in operation manager 4eee26b7b [senmiaoliu] rename conf key 4d8b74796 [senmiaoliu] add conf for JDBC engine default fetch size Authored-by: senmiaoliu <senmiaoliu@trip.com> Signed-off-by: Shaoyun Chen <csy@apache.org> |
||
|
mrtisttt
|
50cd91b59e |
[KYUUBI #5683] Add kyuubi.session.proxy.user as an alternative to hive.server2.proxy.user
### _Why are the changes needed?_ We need a configuration to be an alternative to `hive.server2.proxy.user`. `hive.server2.proxy.user` is derived from the hive server, and it's perfect for keeping the behavior the same as the hive. But hive thrift protocol is only one of the FrontendServices. There are also other FrontendServices not necessary to follow `hive.server2.proxy.user`. So we need `kyuubi.proxy.user`. In this commit, support for using the `kyuubi.proxy.user` configuration has been implemented in scenarios where the original `hive.server2.proxy.user` configuration was used.In the future, this feature will continue to be improved according to other scenarios where it is used. Close #5683 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No. Closes #5653 from mrtisttt/add-kyuubi-proxy-user. Closes #5683 6d60e7632 [mrtisttt] Rollback unnecessary modifications 7298e4137 [mrtisttt] fix doc violations 19f6f4084 [mrtisttt] For API side, shorten the name of kyuubi.session.proxy.user and change it to proxyUser. 0fbcc704d [mrtisttt] Change kyuubi.proxy.user to kyuubi.session.proxy.user and improve documentation c2a810ce2 [mrtisttt] fix format violations a6413e8aa [mrtisttt] Compatible with scenarios where hive.server2.proxy.user is used b687c21b9 [mrtisttt] Add kyuubi.proxy.user as an alternative to hive.server2.proxy.user Authored-by: mrtisttt <mrtisttt@126.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |