e6b1bf76cb
728 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Cheng Pan
|
e6b1bf76cb
|
[KYUUBI #5933] Happy New Year 2024
# 🔍 Description ## Issue References 🔗 Update the NOTICE files for coming 2024. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Review --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #5933 from pan3793/notice-2024. Closes #5933 25e85e5f5 [Cheng Pan] notice Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
5d59cf1d6a
|
[KYUUBI #5867][FOLLOWUP] Fix generated configuration docs and typo
# 🔍 Description ## Issue References 🔗 This pull request is a follow-up of #5867, to fix generated configuration docs and typo to recover CI. ## Describe Your Solution 🔧 Regenerated docs by performing `dev/gen/gen_all_config_docs.sh` Address comment https://github.com/apache/kyuubi/pull/5868#discussion_r1438080698 ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Pass GA. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #5932 from pan3793/fix-docs. Closes #5867 93b8f97b0 [Cheng Pan] docs 0ad0d29f3 [Cheng Pan] typo Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
yikaifei
|
679aca5a6c
|
[KYUUBI #5867] HiveEngine support run on YARN mode
# 🔍 Description ## Issue References 🔗 This PR aims to support hive engine run on yarn mode, close https://github.com/apache/kyuubi/issues/5867 ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [ ] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [x] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5868 from Yikf/hive-on-yarn. Closes #5867 44f7287f5 [yikaifei] fix 3c17d2c4a [yikaifei] fix test 5474ebfba [yikaifei] parse classpath 6b97c4213 [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala 34a67b452 [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala 5e5045e66 [yikaifei] fix app type d1eb5aea7 [yikaifei] fix d89d09cfe [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala 1fa18ba1b [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/ApplicationMaster.scala 1b0b77f4d [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/ApplicationMaster.scala 2ed1d4492 [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala 98ff19ce6 [yikaifei] HiveEngine support run on YARN mode Lead-authored-by: yikaifei <yikaifei@apache.org> Co-authored-by: Cheng Pan <pan3793@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Bowen Liang
|
86b88b7e49 |
[KYUUBI #5906] [JDBC] Rebase Doris dialect implementation
# 🔍 Description ## Issue References 🔗 As decribed. ## Describe Your Solution 🔧 As Doris is MySQL protocol compatible , use MySQLDialect as base class to simplify Doris dialect implementation ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ No behaviour changes. #### Behavior With This Pull Request 🎉 No behaviour changes. #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5906 from bowenliang123/jdbc-doris-simp. Closes #5906 2e1420a88 [liangbowen] skip e095d28d7 [liangbowen] Revert "simplify MySQLTRowSetGenerator" 7d59e2d28 [liangbowen] simplify MySQLTRowSetGenerator 9214388f7 [Bowen Liang] simplify StarRocksSchemaHelper 5f910426b [Bowen Liang] simplify DorisSchemaHelper b90f26d04 [Bowen Liang] simplify DorisSchemaHelper 45ccdfeab [Bowen Liang] update 5b4a2c2ba [Bowen Liang] import f7fdd35ca [Bowen Liang] simplify DorisDialect Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn> Co-authored-by: liangbowen <liangbowen@gf.com.cn> Signed-off-by: liangbowen <liangbowen@gf.com.cn> |
||
|
yikaifei
|
d7ea20236b
|
[KYUUBI #5878] Hive engine should recognize hiveserver2-site.xml
# 🔍 Description ## Issue References 🔗 This pull request fixes https://github.com/apache/kyuubi/issues/5878, It aims to incorporate the `hiveserver2-site.xml` configuration into hiveConf for the Hive engine ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [ ] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5907 from Yikf/hiveserver2-site. Closes #5878 c4a9268c7 [Cheng Pan] Update externals/kyuubi-hive-sql-engine/src/main/scala/org/apache/kyuubi/engine/hive/HiveSQLEngine.scala ba59186b6 [yikaifei] add hiveserver2-site to hiveconf Lead-authored-by: yikaifei <yikaifei@apache.org> Co-authored-by: Cheng Pan <pan3793@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Fei Wang
|
58db6ca225 |
[KYUUBI #5877][FOLLOWUP] Add spark output mode to better support PySpark notebook
# 🔍 Description ## Issue References 🔗 This pull request fixes # ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [ ] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5898 from turboFei/notebook_mode. Closes #5877 7f1c607b9 [Fei Wang] PySpark 644d036bc [Fei Wang] docs 7c68b7742 [Fei Wang] add option to support notebook well Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: Fei Wang <fwang12@ebay.com> |
||
|
Fei Wang
|
42b08ba840 |
[KYUUBI #5899] Refine the python/scala interpreter error prompt
# 🔍 Description ## Issue References 🔗 Refine the interpreter error message format for python/scala. This pull request fixes # ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [ ] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5899 from turboFei/json_utils. Closes #5899 8460f18cc [Fei Wang] refine Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: Fei Wang <fwang12@ebay.com> |
||
|
senmiaoliu
|
19701efc3c |
[KYUUBI #5377][FOLLOWUP] Spark engine query result save to file
# 🔍 Description ## Issue References 🔗 https://github.com/apache/kyuubi/pull/5591#discussion_r1432281147 ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5895 from lsm1/branch-kyuubi-5377-followup. Closes #5377 4219d28ba [Fei Wang] nit 31d4fc15f [senmiaoliu] use zlib when SPARK version less than 3.2 Lead-authored-by: senmiaoliu <senmiaoliu@trip.com> Co-authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: Fei Wang <fwang12@ebay.com> |
||
|
Bowen Liang
|
4be20e1ca7 |
[KYUUBI #5404] JDBC Engine supports StarRocks
# 🔍 Description ## Issue References 🔗 This pull request fixes #5404 ## Describe Your Solution 🔧 - Introduce StarRocks support in JDBC Engine - Adding dialects and tests for StarRocks - Tested with StarRocks 3.x official open-sourced version docker image, with a mini cluster with FE and BE setup. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 Supported StarRocks in JDBC engine. #### Related Unit Tests Add tests of `StarRocksOperationSuite`, `StarRocksOperationWithEngineSuite`, `StarRocksSessionSuite` and `StarRocksStatementSuite`. --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5882 from bowenliang123/jdbc-sr. Closes #5404 54927d341 [Bowen Liang] update 5b1bbf71f [Bowen Liang] update doc f01da74ea [Bowen Liang] update StarRocksStatementSuite 1018bc95a [Bowen Liang] MySQL8ConnectionProvider 59cba957f [Bowen Liang] simplify StarRocksDialect 4bf55bba7 [Bowen Liang] simplify StarRocksTRowSetGenerator c85722481 [Bowen Liang] jdbc starrocks Authored-by: Bowen Liang <liangbowen@gf.com.cn> Signed-off-by: Bowen Liang <liangbowen@gf.com.cn> |
||
|
Bowen Liang
|
aac4503bba
|
[KYUUBI #5870] Directly mapping engine's data type to Java type for TRowSet generation
# 🔍 Description ## Issue References 🔗 As described. ## Describe Your Solution 🔧 Previously, the TRowSetGenerator built a chain of `Engine's data type` to `Thrift data type` and then to `Java data type`. This limits the possibility of overriding the implementation of Java data types handlers and coupling the `Engine's data type` to specific `Thrift data type`. 1. This PR directly mapping `Engine's data type` to `Java data type` focusing converting the upstream Engine result set's value to real data type. 2. it also decoupled the `Thrift data type` to Java data types, eg. Thrift's SMALL_INT and TINY_INT can be now ignored, and making the engines data type mapped to Java's Short data type. 3. it's now able to distinguish similar terms of data types with different meanings, like Java's `byte` or `byte array` and `tiny int` or `samll int` in engines and thrift. Only Java data types are represented in `TRowSetColumnGenerator` and `TRowSetColumnValueGenerator` ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ No behaviour changes. #### Behavior With This Pull Request 🎉 No behaviour changes. #### Related Unit Tests CI tests. --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [x] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5870 from bowenliang123/rowset-map. Closes #5870 51740a00a [Bowen Liang] update 3dc48f4b9 [liangbowen] type mapping Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn> Co-authored-by: liangbowen <liangbowen@gf.com.cn> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Fei Wang
|
50910ae6c4 |
[KYUUBI #5877] Support Python magic syntax for notebook usage
# 🔍 Description ## Issue References 🔗 Support python magic syntax, for example: ``` %table %json %matplot ``` Refer: https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-studio-magics.html https://github.com/jupyter-incubator/sparkmagic https://github.com/apache/incubator-livy/blob/master/repl/src/main/resources/fake_shell.py This pull request fixes #5877 ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Testing with python code: ``` import matplotlib.pyplot as plt plt.plot([3,4,5],[6,7,8]) %matplot plt; ``` <img width="1723" alt="image" src="https://github.com/apache/kyuubi/assets/6757692/9a1176c0-8eb0-4a64-83e4-35e74e33d2f0"> Decode the "image/png" and save to png.  #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [ ] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5881 from turboFei/magic_command. Closes #5877 6f2b193a9 [Fei Wang] ut 877c7d108 [Fei Wang] internal config 012dfe44f [Fei Wang] nit 3e0f324f4 [Fei Wang] except other exceptions 24352d2c6 [Fei Wang] raise execution error 085316111 [Fei Wang] raise ExecutionError instead of execute_reply_error c058defc5 [Fei Wang] add more ut 4da52153b [Fei Wang] Dumps python object to json at last 35127537b [Fei Wang] add ut for json and table 48735ebd9 [Fei Wang] the data should be Map[String, Object] 3a3ba0a49 [Fei Wang] return other data fields 54d680090 [Fei Wang] reformat 87ded6e8d [Fei Wang] add config to disable 44f88ef74 [Fei Wang] add magic node back Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: Fei Wang <fwang12@ebay.com> |
||
|
yikaifei
|
667b5ab6b8 |
[KYUUBI #5865] Support get SQL keywords from Hive engine through API
# 🔍 Description ## Issue References 🔗 This pull request implement the https://github.com/apache/kyuubi/issues/5865, it support get SQL keywords from Hive engine through API. ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [x] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5866 from Yikf/hive-keywords. Closes #5865 e54945d94 [yikaifei] Hive engine CLI_ODBC_KEYWORDS Authored-by: yikaifei <yikaifei@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Bowen Liang
|
09febce0d8 |
[KYUUBI #5861] Generalize TRowSet generator for JDBC engine with dialects
# 🔍 Description ## Issue References 🔗 As described. ## Describe Your Solution 🔧 - Introduced JdbcTRowSetGenerator extending `AbstractTRowSetGenerator ` introduced in #5851 in JDBC engine. - Provide a DefaultJdbcTRowSetGenerator as default implementation for mapping the JDBC data types to TRowSet generation - Make JDBC dialect providing TRowSetGenerator extending DefaultJdbcTRowSetGenerator to adapt detailed differences ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5861 from bowenliang123/jdbc-rowgen. Closes #5861 7f8658df1 [Bowen Liang] generalize jdbc TRowSet generator Authored-by: Bowen Liang <liangbowen@gf.com.cn> Signed-off-by: liangbowen <liangbowen@gf.com.cn> |
||
|
Bowen Liang
|
acdd74d5ca |
[KYUUBI #5862] Use TestContainerForAll for testing JDBC engine with testcontainers
# 🔍 Description ## Issue References 🔗 As described. ## Describe Your Solution 🔧 - replacing the usage of `ForAllTestContainer` with `TestContainerForAll`, simplifying the lifecycle for starting / stopping the containers and fetching the configs from the containers - use `testcontainers-scala-postgresql` for testing with PostgreSQL containers - add version 16 for PostgreSQL image tag ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ No behaviour changes. #### Behavior With This Pull Request 🎉 No behaviour changes. #### Related Unit Tests JDBC Engine IT. --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5862 from bowenliang123/jdbc-container. Closes #5862 29e85121c [Bowen Liang] TestContainerForAll Authored-by: Bowen Liang <liangbowen@gf.com.cn> Signed-off-by: liangbowen <liangbowen@gf.com.cn> |
||
|
Bowen Liang
|
9047151d8d |
[KYUUBI #5851] Generalize TRowSet generators
# 🔍 Description ## Issue References 🔗 As described. ## Describe Your Solution 🔧 - Introduced a generalized RowSet generator `AbstractTRowSetGenerator[SchemaT, RowT, ColumnT]` - extract common methods for looping and assembling the rows to TRowSet - support generation for either column-based or row-based TRowSet - Each engine creates a sub-generator of `AbstractTRowSetGenerator` - focus on mapping and conversion from the engine's data type to the relative Thrift type - implements the schema data type and column value methods - create a generator instance instead of the previously used `RowSet` object, for isolated session-aware or thread-aware configs or context, eg. Timezone ID for Flink, and the Hive time formatters for Spark. - This PR covers the TRowSet generation for the server and the engines of Spark/Flink/Trino/Chat, except the JDBC engine which will be supported in the follow-ups with JDBC dialect support. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ No behavior changes. #### Behavior With This Pull Request 🎉 No behavior changes. #### Related Unit Tests CI tests. --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5851 from bowenliang123/rowset-gen. Closes #5851 1d2f73ab4 [Bowen Liang] common RowSetGenerator Authored-by: Bowen Liang <liangbowen@gf.com.cn> Signed-off-by: Bowen Liang <liangbowen@gf.com.cn> |
||
|
senmiaoliu
|
4c029f991f |
[KYUUBI #5377] Spark engine query result save to file
### _Why are the changes needed?_ close #5377 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ NO Closes #5591 from lsm1/branch-kyuubi-5377. Closes #5377 9d1a18c15 [senmiaoliu] ignore empty file 3c70a1e9e [LSM] fix doc 73d3c3abf [senmiaoliu] fix style and add some comment 80e1f0d70 [senmiaoliu] Close orc fetchOrcStatement and remove result save file when ExecuteStatement close 42634a1d0 [senmiaoliu] fix style 979125d9b [senmiaoliu] fix style 1dc07a553 [senmiaoliu] spark engine save into hdfs file Lead-authored-by: senmiaoliu <senmiaoliu@trip.com> Co-authored-by: LSM <senmiaoliu@trip.com> Signed-off-by: Fu Chen <cfmcgrady@gmail.com> |
||
|
hezhao2
|
8ab4763ce4 |
[KYUUBI #5828] [CHAT] Kyuubi chat engine supports ernie bot(文心一言)
# 🔍 Description ## Issue References 🔗 This pull request fixes #5386 ## Describe Your Solution 🔧 add a new backend(ernie bot) for the Chat engine. <img width="1672" alt="Screenshot 2023-12-07 at 16 20 56" src="https://github.com/apache/kyuubi/assets/32693629/9850916c-3c4a-433c-8278-3068e7b37314"> ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [ ] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [ ] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5828 from zhaohehuhu/dev-1207. Closes #5828 c7314d4a8 [Cheng Pan] Update externals/kyuubi-chat-engine/src/main/scala/org/apache/kyuubi/engine/chat/ernie/service/ErnieBotService.scala 0c4e01007 [hezhao2] update doc 78e51b3d1 [hezhao2] add ernie into doc 2f4a63845 [hezhao2] refactor 832bc0453 [hezhao2] delete enum model 67214c575 [hezhao2] get rid of some java code 567772679 [hezhao2] java bean to scale case class 4d5c5940d [hezhao2] refactor some params 7c44eb83f [hezhao2] refactor some params 56b9ad13a [hezhao2] refactor a8e3d6cf6 [hezhao2] refactor 7376d800d [hezhao2] Kyuubi chat engine supports ernie bot 4b72a09c0 [hezhao2] Kyuubi chat engine supports ernie bot Lead-authored-by: hezhao2 <hezhao2@cisco.com> Co-authored-by: Cheng Pan <pan3793@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Bowen Liang
|
1d6e65356e |
[KYUUBI #5831] Pre-allocate array list capacity for TColumns and TColumnValues in TRowSet generation
# 🔍 Description ## Issue References 🔗 Subtask of #5808. ## Describe Your Solution 🔧 To avoid unnecessary possible array copy in ensuring capacity of the array list, pre-allocate array list capacity for TColumns and TColumnValues in TRowSet generation. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 No behaviour changes. #### Related Unit Tests RowSetSuite of Spark Engine. --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [ ] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [x] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5831 from bowenliang123/rowset-inplace. Closes #5831 c1c6c0f84 [liangbowen] avoid possible array copying with growing of array list in TRowSet generation Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn> Co-authored-by: liangbowen <liangbowen@gf.com.cn> Signed-off-by: Bowen Liang <liangbowen@gf.com.cn> |
||
|
Bowen Liang
|
4463cc8f97 |
[KYUUBI #5811] [SPARK] Reuse time formatters instance in value serialization of TRowSet generation
# 🔍 Description ## Issue References 🔗 Subtask of #5808. ## Describe Your Solution 🔧 Value serialization to Hive style string by `HiveResult.toHiveString` requires a `TimeFormatters` instance handling the date/time data types. Reusing the pre-created time formatters's instance, it dramatically reduces the overhead and improves the TRowset generation. This may help to reduce memory footprints and fewer operations for TRowSet generation performance. This is also aligned to the Spark's `RowSetUtils` in the implementation of RowSet generation for [SPARK-39041](https://issues.apache.org/jira/browse/SPARK-39041) by yaooqinn , with explicitly declared TimeFormatters instance. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [x] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5811 from bowenliang123/rowset-timeformatter. Closes #5811 22709914e [Bowen Liang] Reuse time formatters instance in value serialization of TRowSet Authored-by: Bowen Liang <liangbowen@gf.com.cn> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
9be0c65fe9
|
[KYUUBI #5783] Switch to kyuubi-relocated-hive-service-rpc
# 🔍 Description ## Issue References 🔗 TL;DR there are some issues with shading Thrift RPC classes during the engine packaging phase, see details in the PR description of https://github.com/apache/kyuubi-shaded/pull/20. ## Describe Your Solution 🔧 This PR aims to migrate from vanilla `hive-service-rpc`, `libfb303`, `libthrift` to `kyuubi-relocated-hive-service-rpc` introduced in https://github.com/apache/kyuubi-shaded/pull/20, the detailed works are: - replace imported deps in `pom.xml` and rename the package prefix in all modules, except for - `kyuubi-server` there are a few places use vanilla thrift classes to access HMS to get token - `kyuubi-hive-sql-engine` Hive method invocation - update relocations rules in modules that creates shaded jar - introduce `HiveRpcUtils` in `kyuubi-hive-sql-engine` module for object conversion. As part of the whole change, this PR upgrades from the Kyuubi Shaded 0.1.0 to 0.2.0, which changes the jars name. see https://kyuubi.apache.org/shaded-release/0.2.0.html ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests Pass all Hive UT with Hive 3.1.3, and IT with Hive 3.1.3 and 2.3.9 (also tested with 2.1.1-cdh6.3.2) --- # Checklists ## 📝 Author Self Checklist - [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [x] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5783 from pan3793/rpc-shaded. Closes #5783 b45d4deaa [Cheng Pan] remove staging repo 890076a20 [Cheng Pan] Kyuubi Shaded 0.2.0 RC0 071945d45 [Cheng Pan] Rebase 199794ed9 [Cheng Pan] fix fc128b170 [Cheng Pan] fix 26d313896 [Cheng Pan] fix 632984c92 [Cheng Pan] fix 428305589 [Cheng Pan] fix 6301e28fd [Cheng Pan] fix 955cdb33b [Cheng Pan] Switch to kyuubi-shaded-hive-service-rpc Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
kandy01.wang
|
f3f643a309
|
[KYUUBI #5756] Introduce specified initialized SQL to every engine
# 🔍 Description ## Issue References 🔗 This pull request fixes #5756 ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [x] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5821 from hadoopkandy/KYUUBI-5756. Closes #5756 046fe2a58 [kandy01.wang] [KYUUBI #5756] Introduce specified initialized SQL to every engine Authored-by: kandy01.wang <kandy01.wang@vipshop.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Paul Lin
|
51c3e0372f |
[KYUUBI #5814] [BUG][FLINK] Skip the max-rows limitation if the statement is not DQL
# 🔍 Description ## Issue References 🔗 Currently, Flink max-rows limitation is applied to DDL as well. It's unexpected and should be fixed. ## Describe Your Solution 🔧 Skip max-rows limitation when the result set is not query result. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests FlinkOperationSuite - ensure result max rows --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5814 from link3280/fix/flink_ddl_max_rows. Closes #5814 07c146a77 [Paul Lin] [FLINK] Fix if condition 28981fc60 [Paul Lin] [FLINK] Fix if condition 0c7cce33d [Paul Lin] [FLINK] Fix Flink 1.16 compatiblity 869502e8c [Paul Lin] [FLINK] Update thread name 60e76cc57 [Paul Lin] [FLINK] Replace scala bounded reflection utils with the common ones 2d8b9c163 [Paul Lin] [FLINK] Skip the max-rows limitation if the statement is not DQL Authored-by: Paul Lin <paullin3280@gmail.com> Signed-off-by: Paul Lin <paullin3280@gmail.com> |
||
|
佘志铭
|
88c8cc312b |
[KYUUBI #5807] Simple norm the use of kyuubi.operation.result.arrow.timestampAsString and kyuubi.operation.result.format
# 🔍 Description ## Issue References 🔗 This pull request norm the arrow conf. ## Describe Your Solution 🔧 Using `kyuubi.operation.result.arrow.timestampAsString` and `kyuubi.operation.result.format` string to get conf . To replace as `ARROW_BASED_ROWSET_TIMESTAMP_AS_STRING.key` and `OPERATION_RESULT_FORMAT`. Make it more norm. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Other ## Test Plan 🧪 No need test --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5807 from zzzzming95/norm-conf. Closes #5807 9e65bfafe [佘志铭] norm the arrow conf Authored-by: 佘志铭 <shezhiming@corp.netease.com> Signed-off-by: liangbowen <liangbowen@gf.com.cn> |
||
|
liangbowen
|
13af6aeb70 |
[KYUUBI #5767] Extract common utils for assembling key value pairs with config option prefix in processbuilder
# 🔍 Description ## Issue References 🔗 As described. ## Describe Your Solution 🔧 - Focus on key points for configuration option assembling, instead of repeating manually command configs assembling - Avoid using magic string value "--conf" / "-cp" in each processbuilder - Extract common utils for assembling key value pairs with config option prefix in processbuilder - Use `mutable.ListBuffer` for command assembling - Extract common method for redact config value by key names - NO changes in expected string value for processbuilder command assertions in test suites ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ No behavior changes. #### Behavior With This Pull Request 🎉 No behavior changes. #### Related Unit Tests Added `CommandUtilsSuite`. --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5767 from bowenliang123/config-option. Closes #5767 b043888d6 [liangbowen] use ++ for command configs 16a3c27d1 [liangbowen] .key bc285004e [liangbowen] use raw literal in test suites ab018cf2d [Bowen Liang] config option Lead-authored-by: liangbowen <liangbowen@gf.com.cn> Co-authored-by: Bowen Liang <liangbowen@gf.com.cn> Signed-off-by: liangbowen <liangbowen@gf.com.cn> |
||
|
Paul Lin
|
8f529aacf1 |
[KYUUBI #5799] [FLINK] Fix fetch timeout in session conf doesn't support ISO-8601
# 🔍 Description ## Issue References 🔗 Currently, Flink engine supports overwriting result fetch timeout in session conf, but in that way IOS-8601 time format is not supported. This PR fixes the problem. This pull request fixes # ## Describe Your Solution 🔧 Apply ConfigOption's time value conf parsing in session conf parsing. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [x] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5799 from link3280/timeconf_parsing. Closes #5799 417898a63 [Paul Lin] [FLINK] Use ISO-8601 time conf in unit test 99a496419 [Paul Lin] [FLINK] Fix fetch timeout in session conf doesn't support ISO-8601 Authored-by: Paul Lin <paullin3280@gmail.com> Signed-off-by: Paul Lin <paullin3280@gmail.com> |
||
|
junjie.miao
|
fc3a215c16 |
[KYUUBI #5789] Flink engine kyuubi.session.engine.flink.fetch.timeout parameter on the server is not effective
# 🔍 Description ## Issue References 🔗 This pull request fixes #5789 ## Describe Your Solution 🔧 First initialize the server 'kyuubi.session.engine.flink.fetch.timeout' parameters, if the server parameter does not exist, it is set to unlimited duration; And then obtain the client 'kyuubi.session.engine.flink.fetch.timeout' parameters, if the client parameters use the server does not exist. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [x] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested Closes #5790 from junjiem/fix-5789. Closes #5789 925ac25f3 [junjie.miao] fix flink engine kyuubi.session.engine.flink.fetch.timeout parameter on the server is not effective bug Authored-by: junjie.miao <junjie.miao@goupwith.com> Signed-off-by: Paul Lin <paullin3280@gmail.com> |
||
|
waywtdcc
|
eb9e88be17
|
[KYUUBI #5750] [Spark] Fix the inaccurate issue of obtaining COLUMN_SIZE in the decimal field jdbc of spark engine
# 🔍 Description ## Issue References 🔗 This pull request fixes # ## Describe Your Solution 🔧 Fix the inaccuracy in getting COLUMN_SIZE from spark engine's decimal field jdbc. The current jdbc's get column size gets the decimal field as default size, which is inaccurate; if it is decimal(20,3), the obtained column size is 16; the actual is 20. ## Types of changes 🔖 - [X] Bugfix (non-breaking change which fixes an issue) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- Closes #5750 from waywtdcc/support_spark_decimal2. Closes #5750 2d288f5aa [waywtdcc] [Spark] Fix the inaccurate issue of obtaining COLUMN_SIZE in the decimal field jdbc of spark engine 4286354c2 [waywtdcc] Support flink engine under the select statement, the results can be read in a stream e5b74b0c8 [waywtdcc] Support flink engine under the select statement, the results can be read in a stream Authored-by: waywtdcc <waywtdcc@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
zhouyifan279
|
372b6e2031 |
[KYUUBI #5784] Implement HiveTBinaryFrontendService#RenewDelegationToken
# 🔍 Description ## Issue References 🔗 We had a KyuubiServer with `kyuubi.kinit.principal=hive/xxxxxxx.xxx` and connected to it using beeline: ``` ./bin/beeline -u "jdbc:hive2://xxxxxxx:10009/;principal=hive/_HOSTxxx.xxx;hive.server2.proxy.user=zhouyifan03#kyuubi.engine.type=hive_sql;hive.server2.enable.doAs=false;" ``` When we execute SQL `select 1`, it failed with error: ``` 0: jdbc:hive2://xxxxxxx:10009/> select 1; Error: org.apache.kyuubi.KyuubiSQLException: Failed to get metastore connection (state=,code=0) ``` HiveSQLEngine log: ``` 2023-11-27 15:19:09.217 ERROR HiveTBinaryFrontendHandler-Pool: Thread-27 org.apache.thrift.transport.TSaslTransport: SASL negotiation failure javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:545) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:303) at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1773) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3819) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3871) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3851) at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4105) at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:254) at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:237) at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:394) at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:338) at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:318) at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:294) at org.apache.hive.service.cli.session.HiveSessionImpl.open(HiveSessionImpl.java:181) at org.apache.kyuubi.engine.hive.session.HiveSessionImpl.open(HiveSessionImpl.scala:51) at org.apache.kyuubi.session.SessionManager.openSession(SessionManager.scala:109) at org.apache.kyuubi.service.AbstractBackendService.openSession(AbstractBackendService.scala:46) at org.apache.kyuubi.service.TFrontendService.getSessionHandle(TFrontendService.scala:182) ``` ## Describe Your Solution 🔧 In order to pass HiveMetaStoreClient authentication, we need to add a HIVE_DELEGATION_TOKEN into proxy user `zhouyifan03`. It can be achieved by implementing HiveTBinaryFrontendService#RenewDelegationToken. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ <img width="1009" alt="image" src="https://github.com/apache/kyuubi/assets/88070094/9a22232f-dc1b-4557-acd7-1c16463d651f"> #### Behavior With This Pull Request 🎉 <img width="1542" alt="image" src="https://github.com/apache/kyuubi/assets/88070094/9e0658e1-af3a-4970-8c47-9629c183ea9e"> #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5784 from zhouyifan279/hive-engine-renew-token. Closes #5784 d0e79172c [Cheng Pan] Update externals/kyuubi-hive-sql-engine/src/main/scala/org/apache/kyuubi/engine/hive/HiveTBinaryFrontendService.scala 9f6c45239 [zhouyifan279] Implement HiveTBinaryFrontendService#RenewDelegationToken Lead-authored-by: zhouyifan279 <zhouyifan279@gmail.com> Co-authored-by: Cheng Pan <pan3793@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
yikaifei
|
67bf8e135b
|
[KYUUBI #5782] Flink Engine GetInfo support CLI_ODBC_KEYWORDS
# 🔍 Description ## Issue References 🔗 This PR aims to support `CLI_ODBC_KEYWORDS` on flink engine to avoid https://github.com/apache/kyuubi/issues/2637 ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests Adjusted existing test #org.apache.kyuubi.it.flink.operation.FlinkOperationSuite --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5782 from Yikf/flink-CLI_ODBC_KEYWORDS. Closes #5782 ef0dc049a [yikaifei] Flink GetInfo support CLI_ODBC_KEYWORDS Authored-by: yikaifei <yikaifei@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
933978e4e1 |
[KYUUBI #5778] Hive engine shall respect hive.server2.enable.doAs
# 🔍 Description ## Issue References 🔗 Currently, the Hive engine ignores `hive.server2.enable.doAs` and always constructs `HiveSessionImplwithUGI` ## Describe Your Solution 🔧 Hive engine shall respect `hive.server2.enable.doAs`, to align the behavior with HiveServer2 ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ The Hive engine always constructs `HiveSessionImplwithUGI` #### Behavior With This Pull Request 🎉 When `hive.server2.enable.doAs` is `true`, the Hive engine constructs `HiveSessionImplwithUGI`; When `hive.server2.enable.doAs` is `false`, the Hive engine constructs `HiveSessionImpl`; #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [ ] Test coverage is ok - [x] Assignees are selected. - [ ] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5778 from pan3793/hive-doas. Closes #5778 4ac97a79d [Cheng Pan] Hive engine shall respect `hive.server2.enable.doAs` Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
zhouyifan279
|
65ba3b1d63 |
[KYUUBI #5777] Include hive-service-rpc in Hive engine jar
# 🔍 Description ## Issue References 🔗 Exception was thrown when execute SQL on HiveSQLEngine when HIVE_HOME set to Hive 2.1.1-cdh6.3.2.  ## Describe Your Solution 🔧 hive-service-rpc-2.1.1-cdh6.3.2.jar is not compatible with Kyuubi Server's hive-service-rpc. We should use Kyuubi Server's hive-service-rpc at HiveSQLEngine side. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ Exception was thrown  #### Behavior With This Pull Request 🎉 SQL execute successfully <img width="1545" alt="image" src="https://github.com/apache/kyuubi/assets/88070094/8bb7cfa0-0af0-42cf-acf5-40cff39efd1f"> #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [ ] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5777 from zhouyifan279/hive-sql-engine. Closes #5777 c55903a52 [zhouyifan279] Include org.apache.hive:hive-service-rpc in HiveSQLEngine jar Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
wangjunbo
|
5481bf54f2 |
[KYUUBI #5464] JDBC Engine supports MySQL
### _Why are the changes needed?_ To close https://github.com/apache/kyuubi/issues/5464. To support JDBC engine use MySQL Dialect (kyuubi.engine.jdbc.type=mysql). ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No. Closes #5588 from Kwafoor/kyuubi_5464. Closes #5464 1019a6118 [wangjunbo] [KYUUBI #5464]rename function name `getProviderClass` to `getDriverClass` 9901bbad4 [wangjunbo] [KYUUBI #5464]handle properly to keep compatiblity b33d79ed2 [wangjunbo] [KYUUBI #5464]handle properly to keep compatiblity 86e6ee2b3 [wangjunbo] [KYUUBI #5464]handle properly to keep compatiblity d76cb3275 [wangjunbo] [KYUUBI #5464]update the docs 4a1acffd1 [wangjunbo] [KYUUBI #5464]update the docs 1aff55ecd [wangjunbo] [KYUUBI #5464]update the docs of kyuubi.engine.type 84202ea0c [wangjunbo] [KYUUBI #5464] update the docs of kyuubi.engine.type e3c1e94db [wangjunbo] [KYUUBI #5464] fix check cdf820da0 [wangjunbo] [KYUUBI #5464] fix check ff0f30ad8 [wangjunbo] [KYUUBI #5464] fix check 01321dc44 [wangjunbo] [KYUUBI #5464] JDBC Engine supports MySQL 756f5303c [wangjunbo] [KYUUBI #5464] JDBC Engine supports MySQL Authored-by: wangjunbo <wangjunbo@qiyi.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
1e4d91e43e |
[KYUUBI #5280] Hive engine compatible with Hive 2.3
### _Why are the changes needed?_ Try to make the Hive engine compatible with Hive 2.3. It is not easy to align all dependencies with Hive 2.3, thus this PR just added a new integration test to make sure that `kyuubi hive engine` compiled against Hive 3.1.3 could work with Hive 2.3.9 runtime, instead of introducing new a profile to make it pass building with Hive 2.3.9 There are potential class conflict issues, because the Apache Hive 2.3.9 and 3.1.3 official binary artifacts ship Scala 2.11 jars but Kyuubi uses Scala 2.12. Also tested with `apache-hive-2.1.1-cdh6.3.2-bin.tar.gz` locally, which can successfully bootstrap and run queries. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes #5280 from pan3793/hive-2.3. Closes #5280 8ca02a351 [Cheng Pan] corss verify hive engine on hive 2.3 Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Bowen Liang
|
8480605463 |
[KYUUBI #5730] Tolerate execeptions for periodical scheduled tasks
# 🔍 Description ## Issue References 🔗 As the PR https://github.com/apache/kyuubi/pull/5727 fixed a problem caused by leaking exception in a scheduled task, the exception thrown should be properly handled to prevent suspension in `ScheduledExecutorService.scheduleWithFixedDelay`. ## Describe Your Solution 🔧 Introducing a util method `scheduleTolerableRunnableWithFixedDelay` for catching possible exceptions in scheduled tasks. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ No behaviour change. #### Behavior With This Pull Request 🎉 This is a fallback measure to ensure all the exceptions handled. Currently all the occurrences are properly handled. #### Related Unit Tests Pass all the CI tests. --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5730 from bowenliang123/tolerate-runnable. Closes #5730 a38eda075 [Bowen Liang] use thread name for error message 6bc13b775 [Bowen Liang] comment 87e2bf253 [Bowen Liang] schedule tolerable runnable Authored-by: Bowen Liang <liangbowen@gf.com.cn> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
senmiaoliu
|
e498bdba00
|
[KYUUBI #5582] JDBC Engine supports configurable default fetchSize
### _Why are the changes needed?_ close #5582 JDBC Engine supports configurable default fetchSize ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ NO Closes #5614 from lsm1/branch-kyuubi-5582. Closes #5582 dd58a6d51 [senmiaoliu] fix style d0eb7c5dc [senmiaoliu] set jdbc fetch size in operation manager 4eee26b7b [senmiaoliu] rename conf key 4d8b74796 [senmiaoliu] add conf for JDBC engine default fetch size Authored-by: senmiaoliu <senmiaoliu@trip.com> Signed-off-by: Shaoyun Chen <csy@apache.org> |
||
|
liupeiyue
|
7c5f583842 |
[KYUUBI #5331] Spark engine should throw an exception when it fails to start
### _Why are the changes needed?_ Close #5331 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No. Closes #5332 from ASiegeLion/master. Closes #5331 21342f546 [sychen] wrap InterruptedException 1f2542c82 [sychen] fix UT e433b544f [liupeiyue] [KYUUBI #5331]Spark Engine should throw an exception to let K8s know when the engine fails to start Lead-authored-by: liupeiyue <liupeiyue@yy.com> Co-authored-by: sychen <sychen@ctrip.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Bowen Liang
|
bad0ccb5ed |
[KYUUBI #5660] [TEST] Ensure SPI provider file content sorted in alphabet order
### _Why are the changes needed?_ Check the content of the SPI provider files sorted in alphabet order. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No. Closes #5660 from bowenliang123/spi-sort. Closes #5660 30c7028f6 [Bowen Liang] ensure SPI provider file sorted Authored-by: Bowen Liang <liangbowen@gf.com.cn> Signed-off-by: Bowen Liang <liangbowen@gf.com.cn> |
||
|
David Yuan
|
9615db55ce |
[KYUUBI #5438] Add common method to get session level config
### _Why are the changes needed?_ Current now, in spark-engine module, some session-level configurations are ignored due to the complexity of get session-level configurations in kyuubi spark engine, so As discussed in https://github.com/apache/kyuubi/pull/5410#discussion_r1360164253. If we need unit test use withSessionConf method, we need make the code get configuration from the right session The PR is unfinished, it need wait the pr https://github.com/apache/kyuubi/pull/5410 success so that i can use the new change in unit test closes #5438 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes #5487 from davidyuan1223/5438_add_common_method_to_support_session_config. Closes #5438 e1ded3654 [davidyuan] add more optional session level to get conf 84c4568d9 [davidyuan] add more optional session level to get conf 4d709023e [davidyuan] add more optional session level to get conf 96d7cde05 [davidyuan] Revert "add more optional session level to get conf" 940f8f878 [davidyuan] add more optional session level to get conf 15641e8ec [davidyuan] add more optional session level to get conf d83893119 [davidyuan] Merge branch '5438_add_common_method_to_support_session_config' of https://github.com/davidyuan1223/kyuubi into 5438_add_common_method_to_support_session_config 2de96b5f8 [davidyuan] add common method to get session level config 3ec73adf8 [liangbowen] [KYUUBI #5522] [BATCH] Ignore main class for PySpark batch job submission d8b808dbe [Cheng Pan] [KYUUBI #5523] [DOC] Update the Kyuubi supported components version c7d15aed0 [Cheng Pan] [KYUUBI #5483] Release Spark TPC-H/DS Connectors with Scala 2.13 4a1db4206 [zwangsheng] [KYUUBI #5513][BATCH] Always redirect delete batch request to Kyuubi instance that owns batch session b06e04485 [labbomb] [KYUUBI #5517] [UI] Initial implement the SQL Lab page 88bb6b4a8 [liangbowen] [KYUUBI #5486] Bump Kafka client version from 3.4.0 to 3.5.1 538a648cd [davidyuan] [KYUUBI #4186] Spark showProgress with JobInfo 682e5b5e3 [Xianxun Ye] [KYUUBI #5405] [FLINK] Support Flink 1.18 c71528ea3 [Cheng Pan] [KYUUBI #5484] Remove legacy Web UI ee52b2a69 [Angerszhuuuu] [KYUUBI #5446][AUTHZ] Support Create/Drop/Show/Reresh index command for Hudi 6a5bb1026 [weixi] [KYUUBI #5380][UT] Create PySpark batch jobs tests for RESTful API 86f692d73 [Kent Yao] [KYUUBI #5512] [AuthZ] Remove the non-existent query specs in Deletes and Updates dfdd7a3f4 [fwang12] [KYUUBI #5499][KYUUBI #2503] Catch any exception when closing idle session b7b354485 [伟程] [KYUUBI #5212] Fix configuration errors causing by helm charts of prometheus services d123a5a1e [liupeiyue] [KYUUBI #5282] Support configure Trino session conf in `kyuubi-default.conf` 075043754 [yangming] [KYUUBI #5294] [DOC] Update supported dialects for JDBC engine 9c75d8252 [zwangsheng] [KYUUBI #5435][INFRA][TEST] Improve Kyuubi On Kubernetes IT 1dc264add [Angerszhuuuu] [KYUUBI #5479][AUTHZ] Support Hudi CallProcedureHoodieCommand for stored procedures bc3fcbb4d [Angerszhuuuu] [KYUUBI #5472] Permanent View should pass column when child plan no output a67b8245a [Fantasy-Jay] [KYUUBI #5382][JDBC] Duplication cleanup improvement in JdbcDialect and schema helpers c039e1b6f [Kent Yao] [KYUUBI #5497] [AuthZ] Simplify debug message for missing field/method in ReflectUtils 0c8be79ab [Angerszhuuuu] [KYUUBI #5475][FOLLOWUP] Authz check permanent view's subquery should check view's correct privilege 1293cf208 [Kent Yao] [KYUUBI #5500] Add Kyuubi Code Program to Doc e2754fedd [Angerszhuuuu] [KYUUBI #5492][AUTHZ] saveAsTable create DataSource table miss db info 0c53d0079 [Angerszhuuuu] [KYUUBI #5447][FOLLOWUP] Remove unrelated debug prints in TableIdentifierTableExtractor 119c393fc [Angerszhuuuu] [KYUUBI #5447][AUTHZ] Support Hudi DeleteHoodieTableCommand/UpdateHoodieTableCommand/MergeIntoHoodieTableCommand 3af5ed13c [yikaifei] [KYUUBI #5427] [AUTHZ] Shade spark authz plugin 503c3f7fd [davidyuan] Merge remote-tracking branch 'origin/5438_add_common_method_to_support_session_config' into 5438_add_common_method_to_support_session_config 7a67ace08 [davidyuan] add common method to get session level config 3f4231735 [davidyuan] add common method to get session level config bb5d5ce44 [davidyuan] add common method to get session level config 623200ff9 [davidyuan] Merge remote-tracking branch 'origin/5438_add_common_method_to_support_session_config' into 5438_add_common_method_to_support_session_config 8011959da [davidyuan] add common method to get session level config 605ef16bc [davidyuan] Merge remote-tracking branch 'origin/5438_add_common_method_to_support_session_config' into 5438_add_common_method_to_support_session_config bb63ed87c [davidyuan] add common method to get session level config d9cf248d4 [davidyuan] add common method to get session level config c8647ef53 [davidyuan] add common method to get session level config 618c0f65e [david yuan] Merge branch 'apache:master' into 5438_add_common_method_to_support_session_config c1024bded [david yuan] Merge branch 'apache:master' into 5438_add_common_method_to_support_session_config 32028f99f [davidyuan] add common method to get session level config 03e28874c [davidyuan] add common method to get session level config Lead-authored-by: David Yuan <yuanfuyuan@mafengwo.com> Co-authored-by: davidyuan <yuanfuyuan@mafengwo.com> Co-authored-by: Angerszhuuuu <angers.zhu@gmail.com> Co-authored-by: Cheng Pan <chengpan@apache.org> Co-authored-by: Kent Yao <yao@apache.org> Co-authored-by: liangbowen <liangbowen@gf.com.cn> Co-authored-by: david yuan <51512358+davidyuan1223@users.noreply.github.com> Co-authored-by: zwangsheng <binjieyang@apache.org> Co-authored-by: yangming <261635393@qq.com> Co-authored-by: 伟程 <cheng1483x@gmail.com> Co-authored-by: weixi <weixi62961@outlook.com> Co-authored-by: fwang12 <fwang12@ebay.com> Co-authored-by: Xianxun Ye <yesorno828423@gmail.com> Co-authored-by: liupeiyue <liupeiyue@yy.com> Co-authored-by: Fantasy-Jay <13631435453@163.com> Co-authored-by: yikaifei <yikaifei@apache.org> Co-authored-by: labbomb <739955946@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
zhaoyangming
|
456b3c3f06 |
[KYUUBI #5375] JDBC Engine supports PostgreSQL
### _Why are the changes needed?_ Close #5375 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes #5416 from ymZhao1001/jdbc-pg-dialect. Closes #5375 d81988d84 [zhaoyangming] postgreSQL 915f9fb0a [yangming] change to like d8da12af5 [yangming] reformat 29c63e38e [zhaoyangming] add postgresql dependency ec328ad93 [zhaoyangming] add postgresql dependency a8944fed5 [zhaoyangming] update postgresql to postgreSQL cf7b69107 [zhaoyangming] Merge remote-tracking branch 'origin/jdbc-pg-dialect' into jdbc-pg-dialect c127aa3d3 [zhaoyangming] update postgresql to postgreSQL a693d6c34 [yangming] reformat 0d12a6ceb [zhaoyangming] add postgresql dependency c7d3fa3da [yangming] fix conflict dde1564b6 [zhaoyangming] add test info 2a49b338a [zhaoyangming] style c8ce15f29 [zhaoyangming] StringBuilder is redundant. 5d70173cf [yangming] JDBC Engine supports PostgreSQL Lead-authored-by: zhaoyangming <zhaoyangming@deepexi.com> Co-authored-by: yangming <261635393@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Xianxun Ye
|
d4320e7a6b |
[KYUUBI #5505][FLINK] Support HELP command
### _Why are the changes needed?_ resolve: #5505 Show available commands when users type 'HELP;' in the beeline. #### Solution: - Using `ExtendedParser` parse statement and return Operation of Flink engine. - Check whether the operation is HelpOperation or not. - dependency on `flink-table-planner-loader.jar`. #### **Why not using Flink SQL Client Parser(SqlCommandParserImpl) to obtain the Command enum?** Flink 1.16's approach: ``` val opt:Optional[Operation] = org.apache.flink.table.client.cli.SqlCommandParserImpl.parseCommand() check opt.get() instance of HelpOperation or not if yes return CliStrings.MESSAGE_HELP ``` Flink 1.17 & 1.18 ``` val opt: Optional[Command] = org.apache.flink.table.client.cli.parser.SqlCommandParserImpl.parseStatement() check opt.get() is Command.HELP or not if yes return CliStrings.MESSAGE_HELP ``` The `Command` class is added from Flink1.17; The `SqlCommandParserImpl` package is changed, and the method name is changed from Flink1.17; This approach requires distinguishing between different Flink versions and maintaining both implementations. It's more complicated, so abandoned. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ Closes #5585 from YesOrNo828/help. Closes #5505 e73b15e43 [Xianxun Ye] [KYUUBI #5505] [FLINK] Add job,update,delete,truncate and call statements 5943dd072 [Xianxun Ye] [KYUUBI #5505] [FLINK] Add help messages fdc2db6ab [Xianxun Ye] [KYUUBI #5505] [FLINK] Support HELP command, fixed Pan's comments a728048fc [Xianxun Ye] [KYUUBI #5505] [FLINK] Support HELP command, fixed Pan's comments 6323cfb85 [Xianxun Ye] [KYUUBI #5505] [FLINK] Support HELP command Authored-by: Xianxun Ye <yesorno828423@gmail.com> Signed-off-by: Paul Lin <paullin3280@gmail.com> |
||
|
Kang Wang
|
4668baf736 |
[KYUUBI #5507][FLINK] Support Initialize SQL in Flink Engine
### _Why are the changes needed?_ We shoud support initialize SQL to init session context the in Flink SQL (e.g. setting up catalogs). ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No. Closes #5518 from hadoopkandy/KYUUBI-5507. Closes #5507 b1720ecde [Cheng Pan] Update externals/kyuubi-flink-sql-engine/src/test/scala/org/apache/kyuubi/engine/flink/operation/FlinkEngineInitializeSuite.scala 078273824 [Paul Lin] [KYUUBI-5507] Improve codestyle 13035f3e9 [Paul Lin] Update externals/kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/FlinkSQLEngine.scala 2ce803181 [Paul Lin] [KYUUBI-5507] Improve tests a29ac3860 [Paul Lin] [KYUUBI-5507] Run engine initial SQL at Engine start b864af54f [wangkang] Merge branch 'apache:master' into KYUUBI-5507 b37d62d44 [kandy01.wang] [KYUUBI #5507] [FLINK] Support Initialize SQL in Flink Engine Lead-authored-by: Kang Wang <kandy01.wang@vipshop.com> Co-authored-by: Paul Lin <paullin3280@gmail.com> Co-authored-by: kandy01.wang <kandy01.wang@vipshop.com> Co-authored-by: Cheng Pan <pan3793@gmail.com> Co-authored-by: wangkang <525262800@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
wforget
|
0a0088bd72 |
[KYUUBI #5570] Fix memory leak when using incremental collect mode in JDBC engine
### _Why are the changes needed?_ Similar to #3885, there is also memory leak in the jdbc engine when using incremental collect mode. Duplicate of #5161 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes #5570 from wForget/hotfix. Closes #5570 dbeddc9cf [wforget] comment 0c4e499ba [wforget] fix b7c421a37 [wforget] fix 0ef21ce55 [wforget] fix f5f2e8048 [wforget] Fix memory leak when using incremental collect mode in JDBC engine Authored-by: wforget <643348094@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
davidyuan
|
26f614a504 |
[KYUUBI #5531][TEST] Fix flaky FlinkOperationOnYarnSuite by enlarging the max rows setting
### _Why are the changes needed?_ #### 1. know about this pr When we execute flink(1.17+) test case, it may throw exception when the test case is `show/stop job`, the exception desc like this ``` - execute statement - show/stop jobs *** FAILED *** org.apache.kyuubi.jdbc.hive.KyuubiSQLException: Error operating ExecuteStatement: org.apache.flink.table.gateway.service.utils.SqlExecutionException: Could not stop job 4dece26857fab91d63fad1abd8c6bdd0 with savepoint for operation 9ed8247a-b7bd-4004-875b-61ba654ab3dd. at org.apache.flink.table.gateway.service.operation.OperationExecutor.lambda$callStopJobOperation$11(OperationExecutor.java:628) at org.apache.flink.table.gateway.service.operation.OperationExecutor.runClusterAction(OperationExecutor.java:716) at org.apache.flink.table.gateway.service.operation.OperationExecutor.callStopJobOperation(OperationExecutor.java:601) at org.apache.flink.table.gateway.service.operation.OperationExecutor.executeOperation(OperationExecutor.java:434) at org.apache.flink.table.gateway.service.operation.OperationExecutor.executeStatement(OperationExecutor.java:195) at org.apache.kyuubi.engine.flink.operation.ExecuteStatement.executeStatement(ExecuteStatement.scala:64) at org.apache.kyuubi.engine.flink.operation.ExecuteStatement.runInternal(ExecuteStatement.scala:56) at org.apache.kyuubi.operation.AbstractOperation.run(AbstractOperation.scala:171) at org.apache.kyuubi.session.AbstractSession.runOperation(AbstractSession.scala:101) at org.apache.kyuubi.session.AbstractSession.$anonfun$executeStatement$1(AbstractSession.scala:131) at org.apache.kyuubi.session.AbstractSession.withAcquireRelease(AbstractSession.scala:82) at org.apache.kyuubi.session.AbstractSession.executeStatement(AbstractSession.scala:128) at org.apache.kyuubi.service.AbstractBackendService.executeStatement(AbstractBackendService.scala:67) at org.apache.kyuubi.service.TFrontendService.ExecuteStatement(TFrontendService.scala:252) at org.apache.kyuubi.shade.org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557) at org.apache.kyuubi.shade.org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542) at org.apache.kyuubi.shade.org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.kyuubi.shade.org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) at org.apache.kyuubi.shade.org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1[149](https://github.com/apache/kyuubi/actions/runs/6649714451/job/18068699087?pr=5501#step:8:150)) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.CompletionException: java.util.concurrent.CompletionException: org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint triggering task Source: tbl_a[6] -> Sink: tbl_b[7] (1/1) of job 4dece26857fab91d63fad1abd8c6bdd0 is not being executed at the moment. Aborting checkpoint. Failure reason: Not all required tasks are currently running. at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928) at org.apache.flink.table.gateway.service.operation.OperationExecutor.lambda$callStopJobOperation$11(OperationExecutor.java:617) ... 22 more Caused by: java.util.concurrent.CompletionException: java.util.concurrent.CompletionException: org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint triggering task Source: tbl_a[6] -> Sink: tbl_b[7] (1/1) of job 4dece26857fab91d63fad1abd8c6bdd0 is not being executed at the moment. Aborting checkpoint. Failure reason: Not all required tasks are currently running. at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:326) at java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:338) at java.util.concurrent.CompletableFuture.uniRelay(CompletableFuture.java:925) at java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:913) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) at org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$1(AkkaInvocationHandler.java:260) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) at org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1298) at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$null$1(ClassLoadingUtils.java:93) at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68) at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) at org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$1.onComplete(AkkaFutureUtils.java:45) at akka.dispatch.OnComplete.internal(Future.scala:299) at akka.dispatch.OnComplete.internal(Future.scala:297) at akka.dispatch.japi$CallbackBridge.apply(Future.scala:224) at akka.dispatch.japi$CallbackBridge.apply(Future.scala:221) at org.apache.kyuubi.operation.AbstractOperation.run(AbstractOperation.scala:171) ... Cause: java.lang.RuntimeException: org.apache.flink.util.SerializedThrowable:java.util.concurrent.CompletionException: org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint triggering task Source: tbl_a[6] -> Sink: tbl_b[7] (1/1) of job 4dece26857fab91d63fad1abd8c6bdd0 is not being executed at the moment. Aborting checkpoint. Failure reason: Not all required tasks are currently running. at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:326) at java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:338) at java.util.concurrent.CompletableFuture.uniRelay(CompletableFuture.java:925) at java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:913) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) at org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$1(AkkaInvocationHandler.java:260) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ... Cause: java.lang.RuntimeException: org.apache.flink.util.SerializedThrowable:org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint triggering task Source: tbl_a[6] -> Sink: tbl_b[7] (1/1) of job 4dece26857fab91d63fad1abd8c6bdd0 is not being executed at the moment. Aborting checkpoint. Failure reason: Not all required tasks are currently running. at org.apache.flink.runtime.checkpoint.DefaultCheckpointPlanCalculator.checkTasksStarted(DefaultCheckpointPlanCalculator.java:143) at org.apache.flink.runtime.checkpoint.DefaultCheckpointPlanCalculator.lambda$calculateCheckpointPlan$1(DefaultCheckpointPlanCalculator.java:105) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:[160](https://github.com/apache/kyuubi/actions/runs/6649714451/job/18068699087?pr=5501#step:8:161)4) at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRunAsync$4(AkkaRpcActor.java:453) at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68) at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:453) at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:218) at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:84) at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:[168](https://github.com/apache/kyuubi/actions/runs/6649714451/job/18068699087?pr=5501#step:8:169)) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24) ... ``` #### 2. what make the test case failed? If we want know the reason about the exception, we need to understand the process of flink executing stop job, the process line like this code space show(it's source is our bad test case, we can use this test case to solve similar problems) ``` 1. sql 1.1 create table tbl_a (a int) with ('connector' = 'datagen','rows-per-second'='10') 1.2 create table tbl_b (a int) with ('connector' = 'blackhole') 1.3 insert into tbl_b select * from tbl_a 2. start job: it will get 2 tasks abount source sink 3. show job: we can get job info 4. stop job(the main error): 4.1 stop job need checkpoint 4.2 start checkpoint, it need all task state is running 4.3 checkpoint can not get all task state is running, then throw the exception ``` Actually, in a normal process, it should not throw the exception, if this happens to your job, please check your kyuubi conf `kyuubi.session.engine.flink.max.rows`, it's default value is 1000000. But in the test case, we only the the conf's value is 10. It's the reason to make the error, this conf makes when we execute a stream query, it will cancel the when the limit is reached. Because flink's datagen is a streamconnector, so we can imagine, when we execute those sql, because our conf, it will make the sink task be canceled because the query reached 10. So when we execute stop job, flink checkpoint cannot get the tasks about this job is all in running state, then flink throw this exception. #### 3. how can we solve this problem? When your job makes the same exception, please make sure your kyuubi conf `kyuubi.session.engine.flink.max.rows`'s value can it meet your streaming query needs? Then changes the conf's value. close #5531 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes #5549 from davidyuan1223/fix_flink_test_bug. Closes #5531 ce7fd7961 [david yuan] Update externals/kyuubi-flink-sql-engine/src/test/scala/org/apache/kyuubi/engine/flink/operation/FlinkOperationSuite.scala dc3a4b9ba [davidyuan] fix flink on yarn test bug 86a647ad9 [davidyuan] fix flink on yarn test bug cbd4c0c3d [davidyuan] fix flink on yarn test bug 8b51840bc [davidyuan] add common method to get session level config bcb0cf372 [davidyuan] Merge remote-tracking branch 'origin/master' 72e7aea3c [david yuan] Merge branch 'apache:master' into master 57ec746e9 [david yuan] Merge pull request #13 from davidyuan1223/fix 56b91a321 [yuanfuyuan] fix_4186 c8eb9a2c7 [david yuan] Merge branch 'apache:master' into master 2beccb6ca [david yuan] Merge branch 'apache:master' into master 0925a4b6f [david yuan] Merge pull request #12 from davidyuan1223/revert-11-fix_4186 40e80d9a8 [david yuan] Revert "fix_4186" c83836b43 [david yuan] Merge pull request #11 from davidyuan1223/fix_4186 360d183b0 [david yuan] Merge branch 'master' into fix_4186 b61604442 [yuanfuyuan] fix_4186 e244029b8 [david yuan] Merge branch 'apache:master' into master bfa6cbf97 [davidyuan1223] Merge branch 'apache:master' into master 16237c2a9 [davidyuan1223] Merge branch 'apache:master' into master c48ad38c7 [yuanfuyuan] remove the used blank lines 55a0a43c5 [xiaoyuandajian] Merge pull request #10 from xiaoyuandajian/fix-#4057 cb1193576 [yuan] Merge remote-tracking branch 'origin/fix-#4057' into fix-#4057 86e4e1ce0 [yuan] fix-#4057 info: modify the shellcheck errors file in ./bin 1. "$@" is a array, we want use string to compare. so update "$@" => "$*" 2. `tty` mean execute the command, we can use $(tty) replace it 3. param $# is a number, compare number should use -gt/-lt,not >/< 4. not sure the /bin/kyuubi line 63 'exit -1' need modify? so the directory bin only have a shellcheck note in /bin/kyuubi dd39efdeb [袁福元] fix-#4057 info: 1. "$@" is a array, we want use string to compare. so update "$@" => "$*" 2. `tty` mean execute the command, we can use $(tty) replace it 3. param $# is a number, compare number should use -gt/-lt,not >/< Lead-authored-by: davidyuan <yuanfuyuan@mafengwo.com> Co-authored-by: david yuan <51512358+davidyuan1223@users.noreply.github.com> Co-authored-by: yuanfuyuan <1406957364@qq.com> Co-authored-by: yuan <yuanfuyuan@mafengwo.com> Co-authored-by: davidyuan1223 <51512358+davidyuan1223@users.noreply.github.com> Co-authored-by: xiaoyuandajian <51512358+xiaoyuandajian@users.noreply.github.com> Co-authored-by: 袁福元 <yuanfuyuan@mafengwo.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
davidyuan
|
ed0d9976ae |
[KYUUBI #4186] Spark showProgress with JobInfo
### _Why are the changes needed?_ current version, when set `kyuubi.session.engine.spark.showProgress=true`, it will show stage's progress info,but the info only show stage's detail, now we need to add job info in this, just like ``` [Stage 1:> (0 + 1) / 2] ``` to ``` [Job 1 (0 / 1) Stages] [Stage 1:> (0 + 1) / 2] ``` **this update is useful when user want know their sql execute detail** closes #4186 ### _How was this patch tested?_ - [x] Add screenshots for manual tests if appropriate **The photo show match log**  - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes #5410 from davidyuan1223/improvement_add_job_log. Closes #4186 d8d03c4c0 [Cheng Pan] Update externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SQLOperationListener.scala a06e9a17c [david yuan] Update SparkConsoleProgressBar.scala 854408416 [david yuan] Merge branch 'apache:master' into improvement_add_job_log 963ff18b9 [david yuan] Update SparkConsoleProgressBar.scala 9e4635653 [david yuan] Update SparkConsoleProgressBar.scala 8c04dca7d [david yuan] Update SQLOperationListener.scala 39751bffa [davidyuan] fix 4f657e728 [davidyuan] fix deleted files 86756eba7 [david yuan] Merge branch 'apache:master' into improvement_add_job_log 0c9ac27b5 [davidyuan] add showProgress with jobInfo Unit Test d4434a0de [davidyuan] Revert "add showProgress with jobInfo Unit Test" 84b1aa005 [davidyuan] Revert "improvement_add_job_log fix" 66126f96e [davidyuan] Merge remote-tracking branch 'origin/improvement_add_job_log' into improvement_add_job_log 228fd9cf3 [davidyuan] add showProgress with jobInfo Unit Test 055e0ac96 [davidyuan] add showProgress with jobInfo Unit Test e4aac34bd [davidyuan] Merge remote-tracking branch 'origin/improvement_add_job_log' into improvement_add_job_log b226adad8 [davidyuan] Merge remote-tracking branch 'origin/improvement_add_job_log' into improvement_add_job_log a08799ca0 [david yuan] Update externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/StageStatus.scala a991b68c4 [david yuan] Update externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/StageStatus.scala d12046dac [davidyuan] add showProgress with jobInfo Unit Test 10a56b159 [davidyuan] add showProgress with jobInfo Unit Test a973cdde6 [davidyuan] improvement_add_job_log fix 1. provide Option[Int] with JobId e8a510891 [davidyuan] improvement_add_job_log fix 1. provide Option[Int] with JobId 7b9e874f2 [davidyuan] improvement_add_job_log fix 1. provide Option[Int] with JobId 5b4aaa8b5 [davidyuan] improvement_add_job_log fix 1. fix new end line 2. provide Option[Int] with JobId 780f9d15e [davidyuan] improvement_add_job_log fix 1. remove duplicate synchronized 2. because the activeJobs is ConcurrentHashMap, so reduce synchronized 3. fix scala code style 4. change forEach to asScala code style 5. change conf str to KyuubiConf.XXX.key 59340b713 [davidyuan] add showProgress with jobInfo Unit Test af05089d4 [davidyuan] add showProgress with jobInfo Unit Test c07535a01 [davidyuan] [Improvement] spark showProgress can briefly output info of the job #4186 d4bdec798 [yuanfuyuan] fix_4186 9fa8e73fc [davidyuan] add showProgress with jobInfo Unit Test 49debfbe3 [davidyuan] improvement_add_job_log fix 1. provide Option[Int] with JobId 5cf8714e0 [davidyuan] improvement_add_job_log fix 1. provide Option[Int] with JobId 249a422b6 [davidyuan] improvement_add_job_log fix 1. provide Option[Int] with JobId e15fc7195 [davidyuan] improvement_add_job_log fix 1. fix new end line 2. provide Option[Int] with JobId 4564ef98f [davidyuan] improvement_add_job_log fix 1. remove duplicate synchronized 2. because the activeJobs is ConcurrentHashMap, so reduce synchronized 3. fix scala code style 4. change forEach to asScala code style 5. change conf str to KyuubiConf.XXX.key 32ad0759b [davidyuan] add showProgress with jobInfo Unit Test d30492e46 [davidyuan] add showProgress with jobInfo Unit Test 6209c344e [davidyuan] [Improvement] spark showProgress can briefly output info of the job #4186 56b91a321 [yuanfuyuan] fix_4186 Lead-authored-by: davidyuan <yuanfuyuan@mafengwo.com> Co-authored-by: davidyuan <davidyuan1223@gmail.com> Co-authored-by: david yuan <51512358+davidyuan1223@users.noreply.github.com> Co-authored-by: yuanfuyuan <1406957364@qq.com> Co-authored-by: Cheng Pan <pan3793@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Xianxun Ye
|
3646ae08bd
|
[KYUUBI #5405] [FLINK] Support Flink 1.18
### _Why are the changes needed?_ Resolve: #5405 Support the Flink 1.18 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes #5465 from YesOrNo828/flink-1.18. Closes #5405 a0010ca14 [Xianxun Ye] [KYUUBI #5405] [FLINK] Remove flink1.18 rc repo 2a4ae365c [Xianxun Ye] Update .github/workflows/master.yml d4d458dc7 [Xianxun Ye] [KYUUBI #5405] [FLINK] Update the flink1.18-rc3 repo 99172e3da [Xianxun Ye] [KYUUBI #5405] [FLINK] Using the staging repo during the RC stage 4c0cf887b [Xianxun Ye] [KYUUBI #5405] [FLINK] Using the staging repo during the RC stage c74f5c31b [Xianxun Ye] [KYUUBI #5405] [FLINK] fixed Pan's comments. 1933ebadd [Xianxun Ye] [KYUUBI #5405] [FLINK] Support Flink 1.18 Authored-by: Xianxun Ye <yesorno828423@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
liupeiyue
|
9a409956a1 |
[KYUUBI #5282] Support configure Trino session conf in kyuubi-default.conf
### _Why are the changes needed?_ Refer to spark and flink settings conf, support configure Trino session conf in kyuubi-default.conf issue : https://github.com/apache/kyuubi/issues/5282 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes #5283 from ASiegeLion/kyuubi-master-trino. Closes #5282 87a3f57b4 [Cheng Pan] Apply suggestions from code review effdd79f4 [liupeiyue] [KYUUBI #5282]Add trino's session conf to kyuubi-default.xml 399a200f7 [liupeiyue] [KYUUBI #5282]Add trino's session conf to kyuubi-default.xml 7462b32c2 [liupeiyue] [KYUUBI #5282]Add trino's session conf to kyuubi-default.xml--Update documentation 5295f5f94 [liupeiyue] [KYUUBI #5282]Add trino's session conf to kyuubi-default.xml Lead-authored-by: liupeiyue <liupeiyue@yy.com> Co-authored-by: Cheng Pan <pan3793@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Fantasy-Jay
|
81964803c9 |
[KYUUBI #5382][JDBC] Duplication cleanup improvement in JdbcDialect and schema helpers
### _Why are the changes needed?_ To close #5382. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes #5490 from zhuyaogai/issue-5382. Closes #5382 4757445e7 [Fantasy-Jay] Remove unrelated comment. f68c7aa6c [Fantasy-Jay] Refactor JDBC engine to reduce to code duplication. 4ad6b3c53 [Fantasy-Jay] Refactor JDBC engine to reduce to code duplication. Authored-by: Fantasy-Jay <13631435453@163.com> Signed-off-by: liangbowen <liangbowen@gf.com.cn> |
||
|
zml1206
|
8d2c8d1009
|
[KYUUBI #5449] Bump Delta Lake 3.0.0
### _Why are the changes needed?_ close [#5449](https://github.com/apache/kyuubi/issues/5449). Unlike the initial preview release, Delta Spark 3.0.0 is now built on top of Apache Spark™ 3.5. Delta Spark maven artifact has been renamed from delta-core to delta-spark. https://github.com/delta-io/delta/releases/tag/v3.0.0 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes #5450 from zml1206/5449. Closes #5449 a7969ed6a [zml1206] bump Delta Lake 3.0.0 Authored-by: zml1206 <zhuml1206@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
ITzhangqiang
|
e51095edaa
|
[KYUUBI #5365] Don't use Log4j2's extended throwable conversion pattern in default logging configurations
### _Why are the changes needed?_ The Apache Spark Community found a performance regression with log4j2. See https://github.com/apache/spark/pull/36747. This PR to fix the performance issue on our side. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No. Closes #5400 from ITzhangqiang/KYUUBI_5365. Closes #5365 dbb9d8b32 [ITzhangqiang] [KYUUBI #5365] Don't use Log4j2's extended throwable conversion pattern in default logging configurations Authored-by: ITzhangqiang <itzhangqiang@163.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
sychen
|
99789a82c1 |
[KYUUBI #5339] [MINOR] Avoid using kyuubi.session.engine.initialize.timeout string
### _Why are the changes needed?_ Replace string literal with constant variable ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes #5339 from cxzl25/use_engine_init_timeout_key. Closes #5339 bef2eaa4a [sychen] fix Authored-by: sychen <sychen@ctrip.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |