<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Please see #1204 for details.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1213 from wForget/KYUUBI-1204.
Closes#1204
fa88ffbe [wForget] [KYUUBI #1204] Change the default value of kyuubi.ha.zookeeper.auth.type to NONE
13cd7ecd [wForget] Merge branch 'master' of https://github.com/apache/incubator-kyuubi into dev-1.3.0
3a84c0e1 [wForget] [KYUUBI #1204] fix tests and re-generate settings.md
92c331c7 [wForget] [KYUUBI #1204] Adjust the parameters of zookeeper authentication
db016a03 [wForget] [KYUUBI #1204] Adjust the logic of obtaining the keytab file path
8adb812b [wForget] [KYUUBI #1204] Enhance zookeeper authentication and acls
Authored-by: wForget <643348094@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
…
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1256 from yanghua/KYUUBI-1252.
Closes#1252
8a41a298 [yanghua] Address review suggesion
c78a850d [yanghua] Address review suggesiont
e6aa86c2 [yanghua] Refactor code to address review suggestion
ed39d2dd [yanghua] [KYUUBI #1252] Move KyuubiConf#toSparkPrefixedConf out of kyuubi-common module
Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Support Spark 3.2.0.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1228 from pan3793/32.
Closes#1228
8f2dfd07 [Cheng Pan] Skip Interval test
fcc4efc9 [Cheng Pan] Handle SPARK-35378
a7061650 [Cheng Pan] Add maven profile spark-3.2
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
The previous default value of `kyuubi.engine.initialize.sql` and `kyuubi.engine.session.initialize.sql` both are `SHOW DATABASES`.
The intention of `SHOW DATABASES`is to eagerly active HiveClient, it's quite redundant for executing many times.
This PR proposes to change `kyuubi.engine.session.initialize.sql`'s default value to `Nil`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1248 from pan3793/sess-init.
Closes#1248
ad2bdd7d [Cheng Pan] Update doc
0736857c [Cheng Pan] Change kyuubi.engine.session.initialize.sql default value to Nil
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add hudi columns type test.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1239 from simon824/huditest.
Closes#703
7266ff62 [simon] [KYUUBI #703] [FOLLOWUP] [TEST] Enhance hudi test
5ea742c0 [simon] [KYUUBI #703] [FOLLOWUP] [TEST] Enhance hudi test
ce7b4068 [simon] [KYUUBI #703] [FOLLOWUP] [TEST] Enhance hudi test
5582ac3e [simon] huditest
Authored-by: simon <zhangshiming@cvte.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
This partial revert [KYUUBI #1203], recover codecov
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1238 from pan3793/mvn.
Closes#1238
b3ba87b2 [Cheng Pan] remove unnecessary changes
c124340a [Cheng Pan] nit
39c436be [Cheng Pan] pom
28546cbd [Cheng Pan] CI
f22e9c91 [Cheng Pan] nit
72e9542d [Cheng Pan] Rebase master
7cd1cba3 [Cheng Pan] [BUILD] Add back Scala stuff to root POM
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Sub task of #1131
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1152 from pan3793/ut-kyuubi-hive-jdbc.
Closes#1152
2f508356 [Cheng Pan] Avoid hardcode Hive Jdbc class
5032d23d [Cheng Pan] [KYUUBI #1131][TEST] Replace hive-jdbc by kyuubi-hive-jdbc in UT
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Support incremental collection, [SPARK-25224](https://issues.apache.org/jira/browse/SPARK-25224)
Introduce new conf: `kyuubi.operation.incremental.collect`
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1198 from pan3793/inc-col.
Closes#1198
946068e9 [Cheng Pan] Address comments
2798d0d8 [Cheng Pan] Correct conf doc
3720fd41 [Cheng Pan] Incremental collection
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
For more detail, please go to https://github.com/apache/incubator-kyuubi/issues/981
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1187 from timothy65535/ky-1159.
Closes#1159
52daf139 [timothy65535] update conf md
a9b50843 [timothy65535] improve patch
36676f2a [timothy65535] improve patch
892a6333 [timothy65535] [KYUUBI #1159] Add Session stats on Kyuubi Query Engine Page
Authored-by: timothy65535 <timothy65535@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
The detail locates #1206
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1207 from yaooqinn/1206.
Closes#1206
99e324a9 [Kent Yao] [KYUUBI #1206][FEATURE] Support GROUP for engine.share.level
f41fd944 [Kent Yao] Merge branch 'master' into 1206
a7cb80da [Kent Yao] [KYUUBI #1206][FEATURE] Support GROUP for engine.share.level
6ecef063 [Kent Yao] [KYUUBI #1206][FEATURE] Support GROUP for engine.share.level
d7ff7583 [Kent Yao] [KYUUBI #1206][FEATURE] Support GROUP for engine.share.level
7b3b2f53 [Kent Yao] [KYUUBI #1206][FEATURE] Support GROUP for engine.share.level
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Improve test coverage.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1211 from ulysses-you/minikube.
Closes#1211
f02edeca [ulysses-you] test
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Customers might meet exception when launching a spark engine, especially for beginner, now kyuubi only show the last line of engine log and it is difficult to find the root cause in customer side.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1205 from turboFei/show_more_engine_log.
Closes#1205
72002dd6 [fwang12] address comments
a74fa1de [fwang12] update docs
fd967371 [fwang12] refactor
434a093d [fwang12] update docs
880c8cc2 [fwang12] show more engine failure log for debug
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
#1131
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1203 from pan3793/pom.
Closes#1203
8e6b8f9b [Cheng Pan] fix
d31fd198 [Cheng Pan] fix
9da82843 [Cheng Pan] fix
6d150de9 [Cheng Pan] fix
9faf77e5 [Cheng Pan] Remove Scala stuff from kyuubi-hive-jdbc pom
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
bugfix:
#1066
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1183 from zhang1002/auto-create-log-root.
Closes#1066
d686e33c [zhang1002] 1. change logRoot to engineLogRoot 2. add some judge for mkDir
fb6987f5 [zhang1002] create log root automatically
c319127e [张宇翔] test
Lead-authored-by: zhang1002 <zhang1002@126.com>
Co-authored-by: 张宇翔 <zhang1002@126.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Now, restricted and ingored keys must be complete. We can support eding with '.*' to restricted a list of keys.
### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1185 from hddong/support-perfix-ignore.
Closes#1184
dade3301 [hongdongdong] reset
c58b0aa6 [hongdongdong] [KYUUBI#1184] Support ending with .* to restrict and ingore session config
Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Support to config operation log root dir with kyuubi configuration both for server and engine sides.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1160 from turboFei/support_config_operationlog_dir.
Closes#1160
ed424f46 [fwang12] fix ut
5edfd4ae [fwang12] refactor ut
b7026b2f [fwang12] add ut
6ff4d891 [fwang12] save
30cd5952 [fwang12] fix ut
f058619a [fwang12] fix ut
96eb020d [fwang12] refactor
ae0ded96 [fwang12] set config value for testing
83d932f4 [fwang12] exclude
33e3e214 [fwang12] update
6f030cba [fwang12] fix
77ec9ca2 [fwang12] save
711ececf [fwang12] Fix code style
cfa3bfac [fwang12] refactor
b299f8cf [fwang12] complete ut
26de3712 [fwang12] Support to config operation log root dir for kyuubi server and engine side
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Close#1146.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1156 from byyue/feature-1146.
Closes#1146
6b677723 [Brian Yue] Move KinitAuxiliaryService to kyuubi-server module
Authored-by: Brian Yue <code.byyue@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
For more detail, please go to https://github.com/apache/incubator-kyuubi/issues/981
`EngineStatusStore` helps to push events to listener bus
`EngineStatusStore` is a memory store that tracking the number of statements and sessions, it provides:
- stores all elements, and sorted by startTimestamp.
- cleanup the last elements when reach a certain threshold.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1023 from timothy65535/ky-1022.
Closes#1022
b9f355c4 [timothy65535] [KYUUBI #1022] Add basic EngineStatusStore for events
Authored-by: timothy65535 <timothy65535@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
- Kerberos login module outputs lots of debug level log records to stdout, which is annoying at most of the time.
- MiniYARNCluster already outputs most of the log records to unit-tests.log. But some referred classes, such as `GuiceComponentProviderFactory` use java.util.logging.Logging and output log records to stdout.
In this PR, we turn off kerberos logging by default and allow users to turn on it by setting java system property `sun.security.krb5.debug`. java.util.logging.Logging is redirected to slf4j using `SLF4JBridgeHandler`.
**Note:** MiniDFSCluster outputs a few log records using `System.stdout`/`System.stderr`.
```
Formatting using clusterid: testClusterID
isSaslEnabled:false
Opened streaming server at /0.0.0.0:1025
Successfully obtained privileged resources (streaming port = ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=1025] ) (http listener port = 9864)
Opened info server at /0.0.0.0:9864
```
It should be acceptable.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1150 from zhouyifan279/KYUUBI#1145.
Closes#1145
5c91816d [zhouyifan279] [KYUUBI #1145] Suppress MiniDFSCluster/MiniYARNCluster/Kerberos output in tests
1a9f6345 [zhouyifan279] [KYUUBI #1145] Suppress MiniDFS/YARNCluster/ Kerberos output in tests
9877085a [zhouyifan279] [KYUUBI #1145] Suppress MiniDFS/YARNCluster/ Kerberos output in tests
8b05cb02 [zhouyifan279] [KYUUBI #1145] Suppress MiniDFS/YARNCluster/ Kerberos output in tests
435a4715 [zhouyifan279] [KYUUBI #1145] Suppress MiniDFS/YARNCluster/ Kerberos output in tests
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
After this PR, the service named service discovery will be a child service of frontend service and it's optional.
A servalable will contain a backend and a list of frontends, the former one stay AS-IS and the latter is changed from single to multiple, which means a serverable can have multiple interfaces to interact with its different clients
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1144 from yaooqinn/1143.
Closes#1143
f2beb721 [Kent Yao] fix test
128cf282 [Kent Yao] nit
37aaff9c [Kent Yao] lift FrontendService trait
add335c4 [Kent Yao] nit
3a111cba [Kent Yao] [KYUUBI #1143] Refact Serverable/FrontendService/ServiceDiscovery hierarchy
bfa63b81 [Kent Yao] [KYUUBI #1143] Refact Serverable/FrontendService/ServiceDiscovery hierarchy
7e5a5cbb [Kent Yao] Merge branch 'master' into 1143
09263bc0 [Kent Yao] [KYUUBI #1143] Refact Serverable/FrontendService/ServiceDiscovery hierarchy
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
…
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1142 from yanghua/KYUUBI-1137.
Closes#1137
418f3dff [yanghua] [KYUUBI #1137] KyuubiFrontendServices oom hook should call the stop on serverable level
Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Introduce a session-level configuration `operation.plan.only.mode` to change ExecuteStatement to PlanOnlyStatement,
PlanOnlyStatement will only parse, analyze or optimize the statements instead of executing them.
This feature could run all Spark SQLs to verify the semantic compatibility without having side effects, which is very useful for testing, migrating without query rewrite.
This is similar to the `explain` command
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1060 from yaooqinn/1059.
Closes#1059
e0aa1060 [Kent Yao] [KYUUBI #1059] Add Plan Only Operations
4104aa42 [Kent Yao] [KYUUBI #1059] Add Plan Only Operations
aae52f26 [Kent Yao] [KYUUBI #1059] Add Plan Only Operations
575dbf8f [Kent Yao] [KYUUBI #1059] Add Plan Only Operations
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Umbrella issue #915 is finished. Document is needed to tell user how to enable it.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1101 from zhouyifan279/KYUUBI#1090.
Closes#1090
fdd4da8a [zhouyifan279] [KYUUBI #1090] Add deployment document about Hadoop Credentials Manager
418da1dc [zhouyifan279] [KYUUBI #1090] Add deployment document about Hadoop Credentials Manager
0e8e44ba [zhouyifan279] [KYUUBI #1090] Add deployment document about Hadoop Credentials Manager
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
This PR finishes issue #916
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1074 from zhouyifan279/KYUUBI#916.
Closes#916
28176c7e [zhouyifan279] [KYUUBI #916] Send DelegationTokens to SQL engines
b6c931bd [zhouyifan279] [KYUUBI #916] Send DelegationTokens to SQL engines
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
bug:
#1068
This version, event log don't support upload file to hdfs
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1075 from zhang1002/branch-1.3_supportLogUploadHdfs.
Closes#1068
7b1483c5 [张宇翔] Modify some UT
27aa4596 [张宇翔] Modify some UT
c8223914 [张宇翔] Modify some UT
8b3ef67b [张宇翔] Modify some UT
fa06ead9 [张宇翔] Merge branch 'master' into branch-1.3_supportLogUploadHdfs
8384a6e2 [张宇翔] Merge remote-tracking branch 'upstream/master'
8801ece7 [张宇翔] Remove "toString" from ut
3288724b [张宇翔] Modify some UT
0bd7ecbd [张宇翔] Support the eventLog upload to hdfs
714ab1b8 [张宇翔] Merge remote-tracking branch 'upstream/master'
af28520b [张宇翔] Merge remote-tracking branch 'upstream/master'
c328c884 [张宇翔] Merge remote-tracking branch 'upstream/master'
abcaae1f [张宇翔] Merge remote-tracking branch 'upstream/master'
f0a7e566 [张宇翔] Merge remote-tracking branch 'upstream/master'
112aa6b6 [张宇翔] Merge remote-tracking branch 'upstream/master'
c55652fd [张宇翔] Merge remote-tracking branch 'upstream/master'
ef5e8c55 [张宇翔] Merge remote-tracking branch 'upstream/master'
5a75384c [张宇翔] Merge remote-tracking branch 'upstream/master'
8da4f7ed [张宇翔] Merge remote-tracking branch 'upstream/master'
a7ba1bfc [张宇翔] Merge remote-tracking branch 'upstream/master'
b662a989 [张宇翔] Merge remote-tracking branch 'upstream/master'
4c8f3b87 [张宇翔] Merge remote-tracking branch 'upstream/master'
8b686767 [张宇翔] Merge remote-tracking branch 'upstream/master'
cf99e309 [张宇翔] Merge remote-tracking branch 'upstream/master'
0afaa578 [张宇翔] Merge remote-tracking branch 'upstream/master'
b24fea07 [张宇翔] Merge remote-tracking branch 'upstream/master'
e517cfc5 [张宇翔] Merge remote-tracking branch 'upstream/master'
18aebe76 [张宇翔] Merge remote-tracking branch 'upstream/master'
f248bef7 [张宇翔] Merge remote-tracking branch 'upstream/master'
5ffb54f3 [张宇翔] Add kyuubi-spark-monitor module for nightly.yml
Authored-by: 张宇翔 <zhang1002@126.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Move stringifyException from KyuubiSQLException to Utils, make it more general.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1081 from timothy65535/ky-1080.
Closes#1080
ce3015d5 [timothy65535] [KYUUBI #1080] Move stringifyException to Utils
Authored-by: timothy65535 <timothy65535@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Reduce hudi spark extraJars deps , it is unnecessary to submit all jars.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1072 from simon824/master.
Closes#703
2bb4af06 [simon] 1
e2164fa6 [simon] 1
dfc79ba6 [simon] && -> ||
6791f95b [simon] reduce hudi spark extraJars deps
ed4dc4b5 [Simon] Merge branch 'apache:master' into master
ea5db291 [simon] remove space
3d002afd [simon] fix "File line length exceeds 100 characters"
1939038e [simon] roll back
d25d3721 [simon] roll back
41232e55 [simon] remove dataLakeSuiteMixin tableOptions
68d6e891 [simon] fix error commit
b802603c [simon] add getTableOption
f6d37426 [Simon] Merge branch 'apache:master' into master
8b3b4b85 [simon] add tableOptions
Lead-authored-by: simon <zhangshiming@cvte.com>
Co-authored-by: Simon <3656562@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This PR finishes the work of issue #1009
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1053 from zhouyifan279/KYUUBI#1009.
Closes#1009
342fca91 [zhouyifan279] [KYUUBI #1009] Implement Hive metastore server Delegation Token Provider
a9e1b34d [zhouyifan279] [KYUUBI #1009] Implement Hive metastore server Delegation Token Provider
82da9007 [zhouyifan279] [KYUUBI #1009] Implement Hive metastore server Delegation Token Provider
2c00dd3a [zhouyifan279] [KYUUBI #1009] Implement Hive metastore server Delegation Token Provider
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1062 from qiuliang988/1034.
Closes#1034
41d510f5 [qiuliang] add comments
dbd7c770 [qiuliang] [KYUUBI #1034]Engine may deadlock when close operationLog
Authored-by: qiuliang <qlcumt@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
Add create table options for DataLakeSuiteMixin
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1040 from simon824/master.
Closes#1040
ea5db291 [simon] remove space
3d002afd [simon] fix "File line length exceeds 100 characters"
1939038e [simon] roll back
d25d3721 [simon] roll back
41232e55 [simon] remove dataLakeSuiteMixin tableOptions
68d6e891 [simon] fix error commit
b802603c [simon] add getTableOption
f6d37426 [Simon] Merge branch 'apache:master' into master
8b3b4b85 [simon] add tableOptions
Lead-authored-by: simon <zhangshiming@cvte.com>
Co-authored-by: Simon <3656562@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This PR finishes the work of task #1008
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1029 from zhouyifan279/KYUUBI#1008.
Closes#1008
695cd955 [zhouyifan279] [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
da50aaef [zhouyifan279] [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
bf7c628f [zhouyifan279] [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
ed9739ab [zhouyifan279] [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
2512abf9 [zhouyifan279] [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
9c88f721 [zhouyifan279] [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Remove session hande in `AbstractSession`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1046 from ulysses-you/kyuubi-1032-f.
Closes#1032
1d7543a9 [ulysses-you] order
36a0a1cc [ulysses-you] refine
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
HBase deps is optional for creating hudi table and query, but required for writing data to hudi table, as currently we only test create table and query for data lake formats, it's reasonable to remove the hbase deps to reduce the deps pulled from hudi.
This PR also change all data lake test suites in spark engine from `hive` catalog to `in-memory` catalog.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1028 from pan3793/hudi.
Closes#703
56c41a30 [Cheng Pan] Use in-memory catalog for DataLake test
b582a865 [Cheng Pan] Revert "Default use in-memory in Spark engine test"
e62c403a [Cheng Pan] Default use in-memory in Spark engine test
e2b8e0fd [Cheng Pan] Remove unused hudi spark conf
069fbacc [Cheng Pan] Revert "HFileBootstrapIndex"
ca1b20e1 [Cheng Pan] HFileBootstrapIndex
da37b0fc [Cheng Pan] Remove log4j
41ce8526 [Cheng Pan] Align jackson version
f56f5e89 [Cheng Pan] Remove HBase dependencies
c2b76b54 [Cheng Pan] Exclude hudi-timeline-service
56641762 [Cheng Pan] nit
e588c12f [Cheng Pan] [DEPS][TEST] Reduce Hudi dependencies
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Hudi has add Spark Sql Support in version 0.9.0.
This PR add Hudi-0.9.0 dependency and Unit Testing #703
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#994 from simon824/master.
Closes#703
a04b2e42 [simon] exclude jdk.tools dep
50e1ab01 [simon] fix jdk11 dep
f33ebdf1 [simon] fix hudi dep conflicts
ff2585c2 [simon] fix yarn dep conflicts
32dd1ea8 [simon] fix scala version conflicts
b8a37401 [simon] add spark.sql.catalogImplementation
82f5422f [simon] fix spark3 dependency
2a6c497d [simon] fix spark3 dependency
4dbec8be [Simon] Merge branch 'apache:master' into master
3f180157 [simon] fix maven
73e48d0a [simon] add spark3 support maven dependency
3def658b [simon] fix missing spark-sql-engine hudi maven dependency
524132d3 [simon] bugfix
e98998a6 [simon] fix pom error
59fc6669 [Simon] Merge branch 'apache:master' into master
d9e17ebf [simon] fix conflict
2466ece3 [simon] add sparkSQL test on hudi-0.9.0
Lead-authored-by: simon <zhangshiming@cvte.com>
Co-authored-by: Simon <3656562@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1003 from yanghua/KYUUBI-999.
Closes#999
78fdf542 [yanghua] Changed rest frontend service port from 10009 to 10099
73929d5d [yanghua] [KYUUBI #999] Build the basic framework for rest frontend service
Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
This PR finishes sub-task #1007 under umbrella issue #913.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1015 from zhouyifan279/KYUUBI#1007.
Closes#1007
94263804 [zhouyifan279] [KYUUBI #1007] Implement delegation token renewal framework
6cea96d0 [zhouyifan279] [KYUUBI #1007] Implement delegation token renewal framework
dbffcf15 [zhouyifan279] [KYUUBI #1007] Implement delegation token renewal framework
57100271 [zhouyifan279] [KYUUBI #1007] Implement delegation token renewal framework
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1016 from yanghua/KYUUBI-1011.
Closes#1011
78107b76 [yanghua] Addressed review suggestion
651b5989 [yanghua] [KYUUBI #1011] Fallback some frontend service config option about thrift
Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
`subdomain` is a widely used term in computer science
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1010 from yaooqinn/subdomain.
Closes#1010
72ed19a3 [Kent Yao] address comments
4dcae414 [Kent Yao] fix test
c3bb788d [Kent Yao] Rename engine.share.level.sub.domain to engine.share.level.subdomain
b7762357 [Kent Yao] Rename engine.share.level.sub.domain to engine.share.level.subdomain
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Use underscore instead hyphen in path, friendly to sql.
### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1000 from hddong/fix-underscore.
Closes#1000
8b3459a5 [hongdongdong] Use underscore instead hyphen in path
Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
closes https://github.com/apache/incubator-kyuubi/issues/988
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#989 from ulysses-you/kyuubi-session-event.
Closes#988
baebf2eb [ulysses-you] address comment
283d574c [ulysses-you] fix
c5d1d5f7 [ulysses-you] adddress comment
44cfd06f [ulysses-you] docs
43639993 [ulysses-you] simplify session
2618b764 [ulysses-you] config name
a2d4608b [ulysses-you] docs
5adbe5c8 [ulysses-you] config
ea5c70e1 [ulysses-you] nit
eea73ed5 [ulysses-you] config
cda6d38b [ulysses-you] test
a754c76f [ulysses-you] init
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
…
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#984 from yanghua/KYUUBI-983.
Closes#983
db953b8a [yanghua] Rename KyuubiFrontendService to be KyuubiFrontendServices
8c3d7f2a [yanghua] [KYUUBI #983] Refactor the basic abstraction of frontend service
Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
For detail, please go to _Originally posted by yaooqinn in https://github.com/apache/incubator-kyuubi/issues/962#issuecomment-905116261_
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
```
./bin/beeline -u 'jdbc:hive2://localhost:10009/;?kyuubi.engine.pool.enabled=true;kyuubi.engine.pool.size=3;'
```
Closes#986 from timothy65535/ky-962-v2.
Closes#962
b79f8778 [timothy65535] fix test name
9a435607 [timothy65535] fix style
abc746b8 [timothy65535] update tests
71b6933d [timothy65535] revert changes
018dee8b [timothy65535] revert changes
e3492934 [timothy65535] revert changes
67fc8fd9 [timothy65535] improve threshold
fa52acb2 [timothy65535] update pool threshold
d8b5977d [timothy65535] remove dup get
7f39b8d7 [timothy65535] fix test error
4fdf8d76 [timothy65535] update thread pool size to 1 -> threshold
2477348a [timothy65535] rich engine pool tests
8eee427f [timothy65535] fix update error msg
104be5ea [timothy65535] fix error tests
8c2d1441 [timothy65535] fix doc
ddd0eda9 [timothy65535] improve doc & tests
83d7c5cf [timothy65535] update doc
4b6c7ee3 [timothy65535] improve subdomain
169d9d06 [timothy65535] update todo
1424e565 [timothy65535] increase pool size range
3523cd6c [timothy65535] update todo
2cbeafb2 [timothy65535] remove enable option
424b5364 [timothy65535] revert ServiceDiscovery
6fc1c510 [timothy65535] remove random policy
9feef8f3 [timothy65535] [KYUUBI #962] Support engine pool feature
Authored-by: timothy65535 <timothy65535@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Track operation state change and support output to custom system.
### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#968 from hddong/add-server-events.
Closes#922
8be8f268 [hongdongdong] Add prefix for StatementEvent
9fdcc504 [hongdongdong] fix comment
a0cdb35d [hongdongdong] fix test
4d340c67 [hongdongdong] add dependency
00920751 [hongdongdong] [KYUUBI#922]Add OperationEvent for server to track operation state
Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#980 from yanghua/KPIP-1.
Closes#979
e67e37a4 [yanghua] [KYUUBI 979] Introduce AbstractFrontendService and KyuubiFrontendService
Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
Align with `sessionManager.operationManager.closeOperation(operationHandle)` and remove first
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#975 from iodone/fix-opHanle-operation-not-release.
Closes#961
bd798e0d [odone] [KYUUBI #961] fixed: opHanle's operation may not be release
Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
- Remove the `scala.util.Using`
- Remove redundant license declaration
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#973 from pan3793/license.
Closes#973
b02206ea [Cheng Pan] Remove redundant license declaration
3a1401d9 [Cheng Pan] [LICENSE] Remove Scala Using
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Workaround to recover Minikube test until #937 is solved.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#969 from pan3793/747-ignore.
Closes#659
e26153f6 [Cheng Pan] [KYUUBI #659][FOLLOWUP] Temporarily ignore udf engine_name test
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Add UDF engine_name to show the runtime information about execution engine
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#747 from zhaomin1423/udf.
Closes#659
8c3b3c16 [Min Zhao] Merge branch 'master' into udf
f184a661 [Min Zhao] Merge branch 'master' into udf
9ea0c190 [Min Zhao] get appName from driver side
09f07523 [Min Zhao] get conf by SparkEnv.get.conf in the engine_name udf
6c000784 [Min Zhao] update the description of the engine_name udf
63ef2762 [Min Zhao] update class to org.apache.kyuubi.engine.spark.udf.KyuubiDefinedFunctionSuite in functions.md
db02884c [Min Zhao] [KYUUBI #659] Add UDF engine_name
Lead-authored-by: Min Zhao <zhaomin1423@163.com>
Co-authored-by: Min Zhao <49054376+zhaomin1423@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
