Wanqiang Ji
9454aa96f9
[KYUUBI #476 ] Add ignore list and and restrict list in server
...
 [](https://github.com/yaooqinn/kyuubi/pull/505 )     [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png ">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info )<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
### _Why are the changes needed?_
Add ignore list and and restrict list in server
### _How was this patch tested?_
Run local test
Closes #505 from jiwq/476.
Closes #476
fcd7a22 [Wanqiang Ji] Add ignore list and and restrict list in server
Authored-by: Wanqiang Ji <wanqiang.ji@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-04-11 13:02:50 +08:00
fwang12
5b9ef3c3df
[KYUUBI #483 ] Add configEntry for zookeeper node timeout
...
 [](https://github.com/yaooqinn/kyuubi/pull/484 )    [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png ">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info )<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues , add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
Add configEntry for zookeeper node timeout instead of using constant value
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests ) locally before make a pull request
Closes #484 from turboFei/KYUUBI_304_znode_timeout.
Closes #483
786b915 [fwang12] address comments
5c8dbc1 [fwang12] [KYUUBI #483 ] Add configEntry for zookeeper node timeout
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-04-05 20:00:46 +08:00
fwang12
1b7762e5f9
[KYUUBI #457 ] Support configurable initialize sql statement for engine startup
...
 [](https://github.com/yaooqinn/kyuubi/pull/462 )      [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png ">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info )<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues , add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
Currently, the engine uses hardcoded show databases to force initialization with catalog eagerly bootstrapping too.
We can make it configurable with a comma separate query list and select current_database() might be used as default since it's more lightweight than show databases in real-world cases
### _How was this patch tested?_
Existing UT.
Closes #462 from turboFei/KYUUBI-457.
Closes #457
e7bcf52 [fwang12] add ut
7446cd5 [fwang12] [KYUUBI#457】Support configurable initialize sql statement for engine startup
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-04-01 01:31:58 +08:00
Kent Yao
c6585b4708
[KYUUBI #471 ] Separate Embedded Zookeeper server from ha
...
 [](https://github.com/yaooqinn/kyuubi/pull/471 )     [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png ">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info )<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues , add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
1. Separated embedded zk from ha module, as it's only used for setup a zk from Kyuubi server for now if there is not an external one
2. Support more configuration for the embedded zookeeper, for example, HOSTNAME, then it's more close to prod usage.
3. TODO: Make the embedded zookeeper of our own, which should support as quorum peers
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests ) locally before make a pull request
Closes #471 from yaooqinn/qp.
Closes #471
1c24d02 [Kent Yao] address comments
9666a40 [Kent Yao] refine
f069fc0 [Kent Yao] refine
8bb2f3a [Kent Yao] refine
c20593a [Kent Yao] refine
8fa9a40 [Kent Yao] Merge branch 'master' into qp
1e2e1a3 [Kent Yao] tmp
7d66d05 [Kent Yao] tmp
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-30 18:34:01 +08:00
Kent Yao
a66a9054ef
[KYUUBI #470 ] Add ConfigEntry with fallback and deprecate session.timeout
...
 [](https://github.com/yaooqinn/kyuubi/pull/470 )      [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png ">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info )<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues , add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests ) locally before make a pull request
Closes #470 from yaooqinn/fallback.
Closes #470
40d3781 [Kent Yao] add unit test
74abc4b [Kent Yao] Add ConfigEntry with fallback and deprecate session.timeout
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-29 11:23:12 +08:00
Kent Yao
77ecb0d527
[KYUUBI #469 ] Using Optional Config for query timeout
...
 [](https://github.com/yaooqinn/kyuubi/pull/469 )      [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png ">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info )<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues , add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
1. bugfix for comparing a client-side value with a second unit to a server-side value with a millisecond unit
2. using OptionalConfigEntry for query timeout, it's more simple, it will take effect only when set with a valid value
3. add tests for system query timeout
4. move timeout conf getter to `initialize` to detect during Kyuubi server startup, the current behavior is a bit dangerous for deploying.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests ) locally before make a pull request
Closes #469 from yaooqinn/querytimeout.
Closes #469
a524771 [Kent Yao] Using Optional Config for query timeout
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-28 21:12:19 +08:00
Kent Yao
4bb802a146
[KYUUBI #463 ] Time configs can use long to define milliseconds directly
...
 [](https://github.com/yaooqinn/kyuubi/pull/463 )     [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png ">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info )<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues , add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Allow a time config with a long string representation directly, which is more convenient for end-users
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests ) locally before make a pull request
Closes #463 from yaooqinn/timeconf.
a42759e [Kent Yao] tests
def0596 [Kent Yao] tests
acc96e3 [Kent Yao] update tests
a5d0827 [Kent Yao] Time configs can use long to define milliseconds directly
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-27 16:58:50 +08:00
Kent Yao
768d583767
[KYUUBI #459 ] Enable Configuration Checker
...
 [](https://github.com/yaooqinn/kyuubi/pull/459 )      [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png ">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info )<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues , add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Whenever ,there are new configs, the corresponding doc should also be updated
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests ) locally before make a pull request
Closes #459 from yaooqinn/settinglock.
6e1e4d3 [Kent Yao] Enable Configuration Checker
30e7100 [Kent Yao] Enable Configuration Checker
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-26 17:43:48 +08:00
ulysses-you
fecdba32a7
[KYUUBI #451 ] Support query auto timeout cancel on thriftserver
...
 [](https://github.com/yaooqinn/kyuubi/pull/451 )       [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png ">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info )<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues , add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Manual cherry-pick some Spark patch into Kyuubi.
1. [Support query auto timeout cancel on thriftserver](https://github.com/apache/spark/pull/29933 )
2. [Add config to control if cancel invoke interrupt task on thriftserver](https://github.com/apache/spark/pull/30481 )
In order to keep backward with early Spark version, we hard code the config key instead of refer to Spark SQLConf.
Note that, the exists timeout of operator (`kyuubi.operation.idle.timeout`) is to cancel that client has no access with engine. That said if a query run a long time and the client is alive, the query would not be cancelled. Then the new added config `spark.sql.thriftServer.queryTimeout` can handle this case.
### _How was this patch tested?_
Add new test.
Closes #451 from ulysses-you/query-timeout.
212f579 [ulysses-you] docs
9206538 [ulysses-you] empty flaky test
ddab9bf [ulysses-you] flaty test
1da02a0 [ulysses-you] flaty test
edfadf1 [ulysses-you] nit
3f9920b [ulysses-you] address comment
9492c48 [ulysses-you] correct timeout
5df997e [ulysses-you] nit
2124952 [ulysses-you] address comment
192fdcc [ulysses-you] fix tets
d684af6 [ulysses-you] global config
1d1adda [ulysses-you] empty
967a63e [ulysses-you] correct import
128948e [ulysses-you] add session conf in session
144d51b [ulysses-you] fix
a90248b [ulysses-you] unused import
c90386f [ulysses-you] timeout move to operation manager
d780965 [ulysses-you] update docs
a5f7138 [ulysses-you] fix test
f7c7308 [ulysses-you] config name
7f3fb3d [ulysses-you] change conf place
97a011e [ulysses-you] unnecessary change
0953a76 [ulysses-you] move test
38ac0c0 [ulysses-you] Merge branch 'master' of https://github.com/yaooqinn/kyuubi into query-timeout
71bea97 [ulysses-you] refector implementation
35ef6f9 [ulysses-you] update conf
0cad8e2 [ulysses-you] Support query auto timeout cancel on thriftserver
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-26 14:04:09 +08:00
watermen
4fe409562a
[KYUUBI #454 ] Fix metrics Versions
...
 [](https://github.com/yaooqinn/kyuubi/pull/454 )      [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png ">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info )<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
Closes #454 from watermen/version.
c141344 [watermen] Fix metrics Versions
Authored-by: watermen <qiyadong2010@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-26 11:56:20 +08:00
watermen
cfa2d043f6
[KYUUBI #452 ] Remove the unclearly defined group share level engines
...
 [](https://github.com/yaooqinn/kyuubi/pull/452 )      [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png ">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info )<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
Closes #452 from watermen/group.
1b89b1c [Kent Yao] Remove the unclearly defined group share level engines
Lead-authored-by: watermen <qiyadong2010@gmail.com>
Co-authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-25 15:11:49 +08:00
Kent Yao
9e7649296d
[KYUUBI #449 ] Add Basic Metrics Service framework
...
 [](https://github.com/yaooqinn/kyuubi/pull/450 )    [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png ">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info )<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues , add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
This PR adds a future-proofing Metrics Service framework close #499
add some reporters, like json file, jmx, etc..
add some basic metrics, like statement.error
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
WIP
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests ) locally before make a pull request
Closes #450 from yaooqinn/449.
ea21a43 [Kent Yao] add tests
6513d07 [Kent Yao] update codecov
7cfe318 [Kent Yao] update docs
a8454d0 [Kent Yao] update docs
91c33f4 [Kent Yao] update deps
05b13fb [Kent Yao] fix dep issues
babc739 [Kent Yao] Add Basic Metrics Service Platform
acc716a [Kent Yao] Add Basic Metrics Service Platform
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-24 18:56:56 +08:00
Kent Yao
3ccbc505b1
logo
2021-03-19 21:08:52 +08:00
yangrong688
03ba80dbcd
[KYUUBI #434 ] [DOCS] Rename BONDED_EXPONENTIAL_BACKOFF to BOUNDED_EXPONENTIAL_BACKOFF
...
 [](https://github.com/yaooqinn/kyuubi/pull/434 )       [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info )<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
change BONDED_EXPONENTIAL_BACKOFF to BOUNDED_EXPONENTIAL_BACKOFF
Closes #434 from yangrong688/master.
492ceb9 [yangrong688] fix typo
8bd6809 [yangrong688] fix typo
Authored-by: yangrong688 <yangrong.jxufe@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-15 11:45:12 +08:00
Kent Yao
183e18f1bc
[KYUUBI #404 ] How To Use Spark Dynamic Resource Allocation (DRA) in Kyuubi
...
 [](https://github.com/yaooqinn/kyuubi/pull/404 )     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info )<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues , add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests ) locally before make a pull request
Closes #404 from yaooqinn/dra.
b8b755c [Kent Yao] How To Use Spark Dynamic Resource Allocation (DRA) in Kyuubi
9bb47b1 [Kent Yao] How To Use Spark Dynamic Resource Allocation (DRA) in Kyuubi
e786787 [Kent Yao] How To Use Spark Dynamic Resource Allocation (DRA) in Kyuubi
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-08 18:27:49 +08:00
Cheng Pan
e49ff88305
[KYUUBI #390 ] Replace abbr in doc
...
 [](https://github.com/yaooqinn/kyuubi/pull/390 )      [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info )<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues , add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests ) locally before make a pull request
Closes #390 from pan3793/doc.
4b7cdf2 [Cheng Pan] replace abbr in doc
Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-03 22:04:32 +08:00
Kent Yao
f83faa8356
[KYUUBI #389 ] Bump up curator to 2.12.0
...
 [](https://github.com/yaooqinn/kyuubi/pull/389 )      [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info )<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues , add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
2.12.0 is the last update for curator-test to support zk v3.4.x
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests ) locally before make a pull request
Closes #389 from yaooqinn/curator.
0b12323 [Kent Yao] Bump up curator to 2.12.0
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Cheng Pan <379377944@qq.com>
2021-03-03 19:37:59 +08:00
Kent Yao
0f62568171
[KYUUBI #379 ] Improve engine log capture
...
 [](https://github.com/yaooqinn/kyuubi/pull/379 )     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info )<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues , add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
1. limit error size with `kyuubi.session.engine.startup.error.max.size` to reduce long time pause or heap overhead
2. update `error` immediately when detecting errors at the very first line ASAP
3. Using `Exception:` instead of `Exception` to search each line to filter better
4. add the engine log address in the exception message in case of error bootstraping
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests ) locally before make a pull request
Closes #379 from yaooqinn/logsize.
7cf58da [Kent Yao] mute in console
b58a3a2 [Kent Yao] update
fbc4a0b [Kent Yao] update
0815f77 [Kent Yao] log
93f5f2f [Kent Yao] log
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-26 17:31:44 +08:00
Kent Yao
4b13f1c551
[KYUUBI #367 ] Engine terminating checker should start after exposed
...
 [](https://github.com/yaooqinn/kyuubi/pull/367 )      [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info )<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues , add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Currently, the engine starts to check its lifecycle during the session manager's startup without any initial delay. Before the engine gets ready finally, it might trigger self-terminating.
In this PR, we move the checker to the end where the engine is exposed to the servers.
Avoiding errors like:
```
org.apache.hive.service.cli.HiveSQLException: Error operating GET_CATALOGS: java.net.SocketException: Connection reset
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:267)
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:258)
at org.apache.hive.jdbc.HiveDatabaseMetaData.getCatalogs(HiveDatabaseMetaData.java:142)
at org.apache.kyuubi.operation.BasicIcebergJDBCTests.$anonfun$$init$$2(BasicIcebergJDBCTests.scala:50)
at org.apache.kyuubi.operation.BasicIcebergJDBCTests.$anonfun$$init$$2$adapted(BasicIcebergJDBCTests.scala:48)
at org.apache.kyuubi.operation.JDBCTestUtils.$anonfun$withMultipleConnectionJdbcStatement$3(JDBCTestUtils.scala:43)
at org.apache.kyuubi.operation.JDBCTestUtils.$anonfun$withMultipleConnectionJdbcStatement$3$adapted(JDBCTestUtils.scala:43)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests ) locally before make a pull request
Closes #367 from yaooqinn/engine.
f6a979e [Kent Yao] fix test
9cc561c [Kent Yao] address comment
3d15ab2 [Kent Yao] nit
a85d2e0 [Kent Yao] Engine terminating checker should start after exposed
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-25 15:10:43 +08:00
jhx1008
ecd1c0e1a6
[KYUUBI #323 ] Fix Kyuubi conf template
...
 [](https://github.com/yaooqinn/kyuubi/pull/323 )     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info )<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://kyuubi.readthedocs.io/en/latest/community/contributions.html
-->
### _Which issue are you going to fix?_
<!--
Replace ${ID} below with the actual issue id from
https://github.com/yaooqinn/kyuubi/issues ,
so that the issue will be linked and automatically closed after merging
-->
Fixes #${ID}
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the user case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
the Hadoop conf can not be recognized in `kyuubi-defaulfs.conf`, use HADOOP_CONF_DIR instead
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests ) locally before make a pull request
Closes #323 from yaooqinn/conf.
a5f8a66 [Kent Yao] nit
7ef0046 [jhx1008] Fix Kyuubi conf template
Lead-authored-by: jhx1008 <jhx1008@gmail.com>
Co-authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-01-30 13:36:45 +08:00
ulysses-you
6dd78652d8
[KYUUBI #284 ] Sesson submit log should be overwritten
...
fixes #284
We will create and write a log during submit a new session (like spark-submit) but we don't delete or overwrite it. This PR aims to add a time config to decide if we need to overwrite a session submit log.
The log path template is `KYUUBI_WORD_DIR/user/$engine-log.$index`. We will increase the `index` automatically if a user create some session concurrently.
After this PR. The log path template is `KYUUBI_WORD_DIR/user/$engine.log.$index`.
Squashed commit of the following:
commit 7794406098df0a3129ccd2ab04a445d2155bef8e
Author: ulysses-you <ulyssesyou18@gmail.com>
Date: Thu Jan 14 19:52:10 2021 +0800
fix
commit 13fd3bbb577c8bdad0ef73851262344d2870fd4d
Author: ulysses-you <ulyssesyou18@gmail.com>
Date: Thu Jan 14 19:02:36 2021 +0800
fix docs
commit 82c192bc6bccf749f7d7232ed4f03f3c0a56c65f
Author: ulysses-you <ulyssesyou18@gmail.com>
Date: Thu Jan 14 18:04:50 2021 +0800
fix
commit 134ea4389b0b9fddc777c79b1a9f43fc59ec3f7e
Author: ulysses-you <ulyssesyou18@gmail.com>
Date: Thu Jan 14 17:20:55 2021 +0800
docs
commit 921ac11b27c028c508e834b88f9f9d9e17eb104a
Author: ulysses-you <ulyssesyou18@gmail.com>
Date: Thu Jan 14 17:16:09 2021 +0800
fix config
commit bfcbd9633a18e9a4cf0b5b2308855ad8af7f7efd
Author: ulysses-you <ulyssesyou18@gmail.com>
Date: Thu Jan 14 16:36:00 2021 +0800
prefix
commit bceb8c59cbd7d7757b2c30058f845fa1e3c7ee9d
Author: ulysses-you <ulyssesyou18@gmail.com>
Date: Thu Jan 14 16:14:20 2021 +0800
doc
commit 4dd556333c7e6e8181f595293bb3877473c00ee7
Author: ulysses-you <ulyssesyou18@gmail.com>
Date: Thu Jan 14 16:11:00 2021 +0800
init
2021-01-14 20:42:11 +08:00
Cheng Pan
57ec15d09e
[MINOR] fix typo
...
fixes #277
Squashed commit of the following:
commit 291cb2c9baaf4072689231a9534c41a5eafd9a7e
Author: Cheng Pan <379377944@qq.com>
Date: Tue Jan 12 23:42:47 2021 +0800
update doc
commit 9f706c49f6c59d67c64d93c197cc963217636444
Author: Cheng Pan <379377944@qq.com>
Date: Tue Jan 12 22:04:30 2021 +0800
[MINOR] fix typo
2021-01-13 11:25:19 +08:00
Kent Yao
d70aec651d
Spark Conf temp
2020-12-31 15:02:21 +08:00
Kent Yao
3ec0db6a40
Improve Session Share Level
2020-12-27 00:05:07 +08:00
Kent Yao
d357203b0e
Add authentication guide
2020-11-30 17:45:20 +08:00
Kent Yao
87e243c596
support query log for async exec
2020-11-27 16:40:07 +08:00
Kent Yao
575e9c9599
Add support for user defaults
2020-11-25 21:37:25 +08:00
Kent Yao
f8ef7c8997
Support ansynchronous exec for Kyuubi Server
2020-11-24 10:55:29 +08:00
Kent Yao
9ca23bcf20
Support ansynchronous exec for Spark SQL engine
2020-11-20 19:52:57 +08:00
Kent Yao
91d0ce302e
add HADOOP_CONF_DIR to classpath
2020-11-19 00:30:25 +08:00
Kent Yao
64b83a48c5
Show possible error even timeout
2020-11-16 22:56:25 +08:00
Kent Yao
15820a2b09
fix
2020-11-16 19:42:06 +08:00
Kent Yao
b3d996917d
fix static and core conf via hiveconf or hivevar
2020-11-13 16:08:47 +08:00
Kent Yao
4396e59abe
Add doc for overview
2020-11-13 14:17:42 +08:00
Kent Yao
fb4bace6a5
Ignore static and other unmodifiable settting in session
2020-11-11 11:15:09 +08:00
Kent Yao
574c119712
JDBCTests added and some related bugs fixed
2020-11-10 21:24:02 +08:00
Kent Yao
f8dc5a9ca3
Using ha.zookeeper.acl.enabled to control ZK acl
2020-11-09 17:34:55 +08:00
Kent Yao
7157370ef6
Add Building Doc
2020-11-05 19:05:26 +08:00
