### _Why are the changes needed?_
Fix typo and wording for JDBCMetadataStoreConf.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2919 from lightning-L/typo-1.
Closes#2919
f02ffb8a [Tianlin Liao] fix typo and wording for JDBCMetadataStoreConf
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Help release in future.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2912 from ulysses-you/release-md.
Closes#2912
3cf2e272 [ulysses-you] release
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Support waitCompletion for submit batch
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2909 from lightning-L/kyuubi-2628-1.
Closes#2628
23fe761f [Tianlin Liao] minor fix
06cd66ff [Tianlin Liao] support waitUntilComplete for submit batch; use kyuubi instance url for log batch/submit batch
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
We need support to retry the jdbc requests.
Because we add new jdbc dependency for batch HA, and the database might be in maintenance window.
And there might be master-slave switch, and the database might be unavailable for few minutes.
We need tolerant this issue and unblock the main thread.
BTW, this pr refactor the metadata store naming.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2834 from turboFei/retrying_jdbc_event.
Closes#2834
109a63e70 [Fei Wang] fix docs
a186484be [Fei Wang] refactor the jdbc pool name
cfac9da53 [Fei Wang] rename metadata table name
bb8710ba3 [Fei Wang] comments
1197a098b [Fei Wang] add private package scope
a95694ba9 [Fei Wang] comments
e2f09e784 [Fei Wang] refactor
057eb73c5 [Fei Wang] refactor
88f3ffb6f [Fei Wang] refactor
e0595bdee [Fei Wang] comments
48270e869 [Fei Wang] refactor conf keys
e4f190c5c [Fei Wang] update docs
ed5e90e37 [Fei Wang] add max queues limitation
c5a03c9e2 [Fei Wang] remove getOrCreate
3e39ca94c [Fei Wang] remove
b79fb0954 [Fei Wang] comments
1d309cee0 [Fei Wang] comments
9108a3f0e [Fei Wang] fix docs
5cdffe710 [Fei Wang] fix rat
2e7228fc7 [Fei Wang] fix ut
520236d7b [Fei Wang] rename more
f66d68bcc [Fei Wang] merge into metadata manager
2f70a2f12 [Fei Wang] rename
80e6100fb [Fei Wang] refactor
09cad092a [Fei Wang] add ut
ea4eea74f [Fei Wang] refactor
6b00eb0b3 [Fei Wang] refactor to MetadataRequest
58f1c0587 [Fei Wang] commements
198feb53f [Fei Wang] add more docs
670a23561 [Fei Wang] refactor
a20d54df2 [Fei Wang] waitStateStoreRetryCompletion after super.close closes operation
97a763aef [Fei Wang] refactor RetryingStateStoreRequest
3e136cf34 [Fei Wang] [SUB-TASK][KPIP-4] Support to retry the session state operations if meet transient jdbc issue
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
To close#2859
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2875 from turboFei/kyuubi_2859_retry_delete.
Closes#2859
7908d827 [Fei Wang] revert commands related
23d357dc [Fei Wang] fix ut
76730014 [Fei Wang] print close resp first
76449dad [Fei Wang] support --conf
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Add v1.5.2-incubating changelog for release
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2846 from ulysses-you/add-changelog.
Closes#2846
19c514b5 [ulysses-you] v1.5.2-incubating changelog
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
refactor the changelog script to fix:
- the nit of Apache Kyuubi(Incubating)
- the format of the output changelog
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2839 from ulysses-you/changelog.
Closes#2839
d9e2b6d0 [ulysses-you] comments
b966dc30 [ulysses-you] comments
5f94c52d [ulysses-you] refactor
18cdde22 [ulysses-you] refactor
009699c3 [ulysses-you] Merge branch 'master' of https://github.com/apache/incubator-kyuubi into changelog
76419a92 [ulysses-you] fix
1a4090fc [ulysses-you] nit
b6154b58 [ulysses-you] changelog
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Add generate_changelog.sh script to simple generate changelog for Apache Incubating Kyuubi.
I've searched there are generaly three ways to generate changelog
1. add change log manually in ervery pr ([Apache Skywalking](https://github.com/apache/skywalking/pull/8881/files#diff-e4d10d406c447edb82b7e7958bebc5ff8ca9123076cf138b4e0ba037fa873055))
2. use github action to generate ([sqlfluff](https://github.com/sqlfluff/sqlfluff/blob/main/.github/release-drafter.yml))
3. use script to generate changelog
I choose to write a script to generate changelog, it does not rely on github and we can use it more conviniently as there maybe mutiple release candidates in the release process.
I add a script to generate changelog once we prepare a new release version, it allows release manager to easy generate changelog between last release tag and current release tag.
And the changelog doc structure is:
$KYUUBI_DIR
    /docs
        /changelog
            /v1.5.1-incubating.md
            ...
### _How was this patch tested?_
use `./generate_changelog.sh v1.5.0-incubating v1.5.1-incubating-rc0` to generate v1.5.1-incubating changelog
Closes#2501 from yangrong688/KYUUBI-2211.
Closes#2211
b3174b4d [yangrong688] Add generate_changelog.sh script to simple generate changelog for Apache Incubating Kyuubi.
Authored-by: yangrong688 <yangrong.jxufe@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
To close#2373
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2790 from turboFei/recovery_batch.
Closes#2373
81632191 [Fei Wang] trigger test
6ac6f9bd [Fei Wang] async recovery
6d9edf4e [Fei Wang] comments
daa6719f [Fei Wang] refactor
0c3a2e14 [Fei Wang] remove waitAppCompletion
04e139ee [Fei Wang] comments
b5d11d8f [Fei Wang] comment for method name
321dfa95 [Fei Wang] refactor
90be2df7 [Fei Wang] address comments
14409582 [Fei Wang] batch recovery
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Add a doc of debugging engine in `debugging.md`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2798 from deadwind4/KYUUBI-2793-debug-engine-doc.
Closes#2793
3f24ca5b [Ada Wang] [KYUUBI #2793][DOCS] Add debugging engine
Authored-by: Ada Wang <wang4luning@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Close#2715Close#2374
Introduce jdbc based session state store to share the batch state information across kyuubi instances.
This pr is based on docs: https://docs.google.com/document/d/1_BMekMT88P2vCHCl4ZacWEeWslpIv4pxsJH32PAnWSI and refactor it during implementation.
In this PR, I introduce `StateStore` interface and implement the default `JDBCStateStore`.
The data structures:
1. BatchState - the batch state info for multiple-HA, including batch operation state and application information
2. BatchMeta - the meta data used for batch recovery, including sessionConf, batchRequest information
For JDBCStateStore:
- support to specify data base type:
- 1. DERBY - it will use derby jdbc driver by default
- 2. MYSQL - it will use mysql jdbc by default
- 3. CUSTOM - the user need specify the jdbc driver
- support to init schema
By default, the jdbc url is a derby in-memory database url for single kyuubi instance.
To support multiple HA in production env, we should config the production jdbc url to share batch info across kyuubi instances.
The datasource is managed by HiKariCP.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2717 from turboFei/kyuubi_2715_hikaricp_v2.
Closes#2717
367b944a3 [Fei Wang] comments
9a7963dbd [Fei Wang] post session type in event
b1c130853 [Fei Wang] move session type to session package
66def0cb6 [Fei Wang] add JDBCStateStoreConf
839b72bf6 [Fei Wang] refactor
644cbcd28 [Fei Wang] filter by session type
58f1c7854 [Fei Wang] refactor
1c0035208 [Fei Wang] add session type
34a087ee5 [Fei Wang] refactor
fba38e473 [Fei Wang] refactor
fcbde25a0 [Fei Wang] refactor
332a67537 [Fei Wang] comments
135ba9b6d [Fei Wang] Revert "revert class utils change"
0e045a483 [Fei Wang] Revert "Revert "move""
05048416e [Fei Wang] Revert "test again"
01a50c35b [Fei Wang] test again
b89159d13 [Fei Wang] Revert "move"
3764a9051 [Fei Wang] move
eef17822c [Fei Wang] revert
4166db261 [Fei Wang] adjust engine idle timeout
5fc95a824 [Fei Wang] revert class utils change
23f88d63d [Fei Wang] refactor
a66cb811d [Fei Wang] refactor
89fa0edb4 [Fei Wang] fix
e5228fefd [Fei Wang] refactor
581908d86 [Fei Wang] exclude sql
3446aaa06 [Fei Wang] UPPER
ab980c677 [Fei Wang] [KYUUBI #2715] Introduce session state store for batch session multiple HA
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
make table width fit browsers with 100% width, and 1st column where contains the kyuubi conf fit the table with 25% width.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
<img width="1716" alt="image" src="https://user-images.githubusercontent.com/8326978/170655840-2309574e-4098-4b0f-b91d-50a30033b9ac.png">
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2764 from yaooqinn/layout.
Closes#2764
9391a638 [Kent Yao] [DOCS] Fix tables in docs being coverd by right toc sidebar
33817406 [Kent Yao] [DOCS] Fix tables in docs being coverd by right toc sidebar
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#2721
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2728 from cxzl25/KYUUBI-2721.
Closes#2721
67051644 [sychen] use placeholders and confOverlay
2acf0a45 [sychen] unused import
8bc40b9f [sychen] fix UT
3495b280 [sychen] fix jdbc
e55bb2e2 [sychen] style
ff9078ab [sychen] fix UT
a0247c7e [sychen] Merge branch 'master' into KYUUBI-2721
f62461ce [sychen] use placeholders
698fbbdd [sychen] set catalog/schema empty resultset
45caf552 [sychen] fix import
7f13ad8d [sychen] fix server spark catalog UT
4982ec71 [sychen] hive engine 1.8
9877038f [sychen] add UT
21e57870 [sychen] set/get catalog/database operators
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2595 from zhaomin1423/jdbc_engine.
Closes#2658
90bdf97c [Min Zhao] add jdbc dialect
d2bfd60c [Min Zhao] [KYUUBI #2486] [FEATURE] Support Apache Doris engine
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Rename high availability config key to support muti discovery client
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2633 from hddong/muti-discovery-config-support.
Closes#2631
bc434ac9 [hongdongdong] [KYUUBI #2631] Rename high availability config key to support muti discovery client
Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
### _Why are the changes needed?_
Support internal rest request authentication to enable redirect http request across kyuubi instances
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2719 from turboFei/internal_access.
Closes#2719
b9e150e9 [Fei Wang] revert config key change
aa378727 [Fei Wang] make it internal
ce96d922 [Fei Wang] comments
6fe8523b [Fei Wang] Support internal rest request authentication to enable redirect http request across kyuubi instances
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Because `kyuubi-server` is a long-running service, its logs should preferably have dates.
When Log4j1 was used in previous versions, the default log and template had dates.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2714 from cxzl25/KYUUBI-log4j-layout-pattern.
Closes#2714
b249c296 [sychen] change layout pattern
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
We want to implement different service discovery client, thus add a config to support different service discovery client class implementation.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2601 from hiboyang/datapunch-DiscoveryClient-Configurable.
Closes#2601
719685f1 [BoYang] Update settings.md
8af4f5da [BoYang] Change name to HA_DISCOVERY_CLIENT_CLASS
22139b6d [BoYang] Update settings.md
c9845cee [BoYang] Revert unrelated changes
1713e050 [BoYang] Add .github files
22222a4d [BoYang] Merge branch 'master' into datapunch-DiscoveryClient-Configurable
c0c293db [BoYang] Merge branch 'master' of https://github.com/apache/incubator-kyuubi
3e53ef48 [BoYang] Do not copy spark files in docker
887be439 [BoYang] Add dot after docker build command
48b70053 [BoYang] Update manual-release.yml
39d0bb21 [BoYang] Update workflow: cd dist
9adae8c1 [BoYang] Update manual-release.yml
da76f568 [BoYang] Use nonEmpty
bf8fe2ff [BoYang] Run mvn spotless:apply
188d71c5 [BoYang] Update per comments
fffa3bfe [BoYang] Create manual-release.yml
65c62c31 [BoYang] Add KyuubiConf.createInstance and reuse some class checking code
30f0ff35 [BoYang] Add a config to support different service discovery client class implementation
1cdf8809 [BoYang] Delete github workflows
Authored-by: BoYang <datapunch2021@gmail.com>
Signed-off-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
### _Why are the changes needed?_
close#2605
`kyuubi.engine.spark.register.operation.listener`
> When set to true, Spark engine registers a SQLOperationListener before executing the statement, logs a few summary statistics when each stage completes.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2617 from cxzl25/KYUUBI-2605.
Closes#2605
526af5eb [sychen] use spark conf
492a2044 [sychen] update settings.md
22fe762f [sychen] add UT
a7b29de1 [sychen] Merge branch 'master' into KYUUBI-2605
e8fa3c6f [sychen] kyuubi.engine.spark.register.operation.listener
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Now the user can see the command to start the engine, which may have some sensitive information.
Introduce a configuration item to support replacing sensitive information.
For example, if you use the `kyuubi.ha.zookeeper.auth.digest` configuration, you can configure `kyuubi.server.redaction.regex` `(?i)zookeeper.auth.digest`
close#2591
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2592 from cxzl25/KYUUBI-2591.
Closes#2591
67567a26 [sychen] update doc
eb1ec9a1 [sychen] redact kv
fbcba2dd [sychen] Merge branch 'master' into KYUUBI-2591
346e21b1 [sychen] kyuubi.server.redaction.regex
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
fix#2554
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2579 from jiaoqingbo/kyuubi2554.
Closes#2333Closes#2554
f0365c91 [jiaoqingbo] code review
1700aab9 [jiaoqingbo] code review
1ca10a65 [jiaoqingbo] fix ut failed
b53dcdd4 [jiaoqingbo] code review
f9ceb72c [jiaoqingbo] [KYUUBI #2554] Configuring Flink Engine heap memory and java opts
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
We can detect engine dead cases after https://github.com/apache/incubator-kyuubi/pull/2122, so set thrift client socket timeout to inf to avoid socket timeout in blocking case, e.g. retrieve data except to 1st partition in increment collection mode, spark scale mode.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2609 from pan3793/timeout.
Closes#2609
bff14cb1 [Cheng Pan] Update settings.md
e92fa442 [Cheng Pan] Set Kyuubi server thrift client socket timeout to inf
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#2493
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [X] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2529 from wForget/KYUUBI-2493.
Closes#2493
55ef323e [wforget] comments
2da6dea3 [wforget] add conf
0260ece9 [wforget] remove unnecessary log
a46236d9 [wforget] remove OperationProgressUpdate
e76cd78e [wforget] remove OperationProgressStatus
68ee1138 [wforget] remove unnecessary column
a2b9956a [wforget] comments
3d916e42 [wforget] comment
de6910fa [wforget] test details
a07336b7 [wforget] fix test
95277ce8 [wforget] add test
79abed96 [wforget] fix
579ffdbc [wforget] comment
f7f7d420 [wforget] [KYUUBI-2493] Implement the progress of statement
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
try to fix#2524
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2525 from yangrong688/2524.
Closes#2524
1e68f3bc [yangrong688] add doc for get_primary_keys and get_cross_reference
Authored-by: yangrong688 <yangrong.jxufe@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
jackson-annotations 2.13 and hive-exec 2.3.9 have class conflict
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2326 from deadwind4/hive-ci.
Closes#2025
0644c564 [Ada Wang] [KYUUBI #2025][HIVE] Add a hive on yarn doc
Authored-by: Ada Wang <wang4luning@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
only change testing.md
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2487 from gabrywu/fixTestingMd.
Closes#2487
3c648a7e [gabrywu] fix module test command
7c6d1df0 [gabrywu] fix Running Tests Fully command
0582b6ec [gabrywu] fix test command to make it runnable
Authored-by: gabrywu <gabrywu@apache.org>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
We submit a hive job to yarn, when we don't set `kyuubi.engine.hive.extra.classpath`, HiveSQLEngine will throw an error.
My improvement is supporting set `HIVE_HADOOP_CLASSPATH`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2446 from deadwind4/hive-cp.
Closes#2433
7caa7fbb [Ada Wang] update template
45598cb7 [Ada Wang] update template
c7d34090 [Ada Wang] [KYUUBI #2433] Load jars from HIVE_HADOOP_CLASSPATH
Authored-by: Ada Wang <wang4luning@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
close#2456
Now only supports the management of engines whose share level is user level.
Add engine share level parameter(`--engine-share-level`).
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2460 from cxzl25/KYUUBI-2456.
Closes#2456
ec60b649 [sychen] support share level
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
close#1987
note that, this pr only for Spark engine
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2435 from ulysses-you/kyuubi-1987.
Closes#1987
85826a14 [ulysses-you] style
9ddcc0c9 [ulysses-you] shutdown
a568054e [ulysses-you] release
d1339056 [ulysses-you] address comment
8e958ff2 [ulysses-you] docs
dd9b4422 [ulysses-you] Support preserve user context in group share level
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
`docker-image-tools.sh` should use the version where the script was added as the trial version.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2410 from zwangsheng/bug/fix_docker_image_build_usage.
Closes#2410
f32ef8e2 [zwangsheng] fix version
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
close#2301
### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2364 from wForget/KYUUBI-2301.
Closes#2301
7a176e0e [wforget] merge
0b41fafa [wforget] comment
7614dbda [wforget] Merge remote-tracking branch 'origin/master' into KYUUBI-2301
9180cf63 [wforget] fix
741139a4 [wforget] fix
1194ccbf [wforget] move limiter from common to server
a6b93b38 [wforget] regenerate settings.md
123f9209 [wforget] [KYUUBI-2301] Limit the maximum number of concurrent connections per user and ipaddress
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Add Flink environments to `kyuubi-env.sh.template`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2407 from deadwind4/flink-env.
Closes#2406
24717fcd [Ada Wang] [KYUUBI #2406] Add Flink environments to template
Authored-by: Ada Wang <wang4luning@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close https://github.com/apache/incubator-kyuubi/issues/2349
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2396 from cxzl25/KYUUBI-2349.
Closes#2349
4aa0a7fb [sychen] put list ahead
b2c49c37 [sychen] add doc
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2395 from yaooqinn/doc2.
Closes#2395
109440bf [Kent Yao] [DOC] Add Documentation for Spark AuthZ Extension
852e7fd5 [Kent Yao] [DOC] Add Documentation for Spark AuthZ Extension
dfeef884 [Kent Yao] [DOC] Add Documentation for Spark AuthZ Extension
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Configuring Hive engine heap memory and java opts
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2371 from zhaomin1423/2360.
Closes#2360
fc372571 [Min Zhao] [KYUUBI #2360] [Subtask] Configuring Hive engine heap memory and java opts
52547d9c [Min Zhao] add unit tests
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
To close#2308
Support to ignore some batch configuration items and pre-define some configuration in server side.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2386 from turboFei/KYUUBI_2308_kpip4_batch_conf.
Closes#2308
2e517ec4 [Fei Wang] [KYUUBI #2308][SUB-TASK][KPIP-4] Batch job configuration ignore list and server predefined configurations
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
update developer.md to describe what append descriptions of new configurations to settings.md
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2372 from zhaomin1423/2369.
Closes#2369
a044c156 [Min Zhao] [KYUUBI #2369] [Improvement] update developer.md to describe what append descriptions of new configurations to settings.md
56f878a9 [Min Zhao] [KYUUBI #2369] [Improvement] update developer.md to describe what append descriptions of new configurations to settings.md
581894b7 [Min Zhao] [KYUUBI #2369] [Improvement] update developer.md to describe what append descriptions of new configurations to settings.md
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Configuring Trino Engine heap memory and java opts
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2387 from zhaomin1423/2361.
Closes#2361
e8ef7f6f [Min Zhao] [KYUUBI #2361] [Improvement] Configuring Trino Engine heap memory and java opts
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Upgrade sphinx dependencies for documentation build
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2388 from zhaomin1423/2338.
Closes#2338
d68d682e [Min Zhao] [KYUUBI #2338] [DOCS] Upgrade sphinx dependencies for documentation build
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
To close#2306 and close#2307
In this PR, I implement BatchJobSubmission operation and introduce basic `KyuubiBatchSessionImpl`.
TODO:
- Normalize/validate the batch request
- batch request fields
- merge with server pre-defined batch conf
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2353 from turboFei/KPIP_4_batch_submission_op.
Closes#2353
9bc6050c [Fei Wang] remove unused conf
ef8e962c [Fei Wang] check application in current thread
8738660b [Fei Wang] dedup code
27f22008 [Fei Wang] use static secret id instead of conf
6794ff7a [Fei Wang] Use Seq instead of java.util.List
2f4f9b15 [Fei Wang] Remove BatchType enumaration
7d380800 [Fei Wang] remove dead code
a94a9e6a [Fei Wang] remove jars,files fileds on BatchRequest
6021a1e7 [Fei Wang] add ut for result set
07a939c9 [Fei Wang] refactor long line
a918a496 [Fei Wang] address comments
73229e70 [Fei Wang] set engine max life time
bbe3f1f4 [Fei Wang] unique the application cehcker thread pool
9643e42c [Fei Wang] refactor
11dd71f7 [Fei Wang] add KyuubiBatchYarnClusterSuite
12169910 [Fei Wang] add ut for batch session
47da8c1a [Fei Wang] add open batch session api
6dcf60d9 [Fei Wang] add ut for static batch secret id
a212e62b [Fei Wang] [SUB-TASK][KPIP-4] Implement BatchJobSubmission operation and basic KyuubiBatchSessionImpl
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2363 from deadwind4/flink-krb-doc.
Closes#2248
50f191fa [Ada Wang] [KYUUBI #2248][DOCS] Add a flink on yarn kerberos doc
Authored-by: Ada Wang <wang4luning@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Access Kyuubi with Kyuubi-Hive-JDBC Documentation is missing
Close#2337
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2339 from zwangsheng/ISSUE/#2337.
Closes#2337
d4154e6d [zwangsheng] fix mvn url
510708ce [zwangsheng] fix
c4bb9366 [zwangsheng] KYUUBI-2337
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Help user upgrade easily.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2345 from ulysses-you/hot-upgrade.
Closes#2345
a5898711 [ulysses-you] typo
84bdf4bc [ulysses-you] address comment
e32fd3dc [ulysses-you] enrich
3c92545c [ulysses-you] hot-upgrade
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
To close#2250
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2295 from lightning-L/kyuubi-2250.
Closes#2250
71851bf8 [Tianlin Liao] [KYUUBI #2250] limit the spark engine max running time
7314df84 [Tianlin Liao] [KYUUBI #2250] add method to shutdown threadpool executor in ThreadUtils
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Move `dev/kyuubi-spark-extension*` to `extension/spark/`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2291 from ulysses-you/unify-spark-extension.
Closes#2292
4e6f4ca6 [ulysses-you] fix
c2cc018d [ulysses-you] unify
2375af0d [ulysses-you] unify
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
