### _Why are the changes needed?_
Add generate_changelog.sh script to simple generate changelog for Apache Incubating Kyuubi.
I've searched there are generaly three ways to generate changelog
1. add change log manually in ervery pr ([Apache Skywalking](https://github.com/apache/skywalking/pull/8881/files#diff-e4d10d406c447edb82b7e7958bebc5ff8ca9123076cf138b4e0ba037fa873055))
2. use github action to generate ([sqlfluff](https://github.com/sqlfluff/sqlfluff/blob/main/.github/release-drafter.yml))
3. use script to generate changelog
I choose to write a script to generate changelog, it does not rely on github and we can use it more conviniently as there maybe mutiple release candidates in the release process.
I add a script to generate changelog once we prepare a new release version, it allows release manager to easy generate changelog between last release tag and current release tag.
And the changelog doc structure is:
$KYUUBI_DIR
    /docs
        /changelog
            /v1.5.1-incubating.md
            ...
### _How was this patch tested?_
use `./generate_changelog.sh v1.5.0-incubating v1.5.1-incubating-rc0` to generate v1.5.1-incubating changelog
Closes#2501 from yangrong688/KYUUBI-2211.
Closes#2211
b3174b4d [yangrong688] Add generate_changelog.sh script to simple generate changelog for Apache Incubating Kyuubi.
Authored-by: yangrong688 <yangrong.jxufe@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
To close#2373
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2790 from turboFei/recovery_batch.
Closes#2373
81632191 [Fei Wang] trigger test
6ac6f9bd [Fei Wang] async recovery
6d9edf4e [Fei Wang] comments
daa6719f [Fei Wang] refactor
0c3a2e14 [Fei Wang] remove waitAppCompletion
04e139ee [Fei Wang] comments
b5d11d8f [Fei Wang] comment for method name
321dfa95 [Fei Wang] refactor
90be2df7 [Fei Wang] address comments
14409582 [Fei Wang] batch recovery
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Add a doc of debugging engine in `debugging.md`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2798 from deadwind4/KYUUBI-2793-debug-engine-doc.
Closes#2793
3f24ca5b [Ada Wang] [KYUUBI #2793][DOCS] Add debugging engine
Authored-by: Ada Wang <wang4luning@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Close#2715Close#2374
Introduce jdbc based session state store to share the batch state information across kyuubi instances.
This pr is based on docs: https://docs.google.com/document/d/1_BMekMT88P2vCHCl4ZacWEeWslpIv4pxsJH32PAnWSI and refactor it during implementation.
In this PR, I introduce `StateStore` interface and implement the default `JDBCStateStore`.
The data structures:
1. BatchState - the batch state info for multiple-HA, including batch operation state and application information
2. BatchMeta - the meta data used for batch recovery, including sessionConf, batchRequest information
For JDBCStateStore:
- support to specify data base type:
- 1. DERBY - it will use derby jdbc driver by default
- 2. MYSQL - it will use mysql jdbc by default
- 3. CUSTOM - the user need specify the jdbc driver
- support to init schema
By default, the jdbc url is a derby in-memory database url for single kyuubi instance.
To support multiple HA in production env, we should config the production jdbc url to share batch info across kyuubi instances.
The datasource is managed by HiKariCP.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2717 from turboFei/kyuubi_2715_hikaricp_v2.
Closes#2717
367b944a3 [Fei Wang] comments
9a7963dbd [Fei Wang] post session type in event
b1c130853 [Fei Wang] move session type to session package
66def0cb6 [Fei Wang] add JDBCStateStoreConf
839b72bf6 [Fei Wang] refactor
644cbcd28 [Fei Wang] filter by session type
58f1c7854 [Fei Wang] refactor
1c0035208 [Fei Wang] add session type
34a087ee5 [Fei Wang] refactor
fba38e473 [Fei Wang] refactor
fcbde25a0 [Fei Wang] refactor
332a67537 [Fei Wang] comments
135ba9b6d [Fei Wang] Revert "revert class utils change"
0e045a483 [Fei Wang] Revert "Revert "move""
05048416e [Fei Wang] Revert "test again"
01a50c35b [Fei Wang] test again
b89159d13 [Fei Wang] Revert "move"
3764a9051 [Fei Wang] move
eef17822c [Fei Wang] revert
4166db261 [Fei Wang] adjust engine idle timeout
5fc95a824 [Fei Wang] revert class utils change
23f88d63d [Fei Wang] refactor
a66cb811d [Fei Wang] refactor
89fa0edb4 [Fei Wang] fix
e5228fefd [Fei Wang] refactor
581908d86 [Fei Wang] exclude sql
3446aaa06 [Fei Wang] UPPER
ab980c677 [Fei Wang] [KYUUBI #2715] Introduce session state store for batch session multiple HA
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
make table width fit browsers with 100% width, and 1st column where contains the kyuubi conf fit the table with 25% width.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
<img width="1716" alt="image" src="https://user-images.githubusercontent.com/8326978/170655840-2309574e-4098-4b0f-b91d-50a30033b9ac.png">
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2764 from yaooqinn/layout.
Closes#2764
9391a638 [Kent Yao] [DOCS] Fix tables in docs being coverd by right toc sidebar
33817406 [Kent Yao] [DOCS] Fix tables in docs being coverd by right toc sidebar
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#2721
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2728 from cxzl25/KYUUBI-2721.
Closes#2721
67051644 [sychen] use placeholders and confOverlay
2acf0a45 [sychen] unused import
8bc40b9f [sychen] fix UT
3495b280 [sychen] fix jdbc
e55bb2e2 [sychen] style
ff9078ab [sychen] fix UT
a0247c7e [sychen] Merge branch 'master' into KYUUBI-2721
f62461ce [sychen] use placeholders
698fbbdd [sychen] set catalog/schema empty resultset
45caf552 [sychen] fix import
7f13ad8d [sychen] fix server spark catalog UT
4982ec71 [sychen] hive engine 1.8
9877038f [sychen] add UT
21e57870 [sychen] set/get catalog/database operators
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2595 from zhaomin1423/jdbc_engine.
Closes#2658
90bdf97c [Min Zhao] add jdbc dialect
d2bfd60c [Min Zhao] [KYUUBI #2486] [FEATURE] Support Apache Doris engine
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Rename high availability config key to support muti discovery client
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2633 from hddong/muti-discovery-config-support.
Closes#2631
bc434ac9 [hongdongdong] [KYUUBI #2631] Rename high availability config key to support muti discovery client
Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
### _Why are the changes needed?_
Support internal rest request authentication to enable redirect http request across kyuubi instances
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2719 from turboFei/internal_access.
Closes#2719
b9e150e9 [Fei Wang] revert config key change
aa378727 [Fei Wang] make it internal
ce96d922 [Fei Wang] comments
6fe8523b [Fei Wang] Support internal rest request authentication to enable redirect http request across kyuubi instances
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Because `kyuubi-server` is a long-running service, its logs should preferably have dates.
When Log4j1 was used in previous versions, the default log and template had dates.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2714 from cxzl25/KYUUBI-log4j-layout-pattern.
Closes#2714
b249c296 [sychen] change layout pattern
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
We want to implement different service discovery client, thus add a config to support different service discovery client class implementation.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2601 from hiboyang/datapunch-DiscoveryClient-Configurable.
Closes#2601
719685f1 [BoYang] Update settings.md
8af4f5da [BoYang] Change name to HA_DISCOVERY_CLIENT_CLASS
22139b6d [BoYang] Update settings.md
c9845cee [BoYang] Revert unrelated changes
1713e050 [BoYang] Add .github files
22222a4d [BoYang] Merge branch 'master' into datapunch-DiscoveryClient-Configurable
c0c293db [BoYang] Merge branch 'master' of https://github.com/apache/incubator-kyuubi
3e53ef48 [BoYang] Do not copy spark files in docker
887be439 [BoYang] Add dot after docker build command
48b70053 [BoYang] Update manual-release.yml
39d0bb21 [BoYang] Update workflow: cd dist
9adae8c1 [BoYang] Update manual-release.yml
da76f568 [BoYang] Use nonEmpty
bf8fe2ff [BoYang] Run mvn spotless:apply
188d71c5 [BoYang] Update per comments
fffa3bfe [BoYang] Create manual-release.yml
65c62c31 [BoYang] Add KyuubiConf.createInstance and reuse some class checking code
30f0ff35 [BoYang] Add a config to support different service discovery client class implementation
1cdf8809 [BoYang] Delete github workflows
Authored-by: BoYang <datapunch2021@gmail.com>
Signed-off-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
### _Why are the changes needed?_
close#2605
`kyuubi.engine.spark.register.operation.listener`
> When set to true, Spark engine registers a SQLOperationListener before executing the statement, logs a few summary statistics when each stage completes.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2617 from cxzl25/KYUUBI-2605.
Closes#2605
526af5eb [sychen] use spark conf
492a2044 [sychen] update settings.md
22fe762f [sychen] add UT
a7b29de1 [sychen] Merge branch 'master' into KYUUBI-2605
e8fa3c6f [sychen] kyuubi.engine.spark.register.operation.listener
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Now the user can see the command to start the engine, which may have some sensitive information.
Introduce a configuration item to support replacing sensitive information.
For example, if you use the `kyuubi.ha.zookeeper.auth.digest` configuration, you can configure `kyuubi.server.redaction.regex` `(?i)zookeeper.auth.digest`
close#2591
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2592 from cxzl25/KYUUBI-2591.
Closes#2591
67567a26 [sychen] update doc
eb1ec9a1 [sychen] redact kv
fbcba2dd [sychen] Merge branch 'master' into KYUUBI-2591
346e21b1 [sychen] kyuubi.server.redaction.regex
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
fix#2554
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2579 from jiaoqingbo/kyuubi2554.
Closes#2333Closes#2554
f0365c91 [jiaoqingbo] code review
1700aab9 [jiaoqingbo] code review
1ca10a65 [jiaoqingbo] fix ut failed
b53dcdd4 [jiaoqingbo] code review
f9ceb72c [jiaoqingbo] [KYUUBI #2554] Configuring Flink Engine heap memory and java opts
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
We can detect engine dead cases after https://github.com/apache/incubator-kyuubi/pull/2122, so set thrift client socket timeout to inf to avoid socket timeout in blocking case, e.g. retrieve data except to 1st partition in increment collection mode, spark scale mode.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2609 from pan3793/timeout.
Closes#2609
bff14cb1 [Cheng Pan] Update settings.md
e92fa442 [Cheng Pan] Set Kyuubi server thrift client socket timeout to inf
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#2493
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [X] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2529 from wForget/KYUUBI-2493.
Closes#2493
55ef323e [wforget] comments
2da6dea3 [wforget] add conf
0260ece9 [wforget] remove unnecessary log
a46236d9 [wforget] remove OperationProgressUpdate
e76cd78e [wforget] remove OperationProgressStatus
68ee1138 [wforget] remove unnecessary column
a2b9956a [wforget] comments
3d916e42 [wforget] comment
de6910fa [wforget] test details
a07336b7 [wforget] fix test
95277ce8 [wforget] add test
79abed96 [wforget] fix
579ffdbc [wforget] comment
f7f7d420 [wforget] [KYUUBI-2493] Implement the progress of statement
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
try to fix#2524
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2525 from yangrong688/2524.
Closes#2524
1e68f3bc [yangrong688] add doc for get_primary_keys and get_cross_reference
Authored-by: yangrong688 <yangrong.jxufe@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
jackson-annotations 2.13 and hive-exec 2.3.9 have class conflict
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2326 from deadwind4/hive-ci.
Closes#2025
0644c564 [Ada Wang] [KYUUBI #2025][HIVE] Add a hive on yarn doc
Authored-by: Ada Wang <wang4luning@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
only change testing.md
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2487 from gabrywu/fixTestingMd.
Closes#2487
3c648a7e [gabrywu] fix module test command
7c6d1df0 [gabrywu] fix Running Tests Fully command
0582b6ec [gabrywu] fix test command to make it runnable
Authored-by: gabrywu <gabrywu@apache.org>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
We submit a hive job to yarn, when we don't set `kyuubi.engine.hive.extra.classpath`, HiveSQLEngine will throw an error.
My improvement is supporting set `HIVE_HADOOP_CLASSPATH`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2446 from deadwind4/hive-cp.
Closes#2433
7caa7fbb [Ada Wang] update template
45598cb7 [Ada Wang] update template
c7d34090 [Ada Wang] [KYUUBI #2433] Load jars from HIVE_HADOOP_CLASSPATH
Authored-by: Ada Wang <wang4luning@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
close#2456
Now only supports the management of engines whose share level is user level.
Add engine share level parameter(`--engine-share-level`).
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2460 from cxzl25/KYUUBI-2456.
Closes#2456
ec60b649 [sychen] support share level
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
close#1987
note that, this pr only for Spark engine
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2435 from ulysses-you/kyuubi-1987.
Closes#1987
85826a14 [ulysses-you] style
9ddcc0c9 [ulysses-you] shutdown
a568054e [ulysses-you] release
d1339056 [ulysses-you] address comment
8e958ff2 [ulysses-you] docs
dd9b4422 [ulysses-you] Support preserve user context in group share level
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
`docker-image-tools.sh` should use the version where the script was added as the trial version.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2410 from zwangsheng/bug/fix_docker_image_build_usage.
Closes#2410
f32ef8e2 [zwangsheng] fix version
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
close#2301
### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2364 from wForget/KYUUBI-2301.
Closes#2301
7a176e0e [wforget] merge
0b41fafa [wforget] comment
7614dbda [wforget] Merge remote-tracking branch 'origin/master' into KYUUBI-2301
9180cf63 [wforget] fix
741139a4 [wforget] fix
1194ccbf [wforget] move limiter from common to server
a6b93b38 [wforget] regenerate settings.md
123f9209 [wforget] [KYUUBI-2301] Limit the maximum number of concurrent connections per user and ipaddress
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Add Flink environments to `kyuubi-env.sh.template`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2407 from deadwind4/flink-env.
Closes#2406
24717fcd [Ada Wang] [KYUUBI #2406] Add Flink environments to template
Authored-by: Ada Wang <wang4luning@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close https://github.com/apache/incubator-kyuubi/issues/2349
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2396 from cxzl25/KYUUBI-2349.
Closes#2349
4aa0a7fb [sychen] put list ahead
b2c49c37 [sychen] add doc
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2395 from yaooqinn/doc2.
Closes#2395
109440bf [Kent Yao] [DOC] Add Documentation for Spark AuthZ Extension
852e7fd5 [Kent Yao] [DOC] Add Documentation for Spark AuthZ Extension
dfeef884 [Kent Yao] [DOC] Add Documentation for Spark AuthZ Extension
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Configuring Hive engine heap memory and java opts
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2371 from zhaomin1423/2360.
Closes#2360
fc372571 [Min Zhao] [KYUUBI #2360] [Subtask] Configuring Hive engine heap memory and java opts
52547d9c [Min Zhao] add unit tests
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
To close#2308
Support to ignore some batch configuration items and pre-define some configuration in server side.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2386 from turboFei/KYUUBI_2308_kpip4_batch_conf.
Closes#2308
2e517ec4 [Fei Wang] [KYUUBI #2308][SUB-TASK][KPIP-4] Batch job configuration ignore list and server predefined configurations
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
update developer.md to describe what append descriptions of new configurations to settings.md
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2372 from zhaomin1423/2369.
Closes#2369
a044c156 [Min Zhao] [KYUUBI #2369] [Improvement] update developer.md to describe what append descriptions of new configurations to settings.md
56f878a9 [Min Zhao] [KYUUBI #2369] [Improvement] update developer.md to describe what append descriptions of new configurations to settings.md
581894b7 [Min Zhao] [KYUUBI #2369] [Improvement] update developer.md to describe what append descriptions of new configurations to settings.md
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Configuring Trino Engine heap memory and java opts
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2387 from zhaomin1423/2361.
Closes#2361
e8ef7f6f [Min Zhao] [KYUUBI #2361] [Improvement] Configuring Trino Engine heap memory and java opts
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Upgrade sphinx dependencies for documentation build
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2388 from zhaomin1423/2338.
Closes#2338
d68d682e [Min Zhao] [KYUUBI #2338] [DOCS] Upgrade sphinx dependencies for documentation build
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
To close#2306 and close#2307
In this PR, I implement BatchJobSubmission operation and introduce basic `KyuubiBatchSessionImpl`.
TODO:
- Normalize/validate the batch request
- batch request fields
- merge with server pre-defined batch conf
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2353 from turboFei/KPIP_4_batch_submission_op.
Closes#2353
9bc6050c [Fei Wang] remove unused conf
ef8e962c [Fei Wang] check application in current thread
8738660b [Fei Wang] dedup code
27f22008 [Fei Wang] use static secret id instead of conf
6794ff7a [Fei Wang] Use Seq instead of java.util.List
2f4f9b15 [Fei Wang] Remove BatchType enumaration
7d380800 [Fei Wang] remove dead code
a94a9e6a [Fei Wang] remove jars,files fileds on BatchRequest
6021a1e7 [Fei Wang] add ut for result set
07a939c9 [Fei Wang] refactor long line
a918a496 [Fei Wang] address comments
73229e70 [Fei Wang] set engine max life time
bbe3f1f4 [Fei Wang] unique the application cehcker thread pool
9643e42c [Fei Wang] refactor
11dd71f7 [Fei Wang] add KyuubiBatchYarnClusterSuite
12169910 [Fei Wang] add ut for batch session
47da8c1a [Fei Wang] add open batch session api
6dcf60d9 [Fei Wang] add ut for static batch secret id
a212e62b [Fei Wang] [SUB-TASK][KPIP-4] Implement BatchJobSubmission operation and basic KyuubiBatchSessionImpl
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2363 from deadwind4/flink-krb-doc.
Closes#2248
50f191fa [Ada Wang] [KYUUBI #2248][DOCS] Add a flink on yarn kerberos doc
Authored-by: Ada Wang <wang4luning@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Access Kyuubi with Kyuubi-Hive-JDBC Documentation is missing
Close#2337
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2339 from zwangsheng/ISSUE/#2337.
Closes#2337
d4154e6d [zwangsheng] fix mvn url
510708ce [zwangsheng] fix
c4bb9366 [zwangsheng] KYUUBI-2337
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Help user upgrade easily.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2345 from ulysses-you/hot-upgrade.
Closes#2345
a5898711 [ulysses-you] typo
84bdf4bc [ulysses-you] address comment
e32fd3dc [ulysses-you] enrich
3c92545c [ulysses-you] hot-upgrade
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
To close#2250
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2295 from lightning-L/kyuubi-2250.
Closes#2250
71851bf8 [Tianlin Liao] [KYUUBI #2250] limit the spark engine max running time
7314df84 [Tianlin Liao] [KYUUBI #2250] add method to shutdown threadpool executor in ThreadUtils
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Move `dev/kyuubi-spark-extension*` to `extension/spark/`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2291 from ulysses-you/unify-spark-extension.
Closes#2292
4e6f4ca6 [ulysses-you] fix
c2cc018d [ulysses-you] unify
2375af0d [ulysses-you] unify
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
In this PR, we narrow the scope of 8f15622d9b to support arbitrary parameters which configured in kyuubi-defaults.conf only.
We shall avoid propagating all `sys.props` of Kyuubi server to engine side, which is a dangerous behavior
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2283 from yaooqinn/2084.
Closes#2084
434db31c [Kent Yao] fix flink
bdd983d4 [Kent Yao] [KYUUBI #2084][FOLLOWUP] Support arbitrary parameters for KyuubiConf
78bada0b [Kent Yao] [KYUUBI #2084][FOLLOWUP] Support arbitrary parameters for KyuubiConf
55e6fdf1 [Kent Yao] [KYUUBI #2084][FOLLOWUP] Support arbitrary parameters for KyuubiConf
b2239bc7 [Kent Yao] [KYUUBI #2084][FOLLOWUP] Support arbitrary parameters for KyuubiConf
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
fix#2266
in 1.5.0 version，the seriviceUri consists of ip:port
<img width="576" alt="e3c7a03dbcbeeaedce52139755fe227" src="https://user-images.githubusercontent.com/14961757/161362120-61e45f93-d4fe-491b-ab06-9f02e78e70f3.png">
in 1.4.1 version ，the seriviceUri consists of hostname:port
<img width="591" alt="88cf334a0a425227ed6be7637fc10bd" src="https://user-images.githubusercontent.com/14961757/161364285-d8841387-6393-4280-9565-7d3ce419aa81.png">
using following command to connect would failed
./bin/beeline -u 'jdbc:hive2://ocdp63.asiainfo.com:2008,ocdp172.asiainfo.com:2008,ocdnfo.com:2008/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi;principal=spark/_HOSTocdp;'
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2269 from jiaoqingbo/2266.
Closes#2266
b77b19d8 [jiaoqingbo] modify doc
e87d9670 [jiaoqingbo] [KYUUBI #2266] The default value of frontend.connection.url.use.hostname should be set to true to be consistent with previous versions
25063a34 [jiaoqingbo] Merge branch 'master' of https://github.com/jiaoqingbo/incubator-kyuubi
0ad3b764 [jiaoqingbo] Merge branch 'master' of https://github.com/jiaoqingbo/incubator-kyuubi
f9d68c7f [jiaoqingbo] [KYUUBI #2244] load-kyuubi-env.sh should print SPARK_ENGINE_HOME for consistency
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Fix missing prefix in trino engine quick start
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2276 from hddong/trino-quick-start-fix.
Closes#2275
156ce2ef [hongdongdong] [KYUUBI #2275][DOCS] Fix missing prefix in trino engine quick start
Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Add quick start for trino engine
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2265 from hddong/trino-quick-start.
Closes#2243
fc4c2743 [hongdongdong] [KYUUBI #2243] Add quick start for trino engine
Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Add flink usage detail document to show how to use in standalone and how to set flink option etc.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2215 from deadwind4/KYUUBI-2209.
Closes#2209
408c332a [Ada Wang] delete flink basics.md
246e550a [Ada Wang] update flink doc links
8eacc137 [Ada Wang] fix docs
5b7911f6 [Ada] [KYUUBI #2209][FLINK][DOCS] Add detail usage
Lead-authored-by: Ada Wang <wang4luning@gmail.com>
Co-authored-by: Ada <wang4luning@gmail.com>
Signed-off-by: KenjiFujima <thanosxnicholas@gmail.com>
…y usage
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2210 from lightning-L/kyuubi-1021.
Closes#1021
6a2654ab [Tianlin Liao] [KYUUBI #1021] Expire CredentialsRef in a proper time to reduce memory usage
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Remove unused deployment documents of Spark.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2242 from KenjiFujima/KYUUBI-2241.
Closes#2241
b9264590 [KenjiFujima] [KYUUBI #2241] Remove unused deployment documents of Spark
Authored-by: KenjiFujima <thanosxnicholas@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Support to set result max rows for spark engine.
And it will only take affect if incremental mode is not enabled.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2225 from turboFei/max_res.
Closes#2225
bb2acd8b [Fei Wang] nit
5d827d11 [Fei Wang] address comments
3193d22e [Fei Wang] address comments
51983c35 [Fei Wang] address comments
2904b4ce [Fei Wang] address comments
f2a8aa62 [Fei Wang] fix docs
232f975c [Fei Wang] add ut
508b3560 [Fei Wang] Add engine spark max rows
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
To close#2102
Support to retry all the internal thrift request calls(except RenewDelegationToken now), and fast fail if the remote engine is not stable or not alive.
In this PR, it supports engine liveness probe.
If it is enabled, a companion thrift client will be created and open a liveness probe session when opening remote engine session.
It will send some simple thrift request(GetInfo) to check whether the remote engine is alive, and fast fail before retry if remote engine is not connectable.
#### Why not use the same thrift client to check engine liveness before retry?
I tried that, but met `out of resp sequence` error.
For example:
1. send getOperationStatus request
2. read time out
3. send GetInfoType request
4. receive getOperationStatus response (out of resp sequence)
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2122 from turboFei/retry_rpc.
Closes#2102
3926ba04 [Fei Wang] adress comments
ade4ede6 [Fei Wang] add timeout
1b7a64f9 [Fei Wang] Only check remote engine alive before retry
98e03f8e [Fei Wang] refactor
fac388cf [Fei Wang] remove unused import
9c6d8737 [Fei Wang] add ut
9b595650 [Fei Wang] Support to retry the thrift request and engine alive probe
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
