### _Why are the changes needed?_
Close#3154
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3155 from zhouyifan279/3154.
Closes#3154
682aaf58 [zhouyifan279] [KYUUBI #3154][Subtask] Connectors for Spark SQL Query Engine -> TiDB
4301ca44 [zhouyifan279] [KYUUBI #3154][Subtask] Connectors for Spark SQL Query Engine -> TiDB
65acabe6 [zhouyifan279] [KYUUBI #3154][Subtask] Connectors for Spark SQL Query Engine -> TiSpark
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Add iceberg connector doc for Trino SQL Engine.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3130 from deadwind4/iceberg-trino-doc.
Closes#3082
2f11524c [Luning Wang] [KYUUBI #3082] Add iceberg connector doc for Trino SQL Engine
Authored-by: Luning Wang <wang4luning@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Add iceberg connector for Flink SQL Engine
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3118 from deadwind4/iceberg-flink-doc.
Closes#3071
bf9d158b [Luning Wang] [KYUUBI #3071][DOC] Add iceberg connector for Flink SQL Engine
Authored-by: Luning Wang <wang4luning@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Add Flink Table Store connector doc for Spark SQL Engine
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3151 from huage1994/3067.
Closes#3067
b2bc67d1 [guanhua.lgh] [KYUUBI #3067][DOC] Add Flink Table Store connector doc for Spark SQL Engine
Authored-by: guanhua.lgh <guanhua.lgh@alibaba-inc.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Close#3126
The error:
```
Running Sphinx v4.5.0
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 231 source files that are out of date
updating environment: [new config] 231 added, 0 changed, 0 removed
reading sources... [ 0%] appendix/index
Extension error (sphinx_markdown_tables):
Handler <function process_tables at 0x102589090> for event 'source-read' threw an exception (exception: TableProcessor.__init__() missing 1 required positional argument: 'config')
make: *** [html] Error 2
```
I checked the lib difference with before, seems the only change is Markdown version.
Before, it was 3.3.7 and now it is 3.4.1.
### _How was this patch tested?_
Re-create the kyuubi virtual env and build docs locally.
Closes#3127 from turboFei/kyuubi_docs.
Closes#3126
ac428ce3 [Fei Wang] nit
38b56bc3 [Fei Wang] Use markdown 3.3.7 for kyuubi documents build
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Add Iceberg connector doc for Spark SQL Engine
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3115 from deadwind4/iceberg-spark-doc.
Closes#3069
4c9adeb0 [Luning Wang] Add merge into
119be819 [Luning Wang] update mulit engine support
eb4180d6 [Luning Wang] [KYUUBI #3069][DOC] Add Iceberg connector doc for Spark SQL Engine
Authored-by: Luning Wang <wang4luning@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add the Hudi connector doc for Spark SQL Query Engine
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3099 from deadwind4/hudi-spark-doc.
Closes#3068
fcd2cf6a [Luning Wang] update doc
0ee870d9 [Luning Wang] [KYUUBI #3068][DOC] Add the Hudi connector doc for Spark SQL Query Engine
Authored-by: Luning Wang <wang4luning@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Fix path errors in the build document
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3109 from deadwind4/K3108.
Closes#3108
23830c6e [Luning Wang] [KYUUBI #3108][DOC] Fix path errors in the build document
Authored-by: Luning Wang <wang4luning@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Build the content for extension points documentation, pre-work for #3100
<img width="1767" alt="image" src="https://user-images.githubusercontent.com/8326978/179930987-1accbbb7-e804-4230-871f-6c4b1152f4a1.png">
1. the extensions are divided into 2: server side and engine side extensions. (Do we have client side extension support?)
2. the server side authentication page is cross-referenced by the security section, see 1 in the picture.
3. the engine side ones are grouped by different compute frameworks.
4. connector is one type of extension, so we cross-reference the connector pages directly, see 2 & 3 in the picture.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3103 from yaooqinn/3101.
Closes#3101
a9ae3e32 [Kent Yao] [KYUUBI #3101] [Subtask][#3100] Build content for extension points documentation
3b7367e9 [Kent Yao] [KYUUBI #3101] [Subtask][#3100] Build content for extension points documentation
b5eda13e [Kent Yao] [KYUUBI #3101] [Subtask][#3100] Build content for extension points documentation
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Because the client can specify any ip in http header, to prevent CVE issue, we do not use it for authentication.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3078 from turboFei/http_real_ip_cve.
Closes#3052
e7c41ea9 [Fei Wang] prevent cve
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Close#3046
Expose the metrics likes:
```
kyuubi.operation.state.BatchJobSubmission.pending
kyuubi.operation.state.BatchJobSubmission.running
kyuubi.operation.state.BatchJobSubmission.finished
kyuubi.operation.state.BatchJobSubmission.error
kyuubi.operation.state.BatchJobSubmission.canceled
```
So that the kyuubi service admin can know that how many BatchJobSubmission operations are pending, running and so on.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3063 from turboFei/batch_metrics.
Closes#3046
89ee21db [Fei Wang] remove statement overwrite
25ae41e9 [Fei Wang] update docs
7287527c [Fei Wang] use meter
a2dde891 [Fei Wang] save
a0c6eade [Fei Wang] remove another bug fix
c250a089 [Fei Wang] add docs
87f27e89 [Fei Wang] comments
6bdc4ca9 [Fei Wang] fix flaky test
b000525d [Fei Wang] fix flaky test
f748093e [Fei Wang] expose more metrics
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
1. The version options should be in zkOptions.
2. Add docs for kyuubi-adminctl.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3066 from turboFei/admin_ctl_docs.
Closes#3045
983d133f [Fei Wang] comments
65675ad7 [Fei Wang] add more
b5b772d5 [Fei Wang] comments
222b409c [Fei Wang] add docs
20c50c00 [Fei Wang] kyuubi-adminctl -> kyuubi-admin
078044e0 [Fei Wang] refactor
4e89b52d [Fei Wang] remove unused
c2d1dc70 [Fei Wang] add admin ctl docs
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Add the docs for batch resource rest api.
Close#3076
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3075 from turboFei/rest_api.
Closes#3076
970498e6 [Fei Wang] Cap
445fde21 [Fei Wang] nit
4b0306df [Fei Wang] refactor
b4a5a74f [Fei Wang] add rest api docs
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Add description of the discovery client in the conf doc.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3054 from hddong/conf-to-etcd.
Closes#3054
d364d099 [hongdongdong] Add description of the discovery client in the conf doc
Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Build content of the connector document section
#### Main changes proposed in this PR
- the `integration` section is renamed to `connector` and lifted up
- connectors are separated by engines
- the usage guide is broken into 2 section: Admin Guide and User Guide
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3061 from yaooqinn/3060.
Closes#3060
6932cbb8 [Kent Yao] [KYUUBI #3060] [Subtask][#3059] Build the content of connector document section
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
To close#3052
If your server is behind a load balancer or other proxy, the server will see this load balancer or proxy IP address as the client IP address, to get around this common issue, most load balancers or proxies offer the ability to record the real remote IP address in an HTTP herader that will be added to the request for other devices to use.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3053 from turboFei/http_real_ip.
Closes#3052
b375d02a [Fei Wang] add ut
23ef1c9c [Fei Wang] refactor
907c61c0 [Fei Wang] also take affect for http
a958c50f [Fei Wang] add conf
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Add etcd discovery client for HA
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2767 from hddong/etcd-support.
Closes#2644
6fa37578 [hongdongdong] Rebase and remove from rat-excludes
fc23a2b4 [hongdongdong] upgrade grpc-1.47.0
61c77d27 [hongdongdong] Fix tests
28abd13e [hongdongdong] fix comments
9e81a6a3 [hongdongdong] Rename HA_ZK_ENGINE_REF_ID to HA_ENGINE_REF_ID
aa2b4260 [hongdongdong] [KYUUBI #2644][WIP] Add etcd discovery client for HA
Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
After #2948 and #2953, the OPERATION_THRIFT_CLIENT_REQUEST_MAX_ATTEMPTS is not used.
In this pr, I remove the attempts config entry and related ut that based on the request attempts.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3012 from turboFei/follow_up_engine_alive.
Closes#3012
3d15e44a [Fei Wang] remove deprecated docs
89c9f2cb [Fei Wang] remove docs
edb6ff2e [Fei Wang] remove ut
696bd853 [Fei Wang] saev
f91c61bf [Fei Wang] remove unused attempt
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Correct spelling and smooth documentation make the community better.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3006 from lxorc/master.
Closes#3005
ca294af5 [lxorc] [DOCS] Correct spelling errors and optimizations in 'Building Kyuubi Documentation'
Authored-by: lxorc <13612100484@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2896 from hddong/trino-result-info.
Closes#2895
757019cf [hongdongdong] fix scala style
f40b1c53 [hongdongdong] Fix
3edbdfb3 [hongdongdong] [KYUUBI #2895] Show final info in trino engine
Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
### _Why are the changes needed?_
Replace log4j2.properties by log4j2.xml
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3001 from pan3793/log4j.
Closes#2850
e9396017 [Cheng Pan] fix
4e06c89c [Cheng Pan] [KYUUBI #2850][FOLLOWUP] Replace log4j2.properties by log4j2.xml
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
To close#2801
Customer might set some parameters to upload the local files, such as spark.files=/etc/security/keytabs/kyuubi.keytab. It is not security for kyuubi server.
In this pr, I introduce a local dir allow list and customer can only upload the files under these dirs.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2991 from turboFei/local_white_list.
Closes#2801
6315067d [Fei Wang] comments
deb9fa8a [Fei Wang] remove unused
3330fa98 [Fei Wang] refactor
6eca2f8b [Fei Wang] move to KyuubiApplicationManager and add ut
1290d9e2 [Fei Wang] add more
f14ec7d4 [Fei Wang] add more path configs and add comment
82357183 [Fei Wang] refactor
f201f577 [Fei Wang] disable relative path
f806e69a [Fei Wang] refactor
4f2faa6a [Fei Wang] save
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Close#2956
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2958 from lightning-L/kyuubi-2956.
Closes#2956
a88de508 [Tianlin Liao] allow 0 timeout for rest client; increase default timeout
94a57cce [Tianlin Liao] update
d569abe8 [Tianlin Liao] [KYUUBI #2956] Support to config the connection/socket timeout of rest client for kyuubi-ctl
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Remove thrift request timeout for KyuubiSyncThriftClient
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2948 from turboFei/remove_request_timeout.
Closes#2948
a9c5c084 [Fei Wang] remove request timeout
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Fix typo and wording for JDBCMetadataStoreConf.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2919 from lightning-L/typo-1.
Closes#2919
f02ffb8a [Tianlin Liao] fix typo and wording for JDBCMetadataStoreConf
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Help release in future.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2912 from ulysses-you/release-md.
Closes#2912
3cf2e272 [ulysses-you] release
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Support waitCompletion for submit batch
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2909 from lightning-L/kyuubi-2628-1.
Closes#2628
23fe761f [Tianlin Liao] minor fix
06cd66ff [Tianlin Liao] support waitUntilComplete for submit batch; use kyuubi instance url for log batch/submit batch
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
We need support to retry the jdbc requests.
Because we add new jdbc dependency for batch HA, and the database might be in maintenance window.
And there might be master-slave switch, and the database might be unavailable for few minutes.
We need tolerant this issue and unblock the main thread.
BTW, this pr refactor the metadata store naming.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2834 from turboFei/retrying_jdbc_event.
Closes#2834
109a63e70 [Fei Wang] fix docs
a186484be [Fei Wang] refactor the jdbc pool name
cfac9da53 [Fei Wang] rename metadata table name
bb8710ba3 [Fei Wang] comments
1197a098b [Fei Wang] add private package scope
a95694ba9 [Fei Wang] comments
e2f09e784 [Fei Wang] refactor
057eb73c5 [Fei Wang] refactor
88f3ffb6f [Fei Wang] refactor
e0595bdee [Fei Wang] comments
48270e869 [Fei Wang] refactor conf keys
e4f190c5c [Fei Wang] update docs
ed5e90e37 [Fei Wang] add max queues limitation
c5a03c9e2 [Fei Wang] remove getOrCreate
3e39ca94c [Fei Wang] remove
b79fb0954 [Fei Wang] comments
1d309cee0 [Fei Wang] comments
9108a3f0e [Fei Wang] fix docs
5cdffe710 [Fei Wang] fix rat
2e7228fc7 [Fei Wang] fix ut
520236d7b [Fei Wang] rename more
f66d68bcc [Fei Wang] merge into metadata manager
2f70a2f12 [Fei Wang] rename
80e6100fb [Fei Wang] refactor
09cad092a [Fei Wang] add ut
ea4eea74f [Fei Wang] refactor
6b00eb0b3 [Fei Wang] refactor to MetadataRequest
58f1c0587 [Fei Wang] commements
198feb53f [Fei Wang] add more docs
670a23561 [Fei Wang] refactor
a20d54df2 [Fei Wang] waitStateStoreRetryCompletion after super.close closes operation
97a763aef [Fei Wang] refactor RetryingStateStoreRequest
3e136cf34 [Fei Wang] [SUB-TASK][KPIP-4] Support to retry the session state operations if meet transient jdbc issue
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
To close#2859
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2875 from turboFei/kyuubi_2859_retry_delete.
Closes#2859
7908d827 [Fei Wang] revert commands related
23d357dc [Fei Wang] fix ut
76730014 [Fei Wang] print close resp first
76449dad [Fei Wang] support --conf
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Add v1.5.2-incubating changelog for release
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2846 from ulysses-you/add-changelog.
Closes#2846
19c514b5 [ulysses-you] v1.5.2-incubating changelog
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
refactor the changelog script to fix:
- the nit of Apache Kyuubi(Incubating)
- the format of the output changelog
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2839 from ulysses-you/changelog.
Closes#2839
d9e2b6d0 [ulysses-you] comments
b966dc30 [ulysses-you] comments
5f94c52d [ulysses-you] refactor
18cdde22 [ulysses-you] refactor
009699c3 [ulysses-you] Merge branch 'master' of https://github.com/apache/incubator-kyuubi into changelog
76419a92 [ulysses-you] fix
1a4090fc [ulysses-you] nit
b6154b58 [ulysses-you] changelog
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Add generate_changelog.sh script to simple generate changelog for Apache Incubating Kyuubi.
I've searched there are generaly three ways to generate changelog
1. add change log manually in ervery pr ([Apache Skywalking](https://github.com/apache/skywalking/pull/8881/files#diff-e4d10d406c447edb82b7e7958bebc5ff8ca9123076cf138b4e0ba037fa873055))
2. use github action to generate ([sqlfluff](https://github.com/sqlfluff/sqlfluff/blob/main/.github/release-drafter.yml))
3. use script to generate changelog
I choose to write a script to generate changelog, it does not rely on github and we can use it more conviniently as there maybe mutiple release candidates in the release process.
I add a script to generate changelog once we prepare a new release version, it allows release manager to easy generate changelog between last release tag and current release tag.
And the changelog doc structure is:
$KYUUBI_DIR
    /docs
        /changelog
            /v1.5.1-incubating.md
            ...
### _How was this patch tested?_
use `./generate_changelog.sh v1.5.0-incubating v1.5.1-incubating-rc0` to generate v1.5.1-incubating changelog
Closes#2501 from yangrong688/KYUUBI-2211.
Closes#2211
b3174b4d [yangrong688] Add generate_changelog.sh script to simple generate changelog for Apache Incubating Kyuubi.
Authored-by: yangrong688 <yangrong.jxufe@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
To close#2373
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2790 from turboFei/recovery_batch.
Closes#2373
81632191 [Fei Wang] trigger test
6ac6f9bd [Fei Wang] async recovery
6d9edf4e [Fei Wang] comments
daa6719f [Fei Wang] refactor
0c3a2e14 [Fei Wang] remove waitAppCompletion
04e139ee [Fei Wang] comments
b5d11d8f [Fei Wang] comment for method name
321dfa95 [Fei Wang] refactor
90be2df7 [Fei Wang] address comments
14409582 [Fei Wang] batch recovery
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Add a doc of debugging engine in `debugging.md`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2798 from deadwind4/KYUUBI-2793-debug-engine-doc.
Closes#2793
3f24ca5b [Ada Wang] [KYUUBI #2793][DOCS] Add debugging engine
Authored-by: Ada Wang <wang4luning@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Close#2715Close#2374
Introduce jdbc based session state store to share the batch state information across kyuubi instances.
This pr is based on docs: https://docs.google.com/document/d/1_BMekMT88P2vCHCl4ZacWEeWslpIv4pxsJH32PAnWSI and refactor it during implementation.
In this PR, I introduce `StateStore` interface and implement the default `JDBCStateStore`.
The data structures:
1. BatchState - the batch state info for multiple-HA, including batch operation state and application information
2. BatchMeta - the meta data used for batch recovery, including sessionConf, batchRequest information
For JDBCStateStore:
- support to specify data base type:
- 1. DERBY - it will use derby jdbc driver by default
- 2. MYSQL - it will use mysql jdbc by default
- 3. CUSTOM - the user need specify the jdbc driver
- support to init schema
By default, the jdbc url is a derby in-memory database url for single kyuubi instance.
To support multiple HA in production env, we should config the production jdbc url to share batch info across kyuubi instances.
The datasource is managed by HiKariCP.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2717 from turboFei/kyuubi_2715_hikaricp_v2.
Closes#2717
367b944a3 [Fei Wang] comments
9a7963dbd [Fei Wang] post session type in event
b1c130853 [Fei Wang] move session type to session package
66def0cb6 [Fei Wang] add JDBCStateStoreConf
839b72bf6 [Fei Wang] refactor
644cbcd28 [Fei Wang] filter by session type
58f1c7854 [Fei Wang] refactor
1c0035208 [Fei Wang] add session type
34a087ee5 [Fei Wang] refactor
fba38e473 [Fei Wang] refactor
fcbde25a0 [Fei Wang] refactor
332a67537 [Fei Wang] comments
135ba9b6d [Fei Wang] Revert "revert class utils change"
0e045a483 [Fei Wang] Revert "Revert "move""
05048416e [Fei Wang] Revert "test again"
01a50c35b [Fei Wang] test again
b89159d13 [Fei Wang] Revert "move"
3764a9051 [Fei Wang] move
eef17822c [Fei Wang] revert
4166db261 [Fei Wang] adjust engine idle timeout
5fc95a824 [Fei Wang] revert class utils change
23f88d63d [Fei Wang] refactor
a66cb811d [Fei Wang] refactor
89fa0edb4 [Fei Wang] fix
e5228fefd [Fei Wang] refactor
581908d86 [Fei Wang] exclude sql
3446aaa06 [Fei Wang] UPPER
ab980c677 [Fei Wang] [KYUUBI #2715] Introduce session state store for batch session multiple HA
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
make table width fit browsers with 100% width, and 1st column where contains the kyuubi conf fit the table with 25% width.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
<img width="1716" alt="image" src="https://user-images.githubusercontent.com/8326978/170655840-2309574e-4098-4b0f-b91d-50a30033b9ac.png">
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2764 from yaooqinn/layout.
Closes#2764
9391a638 [Kent Yao] [DOCS] Fix tables in docs being coverd by right toc sidebar
33817406 [Kent Yao] [DOCS] Fix tables in docs being coverd by right toc sidebar
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#2721
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2728 from cxzl25/KYUUBI-2721.
Closes#2721
67051644 [sychen] use placeholders and confOverlay
2acf0a45 [sychen] unused import
8bc40b9f [sychen] fix UT
3495b280 [sychen] fix jdbc
e55bb2e2 [sychen] style
ff9078ab [sychen] fix UT
a0247c7e [sychen] Merge branch 'master' into KYUUBI-2721
f62461ce [sychen] use placeholders
698fbbdd [sychen] set catalog/schema empty resultset
45caf552 [sychen] fix import
7f13ad8d [sychen] fix server spark catalog UT
4982ec71 [sychen] hive engine 1.8
9877038f [sychen] add UT
21e57870 [sychen] set/get catalog/database operators
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2595 from zhaomin1423/jdbc_engine.
Closes#2658
90bdf97c [Min Zhao] add jdbc dialect
d2bfd60c [Min Zhao] [KYUUBI #2486] [FEATURE] Support Apache Doris engine
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Rename high availability config key to support muti discovery client
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2633 from hddong/muti-discovery-config-support.
Closes#2631
bc434ac9 [hongdongdong] [KYUUBI #2631] Rename high availability config key to support muti discovery client
Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
### _Why are the changes needed?_
Support internal rest request authentication to enable redirect http request across kyuubi instances
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2719 from turboFei/internal_access.
Closes#2719
b9e150e9 [Fei Wang] revert config key change
aa378727 [Fei Wang] make it internal
ce96d922 [Fei Wang] comments
6fe8523b [Fei Wang] Support internal rest request authentication to enable redirect http request across kyuubi instances
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Because `kyuubi-server` is a long-running service, its logs should preferably have dates.
When Log4j1 was used in previous versions, the default log and template had dates.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2714 from cxzl25/KYUUBI-log4j-layout-pattern.
Closes#2714
b249c296 [sychen] change layout pattern
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
We want to implement different service discovery client, thus add a config to support different service discovery client class implementation.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2601 from hiboyang/datapunch-DiscoveryClient-Configurable.
Closes#2601
719685f1 [BoYang] Update settings.md
8af4f5da [BoYang] Change name to HA_DISCOVERY_CLIENT_CLASS
22139b6d [BoYang] Update settings.md
c9845cee [BoYang] Revert unrelated changes
1713e050 [BoYang] Add .github files
22222a4d [BoYang] Merge branch 'master' into datapunch-DiscoveryClient-Configurable
c0c293db [BoYang] Merge branch 'master' of https://github.com/apache/incubator-kyuubi
3e53ef48 [BoYang] Do not copy spark files in docker
887be439 [BoYang] Add dot after docker build command
48b70053 [BoYang] Update manual-release.yml
39d0bb21 [BoYang] Update workflow: cd dist
9adae8c1 [BoYang] Update manual-release.yml
da76f568 [BoYang] Use nonEmpty
bf8fe2ff [BoYang] Run mvn spotless:apply
188d71c5 [BoYang] Update per comments
fffa3bfe [BoYang] Create manual-release.yml
65c62c31 [BoYang] Add KyuubiConf.createInstance and reuse some class checking code
30f0ff35 [BoYang] Add a config to support different service discovery client class implementation
1cdf8809 [BoYang] Delete github workflows
Authored-by: BoYang <datapunch2021@gmail.com>
Signed-off-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
### _Why are the changes needed?_
close#2605
`kyuubi.engine.spark.register.operation.listener`
> When set to true, Spark engine registers a SQLOperationListener before executing the statement, logs a few summary statistics when each stage completes.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2617 from cxzl25/KYUUBI-2605.
Closes#2605
526af5eb [sychen] use spark conf
492a2044 [sychen] update settings.md
22fe762f [sychen] add UT
a7b29de1 [sychen] Merge branch 'master' into KYUUBI-2605
e8fa3c6f [sychen] kyuubi.engine.spark.register.operation.listener
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Kent Yao <yao@apache.org>
