### _Why are the changes needed?_
This PR implements a simple `EngineSecuritySecretProvider` beside the existing zookeeper implementation, which simplifies the user threshold to use RESTful API w/ HA mode, and this PR also allows the user set `kyuubi.engine.security.secret.provider` using short name 'simple' or 'zookeeper' as well as the full class name.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4419 from pan3793/simple-secret-provider.
Closes#4419
32b1f966a [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
1af12b99d [Cheng Pan] nit
28a228eea [Cheng Pan] nit
65f14494c [Cheng Pan] Implement simple EngineSecuritySecretProvider
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
1. get spark engine runtime version instead of compile version
2. moved `PySparkTests` from the module `kyuubi-spark-sql-engine` to `kyuubi-server` to ensure that the python progress loading library PYSPARK has the same version as the launched Spark engine. see https://github.com/apache/kyuubi/pull/4381#issuecomment-1442871106
### _How was this patch tested?_
Pass Github Action.
Closes#4381 from cfmcgrady/spark-3.4.0.
Closes#4381
2711f51b3 [Fu Chen] remove verify spark-3.4 binary
a93b6d13e [Fu Chen] mv PySparkTests and enabled
6d5aad537 [Fu Chen] fix style
2da641561 [Fu Chen] fix style
3c9e300ce [Fu Chen] spark compile version -> runtime version
a8e7b7481 [Fu Chen] unused import
6be502ca6 [Fu Chen] fix ut
c1a1e1a8e [Fu Chen] skip pyspark tests
0049c23b7 [Fu Chen] verify spark-3.4.0 binary
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Close#4376
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4405 from lightning-L/kyuubi-4376.
Closes#4376
1a01a75a8 [Tianlin Liao] rename and refactor
7324cab3d [Tianlin Liao] [KYUUBI #4376] Support to config the kyuubi service administrator with kyuubi conf
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4415 from turboFei/operation_handle_align.
Closes#4415
71721797a [fwang12] refactor
c8b667a89 [fwang12] refactor
d3fbf05f3 [fwang12] stmt handle
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Align the server session handle and engine session handle for Spark engine.
It make it easy to recovery the engine session in any kyuubi instance easy.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4412 from turboFei/server_engine_handle_align.
Closes#4412
a20e0f155 [fwang12] fix
9d590e38b [fwang12] fix
94267e583 [fwang12] save
7012c2bef [fwang12] align
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4398 from turboFei/rename_batch_limit.
Closes#3957
28228e4ed [fwang12] 3957 followup
ef1ad6ea5 [fwang12] save
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4395 from lightning-L/kyuubi-4388.
Closes#4388
4f4ae8e59 [Tianlin Liao] create as optional
6b85f6ae6 [Tianlin Liao] refactor
7c2cd5661 [Tianlin Liao] rename conf
2f00da0f4 [Tianlin Liao] throw error in AbstractBackendService to enable it for all kinds of clients
3feabf78c [Tianlin Liao] [KYUUBI #4388] Limit the max rows for get nextRowSet api
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
As discussed https://github.com/apache/kyuubi/pull/4144#discussion_r1112957149, we should put the conf key into namespace `kyuubi.batch.`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4394 from pan3793/4106-followup.
Closes#4106
27c9ed870 [Cheng Pan] nit
66283ac4b [Cheng Pan] [KYUUBI #4106][FOLLOUP] Rename conf key of uploaded batch resource
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4382 from lightning-L/kyuubi-4352.
Closes#4352
2beb1ef40 [Tianlin Liao] update config name
a9eb126d8 [Tianlin Liao] update config setting
75cdfbb02 [Tianlin Liao] [KYUUBI #4352] Support System.gc() with periodic GC interval
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Filter out the LDAP configurations to suppress potential security issues.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4383 from pan3793/auth-conf.
Closes#4152
032245f8 [Cheng Pan] [KYUUBI #4152] LDAP configurations should be server-only
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
To close#4216
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4370 from turboFei/client_version.
Closes#4216
efd934c6b [fwang12] save
cf8acd54a [fwang12] version properties
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
1. this PR introduces a new configuration called `kyuubi.operation.result.arrow.timestampAsString`, when true, arrow-based rowsets will convert timestamp-type columns to strings for transmission.
2. `kyuubi.operation.result.arrow.timestampAsString` default setting to false for better transmission performance
3. the PR fixes timezone issue in arrow based result format described in #3958
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4326 from cfmcgrady/arrow-string-ts.
Closes#4326
38c7fc9b [Fu Chen] fix style
d864db00 [Fu Chen] address comment
b714b3ee [Fu Chen] revert externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/schema/RowSet.scala
6c4eb507 [Fu Chen] minor
289b6007 [Fu Chen] timstampAsString = false by default
78b7caba [Fu Chen] fix
f5601356 [Fu Chen] debug info
b8e4b288 [Fu Chen] fix ut
87c6f9ef [Fu Chen] update docs
86f6cb73 [Fu Chen] arrow based rowset timestamp as string
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Support to refresh the unlimited users for session limiter, so that we can unblock some customers without restart the kyuubi server.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4360 from turboFei/limier_whitelist.
Closes#4360
c846148bd [fwang12] typo
839e71365 [fwang12] nit
904bc9fbc [fwang12] save
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
This PR proposes to fix#4282, since log4j2 supports async mode, we need to make sure the `Log4j2DivertAppender#append` is thread-safe.
This PR also changes `OperationLog.getCurrentOperationLog` from `OperationLog` to `Option[OperationLog]`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4300 from pan3793/log.
Closes#4282
010e34b0 [Cheng Pan] fix
068405b2 [Cheng Pan] fix compile
c79dedd5 [Cheng Pan] Use write lock instead
3daf8a4d [Cheng Pan] nit
94176a04 [Cheng Pan] [KYUUBI #4282] Fix ConcurrentModificationException when log4j2 async enabled
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
It can help to know the backend pressure if the exec pool is full.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4344 from turboFei/wait_queue.
Closes#4344
161e3808a [fwang12] nit
6d122e238 [fwang12] save
55a4b499d [fwang12] version
668ff8bfe [fwang12] save
9f56b98a8 [fwang12] save
a401771ec [fwang12] wait
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
according to `io.trino.jdbc.ColumnInfo`, there are some type requring signature parameter.
- varchar(n)
- char(n)
- decimal(precision, scale)
It failed with trino jdbc now
<img width="613" alt="image" src="https://user-images.githubusercontent.com/12025282/218707052-a2e9dc91-0333-483c-bc0a-96baec213578.png">
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4328 from ulysses-you/fix-signature.
Closes#4328
aede5cec [ulysses-you] nit
ae1a7968 [ulysses-you] fix test
8ecdb346 [ulysses-you] Make Trino jdbc driver work
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
### _Why are the changes needed?_
This PR proposes to use `org.apache.spark.sql.execution#toHiveString` to replace `org.apache.kyuubi.engine.spark.schema#toHiveString` to get consistent result w/ `spark-sql` and `STS`.
Because of [SPARK-32006](https://issues.apache.org/jira/browse/SPARK-32006), it only works w/ Spark 3.1 and above.
The patch takes effects on both thrift and arrow result format.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
```
➜ ~ beeline -u 'jdbc:hive2://0.0.0.0:10009/default'
Connecting to jdbc:hive2://0.0.0.0:10009/default
Connected to: Spark SQL (version 3.3.1)
Driver: Hive JDBC (version 2.3.9)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 2.3.9 by Apache Hive
0: jdbc:hive2://0.0.0.0:10009/default> select to_timestamp('2023-02-08 22:17:33.123456789');
+----------------------------------------------+
| to_timestamp(2023-02-08 22:17:33.123456789) |
+----------------------------------------------+
| 2023-02-08 22:17:33.123456 |
+----------------------------------------------+
1 row selected (0.415 seconds)
```
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4318 from pan3793/hive-string.
Closes#4316
ba9016f6 [Cheng Pan] nit
8be774b4 [Cheng Pan] nit
bd696fe3 [Cheng Pan] nit
b5cf051c [Cheng Pan] fix
dd6b7021 [Cheng Pan] test
63edd34d [Cheng Pan] nit
37cc70af [Cheng Pan] Fix python ut
c66ad22d [Cheng Pan] [KYUUBI #4316] Fix returned Timestamp values may lose precision
41d94445 [Cheng Pan] Revert "[KYUUBI #3958] Fix Spark session timezone format"
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Close#3935
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4232 from iodone/kyuubi-3935.
Closes#3935
936ea1f8 [odone] address
e7bd01a1 [odone] support trino client connect kyuubi trino server
9ea8b6af [odone] [WIP] trion request/response implementation
Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
### _Why are the changes needed?_
This PR proposes to change the paths of distributed lock and seq_num(used for POLLING engine pool select policy) on the Service Discovery component. The reason is that namespace `${serverSpace}_${KYUUBI_VERSION}_${shareLevel}_${engineType}/` should be dedicated to engine registration, we'd better use the separated namespace for other functionalities.
- lock path
```
# before
${serverSpace}_${shareLevel}_${engineType}/lock/${user}/${subdomain}
# after
${serverSpace}_${KYUUBI_VERSION}_${shareLevel}_${engineType}_lock/${user}/${subdomain}
```
- seq_num
```
# before
${serverSpace}_${KYUUBI_VERSION}_${shareLevel}_${engineType}/seq_num/${user}/${poolName}
# after
${serverSpace}_${KYUUBI_VERSION}_${shareLevel}_${engineType}_seqNum/${user}/${poolName}
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4145 from pan3793/namespace.
Closes#4145
c912b3f66 [Cheng Pan] name
3326b9b95 [Cheng Pan] name
10083db0b [Cheng Pan] Change lock and polloing seq_num path on service discovery
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
`kyuubi.operation.result.max.rows` does not take effect on incremental collect mode because of performance concerns, this PR updates the configuration docs to mention that.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4271 from pan3793/maxrow.
Closes#4271
29b290a3e [Cheng Pan] nit
3d2872352 [Cheng Pan] log
277ebb5ff [Cheng Pan] ifx
091511a91 [Cheng Pan] nit
f15fb2270 [Cheng Pan] nit
1a1259ef5 [Cheng Pan] nit
ba57a2660 [Cheng Pan] fix
0b58d8b1a [Cheng Pan] ut
74b59dcee [Cheng Pan] nit
230defbff [Cheng Pan] Increamental collect mode should respect kyuubi.operation.result.max.rows
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- as pyspark is supported in #3780, exposing PYTHON option for `kyuubi.operation.language` as an experimental feature
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4256 from bowenliang123/doc-language-python.
Closes#4256
d10c0281 [liangbowen] update description
d05cedda [liangbowen] add PYTHON option in kyuubi.operation.language
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Note:
- No output changes to existed generated docs as in `settings.md` and `functions.md`.
Improvement:
- readability in Scala code for doc auto-generation, and easy to maintain docs in group of sections
- reline on markdown linting from `flexmark`, and reducing un-meaningful blank lines or alignment
- declaration over operation by replacing repeated usages of `newOutput` itself by elegantly wrapped `.line`,`.lines`and etc.
- less fragile and more handy for changing in a long single line, as now using auto margin stripping in `.line()` method
- reusable extracted licence and auto-generation hints
- more elegant and safer way to read and appending file content
- possible less memory footprint by apply operators to Stream instead of to ArrayBuffer
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4244 from bowenliang123/config-regen.
Closes#4244
3d90ad304 [liangbowen] make buffer private in MarkdownBuilder
8250a55f3 [liangbowen] remove licence.md
a4c7baf78 [liangbowen] add MarkdownBuilder.apply
248a046a4 [liangbowen] Improvement in auto-generated Markdown docs with MarkdownBuilder
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Before:
```
2023-02-02 22:21:35.507 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: SSL Server Socket enabled protocols: [Ljava.lang.String;2fca3eb5
```
Seems `enabledProtocols` is a java array, the output is not friendly.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4238 from turboFei/ssl_followup.
Closes#3658
999ef90e6 [fwang12] Show enabled ssl protocols
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Address comments: https://github.com/apache/kyuubi/pull/4144#issuecomment-1412078077
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4231 from turboFei/uploading_resource_followup.
Closes#4144
93ccd9534 [fwang12] comments
81a224dce [fwang12] ut
6b226969a [fwang12] prevent temp file leak
17a4e394c [fwang12] Do not remove upload dir
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
This PR proposes to enhance the LDAP support, which mainly referring the code introduced in HIVE-14713.
Currently, Kyuubi has very limited LDAP support, and the implementation is from the early Hive codebase. Hive enhanced the LDAP support in later versions, considering the Hive ecosystem is quite mature, I think we'd better to porting this functionality and keep the same behavior w/ Hive first, and we can improve it if meet certain requirements/issues in the future.
Basically, this PR introduces the following configurations
```
kyuubi.authentication.ldap.url (since 1.0.0)
kyuubi.authentication.ldap.domain (since 1.0.0)
kyuubi.authentication.ldap.guidKey (since 1.2.0)
kyuubi.authentication.ldap.base.dn (since 1.0.0 deprecated)
kyuubi.authentication.ldap.baseDN
kyuubi.authentication.ldap.groupMembershipKey
kyuubi.authentication.ldap.userMembershipKey
kyuubi.authentication.ldap.groupClassKey
kyuubi.authentication.ldap.groupDNPattern
kyuubi.authentication.ldap.userDNPattern
kyuubi.authentication.ldap.groupFilter
kyuubi.authentication.ldap.userFilter
kyuubi.authentication.ldap.customLDAPQuery
kyuubi.authentication.ldap.binddn
kyuubi.authentication.ldap.bindpw
```
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
This PR ports all LDAP-related UT&IT from Hive codebase
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4152 from pan3793/ldap.
Closes#4152
d251c959 [Cheng Pan] nit
6d14f44b [Cheng Pan] nit
6b3d116c [Cheng Pan] nit
ab47d822 [Cheng Pan] nit
a56e8702 [Cheng Pan] nit
4624619a [Cheng Pan] nit
b82c0c05 [Cheng Pan] LDAP test password uses alphanumeric
86a01cca [Cheng Pan] Enhance LDAP authentication
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Show line number and assertion when verifying generated markdown files, e.g. `settings.md` from `AllKyuubiConfiguration`, for quicker locating the place and differences in files.
- updated hint message to `mvn clean test` instead of `mvn clean install` for faster verification or regeneration
Hints with line num as below if assertion fails in line comparison.
<img width="1121" alt="image" src="https://user-images.githubusercontent.com/1935105/215451115-813b90f0-9d9d-4ebd-974e-8a071424aa42.png">
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4212 from bowenliang123/markdown-verify-linenum.
Closes#4212
82791a88 [liangbowen] style
609c3b35 [liangbowen] update hints
ed889915 [liangbowen] nit
c676653b [liangbowen] updated hints of markdown generation for AllKyuubiConfiguration and KyuubiDefinedFunctionSuite
646832a0 [liangbowen] show line number and content of expected and got when verifying markdown files
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- to consolidate styles in markdown files from manual written or auto-generated
- apply markdown formatting rules with flexmark from [spotless-maven-plugin](https://github.com/diffplug/spotless/tree/main/plugin-maven#markdown) to *.md files in `/docs`
- use `flexmark` to format markdown generation in `TestUtils` of common module used by `AllKyuubiConfiguration` and `KyuubiDefinedFunctionSuite`, as the same way in `FlexmarkFormatterFunc ` of `spotless-maven-plugin` using with `COMMONMARK` as `FORMATTER_EMULATION_PROFILE` (https://github.com/diffplug/spotless/blob/maven/2.30.0/lib/src/flexmark/java/com/diffplug/spotless/glue/markdown/FlexmarkFormatterFunc.java)
- using `flexmark` of` 0.62.2`, as the last version requiring Java 8+ (checked from pom file and bytecode version)
```
<markdown>
<includes>
<include>docs/**/*.md</include>
</includes>
<flexmark></flexmark>
</markdown>
```
- Changes applied to markdown doc files,
- no style change or breakings in built docs by `make html`
- removal all the first blank in licences and comments to conform markdown style rules
- tables regenerated by flexmark following as in [GitHub Flavored Markdown](https://help.github.com/articles/organizing-information-with-tables/) (https://github.com/vsch/flexmark-java/wiki/Extensions#tables)
### _How was this patch tested?_
- [x] regenerate docs using `make html` successfully and check all the markdown pages available
- [x] regenerate `settings.md` and `functions.md` by `AllKyuubiConfiguration` and `KyuubiDefinedFunctionSuite`, and pass the checks by both themselves and spotless check via `dev/reformat`
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4200 from bowenliang123/markdown-formatting.
Closes#4200
1eeafce4 [liangbowen] revert minor changes in AllKyuubiConfiguration
4f892857 [liangbowen] use flexmark in markdown doc generation
8c978abd [liangbowen] changes on markdown files
a9190556 [liangbowen] apply markdown formatting rules with `spotless-maven-plugin` to markdown files with in `/docs`
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
to close#4106 .
1. add `POST /batches`API in `BatchesResource` of REST API, which consumes `multipart/form-data` media type to support uploading `resourcefile`. And a `batchRequest` form data part in the JSON format string of `BatchRequest`, as in the required request body of `POST /batches`
2. the uploaded `resourceFile` is saved to a temp local file which will be cleaned up after job execution at the end of `submitAndMonitorBatchJob`
3. the local temp copy of `resourceFile` will be used as `resource` in `BatchJobSubmission`, eg. as <application-jar> for spark-submit
Todos in follow-up:
1. add a related description in Rest API doc
4. add `multipart` media type support to `RestClient` and implement `createBatchWithUploadingResource` in `BatchRestApi`
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4144 from bowenliang123/rest-batch-upload.
Closes#4106
f6723a02 [Bowen Liang] Merge branch 'master' into rest-batch-upload
0dd67245 [liangbowen] correct dependencyList
6365a0cd [liangbowen] introducing resource file upload support in batch creation
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Add catalog tree for `kyuubi.authentication` options
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
<img width="746" alt="image" src="https://user-images.githubusercontent.com/26535726/212236908-294b7fa9-e3db-4730-8c16-41c8fd409ae8.png">
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4159 from pan3793/auth-doc.
Closes#4159
8d65768d [Cheng Pan] [DOCS] Refine doc for kyuubi.authentication
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- commit date is more readable and helpful to determine earlier/later version, especially for snapshot branch
- adding revision commit date from git to `kyuubi-version-info.properties`
- Print revision date on server startup
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4187 from bowenliang123/print-revision-date.
Closes#4187
b0b0a6c8 [liangbowen] print revision date on server startup
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
As Kyuubi graduated as top level project, the setting page will be more often requested and should be increasingly reliable and readable with less grammar and spelling mistakes.
This PR is to
- correct mistakes in grammar, spelling, abbreviation and terminology
- with no config name or essential meanings changed
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4161 from bowenliang123/conf-grammar.
Closes#4161
038edfbea [liangbowen] nit
1ec073a4b [liangbowen] to JSON
4f5259a32 [liangbowen] to Prometheus
523855008 [liangbowen] to K8s
fc7a3a81e [liangbowen] to AUTO-GENERATED
da64f54fa [liangbowen] update
d54f9a528 [liangbowen] fix `comma separated` to `comma-separated`
f1d7cc1f1 [liangbowen] update
d84208844 [liangbowen] update
1b75f011c [liangbowen] correction of grammar and spelling mistakes
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
The output of building and testing is now brought with repeated unuseful warnings from `javac`.
This PR reduces over 500 unhelpful warning message lines from `-Xlint` rules in `javac` compilation output in a single run, by eliminating :
- 74 of `deprecation`, by adding `SuppressWarning` to 6 Java classes
- 6 of `rawtypes`, by adding `SuppressWarning` to methods of Java class
- 4 of `unchecked`, by adding `SuppressWarning` to methods of Java class
- 1 of `fallthrough`, by adding `SuppressWarning` to method of Java class
- 1 of `cast`, by removal of redundant type casting
- repeated `processing` for, by adding `-processing` to disable annotation processor warning for annotations from testing framework :
```
- Warning: Unexpected javac output: warning: No processor claimed any of these annotations: /org.scalatest.TagAnnotation
Warning: Unexpected javac output: warning: No processor claimed any of these annotations: /org.junit.Test,/org.junit.Before,/org.junit.After
```
As 1 [fallthrough] warning message remain, which will be reported in followup issues or PR to fix missing breaks for branches of `switch-case` in Java.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4166 from bowenliang123/java-warnings.
Closes#4166
ec4cfc42 [liangbowen] add @SuppressWarnings("fallthrough") to KyuubiConnection#waitLaunchEngineToComplete
c9a59522 [liangbowen] skip deprecation warnings for SQLResultSet and KyuubiBaseResultSet
cab8c358 [liangbowen] remove redundant casting
f2ce8963 [liangbowen] style
bf6c5936 [liangbowen] add `-processing` to javac Xlint option for eliminating the missing annotation processer warnings
01a89a8c [liangbowen] eliminate warning outputs in java by adding @SuppressWarnings
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Support to ignore subdomain when engine pool conditions are met.
It is difficult to understand the engine routing rule for customers, especially for the engine pool and subdomain.
Maybe a user use subdomain before, and then they want to use engine pool.
Then specify the engine pool name, engine pool size, but the result is not expected.
They need to remove the subdomain specified before, otherwise, the engine pool will not work.
I need tell customer that, the engine subdomain config has higher priority and tell them the routing rule.
The understanding cost is high.
With this change, it is clear for engine pool conditions.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4151 from turboFei/engine_pool_enabled.
Closes#4151
cbb7846ff [fwang12] comments
9f4a696ae [fwang12] refactor
9801dc85e [fwang12] save
e5f769874 [fwang12] engine pool first
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4153 from turboFei/log_user_close.
Closes#4153
d55cbcdc6 [fwang12] logSessionCountInfo
7cdb1f45f [fwang12] log more session info
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
to close issue #4058 and help to close pr #3852.
When execute test `KyuubiOnKubernetesWithClientSparkTestsSuite` with `execute statement - select with variable substitution`, the initial method
`org.apache.kyuubi.operation.HiveJDBCTestHelper#withThriftClient` will overlook the connectionConf in `org.apache.kyuubi.operation.JDBCTestHelper#jdbcUrl`:
```
def withThriftClient[T](user: Option[String] = None)(f: TCLIService.Iface => T): T = {
TClientTestUtils.withThriftClient(
jdbcUrl.stripPrefix(URL_PREFIX).split("/;").head,
user)(f)`
}
```
Here we just aborted the tail of `jdbcUrl.stripPrefix(URL_PREFIX).split("/;")`, in which the connectionConfs are set.
In that case the `org.apache.kyuubi.kubernetes.test.deployment.KyuubiOnKubernetesWithClientSparkTestsSuite#connectionConf` was invalid actually.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4059 from xuefeimiaoao/branch-fix-connectionConf.
Closes#4058
a6395631 [xuefeimiaoao] [KYUUBI #4058] [IT][Test][K8S] Fix the missing of connectionConf of SparkQueryTests
Authored-by: xuefeimiaoao <1255072085@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
After some offline discussions, I propose to change the configuration key from`kyuubi.operation.result.codec` to `kyuubi.operation.result.format`.
### _How was this patch tested?_
Pass CI.
Closes#4075 from cfmcgrady/arrow-conf-rename.
Closes#4075
5ad45507 [Fu Chen] fix
214b43fd [Fu Chen] rename
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close#3982 .
Introduce feature of refresh user defaults config (as `___${user}___.*` which starts with three continuous underscores "___") from config file via `kyuubi-admin` cli and `refresh/user_defaults_conf` Rest API.
1. add `refreshUserDefaultsConf` methond in KyuubiServer to read user defautls configs from property file and apply config changes to server's KyuubiConf
3. add `refresh/user_defaults_conf` api to AdminRestApi calling `refreshUserDefaultsConf` of KyuubiServer
3. add config type `userDefautls` in kyuubi-admin cli refresh command
This feature will
- help to apply user defaults conf without restarting server or losing connections
- load latest config for engine launch, e.g. spark related config `spark.*`
It won't
- affect the components already started and using the clone of server conf
- affect configs for launched engine instance
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3983 from bowenliang123/3982-reload-server-conf.
Closes#3983
a8fb0bf2 [liangbowen] fix typo
b9b80f56 [liangbowen] update
b47a1541 [liangbowen] minor
786cb2a0 [liangbowen] add logging for statistics
0860e3a7 [liangbowen] fix: loadFileDefaults in refreshUserDefaultsConf
8dbbbcb8 [liangbowen] fix typo
619acd2e [liangbowen] import
e405dc8e [liangbowen] fix user defaults key filtering by adding `getAllUserDefaults` to `KyuubiConf`
ac407bd5 [liangbowen] rename config refresh option to `userDefaultsConf` and extracted to RefreshConfigCommandConfigType
c65398b2 [liangbowen] fix redundant loadFileDefaults in refreshUserDefaultsConf
1b046feb [liangbowen] update comments
e5dd5dbb [liangbowen] typo
c0a358ba [liangbowen] change to refresh users' config. rename cli command to `refresh config userDefaults`
bf5448e3 [liangbowen] support reload server config from config file 1. add reloadServerConf in KyuubiServer to read config and put all to config, 2. add "refresh/server_conf" api to AdminRestApi, 3. add config type "serverConf" in kyuubi-admin cli
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4069 from lightning-L/kyuubi-4020.
Closes#4020
97406ca0 [Tianlin Liao] [KYUUBI #4020] remove incubating from kyuubi source code
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Unlike `kyuubi.frontend.bind.port`, which is deprecated and replaced by `kyuubi.frontend.thrift.binary.bind.port`, `kyuubi.frontend.bind.host` is still encouraged to use, and it can be overwritten by `kyuubi.frontend.<protocol>.bind.host`, e.g. `kyuubi.frontend.thrift.binary.bind.host`, `kyuubi.frontend.thrift.http.bind.host`, `kyuubi.frontend.rest.bind.host`, `kyuubi.frontend.mysql.bind.host`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4005 from pan3793/doc.
Closes#4005
9c3b1277 [Cheng Pan] comments
0ef140e4 [Cheng Pan] Remove deprecated from description of kyuubi.frontend.bind.host
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4012 from cxzl25/remove_okStatusWithHint.
Closes#4012
c9eb482e [sychen] remove unused okStatusWithHint method
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
close#3905
### _Why are the changes needed?_
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3977 from iodone/kyuubi-3905.
Closes#3977Closes#3905
4e4a22b1 [odone] trino server init and run
Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
After some thought and some offline discussions, I propose to change the configuration key from `kyuubi.engine.pool.balance.policy` to `kyuubi.engine.pool.selectPolicy`.
And in the future if we introduce properties for specific policy, e.g. supposing we have a engine selection policy named `LOAD`, the properties would be
```
kyuubi.engine.pool.selectPolicy.load.abc=xxxx
kyuubi.engine.pool.selectPolicy.load.xyz=xxxx
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3966 from pan3793/rename.
Closes#2887
7c1eda70 [Cheng Pan] address comments
bf3a3863 [Cheng Pan] [KYUUBI #2887][FOLLOWUP] Rename configuration to kyuubi.engine.pool.selectPolicy
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Followed by issue https://github.com/apache/incubator-kyuubi/issues/3930 and pr https://github.com/apache/incubator-kyuubi/pull/3932.
- correct ut in SessionSigningSuite by fetching session public key and user sign via scala scripts instead of `SET` command in SQL , to prevent accidently redacted by `spark.redaction.regex`
- removed `syncronized` in SignUtils introduced in pr #3932 , with confirming no thread safe problem during the signing and verification
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3974 from bowenliang123/3932-fix2.
Closes#3930
c5f07b95 [liangbowen] comment
1f4ace3a [liangbowen] use KyuubiReservedKeys.KYUUBI_SESSION_USER_SIGN/KYUUBI_SESSION_SIGN_PUBLICKEY for property name
a034cbec [liangbowen] style
1f127c10 [liangbowen] remove `synchronized` from `signWithPrivateKey` and `verifySignWithECDSA` in SignUtils
ff5a21d5 [liangbowen] fix ut by getting `kyuubi.session.sign.publickey` and `kyuubi.session.user.sign` via `spark.sparkContext.getLocalProperty` in scala scripts.
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
to close#3915
This pr adds support for jdbc client detecting result set codec
1. in this PR, hints are added in the `TStatus.getInfoMessages()` to return, and the hints were added when the client retrieves the result set schema from the server
2. the hints mechanism is a general extension when we need to change the client behavior, e.g. adding support for result set compression
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3916 from cfmcgrady/arrow-detect-codec.
Closes#3915
90495c30 [Fu Chen] style
bbeada0a [Fu Chen] address comment
825bc0da [Fu Chen] minor refactor
d0a01ff7 [Fu Chen] address comment
08d21a1c [Fu Chen] fix ut
690126ce [Fu Chen] add hint ut
fd32a317 [Fu Chen] style
a1c2bb6c [Fu Chen] simplify KyuubiConnection
f81336d3 [Fu Chen] refactor
500e766f [Fu Chen] unused import
221bc928 [Fu Chen] fix ut
cf564d0a [Fu Chen] refactor
4b895e45 [Fu Chen] fix compile
3efcc335 [Fu Chen] clean up
95ea29c3 [Fu Chen] Client support detecting ResultSet codec
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Followup for #3957 comments
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3963 from turboFei/unlimited_user.
Closes#3957
987b71f1e [fwang12] refactor
e0c6ca515 [fwang12] [KYUUBI #3957][FOLLOWUP] Refactor the session connection unlimited user list config
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
to close#3930 .
1.make SignUtils.signWithPrivateKey and synchronized
2.show exception with input string when base64 decoding failed
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3932 from bowenliang123/3930-sync-usersign.
Closes#3930
a589486e [liangbowen] 1.make SignUtils.signWithPrivateKey and synchronized, 2.show exception with input string when base64 decoding failed
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
followup for #3941, support to config the interval for updating the same status for a query
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3960 from turboFei/config_exec_state_update.
Closes#3941
57d73190 [fwang12] config the interval for updating the same status for a query
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Prevent batch session & operation closed by kyuubi because of timeout
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3923 from turboFei/batch_idle.
Closes#3923
76b0fe16 [fwang12] use val
310119b0 [fwang12] refactor
61ce2036 [fwang12] use session timeout
52fb0cc4 [fwang12] 1.6.2
7819f2091 [fwang12] save
69ffa316 [fwang12] check batch session timeout
dd210329 [fwang12] add batch op timeout
e304ddb4 [fwang12] never idle batch session and never timeout batch op
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Support dedicated limiter for batch session.
Support user white list.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3957 from turboFei/batch_session_limiter.
Closes#3957
55db79e4 [fwang12] support white list
940b76bd [fwang12] add batch session limiter
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Refer the spark scheduler, warn that batch application maybe starved.
<img width="766" alt="image" src="https://user-images.githubusercontent.com/6757692/206888162-663a3ce0-597a-4818-8f7e-04e7be06cef4.png">
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3959 from turboFei/only_state_change.
Closes#3959
6ab59420 [fwang12] show the batch statvation timeout warn
74be3f48 [fwang12] only state change
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
The Spark session supports setting the time zone through `spark.sql.session.timeZone` and formatting according to the time zone, but `timestamp` does not use timezone, resulting in some incorrect results.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3958 from cxzl25/fix_session_timezone.
Closes#3958
3f2e375c [sychen] ut
e2fd90ac [sychen] session timezone format
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
#3847https://app.travis-ci.com/github/apache/incubator-kyuubi/jobs/590599391
```
SparkOperationSuite:
- get catalogs
- get schemas
- get tables
- get type info
- audit Kyuubi Hive JDBC connection common MetaData *** FAILED ***
"...i Project Hive JDBC []Client" did not equal "...i Project Hive JDBC [Shaded ]Client" (SparkMetadataTests.scala:407)
Analysis:
"...i Project Hive JDBC []Client" -> "...i Project Hive JDBC [Shaded ]Client"
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3955 from cxzl25/jdbc_shaded_followup.
Closes#3847
b229ebe5 [sychen] or
921b44bf [sychen] travis -Pjdbc-shaded
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3946 from turboFei/config_rest_pool_size.
Closes#3946
30641c14 [fwang12] save
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
#3847
Windows users need to generate `kyuubi-version-info.properties` to debug.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3903 from cxzl25/win_gen_version_info.
Closes#3903
089cf9eb [sychen] support windows generate kyuubi-version-info.properties
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Kyuubi supports GROUP engine share level, currently, it just simply delegates the group provider to Hadoop UserGroupInformation, which is not flexible enough for users who want to use other group mapping mechanisms, e.g. LDAP, JDBC.
This PR supplies a pluggable plugin interface `GroupProvider` and provides a built-in `HadoopGroupProvider` which has the same behavior w/ the current implementation.
W/ this change, users can easily implement `LDAPGroupProvider`, `JDBCGroupProvider`, `FileGroupProvider`, `CustomGroupProvider`, etc. then the GROUP engine share level will be more powerful and flexible.
The alternative option is to guide users to learn and extend the Hadoop group mapping system[1].
[1] https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/GroupsMapping.html
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3897 from pan3793/group.
Closes#3897
b100348e [Cheng Pan] nit
3cce6451 [Cheng Pan] Supplying plugable GroupProvider
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
If binary ssl enabled, we will bind the port to ssl server socket.
if we invoke the lazy val serverSocket, it will throw BindException.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3778 from turboFei/ssl_fix.
Closes#3658
5d6f71b7 [fwang12] Fix ssl port binded
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
to close#3839 .
The session signing feature is introduced with asymmetric encryption to prevent manipulation of session user identity or other key session elements, which could cause privilege leaking in scripts.
1. Server: Server creates and holds the singleton `keypair` for session signing if feature enabled
2. Server -> Engine: Server passes the `public key` to Engine when launching
3. Server -> Engine: Server generates session signing on `session user` when opening Kyuubi Session to Engine
4. Session -> Statement: Kyuubi session create Statement with context bringing `publickey` and `session user signature`
5. Engine: Engine verify `session user signature` with signature wherever necessary, e.g. in Authz for session name authentication.
ECDSA, with a shorter key length and better performance than RSA, is the supported asymmetric encryption in the initial implementation, which is widely supported on JDK7+ in HotSpot or OpenJDK. The session signature is generated with `SHA256withECDSA`.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3838 from bowenliang123/check-sessionuser.
Closes#3839
925eee47 [liangbowen] Revert "generalize setSparkLocalProperty"
d726d661 [liangbowen] generalize setSparkLocalProperty
470176db [liangbowen] allow setSparkLocalProperty to set value null
8f2a6e38 [liangbowen] nit
cb5891c9 [liangbowen] nit
edbe3c9c [liangbowen] only set to opensession conf when true, since kyuubi.session.user.sign.enabled is default to false
60546c82 [liangbowen] remove setting setSessionSigningPublicKey in KyuubiSessionManager
9a2a60dd [liangbowen] fix asserts
2e2a51b4 [liangbowen] nit
f37f4dd6 [Bowen Liang] Merge branch 'master' into check-sessionuser
9cd22003 [liangbowen] generalize illegalAccessWithUnverifiedUser in AuthZUtils
c1f27afa [liangbowen] generalize setSessionUserSign and clearSessionUserSign in SparkOperation
3683150f [liangbowen] update error message with `Invalid user identifier`
e143ea77 [liangbowen] fast fail for getting missing session configs
866fc821 [liangbowen] rename param name of base64 encoded pubkey with `Base64` suffix
280a95ef [liangbowen] refactor config key name to new class `ReservedKeys` in Authz
8592070e [liangbowen] nit
bf227e35 [liangbowen] remove redundant ut case
9ed14feb [liangbowen] remove redundant ut case
33a723d0 [liangbowen] add AuthzSessionSigningSuite with session user sign enabled
b77d53e3 [liangbowen] remove config setting in ut
2d56bd30 [liangbowen] remove config setting in ut
a25c1c0b [liangbowen] fail w/ AccessControlException, when kyuubi.session.user.sign.enabled is true and kyuubi.session.user is absent
c8a88fe9 [liangbowen] check not null of userPubKeyStr, userSign
c8bc590a [liangbowen] rename param to publicKeyBase64
b55beb60 [liangbowen] revert to use EC secp256k1 for compatibility
8acfd41d [liangbowen] nit
842b3698 [liangbowen] nit
983585bc [liangbowen] nit
d1003cd7 [liangbowen] nit
f9d6cfb6 [liangbowen] make generateKeyPair return (PrivateKey, PublicKey)
52eaaddc [liangbowen] add ut for SignUtils
b4a44687 [liangbowen] general keypair algorithm
7d40da49 [liangbowen] change to use secp192r1 curve for EC key pair for better performance
1ceed876 [liangbowen] sync settings.md with `false` default value
07d23602 [liangbowen] update ut
eec9d44a [liangbowen] move to _confIgnoreList
b7969446 [liangbowen] create SessionSigningSuite and add ut for 1. verifying user sign , 2. conf kyuubi.session.user.sign.enabled restriction
a50c71a3 [liangbowen] set kyuubi.session.user.sign.enabled to openEngineSessionConf
a2f1ed67 [liangbowen] set kyuubi.session.user.sign.enabled to _confRestrictList
5a12182e [liangbowen] make kyuubi.session.user.sign.enabled default to false, and removed from serverOnlyConfEntries
991a4569 [liangbowen] put SESSION_USER_SIGN_ENABLED in serverOnlyConfEntries
3e863af4 [liangbowen] nit
b232e5c0 [liangbowen] npe
13a046be [liangbowen] nit
2d2a6659 [liangbowen] nit
a0d4721d [liangbowen] move session signing keypair generation to KyuubiSessionImpl on server side. and rename config to KYUUBI_SESSION_SIGN_PUBLICKEY
ab430c39 [liangbowen] make generateKeyPair return Key pair and accept algorithm param
42ee2fe0 [liangbowen] nit
ce5f4af7 [liangbowen] refactoring session pubkey generation on server side (instead of engine side), and passing it to engine via OpenSession op
c2b9d897 [liangbowen] nit: rename to verifySignWithECDSA
d0b2cddb [liangbowen] nit
33a044d0 [liangbowen] add ut
2dc1f57b [liangbowen] change to use spark conf `kyuubi.session.user.sign.enabled` to decide whether verify kyuubi seesion user
b11ba5a5 [liangbowen] clear local prop `kyuubi.session.user.public.key` and `kyuubi.session.user.sign` after execution
2c8b4bf6 [liangbowen] move throwing AccessControlException inside verifyKyuubiSessionUser method
82f5c265 [liangbowen] update conf doc
7cf0d481 [liangbowen] rename config name to `kyuubi.session.user.sign.enabled`
cc2c3570 [liangbowen] typo: fix KYUUBI_SESSION_USER_PUBIC_KEY
bdec509d [liangbowen] update settings.md for config doc
2d00163a [liangbowen] nit
102561a8 [liangbowen] update
af99ea84 [liangbowen] move algorithmSpec to ecKeyPairGenerator
966a327e [liangbowen] update
41064712 [liangbowen] update
9d276799 [liangbowen] add config `kyuubi.session.user.verify.enabled` to control Whether to verify the integrity of session user name in Spark Authz
8a8840f6 [liangbowen] nit
bafd85e9 [liangbowen] replace RSA with ECDSA
2f0c87a5 [liangbowen] KeyPairGenerator init with new SecureRandom instance
7cb31204 [liangbowen] shorten the key size to 1024
5011cf49 [liangbowen] remove unused imports
045fd822 [liangbowen] add to SparkOperation for scala
ce6d394d [liangbowen] move generateRSAKeyPair and signWithRSA to SignUtils.scala
5f295792 [liangbowen] update
1d7f3191 [liangbowen] initial support for signing and verifying `kyuubi.session.user`
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
1. `mvn antrun:runbuild-info -pl kyuubi-common`
2. debug engine suite
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3847 from cxzl25/profile_jdbc_debug.
Closes#3847
e1c32067 [Cheng Pan] Merge branch 'master' into profile_jdbc_debug
f41d3cd9 [sychen] use jdbc-shaded
25b54b18 [sychen] use jdbc-shaded
dd23f864 [sychen] use kyuubi-hive-jdbc as default
79b0ffcb [sychen] add jdbc profile
Lead-authored-by: sychen <sychen@ctrip.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Expose `kyuubi.operation.result.codec` to KyuubiConf
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3866 from cfmcgrady/arrow.
Closes#3794
2047f364 [Fu Chen] update
6f3fe24d [Fu Chen] rename
20748ca7 [Fu Chen] expose `kyuubi.operation.result.codec` to KyuubiConf
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close#3848 .
Sort the config map returned in KyuubiConf.getAll to improve readability of engine launching configs in output.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3849 from bowenliang123/sort-configs.
Closes#3848
75e60c89 [liangbowen] sort the returned map of getAll in KyuubiConf (+4 squashed commits) Squashed commits: [f102b423] update method comment [465d7b25] use TreeMap for sorting [1a66e05b] add getAllSorted [2b3dbf91] sort the returned map of getAll in KyuubiConf
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Submit pyspark batch job by restful api
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3836 from leoluan2009/pyspark-1.
Closes#3836
550021ac [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KyuubiApplicationManager.scala
357691c2 [Xuedong Luan] fix comment
7dfdbe24 [Xuedong Luan] fix comment
31bda178 [Xuedong Luan] [WIP] Support pyspark batch job by restful api
Lead-authored-by: Xuedong Luan <luanxuedong2009@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Forward the server ip in `openSession`, and in the Spark engine we get the server ip to start the engine from the Environment tab.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3844 from cxzl25/forward_server_ip.
Closes#3844
7b2cf95b [sychen] fix conf npe
59e7a2fc [sychen] mysql and rest
f50c36d6 [sychen] forward server ip
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Introduce code style check support for Maven's pom.xml with sortPom in spotless maven plugin.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3843 from bowenliang123/spotless-pom.
Closes#3842
3c654597 [liangbowen] apply to pom.xml
fd1536f7 [liangbowen] set expandEmptyElements to true
e498423f [liangbowen] apply spotless:apply to all pom.xml
e46bcfec [liangbowen] add pom style check support in spotless
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close#3805 .
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3832 from bowenliang123/skip-pyspark-tests.
Closes#3805
02a59bfb [liangbowen] add maven.plugin.scalatest.exclude.tags=org.scalatest.tags.PySparkTest to pom instead
444b6f4e [liangbowen] introduce @PySparkTest annotation and skip PySparkTests for nightly builds
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
* [KYUUBI #3814] Make the duration of sessionConfCache in FileSessionConfAdvisor configurable instead of a fixed 10 minutes
* fix ut
* fix ut
* fix doc
* code review
* update doc
* update
* update
### _Why are the changes needed?_
When a kyuubi service connections to multiple engine clusters, we can only explicitly specify relevant parameters after the URL, for example:
`beeline -u 'jdbc:hive2://127.0.0.1:10009/?kyuubi.engineEnv.FLINK_HOME=/opt/flink;kyuubi.ha.namespace=kyuubi-ns-c;kyuubi.engine.type=FLINK_SQL'`
This method is extremely unfriendly to users.
I implemented a SessionConfAdvisor implementation class `org.apache.kyuubi.session.FileSessionConfAdvisor` based on the conf file that to manage session level configurations, so that we can access this cluster only by specifying a unique engine name, for example:
access cluster-a:
`beeline -u 'jdbc:hive2://127.0.0.1:10009/?kyuubi.session.conf.profile=cluster-a'`
access cluster-c:
`beeline -u 'jdbc:hive2://127.0.0.1:10009/?kyuubi.session.conf.profile=cluster-c'`
`kyuubi-session-cluster-a.conf` configuration is as follows:
```
kyuubi.ha.namespace kyuubi-ns-a
kyuubi.engine.type SPARK_SQL
kyuubi.engine.pool.balance.policy POLLING
kyuubi.engineEnv.SPARK_HOME /opt/spark30
kyuubi.engineEnv.HADOOP_CONF_DIR /opt/hadoop_conf_dir
```
`kyuubi-session-cluster-c.conf` configuration is as follows:
```
kyuubi.ha.namespace kyuubi-ns-a
kyuubi.engine.type SPARK_SQL
kyuubi.engineEnv.SPARK_HOME /opt/spark32
kyuubi.engineEnv.HADOOP_CONF_DIR /opt/hadoop_conf_dir
```
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3742 from ychris78/yuqi_1101.
Closes#3742
e1449867 [yuqi] Style
68bbfecf [yuqi] restructure
1c8daa86 [yuqi] Change the cache expiration time to 10 minutes and remove the manual cache
5b4a6d77 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/session/FileSessionConfAdvisor.scala
d332e2d8 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/session/FileSessionConfAdvisor.scala
cb644ab4 [Cheng Pan] Update docs/deployment/settings.md
1ee3c980 [Cheng Pan] Update kyuubi-server/src/test/resources/kyuubi-session-cluster-a.conf
30f8fde0 [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
5ef5841a [yuqi] Cache upgrade
5c2f7461 [yuqi] add cache and fix bug
1b1f9d67 [yuqi] scalastyle
a0507847 [yuqi] License info amendment
335250fc [yuqi] fix bug
9e8d313a [yuqi] Modify according to suggestions
5f57b637 [yuqi] Modify according to suggestions
5e504252 [yuqi] add org.apache.kyuubi.plugin.FileSessionConfAdvisor When kyuubi.session.conf.advisor=org.apache.kyuubi.plugin.FileSessionConfAdvisor,Specify a session level configuration file, which will be combined with default.conf to have an impact. The corresponding configuration file is conf/kyuubi-session-<profile>.conf
0cb71341 [yuqi] add org.apache.kyuubi.plugin.YamlSessionConfAdvisor
b524c221 [yuqi] add org.apache.kyuubi.plugin.YamlSessionConfAdvisor
51729d15 [yuqi] When a kyuubi service connections to multiple engine clusters, we can only explicitly specify relevant parameters after the URL, for example: beeline -u 'jdbc:hive2://127.0.0.1:10009/?kyuubi.engineEnv.FLINK_HOME=/opt/flink;kyuubi.ha.namespace=kyuubi-ns-c;kyuubi.engine.type=FLINK_SQL' This method is extremely unfriendly to users, and it is extremely unsafe to expose namespaces to users. So I added a yaml file(engine-cluster-env.yaml) to manage the engine clusters, so that we can use this cluster only by specifying a unique engine name, for example: use cluster-a: beeline -u 'jdbc:hive2://127.0.0.1:10009/?kyuubi.engine.cluster.name=cluster-a'
Lead-authored-by: yuqi <1450636878@qq.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Support real user for kyuubi session
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3766 from turboFei/real_user.
Closes#3766
49c996ee [Fei Wang] refactor
894b4720 [Fei Wang] owner -> user
477119bf [Fei Wang] address
c36511ae [Fei Wang] add ut
8efc310c [Fei Wang] save
8ca2d589 [Fei Wang] add ut
89264d90 [Fei Wang] get real user
a431b0ad [Fei Wang] real user
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Now we support multiple frontend services.
We need transfer the connection url into session to identify the connection type.
It can also fix#3575
### _How was this patch tested?_
Pass existing UT.
Closes#3577 from turboFei/connection_url.
Closes#3577
f1591ccd [Fei Wang] refactor the order
70d21ca1 [Fei Wang] add more ut
5eaba4f2 [Fei Wang] comments
a78470c5 [Fei Wang] add ut
72107da3 [Fei Wang] SessionData
37965486 [Fei Wang] style
832922d2 [Fei Wang] remove
7739be65 [Fei Wang] save
5078ec6c [Fei Wang] save
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Expose the engine id when registering the service.
So that we can get more details about the engine when using `list engine` command.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3684 from turboFei/extra_info.
Closes#3684
afbcfdef3 [Fei Wang] refactor
513dd7234 [Fei Wang] split 2
ad3a6ac4f [Fei Wang] comments
cc51c5954 [Fei Wang] fix ut
32e31618a [Fei Wang] check engine id
202bbdb68 [Fei Wang] refactor to details
c5c153b8d [Fei Wang] render extra info
d2642cc8a [Fei Wang] save
1da67c258 [Fei Wang] dto
724ac3af4 [Fei Wang] save
c16c15988 [Fei Wang] expose engine id
e91a61e5a [Fei Wang] save
e38bb0b93 [Fei Wang] save
b1c158ba8 [Fei Wang] parse extra info
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Closing existing seekable reader when adding extra log
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3745 from turboFei/fix_op_log_bug.
Closes#3745
5faf9a99 [Fei Wang] save
3bf78f3e [Fei Wang] recreate seek reader
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
fix#3727
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3729 from jiaoqingbo/kyuubi3727.
Closes#3727
cb494c82 [jiaoqingbo] [KYUUBI #3727] Extract the common variables of GetTables in flink and spark to ResultSetSchemaConstant
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close#3701 .
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3702 from bowenliang123/3701-jdbc-auth.
Closes#3701
7a04d091 [Bowen Liang] skip not empty checking for kyuubi.authentication.jdbc.user config
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Current kyuubi build kubernetes client for `KyuubiApplicationOperator` to find or kill spark engine on kubernetes.
This pr aims to extract the code for building kubernetes client into `KubernetesUtils` and to do auto build following the logic in Spark.
We prefer to building with user-configured parameters; When missing, the kubernetes client logic will build one from environment variables and kubeconfig.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3663 from zwangsheng/feature/perfer_build_from_pod_env.
Closes#3663
0a4c8c79 [zwangsheng] clear properties
ce408db0 [zwangsheng] fix test
fb28ea7a [zwangsheng] add ci test
943c68dd [zwangsheng] add unit test
ddc9a2b2 [zwangsheng] fix scala
75380d13 [zwangsheng] comment
e6826609 [zwangsheng] master
683cc260 [zwangsheng] context provided master
7b42133c [zwangsheng] add doc
b56829c5 [zwangsheng] fix helm
5884475d [zwangsheng] rebuild
a7f0b551 [zwangsheng] ad
e7f9ec05 [zwangsheng] add setting
b77bbd9b [zwangsheng] add doc
3f3253d3 [zwangsheng] init
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Support SSL for kyuubi thrift binary conenction.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
Will add screenshot for IT.
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3658 from turboFei/ssl_binary.
Closes#3658
ffd603ec [Fei Wang] comment
ce652b7e [Fei Wang] revert UT
52e24abb [Fei Wang] ut
f6cae357 [Fei Wang] save
c6b664f1 [Fei Wang] exclude passwords
d34a4ca2 [Fei Wang] comments
11bfa9ad [Fei Wang] refactor
e2ca77ea [Fei Wang] align
86aaf87c [Fei Wang] with alternative
d17e0d11 [Fei Wang] style
eb4ba52a [Fei Wang] Support SSL for Kyuubi thrift binary connection
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
As described in #2887, random policy may cause task-hot-issues or production accident, so a POLLING balance policy is added to avoid this problem, we can "set kyuubi.engine.pool.balance.policy = POLLING" to use it.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3662 from ychris78/epbp_1018.
Closes#2887
90834c1f [yuqi] fix bug
c0ce8a7e [yuqi] Fix typos
bec4ca13 [yuqi] Fix typos
dc4e3637 [yuqi] please add test case to cover the added methods
837c920b [yuqi] rename method getAndInc to getAndIncrement and modify the doc
02a8311f [yuqi] add getAndInc method and refer it
9436937d [yuqi] Improve configuration documentation
21b5d7d1 [yuqi] [KYUUBI #2887] Add a polling balance policy for spark engine pool Improve configuration documentation and delete useless comments
f0699b3c [yuqi] [KYUUBI #2887] Add a polling balance policy for spark engine pool
Authored-by: yuqi <1450636878@qq.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
to close#3671.
- fix assert message for '[SCHEMA_NOT_FOUND]' and '[TABLE_OR_VIEW_NOT_FOUND]'
- introduce SparkVersionUtil in kyuubi-common test for checking Spark version
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3676 from bowenliang123/3671-notfound.
Closes#3671
c505098f [Bowen Liang] update
a950abbf [Bowen Liang] fix typo
56464e84 [Bowen Liang] update SparkVersionUtil
2190a93e [Bowen Liang] fix assert message for '[SCHEMA_NOT_FOUND]' and '[TABLE_OR_VIEW_NOT_FOUND]', introduce SparkVersionUtil for checking spark version in kyuubi-common test
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
to close#3615
steps to reproduce
1. start kyuubi server
2. connect to the kyuubi server with client beeline and run query `select 1`
3. force kill the spark application
4. connect to the kyuubi server with another client **A** immediately
```
Error: org.apache.kyuubi.KyuubiSQLException: Error operating LaunchEngine: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at org.apache.kyuubi.client.KyuubiSyncThriftClient$.createTProtocol(KyuubiSyncThriftClient.scala:466)
at org.apache.kyuubi.client.KyuubiSyncThriftClient$.createClient(KyuubiSyncThriftClient.scala:482)
at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:128)
at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1$adapted(KyuubiSessionImpl.scala:113)
at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36)
at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:113)
at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$2(LaunchEngine.scala:60)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.aache.thrift.transport.TSocket.open(TSocket.java:221)
... 14 more (state=,code=0)
```
If the engine exits without calling the shutdown hook, the ZNODE with EPHEMERAL_SEQUENTIAL mode will be deleted upon the client's disconnect, and note that the engine will be marked as disconnected after `kyuubi.zookeeper.embedded.max.session.timeout` (the default value is 1 minutes)
This PR retries opening the engine when encountering a specialty exception (java.net.ConnectException). Before this PR, client **A** will throw an exception, after this PR, client **A** will work properly.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3618 from cfmcgrady/kyuubi-3615.
Closes#3615
96acd66b [Fu Chen] address comment
a1395a60 [Fu Chen] fix style
870216d1 [Fu Chen] Retry opening the engine when encountering a special error
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
fix#3640
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3641 from jiaoqingbo/kyuubi3640.
Closes#3640
c1b2d4a6 [jiaoqingbo] code review
a65bc132 [jiaoqingbo] code review
ab04f135 [jiaoqingbo] [KYUUBI #3640] Change the default event logger description of hive and trino to json instead of spark
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close#3635 .
- change test suits to to use Utils.createTempDir of kyuubi-common module to create temp dir with delete on exit to prevent disk usage leaking
- change signature Utils.createTempDir by making `prefix` param from second place to the first, to make it friendly to use without repeating param name itself. The name`prefix` is more closer to Java's style in `Files.createTempDirectory`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3660 from bowenliang123/3635-deletetmp.
Closes#3635
8947fb0f [Bowen Liang] remove unused imports
c06769d4 [Bowen Liang] nit
dac266c9 [Bowen Liang] - change test suits to to use Utils.createTempDir of kyuubi-common module to create temp dir with delete on exit to prevent disk usage leaking - change signature Utils.createTempDir by making prefix param from second place to the first, to make it friendly to use without repeating param name itself. The nameprefix is more closer to Java's style in Files.createTempDirectory
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Refer Spark PR: https://github.com/apache/spark/pull/18110
ConfigBuilder builds ConfigEntry which can only read value with one key, if we wanna change the config name but still keep the old one, it's hard to do.
This PR introduce ConfigBuilder.withAlternative, to support reading config value with alternative keys.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3659 from turboFei/conf_alternative.
Closes#3659
e268fef2 [Fei Wang] add ut
a2300b26 [Fei Wang] add ut
53eccf99 [Fei Wang] Support alternative keys in ConfigBuilder
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
to close#3635 .
- change test suits to to use Utils.createTempDir of kyuubi-common module to create temp dir with delete on exit to prevent disk usage leaking
- change signature Utils.createTempDir by making `prefix` param from second place to the first, to make it friendly to use without repeating param name itself. The name`prefix` is more closer to Java's style in `Files.createTempDirectory`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3636 from bowenliang123/3635-tempdir.
Closes#3635
5f84a16f [Bowen Liang] nit
b82a149f [Bowen Liang] rename `namePrefix` param to `prefix` of `Utils.createTempDir` method, and make it in first place
76d33143 [Bowen Liang] delete files on exit in test suits with Utils.createTempDir
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
When Kyuubi runs outside of K8s, and w/o enhanced DNS infrastructure, Kyuubi can not access the Pod by using the hostname of Pod, it blocks the user to run Spark on K8s w/ cluster mode out-of-box.
Kyuubi provided a configuration `kyuubi.frontend.connection.url.use.hostname`, turn it off could address this issue, but we can not change the default value globally because of https://github.com/apache/incubator-kyuubi/issues/2266
To improve user experience, we can detect if the Driver is running inside the Pod, and if yes,
change `kyuubi.frontend.connection.url.use.hostname` default value to `false`.
Close#3578
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3597 from pan3793/k8s.
Closes#3597
8b411781 [Cheng Pan] doc
0df15e79 [Cheng Pan] Engine should prefer to use ip for registing on K8s cluster mode
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3512
Currently, Kyuubi EventBus uses `scala.concurrent.ExecutionContext.Implicits.global` to execute async event handler. Generally, it's discouraged to use that global ec, instead, we should create dedicated ec for each workload, this pr aims to use dedicated ExecutionContext for EventBus async execution
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3524 from Yikf/ec.
Closes#3512
9757372d [yikf] Use dedicated ExecutionContext for EventBus async execution
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3510
Reading an uninitialized log should return empty rowSet , So we can set EmptyRowSet for TFetchResultsResp to avoid client side NPE if FetchResults fail on server side.
For example: Configure log4j for `ERROR` level, causing the engine side no log file output, so server side throw `java.nio.file. NoSuchFileException`, at this time the client get engine log, throws the NPE.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3511 from Yikf/issue-3510.
Closes#3510
3fee6591 [yikf] fix
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3490
Currently, when Kyuubi get or set conf by `ConfigEntry`, it does not check whether `ConfigEntry` have been registered, so this pr aims to two points as follow:
- Require ConfigEntry has been registered;
- Using copy-on-write for Kyuubi#kyuubiConfEntries to reduce contention in concurrent workloads.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3491 from Yikf/copy-on-write-conf.
Closes#3490
6d0d5ed2 [yikf] Use copy-on-write semantics for KyuubiConf registered configurations
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close https://github.com/apache/incubator-kyuubi/issues/3449, this pr aims to change `kyuubi.server.info.provider` default value to "ENGINE", and modify corresponding tests.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3476 from Yikf/default-server-info.
Closes#3449
bc474d3b [Cheng Pan] Update docs/deployment/migration-guide.md
23e7690e [yikf] Change default server info provider to ENGINE
Lead-authored-by: yikf <yikaifei1@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The default value of kyuubi.engine.event.loggers is SPARK, but it can't be used outside the spark engine, so we add the config for each engines.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3309 from zhaomin1423/add_engine_logger_conf.
Closes#3309
37b3c05f [Xiao Zhao] fix
b22356f2 [Min] fix typo
431bb02b [Min] update settings.md
44475f46 [Min] Add configs of engine event logger for each engine
Lead-authored-by: Min <zhaomin1423@163.com>
Co-authored-by: Xiao Zhao <zhaomin1423@163.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Workaround for #3032, close#3323
There are known some ODBC drivers e.g. [Databricks ODBC driver](https://www.databricks.com/spark/odbc-drivers-download) depending on `TGetInfoType.CLI_DBMS_VER` and `TGetInfoType.CLI_DBMS_NAME` to check server compatibilities and abilities.
This PR proposes to introduce a new configuration `kyuubi.server.info.provider=SERVER/ENGINE` to make GetInfo support return either server or engine information.
Since beeline will call GetInfo in the initialization phase, to make sure the beeline fast open experience, in async launch mode, when the engine is not ready, return server info regardless `kyuubi.server.info.provider`.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
Testing w/ PowerBI
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3122 from pan3793/info.
Closes#3122
742bdbe3 [Cheng Pan] nit
bb85d2ba [Cheng Pan] style
fd75238c [Cheng Pan] fix
9ddb2afd [Cheng Pan] nit
fd8f7979 [Cheng Pan] fix ut
840205e5 [Cheng Pan] nit
f9996d53 [Cheng Pan] GetInfo supports returning engine info
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Close#3064
Refer the comments by cxzl25
Spark uses the code
`URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory(hadoopConf))` supports the class path from HDFS.
But because the scala compiler only supports adding file schema urls to the class path, non-file schema urls will cause NPE.
```java
Error: Error operating ExecuteScala: java.lang.NullPointerException
at scala.tools.nsc.classpath.FileUtils$AbstractFileOps$.isJarOrZip$extension(FileUtils.scala:32)
at scala.tools.nsc.classpath.ClassPathFactory$.newClassPath(ClassPathFactory.scala:90)
at scala.tools.nsc.Global.$anonfun$extendCompilerClassPath$1(Global.scala:832)
```
scala.tools.nsc.Global#extendCompilerClassPath
```scala
AbstractFile.getURL(u)
```
scala.reflect.io.AbstractFile#getURL
```scala
def getURL(url: URL): AbstractFile =
if (url.getProtocol == "file") {
val f = new java.io.File(url.toURI)
if (f.isDirectory) getDirectory(f)
else getFile(f)
} else null
```
spark-shell supports --jars hdfs jar. At this time, submit will download the remote jar to the local and pass it to spark-shell through the `spark.repl.local.jars` configuration.
In this pr, I localize the remote jar url at first, and then add it into repl class path.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3292 from turboFei/scala_npe.
Closes#3064
095c40d8 [Fei Wang] filter
37637f76 [Fei Wang] add non-exist
ec87ea85 [Fei Wang] add ut
5a823d3b [Fei Wang] save
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3414
Currently, Kyuubi supports output of planOnly in different modes in different styles, As more and more modes and styles are supported, defining enumerations in kyuubiConf is a bit messy and not easy to extend, so this pr aims to tidy up these
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3415 from Yikf/tidy-up.
Closes#3414
e81142e5 [yikf] Tidy up plan only related modes and styles
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
close https://github.com/apache/incubator-kyuubi/issues/3311
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3324 from lsm1/fix/operation_language_incorrect.
Closes#3311
9192f61f [senmiaoliu] revert delete PlanOnlyOperationSuite.scala
9d0079b1 [senmiaoliu] move ut in SparkQueryTests
e8b160da [senmiaoliu] add unknown for operation language
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3376
This pr aims support planOnly output in different styles (Users can perform SQL scheduling based on the JSON description of TreeNode)
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3377 from Yikf/planOnly-support-json.
Closes#3376
b63ae718 [yikf] Support json_description for PlanOnlyStatement
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Support to get engine id, name, url in KyuubiConnection.
It is helpful and friendly for BI.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3354 from turboFei/lanch_engine_result.
Closes#3354
4966de28 [Fei Wang] comments
984b5d35 [Fei Wang] fix ut
963217af [Fei Wang] filter null
a2c0ce2c [Fei Wang] fix
4f2a8b18 [Fei Wang] get from engine side
61607c7c [Fei Wang] return
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3357
This pr aims to improve doc description for configuration kyuubi.operation.plan.only.mode
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3358 from Yikf/planOnly-doc.
Closes#3357
571f4e0c [yikf] Improve doc description for configuration `kyuubi.operation.plan.only.mode`
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Now the connection path is:
`client`-> `kyuubiServer` -> `kyuubiEngine`
Maybe the connection between `client -> kyuubiServer` is valid, but the connection between `kyuubiServer -> kyuubiEngine` is not.
So we need check the whole path.
BTW, before, if customer invoke KyuubiConnection::isValid, it can make the connection between client and kyuubiServer keep alive.
Now, it can make the whole connection path keep alive.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3318 from turboFei/is_valid.
Closes#3318
af4d6b7b [Fei Wang] Transfer the TGetInfoReq to kyuubi engine side to keep the kyuubi connection alive
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
close https://github.com/apache/incubator-kyuubi/issues/3128
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3304 from Kwafoor/master.
Closes#3128
88a5ebe9 [wangjunbo] [kyuubi-3128] Support CostMode for PlanOnlyStatement
Authored-by: wangjunbo <wangjunbo@qiyi.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
fix https://github.com/apache/incubator-kyuubi/issues/3298
Curently, a serval snippets use `getClass.getClassLoader` to get the classLoader in Apache Kyuubi, this pr aims to unify this behavior in `Utils.getContextOrKyuubiClassLoader`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3299 from yikf/unify-calssloader.
Closes#3298
ae081ad9 [yikf] Unify the approach of get classLoader in Kyuubi
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
To fix the config name and placeholder with `username` introduced in https://github.com/apache/incubator-kyuubi/pull/3235 violate this convention as in JDBC driver use `user` keyword used for connection user rather than `username`,
1. change config name from `kyuubi.authentication.jdbc.username` to `kyuubi.authentication.jdbc.user`
2. change placeholder from `${username}` to `${user}`
3. update docs and config description related to above changes, and sync the update in jdbc auth docs statement details to config docs.
4. fix error in throwing AuthenticationException with auth db password. ut added for the fix.
5. other minor update in docs of custom auth
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3288 from bowenliang123/jdbc-auth-config-update.
Closes#3222
502703fb [Bowen Liang] skip map for placeholder value lookup
3733be41 [liangbowen] nit
ab00525b [liangbowen] nit
2301c4f0 [liangbowen] fix ut of jdbc auth with wrong_password
06f0c1bb [liangbowen] remove redundant docs
ec4565b3 [liangbowen] remove redundant docs
ae1cce2e [liangbowen] fix compilation error of configLog
5d14103b [liangbowen] simplify configLog
6678e657 [liangbowen] reformat
52c1038e [liangbowen] simplify placeholder checking
21c2d5ea [liangbowen] check whether placeholders in supported list before conn establishment or authenticate
7db0adf5 [liangbowen] ut for unknown placeholder
657de6af [liangbowen] nit
736b3f22 [liangbowen] refactoring placeholder value lookup, for preventing setString multiple times with "i+1"
86c89125 [liangbowen] setMaxRows after prepare placeholder, to postpone operation on jdbc conn
115fae50 [liangbowen] increase test code coverage
b45b28cb [liangbowen] resultSet returned by executeQuery is never null
e1c07274 [liangbowen] update ut for redactPassword in JdbcUtils
b4a52e29 [liangbowen] fix typo in docs of custom auth
371c2c6e [liangbowen] move redactPassword method to JdbcUtils and add ut.
a4973c59 [liangbowen] reformat code
486e1503 [liangbowen] fix error in throwing AuthenticationException with auth db password. add ut for the fix.
efced90a [liangbowen] update settings.md
ef97e354 [liangbowen] add SELECT prefix hint for doc of kyuubi.authentication.jdbc.query
025f94c7 [liangbowen] fix username to user in JdbcAuthenticationProviderImpl by 1. use config name `kyuubi.authentication.jdbc.user`, 2. use ${user} placeholder instead of ${username}
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This is the followup of #3235, the main change is introdude `JdbcUtils` to simplify code, and allow empty password for Jdbc auth.
Jdbc connection pool has been removed because `JdbcAuthenticationProviderImpl` will be created on each connection, we can improve to use singleton in the future
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3278 from pan3793/jdbc-followup.
Closes#3222
2863cae7 [Cheng Pan] Update kyuubi-common/src/test/scala/org/apache/kyuubi/service/authentication/JdbcAuthenticationProviderImplSuite.scala
51a9c458 [Cheng Pan] Update kyuubi-common/src/test/scala/org/apache/kyuubi/service/authentication/JdbcAuthenticationProviderImplSuite.scala
eee3c55e [Cheng Pan] Update kyuubi-common/src/test/scala/org/apache/kyuubi/util/JdbcUtilsSuite.scala
d02bb999 [Cheng Pan] nit
e001b5b5 [Cheng Pan] nit
8cf5cd67 [Cheng Pan] nit
032f2df8 [Cheng Pan] nit
8a42f184 [Cheng Pan] nit
c7893fdf [Cheng Pan] JdbcUtilsSuite
f97f2d90 [Cheng Pan] remove pool
a8812d0f [Cheng Pan] move render result set to test
83d7d4cf [Cheng Pan] fix ut
db787a4b [Cheng Pan] nit
864f9ddc [Cheng Pan] nit
b60decf5 [Cheng Pan] nit
8c66e0be [Cheng Pan] nit
2063c438 [Cheng Pan] [KYUUBI #3222][FOLLOWUP] Introdude JdbcUtils to simplify code
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Followup of #3020
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3271 from turboFei/refine_ldap.
Closes#3020
caa35d84 [Fei Wang] comment
65597cb4 [Fei Wang] refine
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
fix https://github.com/apache/incubator-kyuubi/issues/3269
Add docs for JDBC auth provider from https://github.com/apache/incubator-kyuubi/issues/3222 shipping within 1.6.0, add jdbc.rst in /docs/security path
simplify the config docs for kyuubi.authentication.jdbc.query in /docs/deployment/settings.md by moving examples to the docs of jdbc auth provider.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3275 from bowenliang123/doc-jdbc-auth.
Closes#3275
e393f6db [liangbowen] fix config example mistakes
ab70cc8d [liangbowen] update doc for auth query sql
ebd5e194 [liangbowen] update doc description
0b15af59 [liangbowen] typo
953000fe [liangbowen] typo
ab944e73 [liangbowen] typo
e3c3d259 [liangbowen] regenerate conf docs. udpate jdbc.md.
45429ada [liangbowen] typo
3b05338e [liangbowen] add docs for jdbc authentication provider
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Among the kyuubi ldap configuration parameters, there is currently "kyuubi.authentication.ldap.base.dn=", but there is no place corresponding to the CN password setting. similar to hive
"hive.server2.authentication.ldap.binddn",
"hive.server2.authentication.ldap.bindpw".
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3213 from dev-lpq/branch-ldap-1.6.
Closes#3020
31817cc6 [pengqli] upgrade ldap to support ldap before 1.6.0
31094030 [pengqli] upgrade ldap to support ldap before 1.6.0
f62e4c96 [pengqli] upgrade ldap to support ldap before 1.6.0
e03e036e [pengqli] add AUTHENTICATION_LDAP_GUIDKEY deprecated
6f680dfc [pengqli] add ldap config AUTHENTICATION_LDAP_BINDDN
19f8e6ce [pengqli] add ldap config AUTHENTICATION_LDAP_BINDDN
07df56b8 [pengqli] upgrade LdapAuthenticationProviderImpl
1a48f418 [pengqli] upgrade settings.md
8e9b5721 [pengqli] upgrade ldap ldapBindpw
5793a811 [pengqli] upgrade ldap ldapBindpw
822d4c30 [pengqli] upgrade docs/deployment/settings.md
c2b20418 [pengqli] upgrade ldap WithLdapServer UTs
5d7a301c [pengqli] add AUTHENTICATION_LDAP_PASSWORD into serverOnlyConfEntries
10b42b1b [pengqli] upgrade AuthenticationProviderFactorySuite UTs
490bfae8 [pengqli] kyuubi ldap add new config property kyuubi.authentication.ldap.bindnpw and kyuubi.authentication.ldap.attrs
5e071252 [pengqli] upgrafe config kyuubi.authentication.ldap.attrs
4e771ec6 [pengqli] kyuubi ldap add new config property kyuubi.authentication.ldap.bindnpw and kyuubi.authentication.ldap.attrs
Authored-by: pengqli <pengqli@cisco.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Preparing v1.7.0-SNAPSHOT with branch-1.6 cut
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3264 from SteNicholas/prepare-1.7.0-snapshot.
Closes#3264
374d56bf [SteNicholas] preparing v1.7.0-SNAPSHOT with branch-1.6 cut
Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
master branch has supported 5 engine types. the docs of `kyuubi.engine.type` missed hive and jdbc.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3255 from ulysses-you/engine-type.
Closes#3255
a1f3c320 [ulysses-you] add config
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
close#3214
### manual tests
when we set `kyuubi.operation.plan.only.mode` an incorrect value,unset plan only mode
so we can set a correct value
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3216 from lsm1/fix/plan_only_mode.
Closes#3214
bb159ec6 [senmiaoliu] fix ut
f23dea1a [LSM] Merge branch 'apache:master' into fix/plan_only_mode
5204745a [senmiaoliu] implement in OperationModes
65898f40 [senmiaoliu] fix style, added ut
a1df52f8 [senmiaoliu] unset kyuubi.operation.plan.only.mode if mode is invalid
Lead-authored-by: senmiaoliu <senmiaoliu@trip.com>
Co-authored-by: LSM <senmiaoliu@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add JDBC authentication provider as implementation of PasswdAuthenticationProvider out of box.
Compared to currently support authentication methods like Kerberos and LDAP, JDBC source is a much easy and stable source for practical deployment.
The solution should provide:
easy to use and config jdbc connection details
handy to customize query statements for authentication
Adding JDBC to AuthMethods enum and JDBCPasswdAuthenticationProvider into package org.apache.kyuubi.service.authentication with listed features:
- specify JDBC driver name and load the driver class
- configs of JDBC url, username and password
- select query with placeholders for checking user and password , like `SELECT 1 from user_pass_hash where username = ${user} and password = MD5(${password})`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3235 from bowenliang123/feature-jdbc-auth-provider.
Closes#3222
17403b33 [liangbowen] cleanup docs
d5f43e0e [liangbowen] remove unuseful logs for unrecognized placeholder error
e9af0966 [liangbowen] use clone instead of repeatly generating configs
6fc42bf3 [liangbowen] code styling
a9404fa3 [liangbowen] use {} for intercept
77f5f86b [liangbowen] remove unuseful comment
6765affb [liangbowen] changed to use in-memory derby db for test
543c66cb [liangbowen] prefer scala style string usage
a4fe582f [liangbowen] refactor connection creation on using HikariDataSource in HikariCP. add HikariCP dependencies to kyuubi-common
3a4d5fe9 [liangbowen] update checkConfigs() signature
5a0ac495 [liangbowen] output password length only in checkConfigs
1c956dfb [liangbowen] update KyuubiAuthenticationFactorySuite
4ebe12e2 [liangbowen] add JDBC value to AuthTypes enum
9885f813 [liangbowen] add JDBC condition for getValidPasswordAuthMethod
b9ffac3c [liangbowen] Merge branch 'master' into feature-jdbc-auth-provider
aeb19ce5 [liangbowen] update doc
653bc126 [liangbowen] add more checks for query sql
cdec2066 [liangbowen] more test cases
36729197 [liangbowen] fix ddl statement and remove truncate statement in test
30974d16 [liangbowen] update format
575301ca [liangbowen] update options usage
1dc4187b [liangbowen] update settings.md config doc
cd2c7c2b [liangbowen] update settings.md config doc
15176b2b [liangbowen] fix import orders
46cc1dda [liangbowen] add config docs in docs/deployment/settings.md
7025330b [liangbowen] fix derby startup error in test
df4be568 [liangbowen] update code style
49c18c25 [liangbowen] update
996f7969 [liangbowen] add unit test in JdbcAuthenticationProviderImplSuite
0e7f0ad0 [liangbowen] refactor config and init process.remove unused import.
0dc75fee [liangbowen] implement JDBC Authentication Method
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
If we use scala enumeration, we need to use logging, and an error will be reported in an environment where log4j2 is not packaged.
This PR just moved the `initializeLogging` method of trait Logging to object Logging.
```scala
XX extends Enumeration with Logging
```
```
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetPublicMethods(Class.java:2902)
at java.lang.Class.privateGetPublicMethods(Class.java:2911)
at java.lang.Class.getMethods(Class.java:1615)
at scala.Enumeration.populateNameMap(Enumeration.scala:193)
at scala.Enumeration.$anonfun$nameOf$1(Enumeration.scala:212)
Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.Level
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
```
Scala enumeration uses reflection, resulting in class not found in some scenarios.
scala.Enumeration#populateNameMap
685017051a/src/library/scala/Enumeration.scala (L188-L193)
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3243 from cxzl25/logging_trait_object.
Closes#3243
330ad95f [sychen] move initializeLogging
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
To make sure flink sql engine build against flink-1.15 can run on flink-1.14.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3230 from pan3793/flink.
Closes#3230
5c3f859c [Cheng Pan] updated
e776ee63 [Cheng Pan] review
a5cfb0c8 [Cheng Pan] review
004c05d9 [Cheng Pan] nit
9c019554 [Cheng Pan] nit
7c3f553c [Cheng Pan] nit
8e1fb04b [Cheng Pan] nit
6adc7dbe [Cheng Pan] nit
bc10739c [Cheng Pan] fix
cf98111a [Cheng Pan] workflow
a8895e68 [Cheng Pan] nit
8cab5188 [Cheng Pan] Flink SQL engine supports run across versions
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3229 from lightning-L/kyuubi-3170.
Closes#3170
ccb14571 [Tianlin Liao] [KYUUBI #3170] Expose thrift binary connection metrics
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Spark SQL Engine failed to get issue date from some DelegationTokenIdentifiers, such as OzoneTokenIdentifier.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3225 from zhouyifan279/3219.
Closes#3219
0ffdcf98 [zhouyifan279] [KYUUBI #3219] Error renew delegation tokens: Unknown version of delegation token 8
c5a37a0c [zhouyifan279] [KYUUBI #3219] Error renew delegation tokens: Unknown version of delegation token 8
594c5a0f [zhouyifan279] [KYUUBI #3219] Error renew delegation tokens: Unknown version of delegation token 8
cbbafd51 [zhouyifan279] [KYUUBI #3219] Error renew delegation tokens: Unknown version of delegation token 8
394308e6 [zhouyifan279] [KYUUBI #3219] Error renew delegation tokens: Unknown version of delegation token 8
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
fix#3204
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3205 from jiaoqingbo/kyuubi-3204.
Closes#3204
185dd427 [jiaoqingbo] [KYUUBI #3204]In LdapAuthenticationProviderImplSuite afterAll() method，ldapServer.close() should be called only once
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add component version util class
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3181 from deadwind4/engine-version-util.
Closes#3180
12dcfbed [Luning Wang] Delete method in companion object
222a4ccc [Luning Wang] Change name to SemanticVersion
d254c0e5 [Luning Wang] Use case class and add method in companion class
f943ebd0 [Luning Wang] Add test case
e267a1fd [Luning Wang] Copy ComponentVersion class to authz module
343a2ded [Luning Wang] [KYUUBI #3180] Add component version util
Authored-by: Luning Wang <wang4luning@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
1. add a warning message when the user sets the deprecated configuration
```
10:49:58.558 WARN org.apache.kyuubi.config.KyuubiConf: The Kyuubi config 'kyuubi.frontend.bind.port' has been deprecated in Kyuubi v1.4.0 and may be removed in the future. Use kyuubi.frontend.thrift.binary.bind.port instead
```
2. make the annotation `deprecated` arguments only contain constants to fix the error when compiling for scala-2.13
```
[ERROR] /home/chenfu/workspace/incubator-kyuubi/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala:353: annotation argument needs to be a constant; found: "using ".+(KyuubiConf.this.FRONTEND_THRIFT_BINARY_BIND_PORT.key).+(" instead")
```
### _How was this patch tested?_
Pass CI.
Closes#3195 from cfmcgrady/log-deprecated-config.
Closes#3194
2c7567fc [Fu Chen] revert log4j2-test.xml
b97306d5 [Fu Chen] deprecated config
264d1276 [Fu Chen] log deprecated config
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The current JDBC Kerberos authentication and Hadoop UGI classes are strongly coupled, it's not friendly for downstream projects which do not have Hadoop dependencies, and users who are not familiar w/ Hadoop security mechanism.
This PR proposes to replace the UGI-based Kerberos authentication with JAAS-based.
The main logic is
```
if (param.auth == 'noSasl') {
// no-sasl
raw socket/http requrest
} else if (!param.principal) {
// sasl/plain
auth by param.username, param.password
} else if (param.principal && param.kerberosAuthType == 'fromSubject') {
[0]
} else if (param.principal && param.kyuubiClientPrincipal && param.kyuubiClientKeytab) {
[1]
} else if (param.principal && !param.kyuubiClientPrincipal && !param.kyuubiClientKeytab) {
[3]
}
- [0][JDK] subject from context
- [1][JDK] principle + keytab
- [2][UGI] principle + keytab (no required, use [1] instead)
- [3][JDK] principle + tgt cache
- [4][UGI] principle + tgt cache (no required, use [3] instead)
```
To achieve this, we need to introduce new JDBC parameters `kyuubiClientPrincipal` and `kyuubiClientKeytab`.
The above description is pure JDK-based and totally removed Hadoop dependencies from Kyuubi Hive JDBC driver.
To minimize the scope of this PR, I separate the support of delegation token to a follow-up PR #3096, and it still needs UGI.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3023 from pan3793/ugi.
Closes#3023
7b45f471 [Cheng Pan] Remove UGI in JDBC driver and use JAAS-based Kerberos authentication instead
3876c3d6 [Cheng Pan] Add util method isCommandAvailable
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
To fix the NoClassDefFoundError
```
Error: org.apache.kyuubi.KyuubiSQLException: org.apache.kyuubi.KyuubiSQLException: Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/bridge/SLF4JBridgeHandler
at org.apache.kyuubi.Logging.initializeLogging(Logging.scala:97)
at org.apache.kyuubi.Logging.initializeLoggerIfNecessary(Logging.scala:89)
at org.apache.kyuubi.Logging.initializeLoggerIfNecessary$(Logging.scala:85)
at org.apache.kyuubi.engine.flink.FlinkSQLEngine$.initializeLoggerIfNecessary(FlinkSQLEngine.scala:62)
at org.apache.kyuubi.Logging.logger(Logging.scala:43)
at org.apache.kyuubi.Logging.logger$(Logging.scala:41)
at org.apache.kyuubi.engine.flink.FlinkSQLEngine$.logger(FlinkSQLEngine.scala:62)
at org.apache.kyuubi.engine.flink.FlinkSQLEngine$.main(FlinkSQLEngine.scala:72)
at org.apache.kyuubi.engine.flink.FlinkSQLEngine.main(FlinkSQLEngine.scala)
Caused by: java.lang.ClassNotFoundException: org.slf4j.bridge.SLF4JBridgeHandler
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
See more: /Users/chengpan/app/kyuubi/work/anonymous/kyuubi-flink-sql-engine.log.0
at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69)
at org.apache.kyuubi.engine.ProcBuilder.$anonfun$start$1(ProcBuilder.scala:219)
at java.lang.Thread.run(Thread.java:750)
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3143 from pan3793/bridge.
Closes#3143
266c9bd1 [Cheng Pan] Check class loadable before applying SLF4JBridgeHandler
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
https://github.com/apache/iceberg/releases/tag/apache-iceberg-0.14.0
Notable changes:
- Support Spark 3.3
- Unified package name
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3050 from pan3793/iceberg.
Closes#3050
3e700a11 [Cheng Pan] Fix ut caused by SPARK-37931
039c4e2d [Cheng Pan] Enable Iceberg test w/ Spark 3.3
53cac396 [Cheng Pan] Bump Apache Iceberg 0.14.0
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Because the client can specify any ip in http header, to prevent CVE issue, we do not use it for authentication.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3078 from turboFei/http_real_ip_cve.
Closes#3052
e7c41ea9 [Fei Wang] prevent cve
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Close#3051
We deploy kyuubi server on k8s and using k8s lb and the connection path is:
client -> lb -> kyuubi server.
Now the client ip address is always the lb ip and is meaning less.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3048 from turboFei/session_ip.
Closes#3051
97f9b004 [Fei Wang] fix
524ddbb9 [Fei Wang] comments
ebf02113 [Fei Wang] comments
501d2177 [Fei Wang] comments
25da7e5e [Fei Wang] comments
f634321d [Fei Wang] remove unused new line
faed580a [Fei Wang] comments
403aafc6 [Fei Wang] reserve the kyuubi session ip to engine side
95bd4ec9 [Fei Wang] remove static
6c09fa80 [Fei Wang] refactor
20d269bd [Fei Wang] only in server side
dba85f36 [Fei Wang] get from session ip
591e7fb0 [Fei Wang] add key
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
To close#3052
If your server is behind a load balancer or other proxy, the server will see this load balancer or proxy IP address as the client IP address, to get around this common issue, most load balancers or proxies offer the ability to record the real remote IP address in an HTTP herader that will be added to the request for other devices to use.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3053 from turboFei/http_real_ip.
Closes#3052
b375d02a [Fei Wang] add ut
23ef1c9c [Fei Wang] refactor
907c61c0 [Fei Wang] also take affect for http
a958c50f [Fei Wang] add conf
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Handles configuring the JUL -> SLF4J bridge, otherwise, the components which use JUL like Jersey will be not controlled by log4j2 configurations
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3037 from pan3793/jul.
Closes#3037
fda05ff6 [Cheng Pan] update license
31f579fd [Cheng Pan] nit
7da0f9fb [Cheng Pan] Fix dependencyList
3179e6bd [Cheng Pan] Handles configuring the JUL -> SLF4J bridge
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
After #2948 and #2953, the OPERATION_THRIFT_CLIENT_REQUEST_MAX_ATTEMPTS is not used.
In this pr, I remove the attempts config entry and related ut that based on the request attempts.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3012 from turboFei/follow_up_engine_alive.
Closes#3012
3d15e44a [Fei Wang] remove deprecated docs
89c9f2cb [Fei Wang] remove docs
edb6ff2e [Fei Wang] remove ut
696bd853 [Fei Wang] saev
f91c61bf [Fei Wang] remove unused attempt
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2896 from hddong/trino-result-info.
Closes#2895
757019cf [hongdongdong] fix scala style
f40b1c53 [hongdongdong] Fix
3edbdfb3 [hongdongdong] [KYUUBI #2895] Show final info in trino engine
Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
### _Why are the changes needed?_
To close#2801
Customer might set some parameters to upload the local files, such as spark.files=/etc/security/keytabs/kyuubi.keytab. It is not security for kyuubi server.
In this pr, I introduce a local dir allow list and customer can only upload the files under these dirs.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2991 from turboFei/local_white_list.
Closes#2801
6315067d [Fei Wang] comments
deb9fa8a [Fei Wang] remove unused
3330fa98 [Fei Wang] refactor
6eca2f8b [Fei Wang] move to KyuubiApplicationManager and add ut
1290d9e2 [Fei Wang] add more
f14ec7d4 [Fei Wang] add more path configs and add comment
82357183 [Fei Wang] refactor
f201f577 [Fei Wang] disable relative path
f806e69a [Fei Wang] refactor
4f2faa6a [Fei Wang] save
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Remove `conf/log4j2.properties.template` as it has been replaced by `conf/log4j2.xml.template`
Remove `<ThresholdFilter level="FATAL"/>` in default stdout appender.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3000 from pan3793/log4j-followup.
Closes#2850
ccb9dd6e [Cheng Pan] [KYUUBI #2850][FOLLOWUP] Fix default log4j2 configuration
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix typo in KyuubiConf and mark more config entries server only.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2993 from turboFei/fix_typo.
Closes#2993
002e911f [Fei Wang] add more
53ef59a0 [Fei Wang] save
d5a55683 [Fei Wang] add into server only
fea3daa4 [Fei Wang] Fix typo
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
This is a follow up of #2850
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2971 from turboFei/remove_on_match_from_default.
Closes#2850
10e6067e [Fei Wang] refactor
929fde7d [Fei Wang] comments
4ad12f11 [Fei Wang] add log4j1
91168b6c [Fei Wang] fix
e5ac285e [Fei Wang] fix
efe789eb [Fei Wang] [KYUUBI #2850][FOLLOWUP] Use log4j2 xml to take replace of log4j2 properties
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Remove TProtocolVersion from SessionHandle/OperationHandle/Operation classes, and now it only kept in Session
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2966 from yaooqinn/proto.
Closes#2966
f698d7d3 [Kent Yao] [WIP] Remove TProtocolVersion from SessionHandle/OperationHandle
83f3676a [Kent Yao] [WIP] Remove TProtocolVersion from SessionHandle/OperationHandle
10024e9e [Kent Yao] [WIP] Remove TProtocolVersion from SessionHandle/OperationHandle
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
1. bugfix for issue #2960
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2961 from CavemanIV/kyuubi-2960.
Closes#2960
8aa38eb9 [zhangliang] TFrontendService always use newest hive cli TProtocolVersion
25e5110e [zhangliang] [KYUUBI #2960] TFrontendService.SERVER_VERSION shall be HIVE_CLI_SERVICE_PROTOCOL_V11
Authored-by: zhangliang <zhangliang@trip.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
We have to use canonical host name as kyuubi instance, otherwise, the ip is always change after pod restart, and the kyuubi instance can not get recovered.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2957 from turboFei/kyuubi_instance_url.
Closes#2957
ac8d9df4 [Fei Wang] respect FRONTEND_CONNECTION_URL_USE_HOSTNAME
c4dcd7d3 [Fei Wang] use getCanonicalHostName for EmbeddedZookeeper
c183b41f [Fei Wang] use getCanonicalHostName for TFrontendService
59f3c0a6 [Fei Wang] Use getCanonicalHostName
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
The Operation Type is only used for converting `TOperationType` through specific Thrift Operation APIs. So we can TOperationType in these APIs directly, such as, during calling TFrontendService.ExecuteStatement, we can set TOperationType.EXECUTE_STATEMENT at the end. There is no need to add it to OperationType.
This PR removes `OperationType` completely to simply `OperationHandle` which benefits other `kyuubi.frontend.protocols`.
The thrift protocol shall be able to be removed in following PRs.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2952 from yaooqinn/opType.
Closes#2952
af00c04a [Kent Yao] address comments
4e4f4341 [Kent Yao] address comments
54a2c4ff [Kent Yao] [WIP] Remove Operation Type
ba08ccaf [Kent Yao] [WIP] Remove Operation Type
1737176b [Kent Yao] [WIP] Remove Operation Type
8fe00820 [Kent Yao] [WIP] Remove Operation Type
f969d73d [Kent Yao] [WIP] Remove Operation Type
778f8f4f [Kent Yao] [WIP] Remove Operation Type
54f6573c [Kent Yao] [WIP] Remove Operation Type
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
to close#2949
Unquoted the function name in the error `SECOND_FUNCTION_ARGUMENT_NOT_INTEGER` since Spark-3.4.0, for more details, see https://github.com/apache/spark/pull/36693
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2950 from cfmcgrady/kyuubi-2949.
Closes#2949
4d4da0ba [Fu Chen] fix
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Remove thrift request timeout for KyuubiSyncThriftClient
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2948 from turboFei/remove_request_timeout.
Closes#2948
a9c5c084 [Fei Wang] remove request timeout
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
The cipher is not thread-safe.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2933 from turboFei/cipher_not_thread_safe.
Closes#2894
ea3f79c03 [Fei Wang] not thread safe
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
fix#2927
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2928 from iodone/dev-2.
Closes#2927
a5cb1908 [odone] [KYUUBI #2927] fixed
Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Fix typo for mysql metadata schema.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2920 from lightning-L/typo-2.
Closes#2920
5f54b6f5 [Tianlin Liao] fix minor typos
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Now for the batch state, we rely on the metadata store, which is only updated by the local kyuubi instance.
But if the kyuubi instance is down or unreachable sometimes, the metadata state will not change and we can not get the correct state, it might cause the client side stuck if it is waiting the batch completion.
So, if the kyuubi instance is not reachable, we need backfill the batch state according to resource manager batch application state it the batch was in RUNNING state.
If the batch was stuck in PENDING state and the kyuubi instance can not get recovered in short-time, we need support to close the batch session and prevent the batch job re-submitted if it get recovered.
In this pr, I introduce a new field `remoteClosed` to present whether it has been marked as CLOSED state by remote kyuubi instance.
The `remoteClosed` should only be updated by remote kyuubi instance and set its value to true, so there is no race condition.
- When opening the batch session in recovery mode, if `remoteClosed` is true, set the state to CANCELED directly
- check the local batch sessions that is marked as remoteClosed periodically and close them.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2911 from turboFei/across_kill_one.
Closes#2911
d582459c [Fei Wang] remoteClosed => peerInstanceClosed
cd4207c2 [Fei Wang] unify the state
ede40d0a [Fei Wang] Redirect the get batch request
f4b86868 [Fei Wang] save
5faee761 [Fei Wang] save
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
close https://github.com/apache/incubator-kyuubi/issues/2745#issue-1248428722
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2776 from zhaomin1423/doris_get_tables.
Closes#2745
d0141dde [Min Zhao] fix
de6ea21b [Min Zhao] fix
0ed24bad [zhaomin3] add get tables
86ac8c61 [Min Zhao] get tables
7e7e5479 [Min Zhao] [KYUUBI #2745] [Subtask] DorisSQLEngine - GetTables Operation
Lead-authored-by: Min Zhao <zhaomin1423@163.com>
Co-authored-by: zhaomin3 <zhaomin3@corp.netease.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
This PR moves engine integration tests except to spark engine from `kyuubi-server` module to dedicated IT modules e.g. `kyuubi-hive-it`, then we can spilt different engine test into jobs to speed up CI tes
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
before
<img width="836" alt="image" src="https://user-images.githubusercontent.com/26535726/173265839-5d0f1ce6-e133-4d60-9a95-c95aa16f817f.png">
after
<img width="825" alt="image" src="https://user-images.githubusercontent.com/26535726/173265903-fb01e910-8f78-4091-91da-dca25be4357d.png">
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2827 from pan3793/hive-it.
Closes#2827
df3798f1 [Cheng Pan] Remove Java 11 test for Hive
e1645338 [Cheng Pan] nit
66524226 [Cheng Pan] nit
3aef2cef [Cheng Pan] review
500c9cff [Cheng Pan] MVN_OPT
3b4ad142 [Cheng Pan] nit
daa6e0ab [Cheng Pan] review
3383d4c1 [Cheng Pan] [BUILD][TEST] Decouple integration tests from kyuubi-server
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
We need support to retry the jdbc requests.
Because we add new jdbc dependency for batch HA, and the database might be in maintenance window.
And there might be master-slave switch, and the database might be unavailable for few minutes.
We need tolerant this issue and unblock the main thread.
BTW, this pr refactor the metadata store naming.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2834 from turboFei/retrying_jdbc_event.
Closes#2834
109a63e70 [Fei Wang] fix docs
a186484be [Fei Wang] refactor the jdbc pool name
cfac9da53 [Fei Wang] rename metadata table name
bb8710ba3 [Fei Wang] comments
1197a098b [Fei Wang] add private package scope
a95694ba9 [Fei Wang] comments
e2f09e784 [Fei Wang] refactor
057eb73c5 [Fei Wang] refactor
88f3ffb6f [Fei Wang] refactor
e0595bdee [Fei Wang] comments
48270e869 [Fei Wang] refactor conf keys
e4f190c5c [Fei Wang] update docs
ed5e90e37 [Fei Wang] add max queues limitation
c5a03c9e2 [Fei Wang] remove getOrCreate
3e39ca94c [Fei Wang] remove
b79fb0954 [Fei Wang] comments
1d309cee0 [Fei Wang] comments
9108a3f0e [Fei Wang] fix docs
5cdffe710 [Fei Wang] fix rat
2e7228fc7 [Fei Wang] fix ut
520236d7b [Fei Wang] rename more
f66d68bcc [Fei Wang] merge into metadata manager
2f70a2f12 [Fei Wang] rename
80e6100fb [Fei Wang] refactor
09cad092a [Fei Wang] add ut
ea4eea74f [Fei Wang] refactor
6b00eb0b3 [Fei Wang] refactor to MetadataRequest
58f1c0587 [Fei Wang] commements
198feb53f [Fei Wang] add more docs
670a23561 [Fei Wang] refactor
a20d54df2 [Fei Wang] waitStateStoreRetryCompletion after super.close closes operation
97a763aef [Fei Wang] refactor RetryingStateStoreRequest
3e136cf34 [Fei Wang] [SUB-TASK][KPIP-4] Support to retry the session state operations if meet transient jdbc issue
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
To close#2708
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2856 from turboFei/open_in_one.
Closes#2708
2fd77951 [Fei Wang] comments
35ae00d4 [Fei Wang] comment
e02c13e3 [Fei Wang] use kyuubiconf
70d96775 [Fei Wang] refactor
cfb1a36b [Fei Wang] refresh token for startup
f55912f5 [Fei Wang] comments
130a3bb3 [Fei Wang] remove credentials to prevent escalate to session conf
d14ebf78 [Fei Wang] fix ut
71ec3c9e [Fei Wang] renew token when opening session
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
DS v2 does not support view yet.
<img width="1916" alt="image" src="https://user-images.githubusercontent.com/26535726/172974844-f1347cb0-ffcd-49fc-99f2-2669f8f945d1.png">
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2848 from pan3793/global_temp.
Closes#2848
4263131f [Cheng Pan] ut
50a51590 [Cheng Pan] ut
9e476c1e [Cheng Pan] Global temp view should only exist in session catalog
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Log session name when opening/closing session
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2851 from turboFei/session_name.
Closes#2851
ca855deab [Fei Wang] comments
2fc4ab770 [Fei Wang] make name lazy
31bb87b98 [Fei Wang] move to behind handle
dc08687c0 [Fei Wang] log session Name
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
- change log4j2-test.properties to log4j2-test.xml
- add the unit test log4j2.xml for spark relative submodule, and remove the log4j.properties
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2850 from cfmcgrady/kyuubi-2247.
Closes#2247
a33d4d80 [Fu Chen] style
f99dadac [Fu Chen] fix style
49c99dea [Fu Chen] add log4j2.xml for spark relative submodule
a8a38561 [Fu Chen] change log4j2-test.properties to log4j2-test.xml
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
1. remove `thrift protocol version` from REST req and resp.
- it's useless for this REST. It's not used for version check and not for result set determination.
- it's not a part of TSessionHandle/TOperationHandle either. The protocol version is determined in open session phase and used both by the session and the within operations. It should not be set per request.
2. Make secret-id static across rest/batch/jdbc APIs
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2829 from yaooqinn/handle.
Closes#2829
eb639975 [Kent Yao] Make secret id static and move thrift protocol from RPC handles
aefc2f25 [Kent Yao] Make secret id static and move thrift protocol from RPC handles
dc404af6 [Kent Yao] Merge branch 'master' into handle
cec28a19 [Kent Yao] Make secret id static and move thrift protocol from RPC handles
dfa749dd [Kent Yao] Make secret id static and move thrift protocol from RPC handles
8774be3f [Kent Yao] Make secret id static and move thrift protocol from RPC handles
4d6bd4a3 [Kent Yao] Make secret id static and move thrift protocol from RPC handles
386bfa82 [Kent Yao] Make secret id static and move thrift protocol from RPC handles
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
To close#2820
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2819 from turboFei/kpip_redirect_batch_rest.
Closes#2820
0346d1c0 [Fei Wang] revert server delete batch retry
eea42917 [Fei Wang] comments for timeout
0f4e2411 [Fei Wang] make default constructor private
542b89ac [Fei Wang] remove AuthUtil
772adeb6 [Fei Wang] add AuthHeaderGenerator
9f8efb83 [Fei Wang] Add internal batch rest client
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
To close#2373
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2790 from turboFei/recovery_batch.
Closes#2373
81632191 [Fei Wang] trigger test
6ac6f9bd [Fei Wang] async recovery
6d9edf4e [Fei Wang] comments
daa6719f [Fei Wang] refactor
0c3a2e14 [Fei Wang] remove waitAppCompletion
04e139ee [Fei Wang] comments
b5d11d8f [Fei Wang] comment for method name
321dfa95 [Fei Wang] refactor
90be2df7 [Fei Wang] address comments
14409582 [Fei Wang] batch recovery
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Currently, we define a `ExtendedSQLTest` in `kyuubi-common` test sources, and use it to mark some slow test. This PR propose to replace `ExtendedSQLTest` by `Slow` which is provided by scalatest, then we can use the same tag to mark slow tests in some modules which does not depends on `kyuubi-common`, and then seperate them into independent CI job to reduce the CI time
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2824 from pan3793/scalatest-tag.
Closes#2824
2597863c [Cheng Pan] [TEST] Replace test tag ExtendedSQLTest by Slow
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Have seen run out of disk spaces in GitHub Action, set log level to info to reduce log size
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2822 from pan3793/log.
Closes#2822
b355452a [Cheng Pan] followup
541ca5fe [Cheng Pan] remove unsed filter
41f656ca [Cheng Pan] [GA] Set log level to info
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Return more batch info for insight: `user`, `name`, `createTime` and `endTime`.
Support to list batches with filter conditions:
- batchState
- batchUser
- createTime
- endTime
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2812 from turboFei/return_more_info.
Closes#2812
972fe888 [Fei Wang] comment
da2ea29c [Fei Wang] return more info for batch
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Now Trino, Hive and JDBC Engine use session manager configuration in `newExecuteStatementOperation`, ignoring session level configuration.
Supporting session conf also helps with testing `withSessionConf`.
close#2807
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2808 from cxzl25/KYUUBI-2807.
Closes#2807
29d6c51e [sychen] support session conf jdbc
7bd560fe [sychen] support session conf trino/hive
ce5e4735 [sychen] closeExpiredOperations add parentheses
2d402fb0 [sychen] trigger test
3e7ae439 [sychen] trigger test
d0f47983 [sychen] sessionKyuubiConf
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
1. rename `Metadata` to `SessionMetadata`
2. wrap `CloseBatchResponse` for closeBatchSession
3. remove killApp in closeBatchSession, so every request will kill the app by default
4. correct when to update metadata
5. cleanupMetadataByAge should check state and only cleanup terminated metadata
6. correct the batch session state, we should never get a closed state
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2800 from ulysses-you/followup.
Closes#2800
a132439e1 [ulysses-you] comments
0f1c3aadb [ulysses-you] fix
e79ec501d [ulysses-you] fix
e8862f844 [ulysses-you] improve
facad1757 [ulysses-you] comments
777f00ce5 [ulysses-you] Merge branch 'master' of https://github.com/apache/incubator-kyuubi into followup
df71d5962 [ulysses-you] address comment
28010cb53 [ulysses-you] docs
dacdc8718 [ulysses-you] lock
6516d324c [ulysses-you] cleanup
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
To close#2742
For impersonation, it depends on local hadoop configuration.
If we want to refresh the super user group, we have to restart kyuubi server.
And hadoop namenode/resource manager support refresh the super user groups without restart service.
So, it is great that kyuubi also support refreshSuperUserGroupsConfiguration function.
With this pr, we can refresh the frontend service hadoop conf(used for proxyuser verfication) without restarting kyuubi server.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2762 from turboFei/admin_resource.
Closes#2742
208f96800 [Fei Wang] comment
99f1148ec [Fei Wang] fix ut
6a7ed3a9a [Fei Wang] fix ut
c8e4d1ec4 [Fei Wang] fix ut
49d2e419e [Fei Wang] fix ut
890e2adb2 [Fei Wang] fix ut
3e026149d [Fei Wang] comment
cc3df379a [Fei Wang] comment
80e49c574 [Fei Wang] refactor
20b7afde9 [Fei Wang] move to kyuubi server
555747714 [Fei Wang] [KYUUBI #2742] Introduce admin resource for service admin - refresh frontend hadoop conf without restart
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Close#2715Close#2374
Introduce jdbc based session state store to share the batch state information across kyuubi instances.
This pr is based on docs: https://docs.google.com/document/d/1_BMekMT88P2vCHCl4ZacWEeWslpIv4pxsJH32PAnWSI and refactor it during implementation.
In this PR, I introduce `StateStore` interface and implement the default `JDBCStateStore`.
The data structures:
1. BatchState - the batch state info for multiple-HA, including batch operation state and application information
2. BatchMeta - the meta data used for batch recovery, including sessionConf, batchRequest information
For JDBCStateStore:
- support to specify data base type:
- 1. DERBY - it will use derby jdbc driver by default
- 2. MYSQL - it will use mysql jdbc by default
- 3. CUSTOM - the user need specify the jdbc driver
- support to init schema
By default, the jdbc url is a derby in-memory database url for single kyuubi instance.
To support multiple HA in production env, we should config the production jdbc url to share batch info across kyuubi instances.
The datasource is managed by HiKariCP.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2717 from turboFei/kyuubi_2715_hikaricp_v2.
Closes#2717
367b944a3 [Fei Wang] comments
9a7963dbd [Fei Wang] post session type in event
b1c130853 [Fei Wang] move session type to session package
66def0cb6 [Fei Wang] add JDBCStateStoreConf
839b72bf6 [Fei Wang] refactor
644cbcd28 [Fei Wang] filter by session type
58f1c7854 [Fei Wang] refactor
1c0035208 [Fei Wang] add session type
34a087ee5 [Fei Wang] refactor
fba38e473 [Fei Wang] refactor
fcbde25a0 [Fei Wang] refactor
332a67537 [Fei Wang] comments
135ba9b6d [Fei Wang] Revert "revert class utils change"
0e045a483 [Fei Wang] Revert "Revert "move""
05048416e [Fei Wang] Revert "test again"
01a50c35b [Fei Wang] test again
b89159d13 [Fei Wang] Revert "move"
3764a9051 [Fei Wang] move
eef17822c [Fei Wang] revert
4166db261 [Fei Wang] adjust engine idle timeout
5fc95a824 [Fei Wang] revert class utils change
23f88d63d [Fei Wang] refactor
a66cb811d [Fei Wang] refactor
89fa0edb4 [Fei Wang] fix
e5228fefd [Fei Wang] refactor
581908d86 [Fei Wang] exclude sql
3446aaa06 [Fei Wang] UPPER
ab980c677 [Fei Wang] [KYUUBI #2715] Introduce session state store for batch session multiple HA
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
close#2721
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2728 from cxzl25/KYUUBI-2721.
Closes#2721
67051644 [sychen] use placeholders and confOverlay
2acf0a45 [sychen] unused import
8bc40b9f [sychen] fix UT
3495b280 [sychen] fix jdbc
e55bb2e2 [sychen] style
ff9078ab [sychen] fix UT
a0247c7e [sychen] Merge branch 'master' into KYUUBI-2721
f62461ce [sychen] use placeholders
698fbbdd [sychen] set catalog/schema empty resultset
45caf552 [sychen] fix import
7f13ad8d [sychen] fix server spark catalog UT
4982ec71 [sychen] hive engine 1.8
9877038f [sychen] add UT
21e57870 [sychen] set/get catalog/database operators
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2595 from zhaomin1423/jdbc_engine.
Closes#2658
90bdf97c [Min Zhao] add jdbc dialect
d2bfd60c [Min Zhao] [KYUUBI #2486] [FEATURE] Support Apache Doris engine
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
1. Change to a true iterator implementation to get the results of Trino Server to avoid caching data in memory.
2. Supports FETCH_NEXT mode only, consistent with Trino Server's capabilities
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2678 from iodone/kyuubi-2285.
Closes#2285
f0ed8085 [odone] [KYUUBI #2285] update trino get resultset to stream fetch
Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Support internal rest request authentication to enable redirect http request across kyuubi instances
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2719 from turboFei/internal_access.
Closes#2719
b9e150e9 [Fei Wang] revert config key change
aa378727 [Fei Wang] make it internal
ce96d922 [Fei Wang] comments
6fe8523b [Fei Wang] Support internal rest request authentication to enable redirect http request across kyuubi instances
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Because `kyuubi-server` is a long-running service, its logs should preferably have dates.
When Log4j1 was used in previous versions, the default log and template had dates.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2714 from cxzl25/KYUUBI-log4j-layout-pattern.
Closes#2714
b249c296 [sychen] change layout pattern
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Stop NoopServer should not throw exception
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2696 from pan3793/noop.
Closes#2696
02c98503 [Cheng Pan] [TEST] Stop NoopServer should not throw exception
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
`LaunchEngineOp` will never close until session is closed, the client should not assume launch engine has completed on exception.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2641 from pan3793/close.
Closes#2641
986104b0 [Cheng Pan] reserved
970d90ea [Cheng Pan] ut
ff8f75cf [Cheng Pan] import
0247117b [Cheng Pan] ut
e490344b [Cheng Pan] Client should not assume launch engine has completed on exception
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix#2637
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2640 from pan3793/odbc.
Closes#2640
a798271b [Cheng Pan] Implement TGetInfoType CLI_ODBC_KEYWORDS
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
We want to implement different service discovery client, thus add a config to support different service discovery client class implementation.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2601 from hiboyang/datapunch-DiscoveryClient-Configurable.
Closes#2601
719685f1 [BoYang] Update settings.md
8af4f5da [BoYang] Change name to HA_DISCOVERY_CLIENT_CLASS
22139b6d [BoYang] Update settings.md
c9845cee [BoYang] Revert unrelated changes
1713e050 [BoYang] Add .github files
22222a4d [BoYang] Merge branch 'master' into datapunch-DiscoveryClient-Configurable
c0c293db [BoYang] Merge branch 'master' of https://github.com/apache/incubator-kyuubi
3e53ef48 [BoYang] Do not copy spark files in docker
887be439 [BoYang] Add dot after docker build command
48b70053 [BoYang] Update manual-release.yml
39d0bb21 [BoYang] Update workflow: cd dist
9adae8c1 [BoYang] Update manual-release.yml
da76f568 [BoYang] Use nonEmpty
bf8fe2ff [BoYang] Run mvn spotless:apply
188d71c5 [BoYang] Update per comments
fffa3bfe [BoYang] Create manual-release.yml
65c62c31 [BoYang] Add KyuubiConf.createInstance and reuse some class checking code
30f0ff35 [BoYang] Add a config to support different service discovery client class implementation
1cdf8809 [BoYang] Delete github workflows
Authored-by: BoYang <datapunch2021@gmail.com>
Signed-off-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
### _Why are the changes needed?_
In scala code mode implementation, we need to call `repl.addUrlsToClassPath` to set the classpath of the current thread to repl before entering `repl`
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2475 from iodone/dev-1.
Closes#2471
21b92cc4 [odone] [KYUUBI #2471] add dynamic jar generator
3542786e [odone] [KYUUBI #2471] Fixed
3db2a7df [odone] [KYUUBI #2471] Fixed
65303683 [odone] [KYUUBI #2471] Fixed
Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
close#2605
`kyuubi.engine.spark.register.operation.listener`
> When set to true, Spark engine registers a SQLOperationListener before executing the statement, logs a few summary statistics when each stage completes.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2617 from cxzl25/KYUUBI-2605.
Closes#2605
526af5eb [sychen] use spark conf
492a2044 [sychen] update settings.md
22fe762f [sychen] add UT
a7b29de1 [sychen] Merge branch 'master' into KYUUBI-2605
e8fa3c6f [sychen] kyuubi.engine.spark.register.operation.listener
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Add Multi-tenancy support for hive engine.
With this feature, hive engine will talk to hive metatore, resource managers, hdfs with the correct user name.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2606 from yaooqinn/2604.
Closes#2604
8ce6fa01 [Kent Yao] [KYUUBI #2604] Hive Backend Engine - Multi tenancy support
86c92121 [Kent Yao] [KYUUBI #2604] Hive Backend Engine - Multi tenancy support
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
There are some deps has low version hadoop deps, we should exclude them and use hadoop shaded client instead.
For engine modules, the shaded client will not be packaged into shaded jar, then nothing changes in the final shaded jar.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2621 from pan3793/hdp.
Closes#2621
88399979 [Cheng Pan] nit
d75a5806 [Cheng Pan] Always use Hadoop shaded client
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Now the user can see the command to start the engine, which may have some sensitive information.
Introduce a configuration item to support replacing sensitive information.
For example, if you use the `kyuubi.ha.zookeeper.auth.digest` configuration, you can configure `kyuubi.server.redaction.regex` `(?i)zookeeper.auth.digest`
close#2591
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2592 from cxzl25/KYUUBI-2591.
Closes#2591
67567a26 [sychen] update doc
eb1ec9a1 [sychen] redact kv
fbcba2dd [sychen] Merge branch 'master' into KYUUBI-2591
346e21b1 [sychen] kyuubi.server.redaction.regex
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
fix#2554
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2579 from jiaoqingbo/kyuubi2554.
Closes#2333Closes#2554
f0365c91 [jiaoqingbo] code review
1700aab9 [jiaoqingbo] code review
1ca10a65 [jiaoqingbo] fix ut failed
b53dcdd4 [jiaoqingbo] code review
f9ceb72c [jiaoqingbo] [KYUUBI #2554] Configuring Flink Engine heap memory and java opts
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Taking over #2174, all credits belongs to KenjiFujima
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2586 from yaooqinn/PR_2174.
Closes#2029
6b45a4f6 [Kent Yao] followup
7861fb20 [KenjiFujima] [KYUUBI #2029] Hive Backend Engine - Operation Logs
Lead-authored-by: Kent Yao <yao@apache.org>
Co-authored-by: KenjiFujima <thanosxnicholas@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
We can detect engine dead cases after https://github.com/apache/incubator-kyuubi/pull/2122, so set thrift client socket timeout to inf to avoid socket timeout in blocking case, e.g. retrieve data except to 1st partition in increment collection mode, spark scale mode.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2609 from pan3793/timeout.
Closes#2609
bff14cb1 [Cheng Pan] Update settings.md
e92fa442 [Cheng Pan] Set Kyuubi server thrift client socket timeout to inf
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Why are the changes needed?
upgrade hive to 3.1.3 in the kyuubi-hive-jdbc module
### change details
file|comment
--- | ---
kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/HttpBasicAuthInterceptor.java | add customCookies as a construct parameter
kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/HttpKerberosRequestInterceptor.java | replace assumeSubject field with `Subject loggedInSubject` ; add customCookies as a construct parameter
kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/HttpRequestInterceptorBase.java | add customCookies as a construct parameter; add customCookies initialization codes
kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/HttpTokenAuthInterceptor.java | add customCookies as a construct parameter
kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/JdbcColumn.java | support TIMESTAMP_WITH_TIMEZONE & UNION_TYPE type
kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiBaseResultSet.java | support TIMESTAMPLOCALTZ_TYPE
kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiConnection.java | add loggedInSubject,wmPool & wmApp fields and their initialization codes; add getAllUrls function; use Utils.getCanonicalHostName to extract host from connParams; clear hive conf in embedded mode; use SessionUtils.getTokenStrForm to get token string; remove isZkDynamicDiscoveryMode
kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiDatabaseMetaData.java | support getSQLKeywords; add Object class type parameter to KyuubiMetaDataResultSet
kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiMetaDataResultSet.java | optimize initialization logic of data & columnTypes field; add normalizeColumnNames function
kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiStatement.java | split closeClientOperation into 2 functions(closeStatementIfNeeded & closeClientOperation) & add reInitState function to replace initFlags & ad error messages to exceptions in CANCELED_STATE
kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/Utils.java | add dummyAuthorityString, SERVICE_DISCOVERY_MODE_ZOOKEEPER_HA, ZOOKEEPER_ACTIVE_PASSIVE_HA_DEFAULT_NAMESPACE, WM_POOL, HTTP_COOKIE_PREFIX, APPLICATION fields; replace authorityList field with suppliedURLAuthority in JdbcConnectionParams;add a construct function to JdbcConnectionParams; replace getAuthorityList function with getSuppliedURLAuthority; split parseURL into 2 different functions(parseURL, extractURLComponents); add configureConnParamsFromZooKeeper, handleAllDeprecations, getAuthorityFromJdbcURL functions; remove configureConnParams;remove joinStringArray;add getCanonicalHostName function
kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/ZooKeeperHiveClientHelper.java | add more functions
kyuubi-server/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier | change package name of DelegationTokenIdentifier
kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HiveDelegationTokenProvider.scala | change package name of DelegationTokenIdentifier
kyuubi-server/src/test/scala/org/apache/kyuubi/credentials/HiveDelegationTokenProviderSuite.scala | add configs, METASTORE_CONNECTION_POOLING_TYPE = none,METASTORE_AUTO_CREATE_ALL = true, METASTORE_SCHEMA_VERIFICATION = false; add construct parameter clientConf to HadoopThriftAuthBridgeWithServerContextClassLoader
kyuubi-common/src/test/scala/org/apache/kyuubi/operation/SparkMetadataTests.scala | remove getSQLKeywords from unsupported list
externals/kyuubi-hive-sql-engine/pom.xml | add HikariCP to test scope
pom.xml | update hive version to 3.1.3
kyuubi-hive-jdbc-shaded/pom.xml | add hive-standalone-metastore dependency; add hive-llap-client dependency
kyuubi-hive-jdbc/pom.xml | add hive-standalone-metastore dependency;add hive-llap-client dependency
kyuubi-server/pom.xml | add hive-standalone-metastore dependency
dev/dependencyList | add hive-standalone-metastore
kyuubi-hive-jdbc-shaded/src/main/resources/META-INF/LICENSE | update content because of hive-standalone-metastore & hive-llap-client
kyuubi-hive-jdbc-shaded/src/main/resources/META-INF/NOTICE | update content because of hive-standalone-metastore & hive-llap-client
### How was this patch tested?
use existing UTs to test this patch
Closes#2560 from gabrywu/jdbc-3.1.
Closes#2560
9c5e3eda [gabry.wu] Merge branch 'master' into jdbc-3.1
c23cefc5 [gabrywu] update NOTICE rollback unnecessary change
b9394144 [gabrywu] change hikaricp.version position to satisfy alphabetical order
59ebed37 [gabrywu] add hive-storage-api explicitly
d22dc585 [gabrywu] rollback to minimize diff of this PR
33c13f0f [gabrywu] update LICENSE, NOTICE, LICENSE add hive-llap-client to kyuubi-hive-jdbc-shaded module
d1ce2c6c [gabrywu] manage hikaricp.version under dependencyManagement
1f4b7c72 [gabrywu] Merge remote-tracking branch 'origin/jdbc-3.1' into jdbc-3.1
fd309054 [gabrywu] manage hikaricp.version under dependencyManagement
39aa7ae6 [gabrywu] manage hikaricp.version under dependencyManagement
85dc983a [gabrywu] manage hikaricp.version under dependencyManagement
545578c5 [gabrywu] update dependencyList file
f479f642 [gabrywu] remove optional flag of hive-standalone-metastore in kyuubi-server
5ee8f89b [gabrywu] 1. add hive-storage-api 2. enable METASTORE_AUTO_CREATE_ALL 3. disable METASTORE_SCHEMA_VERIFICATION
f70b0354 [gabrywu] add HikariCP to test scope
c93ab167 [gabrywu] set METASTORE_CONNECTION_POOLING_TYPE to none
feb53d6e [gabrywu] support getSQLKeywords func
4601b9b9 [gabrywu] exclude package.jdo
4f4a19bf [gabrywu] upgrade hive version to 3.3.1 in dependencyList
0b88a77a [gabrywu] upgrade hive to 3.1.3
Lead-authored-by: gabrywu <gabrywu@apache.org>
Co-authored-by: gabry.wu <gabrywu@apache.org>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Support fetch operation log with rest batch mode.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2567 from ulysses-you/KYUUBI-2378.
Closes#2378
d4dd5352 [ulysses-you] comment
fd08362c [ulysses-you] desc
37b6cc83 [ulysses-you] seek
78fb6391 [ulysses-you] rebase
d6b838ac [ulysses-you] style
0440ec29 [ulysses-you] fix
b07a81ce [ulysses-you] Implement BatchesResource GET /batches//log
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
close#2493
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [X] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2529 from wForget/KYUUBI-2493.
Closes#2493
55ef323e [wforget] comments
2da6dea3 [wforget] add conf
0260ece9 [wforget] remove unnecessary log
a46236d9 [wforget] remove OperationProgressUpdate
e76cd78e [wforget] remove OperationProgressStatus
68ee1138 [wforget] remove unnecessary column
a2b9956a [wforget] comments
3d916e42 [wforget] comment
de6910fa [wforget] test details
a07336b7 [wforget] fix test
95277ce8 [wforget] add test
79abed96 [wforget] fix
579ffdbc [wforget] comment
f7f7d420 [wforget] [KYUUBI-2493] Implement the progress of statement
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
A util class for random access operation log
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2573 from ulysses-you/seekable-reader.
Closes#2573
f75dac72 [ulysses-you] address comment
9239ea5f [ulysses-you] seek
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2466 from zhaomin1423/timeoutMonitor.
Closes#2403
3266a02b [Min Zhao] modify statementTimeoutCleaner access modifier
db73a42d [Min Zhao] add shutdownTimeoutMonitor
4f1d0e10 [Min Zhao] clear unused import
acda7e6c [Min Zhao] handle cleanup
eca266a3 [Min Zhao] [KYUUBI #2403] [Improvement] move addTimeoutMonitor to AbstractOperation because it was used in multiple engines
f5256053 [Min Zhao] [KYUUBI #2403] [Improvement] move addTimeoutMonitor to AbstractOperation because it was used in multiple engines
8e7820bc [Min Zhao] [KYUUBI #2403] [Improvement] move addTimeoutMonitor to AbstractOperation because it was used in multiple engines
9b19704e [Min Zhao] [KYUUBI #2403] [Improvement] move addTimeoutMonitor to AbstractOperation because it was used in multiple engines
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Support jdbc url prefix `jdbc:kyuubi://`
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2547 from turboFei/jdbc_kyuubi.
Closes#2547
608f46da [Fei Wang] fix ut
cd267d3e [Fei Wang] add ut
762631bb [Fei Wang] support jdbc:kyuubi://
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Follow up #2450
In PR #2452, I add opHandle to opHandleSet before running, when running exception we need to remove opHandle from opHandleSet.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2504 from wForget/KYUUBI-2450-1.
Closes#2450
a4ef89a7 [wforget] [KYUUBI-2450][FOLLOWUP] Remove Operation handle from opHandleSet when exception occurs
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Hive Backend Engine - new APIs with hive-service-rpc 3.1.2 - TGetQueryId
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2382 from yangrong688/KYUUBI-2020.
Closes#2020
97eccc07 [yangrong688] fix spotless check
efe01165 [yangrong688] add javadoc and add returns null if stmtHandle is null
a9d04e6c [yangrong688] fix
ac0dff2f [yangrong688] fix
3e935802 [yangrong688] skip test when java version beyond JAVA_1_8
2a777e26 [yangrong688] update getQueryId test case
ec12d2a2 [yangrong688] [feat] complete TGetQueryId op, should refined test case later
2996579f [yangrong] [feat] init operation getQueryId, just structure, need to improve quickly
Lead-authored-by: yangrong688 <yangrong.jxufe@gmail.com>
Co-authored-by: yangrong <yangrong.jxufe@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2465 from zhaomin1423/2453.
Closes#2453
8a624991 [Min Zhao] fix tset
54ee09df [Min Zhao] fix suite
3d434a78 [Min Zhao] [KYUUBI #2453] [Improvement] checkValue of TypedConfigBuilder shall also print the config name
f7c7bf27 [Min Zhao] [KYUUBI #2453] [Improvement] checkValue of TypedConfigBuilder shall also print the config name
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
close#1987
note that, this pr only for Spark engine
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2435 from ulysses-you/kyuubi-1987.
Closes#1987
85826a14 [ulysses-you] style
9ddcc0c9 [ulysses-you] shutdown
a568054e [ulysses-you] release
d1339056 [ulysses-you] address comment
8e958ff2 [ulysses-you] docs
dd9b4422 [ulysses-you] Support preserve user context in group share level
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Add KyuubiApplicationManager in SessionManager for application management, currently support kill and get application information.
The underlying cluster manager operation added in this PR are
- local jps with SIG TERM KILL
- YARN-client
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2447 from yaooqinn/2445.
Closes#2445
77810390 [Kent Yao] address comment
aed3f251 [Kent Yao] address comment
0fad9419 [Kent Yao] address comment
67ec2500 [Kent Yao] address comment
800bedb5 [Kent Yao] [KYUUBI #2445] Implement ApplicationManager and Yarn/ JPS-local Application Operation
1a4084ab [Kent Yao] [KYUUBI #2445] Implement ApplicationManager and Yarn/ JPS-local Application Operation
be58583a [Kent Yao] [KYUUBI #2445] Implement ApplicationManager and Yarn/ JPS-local Application Operation
e75e20e7 [Kent Yao] [KYUUBI #2445] Implement ApplicationManager and Yarn/ JPS-local Application Operation
baac7f04 [Kent Yao] [KYUUBI #2445] Implement ApplicationManager and Yarn/ JPS-local Application Operation
e3f5c29a [Kent Yao] [KYUUBI #2445] Implement ApplicationManager and Yarn/ JPS-local Application Operation
c81e5635 [Kent Yao] [KYUUBI #2445] Implement ApplicationManager and Yarn/ JPS-local Application Operation
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
close#2450
1. The client obtains the status of the Operation by calling getOperationStatus cyclically. We need to update the lastAccessTime in the getStatus method to prevent timeouts caused by long-running.
2. We need to add opHandle to opHandleSet before running to avoid losing Operation by calling close during Operation running.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2452 from wForget/KYUUBI-2450.
Closes#2450
35002396 [wforget] add opHandle to opHandleSet before run
dc82f65b [wforget] unnecessary change
cca3377c [wforget] [KYUUBI-2450] Update lastAccessTime in getStatus and catch close launchEngineOp exception
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
fix#2424
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2425 from jiaoqingbo/2424.
Closes#2424
9949d9a5 [jiaoqingbo] Update UtilsSuite.scala
ac9afb46 [jiaoqingbo] Update package.scala
e415345d [jiaoqingbo] Update kyuubi-common/src/main/scala/org/apache/kyuubi/package.scala
081b061f [jiaoqingbo] Update kyuubi-server/src/main/scala/org/apache/kyuubi/server/KyuubiServer.scala
c1ade020 [jiaoqingbo] Update build/kyuubi-build-info
6af3dd08 [jiaoqingbo] [KYUUBI #2424] [Improvement] add Flink compile version and Trino client compile version
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Now lose error stack trace information.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2428 from deadwind4/stack-trace-front.
Closes#2426
c7dde402 [Ada Wang] [KYUUBI #2426] Return complete error stack trace information
Authored-by: Ada Wang <wang4luning@gmail.com>
Signed-off-by: KenjiFujima <thanosxnicholas@gmail.com>
### _Why are the changes needed?_
close#2422
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2423 from wForget/KYUUBI-2422.
Closes#2422
94ac9ed1 [wforget] [KYUUBI-2422] Wrap close session with try-finally
Authored-by: wforget <643348094@qq.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
close#2301
### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2364 from wForget/KYUUBI-2301.
Closes#2301
7a176e0e [wforget] merge
0b41fafa [wforget] comment
7614dbda [wforget] Merge remote-tracking branch 'origin/master' into KYUUBI-2301
9180cf63 [wforget] fix
741139a4 [wforget] fix
1194ccbf [wforget] move limiter from common to server
a6b93b38 [wforget] regenerate settings.md
123f9209 [wforget] [KYUUBI-2301] Limit the maximum number of concurrent connections per user and ipaddress
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Separate events to a submodule - kyuubi-event
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2392 from zhaomin1423/event-module.
Closes#2323
7efa923f [Min Zhao] fix unit test
07126f43 [Min Zhao] [KYUUBI #2323] Separate events to a submodule - kyuubi-event
cccbdf11 [Min Zhao] [KYUUBI #2323] Separate events to a submodule - kyuubi-event
194aca82 [Min Zhao] add shade and log4j to test/resources
f75891a1 [Min Zhao] [KYUUBI #2323] Separate events to a submodule - kyuubi-event
357395fa [Min Zhao] [KYUUBI #2323] Separate events to a submodule - kyuubi-event
b6dfd186 [Min Zhao] [KYUUBI #2323] Separate events to a submodule - kyuubi-event
fc876688 [Min Zhao] [KYUUBI #2323] Separate events to a submodule - kyuubi-event
1b86d41b [Min Zhao] seperate event
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2356 from zhaomin1423/option_parser.
Closes#2021
e07abe55 [Min Zhao] fix unit test
a9bc522b [Min Zhao] fix unit test
be628c3c [Min Zhao] [KYUUBI #2021] Command OptionParser for launching Hive Backend Engine
db88bfef [Min Zhao] [KYUUBI #2021] Command OptionParser for launching Hive Backend Engine
b4f50cee [Min Zhao] [KYUUBI #2021] Command OptionParser for launching Hive Backend Engine
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
#2355 Bump Delta Lake 1.2.0
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2389 from Nick-0723/2355.
Closes#2355
202f448d [宋财礼] remove unnecessary dependency
5fc7cf58 [Nick Song] Bump Delta Lake 1.2.0
3984c6b2 [Nick Song] Bump Delta Lake 1.2.0
Lead-authored-by: Nick Song <chun2184@163.com>
Co-authored-by: 宋财礼 <caili.song@nio.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Configuring Hive engine heap memory and java opts
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2371 from zhaomin1423/2360.
Closes#2360
fc372571 [Min Zhao] [KYUUBI #2360] [Subtask] Configuring Hive engine heap memory and java opts
52547d9c [Min Zhao] add unit tests
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
To close#2308
Support to ignore some batch configuration items and pre-define some configuration in server side.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2386 from turboFei/KYUUBI_2308_kpip4_batch_conf.
Closes#2308
2e517ec4 [Fei Wang] [KYUUBI #2308][SUB-TASK][KPIP-4] Batch job configuration ignore list and server predefined configurations
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Configuring Trino Engine heap memory and java opts
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2387 from zhaomin1423/2361.
Closes#2361
e8ef7f6f [Min Zhao] [KYUUBI #2361] [Improvement] Configuring Trino Engine heap memory and java opts
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
To close#2306 and close#2307
In this PR, I implement BatchJobSubmission operation and introduce basic `KyuubiBatchSessionImpl`.
TODO:
- Normalize/validate the batch request
- batch request fields
- merge with server pre-defined batch conf
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2353 from turboFei/KPIP_4_batch_submission_op.
Closes#2353
9bc6050c [Fei Wang] remove unused conf
ef8e962c [Fei Wang] check application in current thread
8738660b [Fei Wang] dedup code
27f22008 [Fei Wang] use static secret id instead of conf
6794ff7a [Fei Wang] Use Seq instead of java.util.List
2f4f9b15 [Fei Wang] Remove BatchType enumaration
7d380800 [Fei Wang] remove dead code
a94a9e6a [Fei Wang] remove jars,files fileds on BatchRequest
6021a1e7 [Fei Wang] add ut for result set
07a939c9 [Fei Wang] refactor long line
a918a496 [Fei Wang] address comments
73229e70 [Fei Wang] set engine max life time
bbe3f1f4 [Fei Wang] unique the application cehcker thread pool
9643e42c [Fei Wang] refactor
11dd71f7 [Fei Wang] add KyuubiBatchYarnClusterSuite
12169910 [Fei Wang] add ut for batch session
47da8c1a [Fei Wang] add open batch session api
6dcf60d9 [Fei Wang] add ut for static batch secret id
a212e62b [Fei Wang] [SUB-TASK][KPIP-4] Implement BatchJobSubmission operation and basic KyuubiBatchSessionImpl
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
Support GetTypeInfo Operation for hive engine.
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2366 from zhaomin1423/2330.
Closes#2330
fc67e0e9 [Min Zhao] [KYUUBI #2330] [Subtask] Hive Backend Engine - GetTypeInfo Operation
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
close#2331
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2332 from wForget/KYUUBI-2331.
Closes#2331
ce7f32e8 [wforget] fix
b74efd80 [wforget] [KYUUBI-2331] Add createSession method to further abstract openSession
Authored-by: wforget <643348094@qq.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Make the HiveProcBuilder actually work.
- hive 2.3.9 has scala 2.11 deps, so not compatible with kyuubi common, thus, upgrade 3.1.x
- hive 3.1 does not work with java 11 as https://issues.apache.org/jira/browse/HIVE-21237 bug
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2290 from yaooqinn/2024.
Closes#2024
83e07fa5 [Kent Yao] fix
1e7167f8 [Kent Yao] import
ac7853ac [Kent Yao] jdk11
ba9f6c33 [Kent Yao] temp
44049736 [Kent Yao] temp
fd624307 [Kent Yao] temp
a4a5e42f [Kent Yao] temp
b01bb226 [Kent Yao] temp
298fc478 [Kent Yao] temp
5630857a [Kent Yao] temp
98457b6e [Kent Yao] temp
18d8f5cb [Kent Yao] [KYUUBI #2024][FOLLOWUP] Hive Backend Engine - ProcBuilder for HiveEngine
246681dd [Kent Yao] [KYUUBI #2024][FOLLOWUP] Hive Backend Engine - ProcBuilder for HiveEngine
890579b9 [Kent Yao] [KYUUBI #2084][FOLLOWUP] Support arbitrary parameters for KyuubiConf
105b70b6 [Kent Yao] tmp
bcc4a0e2 [Kent Yao] tmp
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
fix#2281
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2286 from jiaoqingbo/2281.
Closes#2281
3dfa7078 [jiaoqingbo] use OK_STATUS
8b98b6cd [jiaoqingbo] code review
b262841a [jiaoqingbo] fix ut failed
83a93937 [jiaoqingbo] HadoopCredentialManager can maintain fewer userCredentials as currently only Spark Engine requires token renewal.
03caa962 [jiaoqingbo] mvn spotless
21fd4155 [jiaoqingbo] [KYUUBI #2281] renewalExecutor should be stopped when Engine's side Delegation token is not supported
25063a34 [jiaoqingbo] Merge branch 'master' of https://github.com/jiaoqingbo/incubator-kyuubi
0ad3b764 [jiaoqingbo] Merge branch 'master' of https://github.com/jiaoqingbo/incubator-kyuubi
f9d68c7f [jiaoqingbo] [KYUUBI #2244] load-kyuubi-env.sh should print SPARK_ENGINE_HOME for consistency
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
fix#2312
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2313 from Nick-0723/2312.
Closes#2312
91e607a8 [Nick Song] Spark data type TimestampNTZ supported version changes as 3.4.0
Authored-by: Nick Song <chun2184@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
If a session is empty which means that it does not contain any operation, then the OperationLog will never be closed.
This bug will happen if we enable `LaunchEngine`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2296 from ulysses-you/fix-file-handler-leak.
Closes#2296
22d07e1a [ulysses-you] fix
bb83b1a9 [ulysses-you] fix
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
To close#2250
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2295 from lightning-L/kyuubi-2250.
Closes#2250
71851bf8 [Tianlin Liao] [KYUUBI #2250] limit the spark engine max running time
7314df84 [Tianlin Liao] [KYUUBI #2250] add method to shutdown threadpool executor in ThreadUtils
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
In this PR, we narrow the scope of 8f15622d9b to support arbitrary parameters which configured in kyuubi-defaults.conf only.
We shall avoid propagating all `sys.props` of Kyuubi server to engine side, which is a dangerous behavior
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2283 from yaooqinn/2084.
Closes#2084
434db31c [Kent Yao] fix flink
bdd983d4 [Kent Yao] [KYUUBI #2084][FOLLOWUP] Support arbitrary parameters for KyuubiConf
78bada0b [Kent Yao] [KYUUBI #2084][FOLLOWUP] Support arbitrary parameters for KyuubiConf
55e6fdf1 [Kent Yao] [KYUUBI #2084][FOLLOWUP] Support arbitrary parameters for KyuubiConf
b2239bc7 [Kent Yao] [KYUUBI #2084][FOLLOWUP] Support arbitrary parameters for KyuubiConf
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Inline `kyuubi.` prefix in `KyuubiConf` to keep consistently with the document and make searching friendly.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2277 from pan3793/conf.
Closes#2277
cca77ccb [Cheng Pan] fixup
9c38925a [Cheng Pan] Inline kyuubi prefix in KyuubiConf
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
fix#2266
in 1.5.0 version，the seriviceUri consists of ip:port
<img width="576" alt="e3c7a03dbcbeeaedce52139755fe227" src="https://user-images.githubusercontent.com/14961757/161362120-61e45f93-d4fe-491b-ab06-9f02e78e70f3.png">
in 1.4.1 version ，the seriviceUri consists of hostname:port
<img width="591" alt="88cf334a0a425227ed6be7637fc10bd" src="https://user-images.githubusercontent.com/14961757/161364285-d8841387-6393-4280-9565-7d3ce419aa81.png">
using following command to connect would failed
./bin/beeline -u 'jdbc:hive2://ocdp63.asiainfo.com:2008,ocdp172.asiainfo.com:2008,ocdnfo.com:2008/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi;principal=spark/_HOSTocdp;'
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2269 from jiaoqingbo/2266.
Closes#2266
b77b19d8 [jiaoqingbo] modify doc
e87d9670 [jiaoqingbo] [KYUUBI #2266] The default value of frontend.connection.url.use.hostname should be set to true to be consistent with previous versions
25063a34 [jiaoqingbo] Merge branch 'master' of https://github.com/jiaoqingbo/incubator-kyuubi
0ad3b764 [jiaoqingbo] Merge branch 'master' of https://github.com/jiaoqingbo/incubator-kyuubi
f9d68c7f [jiaoqingbo] [KYUUBI #2244] load-kyuubi-env.sh should print SPARK_ENGINE_HOME for consistency
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Use case is simple:
```
// scribe sync
EventBus.register[Test0KyuubiEvent](new Test0Handler with EventHandler)
// scribe async
EventBus.registerAsync[Test0KyuubiEvent](new Test0Handler with EventHandler)
// send event
EventBus.post(KyuubiEvent("test0"))
EventBus.post(KyuubiEvent("test1"))
```
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#1853 from iodone/dev.
Closes#1798
7c017006 [odone] Updated
d2ecb428 [odone] [Fixed] address some reviews
09a467fa [odone] [Removed] Remove some mistakenly changed content
5136fe44 [odone] [Update] JsonLoggerEventHandler instead of LoggingService
29b4f8af [odone] [Update] JsonLoggerEventHandler instead of LoggingService
a2f90bbe [odone] [KYUUBI #1798] Added: Add async event handler execution
21be1357 [odone] Added: more test case
10906cfe [odone] Added: eventHandler can scribe subclass event
2c09b758 [odone] Added: Event Bus
bb05f77e [odone] Added: Event Bus
Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
support newly added data types: TimestampNTZType
since Spark3.3.0
### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2224 from Nick-0723/TimestampNTZType.
Closes#2207
56dfb2dd [Nick Song] replace to getSimpleName
10b12f33 [Nick Song] fix
90218ca6 [Nick Song] fix
348d7d0e [Nick Song] fix
54ef78d7 [Nick Song] support DayTimeIntervalType
Authored-by: Nick Song <chun2184@163.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
…y usage
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2210 from lightning-L/kyuubi-1021.
Closes#1021
6a2654ab [Tianlin Liao] [KYUUBI #1021] Expire CredentialsRef in a proper time to reduce memory usage
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
much code about maven options only has `-Pflink-provided,spark-provided` no `hive-provided`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2219 from deadwind4/KYUUBI-2218.
Closes#2218
5e369978 [Ada Wang] fix build dist help
a5de3ba8 [Ada Wang] [KYUUBI #2218] Fix maven options about hive-provided
Authored-by: Ada Wang <wang4luning@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Support to set result max rows for spark engine.
And it will only take affect if incremental mode is not enabled.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2225 from turboFei/max_res.
Closes#2225
bb2acd8b [Fei Wang] nit
5d827d11 [Fei Wang] address comments
3193d22e [Fei Wang] address comments
51983c35 [Fei Wang] address comments
2904b4ce [Fei Wang] address comments
f2a8aa62 [Fei Wang] fix docs
232f975c [Fei Wang] add ut
508b3560 [Fei Wang] Add engine spark max rows
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
To close#2102
Support to retry all the internal thrift request calls(except RenewDelegationToken now), and fast fail if the remote engine is not stable or not alive.
In this PR, it supports engine liveness probe.
If it is enabled, a companion thrift client will be created and open a liveness probe session when opening remote engine session.
It will send some simple thrift request(GetInfo) to check whether the remote engine is alive, and fast fail before retry if remote engine is not connectable.
#### Why not use the same thrift client to check engine liveness before retry?
I tried that, but met `out of resp sequence` error.
For example:
1. send getOperationStatus request
2. read time out
3. send GetInfoType request
4. receive getOperationStatus response (out of resp sequence)
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2122 from turboFei/retry_rpc.
Closes#2102
3926ba04 [Fei Wang] adress comments
ade4ede6 [Fei Wang] add timeout
1b7a64f9 [Fei Wang] Only check remote engine alive before retry
98e03f8e [Fei Wang] refactor
fac388cf [Fei Wang] remove unused import
9c6d8737 [Fei Wang] add ut
9b595650 [Fei Wang] Support to retry the thrift request and engine alive probe
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
#2207
Support newly added datatypes
spark3.2 : DayTimeIntervalType/YearMonthIntervalType
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2212 from Nick-0723/2207.
Closes#2207
c08ee994 [Nick Song] revert DayTimeIntervalType's part
abc56be5 [Nick Song] support newly data types DayTimeIntervalType/YearMonthIntervalType/TimestampNTZType
Authored-by: Nick Song <chun2184@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
In this PR, we handle newly added daytime interval type
TTypeId.INTERVAL_DAY_TIME_TYPE is used for compatible hive thrift type.
jdbc Type is java.sql.Types.OTHER
The data is converted from java duration to hive compatible string representation
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2199 from yaooqinn/2186.
Closes#2186
408954ec [Kent Yao] fix
11ad6311 [Kent Yao] interval day
8c329250 [Kent Yao] interval day
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: SteNicholas <programgeek@163.com>
### _Why are the changes needed?_
Hive Backend Engine - GetCrossReference.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2194 from KenjiFujima/KYUUBI-2033.
Closes#2033
01a0065f [KenjiFujima] [KYUUBI #2033] Hive Backend Engine - GetCrossReference
Authored-by: KenjiFujima <thanosxnicholas@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Based on this https://github.com/apache/incubator-kyuubi/pull/2162
Add several configuration items to control the output of the progress bar.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2168 from cxzl25/KYUUBI-2119-FOLLOWUP.
Closes#2119
36db8c5f [sychen] Add configuration progress bar
Authored-by: sychen <sychen@trip.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
execute simple scala code *** FAILED ***
scala.Predef.augmentString("\d\.\d\.\d").r.pattern.matcher(sparkVer).matches() was false (SparkQueryTests.scala:423)
#2187
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2191 from Nick-0723/fix_sparkversion_test.
Closes#2187
75ab29a0 [Nick Song] fix spark version test, spark master version with '-SNAPSHOT'
Authored-by: Nick Song <chun2184@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Hive Backend Engine - GetPrimaryKeys.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2177 from KenjiFujima/KYUUBI-2034.
Closes#2034
a5c99b6f [KenjiFujima] [KYUUBI #2034] Hive Backend Engine - GetPrimaryKeys
Authored-by: KenjiFujima <thanosxnicholas@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Add ProcBuilder for HiveEngine.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2146 from zhaomin1423/hive_process_builder.
Closes#2024
1baf1397 [Min Zhao] [KYUUBI #2024] Hive Backend Engine - ProcBuilder for HiveEngine
23fdc3b7 [Min Zhao] [KYUUBI #2024] Hive Backend Engine - ProcBuilder for HiveEngine
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Follow up #1936
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2092 from lightning-L/kyuubi-1936.
Closes#1936
2bb2c10d [Tianlin Liao] [KYUUBI #1936][FOLLOWUP] Send credentials when opening session and wait for completion
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
fix#2084
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2091 from jiaoqingbo/2084.
Closes#2084
624216fa [Cheng Pan] permission
c45a71ce [jiaoqingbo] delete empty line
29987bf1 [jiaoqingbo] fix ut failed
2e7f0acc [jiaoqingbo] Merge branch 'master' into 2084
04c0f8e1 [jiaoqingbo] Merge branch 'master' into 2084
1267d195 [jiaoqingbo] Merge branch '2084' of https://github.com/jiaoqingbo/incubator-kyuubi into 2084
1bb2b6d6 [jiaoqingbo] Merge branch 'master' into 2084
db4a4aac [jiaoqingbo] Merge branch 'master' into 2084
82097388 [jiaoqingbo] ignore java/sun prefix
d4640ec3 [jiaoqingbo] add log
d5af758f [jiaoqingbo] modify ut
0f3af6c6 [jiaoqingbo] modify ut case
9ffb4177 [jiaoqingbo] Merge branch 'master' into 2084
52dd5090 [jiaoqingbo] modify ut
3f881748 [jiaoqingbo] modify test conf
705f94af [jiaoqingbo] modify test conf
096fba1d [jiaoqingbo] modify test conf
83f1aba9 [jiaoqingbo] add more example configuration
d2581534 [jiaoqingbo] fix compile problem
65c3198a [jiaoqingbo] [KYUUBI #2084] Support arbitrary parameters for KyuubiConf
Lead-authored-by: jiaoqingbo <1178404354@qq.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
This PR reverts "[KYUUBI #1948] Upgrade thrift version to 0.16.0" in branch-1.5 and some subsequent commits due to issues caused by breaking changes in Thrift, e.g. #2101, #2093
We need more time to evaluate it and add more test cases to ensure stability before the next release.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2103 from pan3793/revert.
Closes#2103Closes#1948
575dfe8d [Cheng Pan] properties order
4c94481e [Cheng Pan] fix deps
bfbdeb89 [Cheng Pan] Add log4j deps for kyuubi-hive-jdbc test
bad4e90e [Cheng Pan] surefire
f2ae8b9b [Cheng Pan] Revert "[KYUUBI #1948] Upgrade thrift version to 0.16.0"
4501f743 [Cheng Pan] Revert "[KYUUBI #1948][FOLLOWUP] Remove TFramedTransport in extension-spark-3-1 and extension-spark-3-2"
cc04df49 [Cheng Pan] Revert "[KYUUBI #1996] Clean up log4j2 conf and enhance surefire plugin conf"
52a20e17 [Cheng Pan] Revert "[KYUUBI #1948][FOLLOWUP] Relocate fb303 classes"
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add description about trino in the config of engine.type.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2109 from zhaomin1423/fix_engine_type_doc.
Closes#2108
adf898c0 [Min Zhao] [Kyuubi #2108] Add description about trino in the config of engine.type
8f11514e [Min Zhao] [Kyuubi #2108] Add description about trino in the config of engine.type
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
We met below issue when fetching result.
```
Caused by: java.lang.RuntimeException: java.lang.ArrayIndexOutOfBoundsException:46
7022 at sun.util.calendar.BaseCalendar.getCalendarDateFromFixedDate(BaseCalendar.java:453)
7023 at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2397)
7024 at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2312)
7025 at java.util.Calendar.setTimeInMillis(Calendar.java:1804)
7026 at java.util.Calendar.setTime(Calendar.java:1770)
7027 at java.text.SimpleDateFormat.format(SimpleDateFormat.java:943)
7028 at java.text.SimpleDateFormat.format(SimpleDateFormat.java:936)
7029 at java.text.DateFormat.format(DateFormat.java:345)
7030 at org.apache.kyuubi.schema.RowSet$.toHiveString(RowSet.scala:245)
7031 at org.apache.kyuubi.schema.RowSet$.$anonfun$toTColumn$3(RowSet.scala:120)
7032 at scala.collection.immutable.List.map(List.scala:290)
7033 at org.apache.kyuubi.schema.RowSet$.toTColumn(RowSet.scala:115)
7034 at org.apache.kyuubi.schema.RowSet$.$anonfun$toColumnBasedSet$1(RowSet.scala:65)
7035 at org.apache.kyuubi.schema.RowSet$.$anonfun$toColumnBasedSet$1$adapted(RowSet.scala:64)
7036 at scala.collection.immutable.List.foreach(List.scala:392)
7037 at org.apache.kyuubi.schema.RowSet$.toColumnBasedSet(RowSet.scala:64)
7038 at org.apache.kyuubi.schema.RowSet$.toTRowSet(RowSet.scala:47)
7039 at org.apache.kyuubi.engine.spark.operation.SparkOperation.getNextRowSet(SparkOperation.scala:183)
7040 at org.apache.kyuubi.operation.OperationManager.getOperationNextRowSet(OperationManager.scala:116)
7041 at org.apache.kyuubi.session.AbstractSession.fetchResults(AbstractSession.scala:197)
7042 at org.apache.kyuubi.service.AbstractBackendService.fetchResults(AbstractBackendService.scala:169)
7043 at org.apache.kyuubi.service.ThriftBinaryFrontendService.FetchResults(ThriftBinaryFrontendService.scala:505)
```
The root cause is that the date time formatter used to convert the result to rowSet is not thread-safe.
In this pr, we use thread-safe FastDateFormat instead of SimpleDateFormat.
### _How was this patch tested?_
Existing UT.
Closes#2075 from turboFei/thread_safe_fat_stimeformatter.
Closes#2075
44ae8fd1 [Fei Wang] Using thread-safe FastDateFormat instead of SimpleDateFormat
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
fix#2066
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2067 from jiaoqingbo/2066.
Closes#2066
331c1812 [jiaoqingbo] [KYUUBI #2066] fix spelling mistake and appropriate naming
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
thanks to the pr original author yaooqinn https://github.com/apache/incubator-kyuubi/pull/1670
After this pr, the endpoint of OpenAPI will change from`http://${ip}:${port}/openapi.json` to `http://${ip}:${port}/api/openapi.json`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2061 from cfmcgrady/pr-1670.
Closes#2061
969b7d3d [Fu Chen] update dev/dependencyList
ff4b6bc2 [Fu Chen] fix ut
207553b6 [Fu Chen] addressed comment
a3baefe0 [Fu Chen] new line at end of file
ff36f957 [Fu Chen] fix
bb6abb3f [Fu Chen] language
b22be2ec [Fu Chen] style
f0c9562f [Fu Chen] Merge remote-tracking branch 'upstream/master' into pr-1670
d4ac4cba [Fu Chen] fix
cd5ee2b8 [Fu Chen] Merge remote-tracking branch 'upstream/master' into pr-1670
1b108a04 [Kent Yao] ui
85e31519 [Kent Yao] ui
0c20fcff [Kent Yao] ui
c2abde93 [Kent Yao] ui
b4dff513 [Kent Yao] ui
43dffab8 [Kent Yao] Merge branch 'master' into web
7cd273a3 [Kent Yao] swagger ui enabled
518882e2 [Kent Yao] swagger ui enabled
Lead-authored-by: Fu Chen <cfmcgrady@gmail.com>
Co-authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Support a lightweight hive engine which has some advantages over hiveserver2
- separate engine/process to the multi-tenancy framework for better stability, without having a heavy server
- more engines introduced in kyuubi, hive may be aged but still widely used
- also good for us to do some cross verifications for spark/flink etc with a stable hive.
In this PR, a basic hive engine backend is introduced.
- only engine side
- only execute statement is supported
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2013 from yaooqinn/hiveengine.
Closes#2011
75d146bc [Kent Yao] Merge branch 'master' into hiveengine
a7152598 [Kent Yao] addressed
b6c8ef59 [Kent Yao] llap/shims -> jdk:tools
07a9843a [Kent Yao] hbase client -> jdk:tools
3acb4aa7 [Kent Yao] Merge branch 'master' into hiveengine
ef5bcb8a [Kent Yao] codecov
b09e46a3 [Kent Yao] hbase client -> jdk:tools
56f74d50 [Kent Yao] hive
dcacffea [Kent Yao] hive
c46ffc41 [Kent Yao] hive
2f883036 [Kent Yao] hive
186dd33d [Kent Yao] hive
5038505f [Kent Yao] hive
5bd2a849 [Kent Yao] hive
180d892d [Kent Yao] hive
568a52b9 [Kent Yao] init hive engine
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
