### _Why are the changes needed?_
- To improve Scala code with corrections, simplification, scala style, redundancy cleaning-up. No feature changes introduced.
Corrections:
- Class doesn't correspond to file name (SparkListenerExtensionTest)
- Correct package name in ResultSetUtil and PySparkTests
Improvements:
- 'var' could be a 'val'
- GetOrElse(null) to orNull
Cleanup & Simplification:
- Redundant cast inspection
- Redundant collection conversion
- Simplify boolean expression
- Redundant new on case class
- Redundant return
- Unnecessary parentheses
- Unnecessary partial function
- Simplifiable empty check
- Anonymous function convertible to a method value
Scala Style:
- Constructing range for seq indices
- Get and getOrElse to getOrElse
- Convert expression to Single Abstract Method (SAM)
- Scala unnecessary semicolon inspection
- Map and getOrElse(false) to exists
- Map and flatten to flatMap
- Null initializer can be replaced by _
- scaladoc link to method
Other Improvements:
- Replace map and getOrElse(true) with forall
- Unit return type in the argument of map
- Size to length on arrays and strings
- Type check can be pattern matching
- Java mutator method accessed as parameterless
- Procedure syntax in method definition
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4959 from bowenliang123/scala-Improve.
Closes#4959
2d36ff351 [liangbowen] code improvement for Scala
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- comment https://github.com/apache/kyuubi/pull/4963#discussion_r1230490326
- simplify reflection calling with unified `invokeAs` / `getField` method for either declared, inherited, or static methods / fields
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4970 from bowenliang123/unify-invokeas.
Closes#4970
592833459 [liangbowen] Revert "dedicate invokeStaticAs method"
ad45ff3fd [liangbowen] dedicate invokeStaticAs method
f08528c0f [liangbowen] nit
42aeb9fcf [liangbowen] add ut case
b5b384120 [liangbowen] nit
072add599 [liangbowen] add ut
8d019ab35 [liangbowen] unified invokeAs and getField
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
It resolves https://github.com/apache/kyuubi/issues/4951.
It would be also possible to resolve it by adding log4j 1 slf4j binding (e.g., adding `slf4j-api-1.7.30.jar` and `slf4j-log4j12-1.7.30`) to the Spark jars directory, but it would be helpful if Kyuubi could support reload4j to avoid log4j 1's security vulnerabilities.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
Assume
- we use Spark SQL engine in YARN cluster deploy mode;
- we use Hadoop-free Spark 3.2.4 and provides the classpath of Hadoop in YARN cluster, which uses reload4j, with `SPARK_DIST_CLASSPATH`.
Before applying the patch, Hive beeline shows operation logs only from Kyuubi server, as follows.
```
0: jdbc:hive2://some_kyuubi_host> select * from some_database.some_table;
2023-06-14 22:55:35.584 INFO org.apache.kyuubi.credentials.HadoopCredentialsManager: Send new credentials with epoch 0 to SQL engine through session 5409e4cb-abe0-457f-b3c8-c3f1c5467c60
2023-06-14 22:55:35.604 INFO org.apache.kyuubi.credentials.HadoopCredentialsManager: Update session credentials epoch from -1 to 0
2023-06-14 22:55:35.662 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing some_user's query[013b8294-b395-4e3d-945e-3594b0603f83]: PENDING_STATE -> RUNNING_STATE, statement:
select * from some_database.some_table
2023-06-14 22:55:38.520 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[013b8294-b395-4e3d-945e-3594b0603f83] in FINISHED_STATE
2023-06-14 22:55:38.520 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing some_user's query[013b8294-b395-4e3d-945e-3594b0603f83]: RUNNING_STATE -> FINISHED_STATE, time taken: 2.857 seconds
+-----+
| id |
+-----+
| 1 |
+-----+
1 row selected (3.113 seconds)
```
After applying the patch, Hive beeline shows operation logs from both Kyuubi server and Spark SQL engine, as follows.
```
0: jdbc:hive2://some_kyuubi_host> select * from some_database.some_table;
2023-06-14 23:05:57.424 INFO org.apache.kyuubi.credentials.HadoopCredentialsManager: Send new credentials with epoch 0 to SQL engine through session 146aa800-6047-471e-a480-5683197a84cd
2023-06-14 23:05:57.453 INFO org.apache.kyuubi.credentials.HadoopCredentialsManager: Update session credentials epoch from -1 to 0
2023-06-14 23:05:57.512 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing some_user's query[1c209bea-6f62-4854-acdc-a1945867a65d]: PENDING_STATE -> RUNNING_STATE, statement:
select * from some_database.some_table
23/06/14 23:05:57 INFO operation.ExecuteStatement: Processing some_user's query[1c209bea-6f62-4854-acdc-a1945867a65d]: PENDING_STATE -> RUNNING_STATE, statement:
select * from some_database.some_table
23/06/14 23:05:57 INFO operation.ExecuteStatement:
Spark application name: kyuubi_CONNECTION_SPARK_SQL_some_user_146aa800-6047-471e-a480-5683197a84cd
application ID: application_1686295570868_0065
application web UI: some_yarn_application_url
master: yarn
deploy mode: cluster
version: 3.2.4
Start time: 2023-06-14T23:05:02.490
User: some_user
(omitted)
2023-06-14 23:06:00.285 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[1c209bea-6f62-4854-acdc-a1945867a65d] in FINISHED_STATE
2023-06-14 23:06:00.286 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing some_user's query[1c209bea-6f62-4854-acdc-a1945867a65d]: RUNNING_STATE -> FINISHED_STATE, time taken: 2.774 seconds
+-----+
| id |
+-----+
| 1 |
+-----+
1 row selected (3.156 seconds)
```
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4964 from touchida/issue-4951-reload4j.
Closes#4951
d7444be17 [toshihiko.uchida] [KYUUBI #4951] Add reload4j support
Authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close https://github.com/apache/kyuubi/issues/4881
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4886 from rhh777/jdbcengine.
Closes#4881
e0439f8e6 [haorenhui] [KYUUBI #4881] update settings.md
d667d8fe8 [haorenhui] [KYUUBI #4881] update conf/docs
cba06b4b9 [haorenhui] [KYUUBI #4881] simplify code
80be4d27c [haorenhui] [KYUUBI #4881] fix style
4f0fa3ab2 [haorenhui] [KYUUBI #4881] JDBCEngine performs initialization sql
Authored-by: haorenhui <haorenhui@kingsoft.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Apache Derby is no longer active. One major drawback of Derby is that it does not support multiple connections to single db files, making it hard to analyze the data on local development.
SQLite may be the most popular embedded DBMS in the world. It lives almost in every smartphone (at least Android and iOS integrate SQLite), which means SQLite is quite stable and may be a good choice for standalone production deployment.
SQLite provides a CLI command `sqlite3` which is easy to use to connect a data file and run queries. Multi connections to a single db file is allowed, which helps a lot to analyze the data when the Kyuubi server is running.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4950 from pan3793/sqlite.
Closes#4950
738c39e6a [Cheng Pan] Update docs/deployment/migration-guide.md
5facdad9c [Cheng Pan] Update docs/deployment/migration-guide.md
b9883489c [Cheng Pan] migrate default metadata store to sqlite
c785e1a77 [Cheng Pan] migrate jdbc auth test to sqlite
23f63b932 [Cheng Pan] introduce sqlite deps
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
For the operation getNextRowSet method, we shall add lock for it.
For example, for spark operation, the result iterator is not thread-safe, it might throw exception(if the jdbc client to kyuubi server connection socket timeout).
For incremental collect mode, the fetchResult might trigger a spark task to collect the incremental result(`self.next().toIterator`).
The jdbc client to kyuubi gateway timeout, but the fetchResult request has been sent to engine.
Then the jdbc client re-send the fetchResult request.
And the getNextResultSet in spark engine side concurrent execute.
And the result iterator is not thread-safe and might cause NPE.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4949 from turboFei/lock_next_rowset.
Closes#4949
8f18f3236 [fwang12] getNextRowSetInternal and withLockRequired
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
- apply the usage of `ReflectUtils` and `Dyn*` to the modules of engines and plugins (eg. Spark engine, Authz plugin, lineage plugin, beeline)
- remove similar redundant methods for calling reflected methods or getting field values
- unified reflection helper methods with type casting support, as `getField[T]` for getting field values from `getFields`, `invokeAs[T]` for invoking methods in `getMethods`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4879 from bowenliang123/reflect-use.
Closes#4879
c685fb67d [liangbowen] bug fix for "Cannot bind static field options" when executing "bin/beeline"
fc1fdf1de [liangbowen] import
59c3dd032 [liangbowen] comment
c435c131d [liangbowen] reflect util usage
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
to close https://github.com/apache/kyuubi/issues/4732
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4813 from huangzhir/operation_log.
Closes#4813
39dcab61d [huangzhir] remove unrelated code
545efaee6 [huangzhir] remove object lock
f0d090f9a [huangzhir] throw exception if user requests FETCH_PRIOR
162e008a1 [huangzhir] remove FETCH_PRIOR test
3b40f6bac [huangzhir] fix style
78e49698c [huangzhir] add extra log test
68154fecb [huangzhir] Merge remote-tracking branch 'origin/master' into operation_log
21c46c06c [huangzhir] code rewritten ,fetch log only support FETCH_NEXT and FETCH_FIRST
cbd714a2b [huangzhir] Add the operationHandle parameter to the /v1/operations/:operationId/log interface.
Authored-by: huangzhir <306824224@qq.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
### _Why are the changes needed?_
- Remove test annotations `org.apache.kyuubi.tags.*` which duplicated with `kyuubi-util-scala`'s in `kyuubi-common` module
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4911 from bowenliang123/remove-test-tags.
Closes#4911
8244410d5 [liangbowen] remove duplicated tags `org.apache.kyuubi.tags.*` in the common module
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- replacing callings to Scala's assert method by Scalatest's `Assertions.assert`
- While Scala's assert method just throws a simple Java's Assertion Error ,
```
def assert(assertion: Boolean) {
if (!assertion)
throw new java.lang.AssertionError("assertion failed")
}
```
the Scalatest's `Assertions.assert` prettifies the error message, eg.,
`assert(a == b || c >= d) // Error message: 1 did not equal 2, and 3 was not greater than or equal to 4`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4912 from bowenliang123/scalatest-assert.
Closes#4912
e1d2ce3e0 [liangbowen] use Scalatest's assert for better error message
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close https://github.com/apache/kyuubi/issues/4897
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4898 from huangzhir/operation_log_bug.
Closes#4898
5d630dd31 [huangzhir] use do while style
87fc2d23d [huangzhir] fix
6cb0e5561 [huangzhir] fix logOperation multiple read with missing line
Authored-by: huangzhir <306824224@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- add unit tests for `ClassUtils`
- refactor `ClassUtils.createInstance` method with DynConstructors
- move `classIsLoadable` method to `ReflectUtils.isClassLoadable`, refeactor to use DynClasses
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4887 from bowenliang123/classutil.
Closes#4887
c39f9a0a4 [liangbowen] simplify ut
633e21ddc [liangbowen] Refactor and add ut for ClassUtils
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- unify config value capitalization for String and Seq[String] configs
- Generalize `transformToUpperCase` and `transformToLowerCase` for config entry
- simplify transformation for configs of `kyuubi.authentication` and `kyuubi.frontend.protocols`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4812 from bowenliang123/conf-upper.
Closes#4812
747c2955c [bowenliang] upper and lower case for config values
Authored-by: bowenliang <bowenliang@apache.org>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Close#4870
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4872 from pan3793/util.
Closes#4870
0b9fe3cba [Cheng Pan] nit
ecc5ee4f2 [Cheng Pan] fix
63be7a20c [Cheng Pan] test
85363c187 [Cheng Pan] style
2227247dd [Cheng Pan] fix package
11d10a081 [Cheng Pan] Add kyuubi-util and kyuubi-util-scala modules
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
It became flaky on my laptop w/ JDK 17, wrap assertion w/ eventually to address it.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4857 from pan3793/flaky-1.
Closes#4857
8d4d3145f [Cheng Pan] Fix flaky test TFrontendServiceSuite
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- change the default EC spec from secp256k1 to secp521r1, as secp256k1 not supported on Java 17
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4850 from bowenliang123/secp384r1.
Closes#4850
73e0dad30 [liangbowen] change EC spec to secp521r1
5b9528ede [liangbowen] change EC spec to secp384r1
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
To provide more insights that the session is closed by requests or because of idle more than timeout.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4841 from turboFei/close_log.
Closes#4841
959a7ba79 [fwang12] log
0e3f3d609 [fwang12] save
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
To provide more insight, if there is uncaught exception.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4836 from turboFei/handler_exception.
Closes#4836
b9d304fb8 [fwang12] comment
3819447a6 [fwang12] un caughtt
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
As title.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4792 from turboFei/remove_unused.
Closes#4792
fe568af7e [fwang12] server conf
97f510020 [fwang12] save
c44e70a58 [fwang12] remove unused code
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
- introduce new event logger type `KAFKA`
- send server events to the Kafka topic with initializing and closing Kafka producer properly with server's lifecyle
- use Kafka 3.4.0 as the client version, and tested with Kakfa servers of 2.8.x and 3.4.x
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4733 from bowenliang123/kafka-logger.
Closes#4733
b5220d234 [liangbowen] introduce kafka server event logger
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
We shall use different engine submit timeout for different resource manager.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4801 from turboFei/engine_submit_timeout.
Closes#4801
e34852a64 [fwang12] nit
ad69008e7 [fwang12] 1.7.2
db11330c5 [fwang12] save
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
The REST enabled judgment will be used in other places, e.g. the developing batch API v2
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4791 from pan3793/conf.
Closes#4791
264566569 [Cheng Pan] Add helper method to simplify REST enabled judgement
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#4752
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4753 from lsm1/features/kyuubi_4752.
Closes#4753
8527a2bca [senmiaoliu] remove log deprecation warning in unset
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
`StaticLoggerBinder.getSingleton.getLoggerFactoryClassStr` is not valid in SLF4J, this is a missing place in the original PR.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4750 from pan3793/4407.
Closes#4407
6b8fe9603 [Cheng Pan] [KYUUBI #4407][FOLLOWUP] Adapt SLF4J2
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
We meet an issue that cause all the operation stuck when closing operation.
Because now all the operations try to lock a Scala Enumeration val.
And if one of them stuck, all the others will be keep stuck.
In this pr, I add a lock for each operation.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4739 from turboFei/op_lock.
Closes#4739
535400a42 [fwang12] revert
a93438927 [fwang12] lockInterruptibly
274abc9db [fwang12] utils
ceda7314f [fwang12] op lock
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
This PR aims to remove Hudi integration tests from the Kyuubi project.
Actually, there is no obvious benefit to running Hudi tests w/ Kyuubi, since the real work happens on the compute engine and Hudi integration. Besides, Hudi's horrible dependency management brings significant maintenance efforts to the Kyuubi community.
This change only affects tests, does not affect any functionality.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4744 from pan3793/remove-hudi.
Closes#4744
ea99f747e [Cheng Pan] Remove Hudi integration tests
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Now `GetCurrentCatalog`/`GetCurrentDatabase`/`SetCurrentCatalog`/`SetCurrentDatabase`is executed through the statement, and the jdbc client will try to obtain the operation log corresponding to the statement.
At present, these operations do not generate operation logs, so the engine log will be throw exception(`failed to generate operation log`).
```java
23/04/10 20:25:23 INFO GetCurrentCatalog: Processing anonymous's query[8218e7ed-b4a4-41ad-a1cc-6f82bf3d55bb]: INITIALIZED_STATE -> RUNNING_STATE, statement:
GetCurrentCatalog
23/04/10 20:25:23 INFO GetCurrentCatalog: Processing anonymous's query[8218e7ed-b4a4-41ad-a1cc-6f82bf3d55bb]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.002 seconds
23/04/10 20:25:23 ERROR SparkTBinaryFrontendService: Error fetching results:
org.apache.kyuubi.KyuubiSQLException: OperationHandle [8218e7ed-b4a4-41ad-a1cc-6f82bf3d55bb] failed to generate operation log
at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69)
at org.apache.kyuubi.operation.OperationManager.$anonfun$getOperationLogRowSet$2(OperationManager.scala:146)
at scala.Option.getOrElse(Option.scala:189)
```
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4688 from cxzl25/op_log_catalog.
Closes#4688
8ebc0f570 [sychen] Fix the failure to read the operation log after executing Catalog and database operation
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Now, with binduser and bindpw enabled, the user authentication pass is incompatible with the domain user.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4669 from dev-lpq/ldap_auth.
Closes#4669
b4d158718 [pengqli] LDAP authentication allows user contains domain UT
d365bdcb4 [pengqli] support LDAP authentication user has domain
Authored-by: pengqli <pengqli@cisco.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#4650
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4653 from lsm1/features/kyuubi_4650.
Closes#4653
79962aa16 [senmiaoliu] reformat
e4bb73281 [senmiaoliu] respect user log pattern
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Close#4623
To reduce the pressure on the Api Server (which use the kubernetes client polls with label to find the `spark driver pod` when multiple Rest Application are running at the same time), use informer, the kubernetes-recommended method of maintaining the application state.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
- [x] Run CI
Closes#4625 from zwangsheng/KYUUBI_4623.
Closes#4623
a415bef7f [Cheng Pan] nit
136d0db4d [Cheng Pan] 171
b5d3c237a [Cheng Pan] re-generate conf
bf14ad870 [Cheng Pan] nit
9ee7e04f9 [Cheng Pan] nit
301162ea0 [Cheng Pan] nit
1d426922b [Cheng Pan] nit
b95d7a650 [Cheng Pan] improve
cc8d2c7f4 [zwangsheng] fix comments
d017bafdf [zwangsheng] Set resycn 0
28f9a70d9 [zwangsheng] Reorder func & slow get app info
22d9c1662 [zwangsheng] fix setting
8e0940334 [zwangsheng] fix comments
10965d3df [zwangsheng] Rename fileter function => isSparkEnginePod
b02677154 [zwangsheng] rename
78c9fdb17 [zwangsheng] fix comments
6d31f70d1 [zwangsheng] Fix IT Test
f43bba2b9 [zwangsheng] fix
17e4f55eb [zwangsheng] debug
be8da790e [zwangsheng] debug
0db45a513 [zwangsheng] retest
a93786abc [zwangsheng] Fix style
652ee837e [zwangsheng] Add Setting & Debug
4add7e4e2 [zwangsheng] improve
1f4341237 [zwangsheng] remove unused import
35acd6106 [zwangsheng] fix compile
05dfc598e [zwangsheng] [KYUUBI #4623][Improvement][K8S] Remove cached app info when out of time
4ab530e99 [zwangsheng] [KYUUBI #4623][Improvement][K8S] kubernetesApplicationOperation Using Informer instead of list
Lead-authored-by: zwangsheng <2213335496@qq.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#4325
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4417 from yehere/kyuubi-4325.
Closes#4325
7b2864b53 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc,update the dependency file, run './build/dependency.sh --replace'
749b1c15c [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc,Code optimization
33ea9ba2b [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc,Code optimization
568418a21 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc,Add test case for DEALLOCATE PREPARE
358a8e3b8 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc,Support DEALLOCATE PREPARE
7d4a32402 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc
91392add6 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc
63bf8c462 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc
f5b7fb786 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc
b0476a79d [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc
1a8f147a0 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc
Authored-by: yehere <867171931@qq.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
Followup #1704, support flink `time` type in query operation
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4635 from yuruguo/support-flink-time-type.
Closes#4635
9f9a3e72d [Ruguo Yu] [Kyuubi #1704] Support flink time type in query operation
Authored-by: Ruguo Yu <jiang7chengzitc@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4541 from turboFei/expose.
Closes#4541
f882b4dda [fwang12] engine register attributes
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4545 from lightning-L/kyuubi-4523.
Closes#4523
2ee634cd5 [Tianlin Liao] modify conf description; add test case
12b930456 [Tianlin Liao] refer to HIVE-13415
c8a9db6a2 [Tianlin Liao] minor fix
9942d2515 [Tianlin Liao] move logic to SessionManager.closeSession
62b990b3e [Tianlin Liao] [KYUUBI #4523] support close engine session gracefully
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
- Make ChatGPT model ID configurable
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4558 from bowenliang123/chatgpt-model.
Closes#4558
63f8ee30d [liangbowen] nit
3012ccaaa [liangbowen] make chatgpt model configurable
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Support Chinese question
- Support proxy settings
- Support setting timeout
<img width="1228" alt="image" src="https://user-images.githubusercontent.com/3898450/225851246-8762a451-9743-4c1d-8a33-cc49a926dfec.png">
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4548 from cxzl25/chatgpt_followup.
Closes#4548
1d5715442 [Cheng Pan] Update externals/kyuubi-chat-engine/src/main/scala/org/apache/kyuubi/engine/chat/provider/ChatGPTProvider.scala
7add6a733 [Cheng Pan] Update externals/kyuubi-chat-engine/src/main/scala/org/apache/kyuubi/engine/chat/provider/ChatGPTProvider.scala
55974f298 [sychen] fix
2d360e102 [sychen] typo
19b5d0814 [sychen] doc
bdf8e29b6 [sychen] 1.utf8;2.proxy;timeout
Lead-authored-by: sychen <sychen@ctrip.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Introduce a brand new CHAT engine, it's supposed to support different backends, e.g. ChatGPT, 文心一言, etc.
This PR implements the following providers:
- ECHO, simply replies a welcome message.
- GPT: a.k.a ChatGPT, powered by OpenAI, which requires a API key for authentication. https://platform.openai.com/account/api-keys
Add the following configurations in `kyuubi-defaults.conf`
```
kyuubi.engine.chat.provider=[ECHO|GPT]
kyuubi.engine.chat.gpt.apiKey=<chat-gpt-api-key>
```
Open an ECHO beeline chat engine.
```
beeline -u 'jdbc:hive2://localhost:10009/?kyuubi.engine.type=CHAT;kyuubi.engine.chat.provider=ECHO'
```
```
Connecting to jdbc:hive2://localhost:10009/
Connected to: Kyuubi Chat Engine (version 1.8.0-SNAPSHOT)
Driver: Kyuubi Project Hive JDBC Client (version 1.7.0)
Beeline version 1.7.0 by Apache Kyuubi
0: jdbc:hive2://localhost:10009/> Hello, Kyuubi!;
+----------------------------------------+
| reply |
+----------------------------------------+
| This is ChatKyuubi, nice to meet you! |
+----------------------------------------+
1 row selected (0.397 seconds)
```
Open a ChatGPT beeline chat engine. (make sure your network can connect the open API and configure the API key)
```
beeline -u 'jdbc:hive2://localhost:10009/?kyuubi.engine.type=CHAT;kyuubi.engine.chat.provider=GPT'
```
<img width="1109" alt="image" src="https://user-images.githubusercontent.com/26535726/225813625-a002e6e2-3b0d-4194-b061-2e215d58ba94.png">
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4544 from pan3793/chatgpt.
Closes#4544
87bdebb6d [Cheng Pan] nit
f7dee18f3 [Cheng Pan] Update docs
9beb55162 [cxzl25] chat api (#1)
af38bdc7c [Cheng Pan] update docs
9aa6d83a6 [Cheng Pan] Initial implement Kyuubi Chat Engine
Lead-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: cxzl25 <cxzl25@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
As title and close#4537
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4538 from zwangsheng/KYUUBI_4537.
Closes#4537
2cae68044 [zwangsheng] [KYUUBI #4537] fix
19fc9bd8b [zwangsheng] [KYUUBI #4537] Set 0.0.0.0 for mac & windows
e8cff7853 [zwangsheng] [KYUUBI #4537] Set 0.0.0.0 for mac & windows
8af446ab0 [zwangsheng] [KYUUBI #4537] Remove test code
2897749ef [zwangsheng] [KYUUBI #4537] Remove test code
e714637e2 [zwangsheng] [KYUUBI #4537][Improvement][UI] Enabled Vite Server Proxy for developer work in local with front and backend server
Lead-authored-by: He Zhao <hezhao2@cisco.com>
Co-authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The following discussion assumes using Spark cluster mode w/ `waitCompletion=false`.
In Spark on Yarn, the application is visible immediately after `spark-submit` is returned, but things are different in Spark on K8s, Driver Pod is ephemerally invisible after submitting, so NOT_FOUND is returned instead of UNKNOWN or PENDING.
To tolerate the above case, `kyuubi.engine.submit.timeout` is introduced, ApplicationManager will report UNKNOWN instead of NOT_FOUND during the Driver Pod scheduling period.
More detail in #4467
1. Remove `KubernetesApplicationOperation`'s `JpsApplicationOperation` for handle Client Deploy Mode(`YarnApplicationOperation` doesn't handle this either)
2. Add engine submit timeout for `KubernetesApplicationOperation` to return Unknown status when not found driver pod in time range.
3. GetApplicationInfo with it's submit time
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4469 from zwangsheng/4467.
Closes#4467
562b67463 [zwangsheng] [KYUUBI #4467] Fix Setting.md
362c43d1b [zwangsheng] [KYUUBI #4467] Fix Setting.md
ac69f4d81 [zwangsheng] [KYUUBI #4467] Add Config Desc
d2b9fb660 [zwangsheng] [KYUUBI #4467] save tab
eac880fcf [zwangsheng] [KYUUBI #4467] Ingnore Kubernetes Operation for client mode test
7a20b97a4 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KubernetesApplicationOperation.scala
aa4c7716a [zwangsheng] [KYUUBI #4467] Ingnore Kubernetes Operation for client mode test
c5bd888ab [zwangsheng] [KYUUBI #4467] note it test
a86dcefba [zwangsheng] [KYUUBI #4467] Using default none
aed7f8794 [Cheng Pan] Update docs/deployment/settings.md
490df7dc0 [zwangsheng] [KYUUBI #4467] fix complie
33f3a5be8 [zwangsheng] [KYUUBI #4467] fix comments
4745790cf [zwangsheng] [KYUUBI #4467] Fix IT Test
924cfe38e [zwangsheng] [KYUUBI #4467] Fix Setting.md
5f8aeaacc [zwangsheng] [KYUUBI #4467] KubernetesApplicationOperation Wait if not fount driver pod in limit time range
Lead-authored-by: zwangsheng <2213335496@qq.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
REST frontend protocol was introduced in Kyuubi 1.4.0 https://github.com/apache/kyuubi/pull/1349, more and more users are using REST in recent days, and to simplify the planned Web UI developments, I think we should enable the REST by default for master(1.8.0-SNAPSHOT).
I do not remove the "experiment" because we don't have strong confident to ensure the API stability in this time, I would like to defer it until the community has confident to mark it as stable.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4483 from pan3793/enable-rest.
Closes#4483
66fef0224 [Cheng Pan] http
91a29b453 [Cheng Pan] nit
29a4826da [Cheng Pan] Enable REST frontend protocol in default
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Notify user fix setting.md with mvn opt ~`-DskipTests`~ `-Dtest=none` to skip unrelated java test.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4471 from zwangsheng/unit/skip_when_modify_setting.
Closes#4471
f0d7993f5 [zwangsheng] Put -Dtest=none befor
58bd24571 [zwangsheng] Skip Java Test Case with Recommand -Dtest=none
3f2ace38d [zwangsheng] Notifiy user fix setting md with skip unit test mvn opts
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Support to disable batch resource upload in server side for service stability.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4482 from turboFei/batches.
Closes#4482
b5cbfb506 [fwang12] version
6219ae2c9 [fwang12] save
e83963675 [fwang12] disable
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
- add dto for operation data, which is programming friendly
- show exception for session data
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4468 from turboFei/session_op_dto.
Closes#4439
fa905e70c [fwang12] fix ut
5c1c7c845 [fwang12] save
2d20215a0 [fwang12] comments
46cd2384e [fwang12] saev
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Fix variable usage issue in `SessionManager#stop`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4462 from Yikf/sessionmanager.
Closes#4462
d4340d4ec [Yikf] fix variable usage issue
Authored-by: Yikf <yikaifei@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
`GetTables` operation is too slow because it queries table details info one by one, but then only a table comment is used to construct a result row, which i think could be optional.
This PR add an optional config which can control this operation. By default, `GetTables` operation queries all message. Otherwise, `GetTables` operation just return table identifiers.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4444 from liaoyt/master.
Closes#4171
af5e60e36 [yeatsliao] rename config
0c9985e32 [yeatsliao] add doc
5e8687cb3 [yeatsliao] Supports ignore table comment when list all tables.
Authored-by: yeatsliao <liaoyt66066@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close#4439
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4440 from lightning-L/kyuubi-4439.
Closes#4439
14d3ebd80 [Tianlin Liao] [KYUUBI #4439] add list/close operation method for AdminResouce
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
This PR proposes to allow the user to provide a batch id on submitting a batch job. If the batch id already existed in metastore, Kyuubi ignores this submission and just returns the existing one, w/ a marker in response, this could avoid duplicated batch job submission.
Talking about the implementation, the key things are
How does the user set the custom batch id?
- User can optionally set the `kyuubi.batch.id` in `conf: Map[String, String]`, and the value must be a UUID, for Java users, it can be generated by `UUID.randomUUID().toString()`
How does the Kyuubi Server detect the duplication?
- It's simple in single Kyuubi Server instance case, Kyuubi just needs to look up the metastore before creating a batch job
- In HA mode, suppose the user requests to create the batch jobs w/ the same batch id concurrently, multiple Kyuubi Servers may process the request and try to insert to metastore DB at the same time, but only the first insertion success, others will fail w/ "duplicated key", Kyuubi Server needs to catch this error and return the existing batch job information instead of creating a new one.
How does the user know if the returned batch job is new created or duplicated?
- a new field `batchInfo: Map[String, String]` is added to the response, and for duplicated batch job, `"kyuubi.batch.duplicated": "true"` will be contained.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4390 from pan3793/batch-id.
Closes#4390
b6917babf [Cheng Pan] move constant to rest client
79ef1b5d8 [Cheng Pan] flaky test
f82228506 [Cheng Pan] it
88bdfa50a [Cheng Pan] ut
fd8bc222a [Cheng Pan] ut
c820f5e43 [Cheng Pan] Support user provided batch id on batch job submission
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The key changes are
1. allow disabling metadata operation retry
2. always fail fast on "duplicated key on unique index" error
Currently, when metadata operations failed, we always do async retry, to tolerate long-time metadata store outages w/o blocking the submission request, but it can not guarantee metadata consistency eventually, e.g. when inserting data violates the unique key restriction, it will never succeed, and block any following update request for the batch job, in such cases, the client gets succeed response but the metadata can not be updated correctly.
We should distinguish between recoverable and unrecoverable errors, for unrecoverable errors, we should fail fast, but the fact is it's hard to enumerate all recoverable nor unrecoverable errors, in this PR, we just enumerate the "duplicated key" as unrecoverable errors, and provide a switch to disable async retry so that the error can propagate to client correctly.
Some configurations are renamed w/ the `async.` prefix(the original key still takes effect) because we may introduce the sync retry logic in the future.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4418 from pan3793/sync-retry.
Closes#4418
ce58ac58c [Cheng Pan] revert migration-guide.md
c2d8377a4 [Cheng Pan] simplify
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This PR implements a simple `EngineSecuritySecretProvider` beside the existing zookeeper implementation, which simplifies the user threshold to use RESTful API w/ HA mode, and this PR also allows the user set `kyuubi.engine.security.secret.provider` using short name 'simple' or 'zookeeper' as well as the full class name.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4419 from pan3793/simple-secret-provider.
Closes#4419
32b1f966a [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
1af12b99d [Cheng Pan] nit
28a228eea [Cheng Pan] nit
65f14494c [Cheng Pan] Implement simple EngineSecuritySecretProvider
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
1. get spark engine runtime version instead of compile version
2. moved `PySparkTests` from the module `kyuubi-spark-sql-engine` to `kyuubi-server` to ensure that the python progress loading library PYSPARK has the same version as the launched Spark engine. see https://github.com/apache/kyuubi/pull/4381#issuecomment-1442871106
### _How was this patch tested?_
Pass Github Action.
Closes#4381 from cfmcgrady/spark-3.4.0.
Closes#4381
2711f51b3 [Fu Chen] remove verify spark-3.4 binary
a93b6d13e [Fu Chen] mv PySparkTests and enabled
6d5aad537 [Fu Chen] fix style
2da641561 [Fu Chen] fix style
3c9e300ce [Fu Chen] spark compile version -> runtime version
a8e7b7481 [Fu Chen] unused import
6be502ca6 [Fu Chen] fix ut
c1a1e1a8e [Fu Chen] skip pyspark tests
0049c23b7 [Fu Chen] verify spark-3.4.0 binary
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Close#4376
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4405 from lightning-L/kyuubi-4376.
Closes#4376
1a01a75a8 [Tianlin Liao] rename and refactor
7324cab3d [Tianlin Liao] [KYUUBI #4376] Support to config the kyuubi service administrator with kyuubi conf
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4415 from turboFei/operation_handle_align.
Closes#4415
71721797a [fwang12] refactor
c8b667a89 [fwang12] refactor
d3fbf05f3 [fwang12] stmt handle
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Align the server session handle and engine session handle for Spark engine.
It make it easy to recovery the engine session in any kyuubi instance easy.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4412 from turboFei/server_engine_handle_align.
Closes#4412
a20e0f155 [fwang12] fix
9d590e38b [fwang12] fix
94267e583 [fwang12] save
7012c2bef [fwang12] align
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4398 from turboFei/rename_batch_limit.
Closes#3957
28228e4ed [fwang12] 3957 followup
ef1ad6ea5 [fwang12] save
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4395 from lightning-L/kyuubi-4388.
Closes#4388
4f4ae8e59 [Tianlin Liao] create as optional
6b85f6ae6 [Tianlin Liao] refactor
7c2cd5661 [Tianlin Liao] rename conf
2f00da0f4 [Tianlin Liao] throw error in AbstractBackendService to enable it for all kinds of clients
3feabf78c [Tianlin Liao] [KYUUBI #4388] Limit the max rows for get nextRowSet api
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
As discussed https://github.com/apache/kyuubi/pull/4144#discussion_r1112957149, we should put the conf key into namespace `kyuubi.batch.`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4394 from pan3793/4106-followup.
Closes#4106
27c9ed870 [Cheng Pan] nit
66283ac4b [Cheng Pan] [KYUUBI #4106][FOLLOUP] Rename conf key of uploaded batch resource
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4382 from lightning-L/kyuubi-4352.
Closes#4352
2beb1ef40 [Tianlin Liao] update config name
a9eb126d8 [Tianlin Liao] update config setting
75cdfbb02 [Tianlin Liao] [KYUUBI #4352] Support System.gc() with periodic GC interval
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Filter out the LDAP configurations to suppress potential security issues.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4383 from pan3793/auth-conf.
Closes#4152
032245f8 [Cheng Pan] [KYUUBI #4152] LDAP configurations should be server-only
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
To close#4216
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4370 from turboFei/client_version.
Closes#4216
efd934c6b [fwang12] save
cf8acd54a [fwang12] version properties
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
1. this PR introduces a new configuration called `kyuubi.operation.result.arrow.timestampAsString`, when true, arrow-based rowsets will convert timestamp-type columns to strings for transmission.
2. `kyuubi.operation.result.arrow.timestampAsString` default setting to false for better transmission performance
3. the PR fixes timezone issue in arrow based result format described in #3958
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4326 from cfmcgrady/arrow-string-ts.
Closes#4326
38c7fc9b [Fu Chen] fix style
d864db00 [Fu Chen] address comment
b714b3ee [Fu Chen] revert externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/schema/RowSet.scala
6c4eb507 [Fu Chen] minor
289b6007 [Fu Chen] timstampAsString = false by default
78b7caba [Fu Chen] fix
f5601356 [Fu Chen] debug info
b8e4b288 [Fu Chen] fix ut
87c6f9ef [Fu Chen] update docs
86f6cb73 [Fu Chen] arrow based rowset timestamp as string
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Support to refresh the unlimited users for session limiter, so that we can unblock some customers without restart the kyuubi server.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4360 from turboFei/limier_whitelist.
Closes#4360
c846148bd [fwang12] typo
839e71365 [fwang12] nit
904bc9fbc [fwang12] save
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
This PR proposes to fix#4282, since log4j2 supports async mode, we need to make sure the `Log4j2DivertAppender#append` is thread-safe.
This PR also changes `OperationLog.getCurrentOperationLog` from `OperationLog` to `Option[OperationLog]`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4300 from pan3793/log.
Closes#4282
010e34b0 [Cheng Pan] fix
068405b2 [Cheng Pan] fix compile
c79dedd5 [Cheng Pan] Use write lock instead
3daf8a4d [Cheng Pan] nit
94176a04 [Cheng Pan] [KYUUBI #4282] Fix ConcurrentModificationException when log4j2 async enabled
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
It can help to know the backend pressure if the exec pool is full.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4344 from turboFei/wait_queue.
Closes#4344
161e3808a [fwang12] nit
6d122e238 [fwang12] save
55a4b499d [fwang12] version
668ff8bfe [fwang12] save
9f56b98a8 [fwang12] save
a401771ec [fwang12] wait
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
according to `io.trino.jdbc.ColumnInfo`, there are some type requring signature parameter.
- varchar(n)
- char(n)
- decimal(precision, scale)
It failed with trino jdbc now
<img width="613" alt="image" src="https://user-images.githubusercontent.com/12025282/218707052-a2e9dc91-0333-483c-bc0a-96baec213578.png">
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4328 from ulysses-you/fix-signature.
Closes#4328
aede5cec [ulysses-you] nit
ae1a7968 [ulysses-you] fix test
8ecdb346 [ulysses-you] Make Trino jdbc driver work
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
### _Why are the changes needed?_
This PR proposes to use `org.apache.spark.sql.execution#toHiveString` to replace `org.apache.kyuubi.engine.spark.schema#toHiveString` to get consistent result w/ `spark-sql` and `STS`.
Because of [SPARK-32006](https://issues.apache.org/jira/browse/SPARK-32006), it only works w/ Spark 3.1 and above.
The patch takes effects on both thrift and arrow result format.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
```
➜ ~ beeline -u 'jdbc:hive2://0.0.0.0:10009/default'
Connecting to jdbc:hive2://0.0.0.0:10009/default
Connected to: Spark SQL (version 3.3.1)
Driver: Hive JDBC (version 2.3.9)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 2.3.9 by Apache Hive
0: jdbc:hive2://0.0.0.0:10009/default> select to_timestamp('2023-02-08 22:17:33.123456789');
+----------------------------------------------+
| to_timestamp(2023-02-08 22:17:33.123456789) |
+----------------------------------------------+
| 2023-02-08 22:17:33.123456 |
+----------------------------------------------+
1 row selected (0.415 seconds)
```
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4318 from pan3793/hive-string.
Closes#4316
ba9016f6 [Cheng Pan] nit
8be774b4 [Cheng Pan] nit
bd696fe3 [Cheng Pan] nit
b5cf051c [Cheng Pan] fix
dd6b7021 [Cheng Pan] test
63edd34d [Cheng Pan] nit
37cc70af [Cheng Pan] Fix python ut
c66ad22d [Cheng Pan] [KYUUBI #4316] Fix returned Timestamp values may lose precision
41d94445 [Cheng Pan] Revert "[KYUUBI #3958] Fix Spark session timezone format"
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Close#3935
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4232 from iodone/kyuubi-3935.
Closes#3935
936ea1f8 [odone] address
e7bd01a1 [odone] support trino client connect kyuubi trino server
9ea8b6af [odone] [WIP] trion request/response implementation
Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
### _Why are the changes needed?_
This PR proposes to change the paths of distributed lock and seq_num(used for POLLING engine pool select policy) on the Service Discovery component. The reason is that namespace `${serverSpace}_${KYUUBI_VERSION}_${shareLevel}_${engineType}/` should be dedicated to engine registration, we'd better use the separated namespace for other functionalities.
- lock path
```
# before
${serverSpace}_${shareLevel}_${engineType}/lock/${user}/${subdomain}
# after
${serverSpace}_${KYUUBI_VERSION}_${shareLevel}_${engineType}_lock/${user}/${subdomain}
```
- seq_num
```
# before
${serverSpace}_${KYUUBI_VERSION}_${shareLevel}_${engineType}/seq_num/${user}/${poolName}
# after
${serverSpace}_${KYUUBI_VERSION}_${shareLevel}_${engineType}_seqNum/${user}/${poolName}
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4145 from pan3793/namespace.
Closes#4145
c912b3f66 [Cheng Pan] name
3326b9b95 [Cheng Pan] name
10083db0b [Cheng Pan] Change lock and polloing seq_num path on service discovery
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
`kyuubi.operation.result.max.rows` does not take effect on incremental collect mode because of performance concerns, this PR updates the configuration docs to mention that.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4271 from pan3793/maxrow.
Closes#4271
29b290a3e [Cheng Pan] nit
3d2872352 [Cheng Pan] log
277ebb5ff [Cheng Pan] ifx
091511a91 [Cheng Pan] nit
f15fb2270 [Cheng Pan] nit
1a1259ef5 [Cheng Pan] nit
ba57a2660 [Cheng Pan] fix
0b58d8b1a [Cheng Pan] ut
74b59dcee [Cheng Pan] nit
230defbff [Cheng Pan] Increamental collect mode should respect kyuubi.operation.result.max.rows
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- as pyspark is supported in #3780, exposing PYTHON option for `kyuubi.operation.language` as an experimental feature
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4256 from bowenliang123/doc-language-python.
Closes#4256
d10c0281 [liangbowen] update description
d05cedda [liangbowen] add PYTHON option in kyuubi.operation.language
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Note:
- No output changes to existed generated docs as in `settings.md` and `functions.md`.
Improvement:
- readability in Scala code for doc auto-generation, and easy to maintain docs in group of sections
- reline on markdown linting from `flexmark`, and reducing un-meaningful blank lines or alignment
- declaration over operation by replacing repeated usages of `newOutput` itself by elegantly wrapped `.line`,`.lines`and etc.
- less fragile and more handy for changing in a long single line, as now using auto margin stripping in `.line()` method
- reusable extracted licence and auto-generation hints
- more elegant and safer way to read and appending file content
- possible less memory footprint by apply operators to Stream instead of to ArrayBuffer
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4244 from bowenliang123/config-regen.
Closes#4244
3d90ad304 [liangbowen] make buffer private in MarkdownBuilder
8250a55f3 [liangbowen] remove licence.md
a4c7baf78 [liangbowen] add MarkdownBuilder.apply
248a046a4 [liangbowen] Improvement in auto-generated Markdown docs with MarkdownBuilder
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Before:
```
2023-02-02 22:21:35.507 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: SSL Server Socket enabled protocols: [Ljava.lang.String;2fca3eb5
```
Seems `enabledProtocols` is a java array, the output is not friendly.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4238 from turboFei/ssl_followup.
Closes#3658
999ef90e6 [fwang12] Show enabled ssl protocols
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Address comments: https://github.com/apache/kyuubi/pull/4144#issuecomment-1412078077
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4231 from turboFei/uploading_resource_followup.
Closes#4144
93ccd9534 [fwang12] comments
81a224dce [fwang12] ut
6b226969a [fwang12] prevent temp file leak
17a4e394c [fwang12] Do not remove upload dir
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
This PR proposes to enhance the LDAP support, which mainly referring the code introduced in HIVE-14713.
Currently, Kyuubi has very limited LDAP support, and the implementation is from the early Hive codebase. Hive enhanced the LDAP support in later versions, considering the Hive ecosystem is quite mature, I think we'd better to porting this functionality and keep the same behavior w/ Hive first, and we can improve it if meet certain requirements/issues in the future.
Basically, this PR introduces the following configurations
```
kyuubi.authentication.ldap.url (since 1.0.0)
kyuubi.authentication.ldap.domain (since 1.0.0)
kyuubi.authentication.ldap.guidKey (since 1.2.0)
kyuubi.authentication.ldap.base.dn (since 1.0.0 deprecated)
kyuubi.authentication.ldap.baseDN
kyuubi.authentication.ldap.groupMembershipKey
kyuubi.authentication.ldap.userMembershipKey
kyuubi.authentication.ldap.groupClassKey
kyuubi.authentication.ldap.groupDNPattern
kyuubi.authentication.ldap.userDNPattern
kyuubi.authentication.ldap.groupFilter
kyuubi.authentication.ldap.userFilter
kyuubi.authentication.ldap.customLDAPQuery
kyuubi.authentication.ldap.binddn
kyuubi.authentication.ldap.bindpw
```
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
This PR ports all LDAP-related UT&IT from Hive codebase
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4152 from pan3793/ldap.
Closes#4152
d251c959 [Cheng Pan] nit
6d14f44b [Cheng Pan] nit
6b3d116c [Cheng Pan] nit
ab47d822 [Cheng Pan] nit
a56e8702 [Cheng Pan] nit
4624619a [Cheng Pan] nit
b82c0c05 [Cheng Pan] LDAP test password uses alphanumeric
86a01cca [Cheng Pan] Enhance LDAP authentication
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- Show line number and assertion when verifying generated markdown files, e.g. `settings.md` from `AllKyuubiConfiguration`, for quicker locating the place and differences in files.
- updated hint message to `mvn clean test` instead of `mvn clean install` for faster verification or regeneration
Hints with line num as below if assertion fails in line comparison.
<img width="1121" alt="image" src="https://user-images.githubusercontent.com/1935105/215451115-813b90f0-9d9d-4ebd-974e-8a071424aa42.png">
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4212 from bowenliang123/markdown-verify-linenum.
Closes#4212
82791a88 [liangbowen] style
609c3b35 [liangbowen] update hints
ed889915 [liangbowen] nit
c676653b [liangbowen] updated hints of markdown generation for AllKyuubiConfiguration and KyuubiDefinedFunctionSuite
646832a0 [liangbowen] show line number and content of expected and got when verifying markdown files
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- to consolidate styles in markdown files from manual written or auto-generated
- apply markdown formatting rules with flexmark from [spotless-maven-plugin](https://github.com/diffplug/spotless/tree/main/plugin-maven#markdown) to *.md files in `/docs`
- use `flexmark` to format markdown generation in `TestUtils` of common module used by `AllKyuubiConfiguration` and `KyuubiDefinedFunctionSuite`, as the same way in `FlexmarkFormatterFunc ` of `spotless-maven-plugin` using with `COMMONMARK` as `FORMATTER_EMULATION_PROFILE` (https://github.com/diffplug/spotless/blob/maven/2.30.0/lib/src/flexmark/java/com/diffplug/spotless/glue/markdown/FlexmarkFormatterFunc.java)
- using `flexmark` of` 0.62.2`, as the last version requiring Java 8+ (checked from pom file and bytecode version)
```
<markdown>
<includes>
<include>docs/**/*.md</include>
</includes>
<flexmark></flexmark>
</markdown>
```
- Changes applied to markdown doc files,
- no style change or breakings in built docs by `make html`
- removal all the first blank in licences and comments to conform markdown style rules
- tables regenerated by flexmark following as in [GitHub Flavored Markdown](https://help.github.com/articles/organizing-information-with-tables/) (https://github.com/vsch/flexmark-java/wiki/Extensions#tables)
### _How was this patch tested?_
- [x] regenerate docs using `make html` successfully and check all the markdown pages available
- [x] regenerate `settings.md` and `functions.md` by `AllKyuubiConfiguration` and `KyuubiDefinedFunctionSuite`, and pass the checks by both themselves and spotless check via `dev/reformat`
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4200 from bowenliang123/markdown-formatting.
Closes#4200
1eeafce4 [liangbowen] revert minor changes in AllKyuubiConfiguration
4f892857 [liangbowen] use flexmark in markdown doc generation
8c978abd [liangbowen] changes on markdown files
a9190556 [liangbowen] apply markdown formatting rules with `spotless-maven-plugin` to markdown files with in `/docs`
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
to close#4106 .
1. add `POST /batches`API in `BatchesResource` of REST API, which consumes `multipart/form-data` media type to support uploading `resourcefile`. And a `batchRequest` form data part in the JSON format string of `BatchRequest`, as in the required request body of `POST /batches`
2. the uploaded `resourceFile` is saved to a temp local file which will be cleaned up after job execution at the end of `submitAndMonitorBatchJob`
3. the local temp copy of `resourceFile` will be used as `resource` in `BatchJobSubmission`, eg. as <application-jar> for spark-submit
Todos in follow-up:
1. add a related description in Rest API doc
4. add `multipart` media type support to `RestClient` and implement `createBatchWithUploadingResource` in `BatchRestApi`
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4144 from bowenliang123/rest-batch-upload.
Closes#4106
f6723a02 [Bowen Liang] Merge branch 'master' into rest-batch-upload
0dd67245 [liangbowen] correct dependencyList
6365a0cd [liangbowen] introducing resource file upload support in batch creation
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Add catalog tree for `kyuubi.authentication` options
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
<img width="746" alt="image" src="https://user-images.githubusercontent.com/26535726/212236908-294b7fa9-e3db-4730-8c16-41c8fd409ae8.png">
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4159 from pan3793/auth-doc.
Closes#4159
8d65768d [Cheng Pan] [DOCS] Refine doc for kyuubi.authentication
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- commit date is more readable and helpful to determine earlier/later version, especially for snapshot branch
- adding revision commit date from git to `kyuubi-version-info.properties`
- Print revision date on server startup
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4187 from bowenliang123/print-revision-date.
Closes#4187
b0b0a6c8 [liangbowen] print revision date on server startup
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
As Kyuubi graduated as top level project, the setting page will be more often requested and should be increasingly reliable and readable with less grammar and spelling mistakes.
This PR is to
- correct mistakes in grammar, spelling, abbreviation and terminology
- with no config name or essential meanings changed
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4161 from bowenliang123/conf-grammar.
Closes#4161
038edfbea [liangbowen] nit
1ec073a4b [liangbowen] to JSON
4f5259a32 [liangbowen] to Prometheus
523855008 [liangbowen] to K8s
fc7a3a81e [liangbowen] to AUTO-GENERATED
da64f54fa [liangbowen] update
d54f9a528 [liangbowen] fix `comma separated` to `comma-separated`
f1d7cc1f1 [liangbowen] update
d84208844 [liangbowen] update
1b75f011c [liangbowen] correction of grammar and spelling mistakes
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
The output of building and testing is now brought with repeated unuseful warnings from `javac`.
This PR reduces over 500 unhelpful warning message lines from `-Xlint` rules in `javac` compilation output in a single run, by eliminating :
- 74 of `deprecation`, by adding `SuppressWarning` to 6 Java classes
- 6 of `rawtypes`, by adding `SuppressWarning` to methods of Java class
- 4 of `unchecked`, by adding `SuppressWarning` to methods of Java class
- 1 of `fallthrough`, by adding `SuppressWarning` to method of Java class
- 1 of `cast`, by removal of redundant type casting
- repeated `processing` for, by adding `-processing` to disable annotation processor warning for annotations from testing framework :
```
- Warning: Unexpected javac output: warning: No processor claimed any of these annotations: /org.scalatest.TagAnnotation
Warning: Unexpected javac output: warning: No processor claimed any of these annotations: /org.junit.Test,/org.junit.Before,/org.junit.After
```
As 1 [fallthrough] warning message remain, which will be reported in followup issues or PR to fix missing breaks for branches of `switch-case` in Java.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4166 from bowenliang123/java-warnings.
Closes#4166
ec4cfc42 [liangbowen] add @SuppressWarnings("fallthrough") to KyuubiConnection#waitLaunchEngineToComplete
c9a59522 [liangbowen] skip deprecation warnings for SQLResultSet and KyuubiBaseResultSet
cab8c358 [liangbowen] remove redundant casting
f2ce8963 [liangbowen] style
bf6c5936 [liangbowen] add `-processing` to javac Xlint option for eliminating the missing annotation processer warnings
01a89a8c [liangbowen] eliminate warning outputs in java by adding @SuppressWarnings
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Support to ignore subdomain when engine pool conditions are met.
It is difficult to understand the engine routing rule for customers, especially for the engine pool and subdomain.
Maybe a user use subdomain before, and then they want to use engine pool.
Then specify the engine pool name, engine pool size, but the result is not expected.
They need to remove the subdomain specified before, otherwise, the engine pool will not work.
I need tell customer that, the engine subdomain config has higher priority and tell them the routing rule.
The understanding cost is high.
With this change, it is clear for engine pool conditions.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4151 from turboFei/engine_pool_enabled.
Closes#4151
cbb7846ff [fwang12] comments
9f4a696ae [fwang12] refactor
9801dc85e [fwang12] save
e5f769874 [fwang12] engine pool first
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4153 from turboFei/log_user_close.
Closes#4153
d55cbcdc6 [fwang12] logSessionCountInfo
7cdb1f45f [fwang12] log more session info
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
to close issue #4058 and help to close pr #3852.
When execute test `KyuubiOnKubernetesWithClientSparkTestsSuite` with `execute statement - select with variable substitution`, the initial method
`org.apache.kyuubi.operation.HiveJDBCTestHelper#withThriftClient` will overlook the connectionConf in `org.apache.kyuubi.operation.JDBCTestHelper#jdbcUrl`:
```
def withThriftClient[T](user: Option[String] = None)(f: TCLIService.Iface => T): T = {
TClientTestUtils.withThriftClient(
jdbcUrl.stripPrefix(URL_PREFIX).split("/;").head,
user)(f)`
}
```
Here we just aborted the tail of `jdbcUrl.stripPrefix(URL_PREFIX).split("/;")`, in which the connectionConfs are set.
In that case the `org.apache.kyuubi.kubernetes.test.deployment.KyuubiOnKubernetesWithClientSparkTestsSuite#connectionConf` was invalid actually.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4059 from xuefeimiaoao/branch-fix-connectionConf.
Closes#4058
a6395631 [xuefeimiaoao] [KYUUBI #4058] [IT][Test][K8S] Fix the missing of connectionConf of SparkQueryTests
Authored-by: xuefeimiaoao <1255072085@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
After some offline discussions, I propose to change the configuration key from`kyuubi.operation.result.codec` to `kyuubi.operation.result.format`.
### _How was this patch tested?_
Pass CI.
Closes#4075 from cfmcgrady/arrow-conf-rename.
Closes#4075
5ad45507 [Fu Chen] fix
214b43fd [Fu Chen] rename
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close#3982 .
Introduce feature of refresh user defaults config (as `___${user}___.*` which starts with three continuous underscores "___") from config file via `kyuubi-admin` cli and `refresh/user_defaults_conf` Rest API.
1. add `refreshUserDefaultsConf` methond in KyuubiServer to read user defautls configs from property file and apply config changes to server's KyuubiConf
3. add `refresh/user_defaults_conf` api to AdminRestApi calling `refreshUserDefaultsConf` of KyuubiServer
3. add config type `userDefautls` in kyuubi-admin cli refresh command
This feature will
- help to apply user defaults conf without restarting server or losing connections
- load latest config for engine launch, e.g. spark related config `spark.*`
It won't
- affect the components already started and using the clone of server conf
- affect configs for launched engine instance
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3983 from bowenliang123/3982-reload-server-conf.
Closes#3983
a8fb0bf2 [liangbowen] fix typo
b9b80f56 [liangbowen] update
b47a1541 [liangbowen] minor
786cb2a0 [liangbowen] add logging for statistics
0860e3a7 [liangbowen] fix: loadFileDefaults in refreshUserDefaultsConf
8dbbbcb8 [liangbowen] fix typo
619acd2e [liangbowen] import
e405dc8e [liangbowen] fix user defaults key filtering by adding `getAllUserDefaults` to `KyuubiConf`
ac407bd5 [liangbowen] rename config refresh option to `userDefaultsConf` and extracted to RefreshConfigCommandConfigType
c65398b2 [liangbowen] fix redundant loadFileDefaults in refreshUserDefaultsConf
1b046feb [liangbowen] update comments
e5dd5dbb [liangbowen] typo
c0a358ba [liangbowen] change to refresh users' config. rename cli command to `refresh config userDefaults`
bf5448e3 [liangbowen] support reload server config from config file 1. add reloadServerConf in KyuubiServer to read config and put all to config, 2. add "refresh/server_conf" api to AdminRestApi, 3. add config type "serverConf" in kyuubi-admin cli
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4069 from lightning-L/kyuubi-4020.
Closes#4020
97406ca0 [Tianlin Liao] [KYUUBI #4020] remove incubating from kyuubi source code
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Unlike `kyuubi.frontend.bind.port`, which is deprecated and replaced by `kyuubi.frontend.thrift.binary.bind.port`, `kyuubi.frontend.bind.host` is still encouraged to use, and it can be overwritten by `kyuubi.frontend.<protocol>.bind.host`, e.g. `kyuubi.frontend.thrift.binary.bind.host`, `kyuubi.frontend.thrift.http.bind.host`, `kyuubi.frontend.rest.bind.host`, `kyuubi.frontend.mysql.bind.host`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4005 from pan3793/doc.
Closes#4005
9c3b1277 [Cheng Pan] comments
0ef140e4 [Cheng Pan] Remove deprecated from description of kyuubi.frontend.bind.host
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4012 from cxzl25/remove_okStatusWithHint.
Closes#4012
c9eb482e [sychen] remove unused okStatusWithHint method
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
close#3905
### _Why are the changes needed?_
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3977 from iodone/kyuubi-3905.
Closes#3977Closes#3905
4e4a22b1 [odone] trino server init and run
Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
After some thought and some offline discussions, I propose to change the configuration key from `kyuubi.engine.pool.balance.policy` to `kyuubi.engine.pool.selectPolicy`.
And in the future if we introduce properties for specific policy, e.g. supposing we have a engine selection policy named `LOAD`, the properties would be
```
kyuubi.engine.pool.selectPolicy.load.abc=xxxx
kyuubi.engine.pool.selectPolicy.load.xyz=xxxx
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3966 from pan3793/rename.
Closes#2887
7c1eda70 [Cheng Pan] address comments
bf3a3863 [Cheng Pan] [KYUUBI #2887][FOLLOWUP] Rename configuration to kyuubi.engine.pool.selectPolicy
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Followed by issue https://github.com/apache/incubator-kyuubi/issues/3930 and pr https://github.com/apache/incubator-kyuubi/pull/3932.
- correct ut in SessionSigningSuite by fetching session public key and user sign via scala scripts instead of `SET` command in SQL , to prevent accidently redacted by `spark.redaction.regex`
- removed `syncronized` in SignUtils introduced in pr #3932 , with confirming no thread safe problem during the signing and verification
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3974 from bowenliang123/3932-fix2.
Closes#3930
c5f07b95 [liangbowen] comment
1f4ace3a [liangbowen] use KyuubiReservedKeys.KYUUBI_SESSION_USER_SIGN/KYUUBI_SESSION_SIGN_PUBLICKEY for property name
a034cbec [liangbowen] style
1f127c10 [liangbowen] remove `synchronized` from `signWithPrivateKey` and `verifySignWithECDSA` in SignUtils
ff5a21d5 [liangbowen] fix ut by getting `kyuubi.session.sign.publickey` and `kyuubi.session.user.sign` via `spark.sparkContext.getLocalProperty` in scala scripts.
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
to close#3915
This pr adds support for jdbc client detecting result set codec
1. in this PR, hints are added in the `TStatus.getInfoMessages()` to return, and the hints were added when the client retrieves the result set schema from the server
2. the hints mechanism is a general extension when we need to change the client behavior, e.g. adding support for result set compression
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3916 from cfmcgrady/arrow-detect-codec.
Closes#3915
90495c30 [Fu Chen] style
bbeada0a [Fu Chen] address comment
825bc0da [Fu Chen] minor refactor
d0a01ff7 [Fu Chen] address comment
08d21a1c [Fu Chen] fix ut
690126ce [Fu Chen] add hint ut
fd32a317 [Fu Chen] style
a1c2bb6c [Fu Chen] simplify KyuubiConnection
f81336d3 [Fu Chen] refactor
500e766f [Fu Chen] unused import
221bc928 [Fu Chen] fix ut
cf564d0a [Fu Chen] refactor
4b895e45 [Fu Chen] fix compile
3efcc335 [Fu Chen] clean up
95ea29c3 [Fu Chen] Client support detecting ResultSet codec
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Followup for #3957 comments
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3963 from turboFei/unlimited_user.
Closes#3957
987b71f1e [fwang12] refactor
e0c6ca515 [fwang12] [KYUUBI #3957][FOLLOWUP] Refactor the session connection unlimited user list config
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
to close#3930 .
1.make SignUtils.signWithPrivateKey and synchronized
2.show exception with input string when base64 decoding failed
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3932 from bowenliang123/3930-sync-usersign.
Closes#3930
a589486e [liangbowen] 1.make SignUtils.signWithPrivateKey and synchronized, 2.show exception with input string when base64 decoding failed
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
followup for #3941, support to config the interval for updating the same status for a query
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3960 from turboFei/config_exec_state_update.
Closes#3941
57d73190 [fwang12] config the interval for updating the same status for a query
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Prevent batch session & operation closed by kyuubi because of timeout
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3923 from turboFei/batch_idle.
Closes#3923
76b0fe16 [fwang12] use val
310119b0 [fwang12] refactor
61ce2036 [fwang12] use session timeout
52fb0cc4 [fwang12] 1.6.2
7819f2091 [fwang12] save
69ffa316 [fwang12] check batch session timeout
dd210329 [fwang12] add batch op timeout
e304ddb4 [fwang12] never idle batch session and never timeout batch op
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Support dedicated limiter for batch session.
Support user white list.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3957 from turboFei/batch_session_limiter.
Closes#3957
55db79e4 [fwang12] support white list
940b76bd [fwang12] add batch session limiter
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Refer the spark scheduler, warn that batch application maybe starved.
<img width="766" alt="image" src="https://user-images.githubusercontent.com/6757692/206888162-663a3ce0-597a-4818-8f7e-04e7be06cef4.png">
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3959 from turboFei/only_state_change.
Closes#3959
6ab59420 [fwang12] show the batch statvation timeout warn
74be3f48 [fwang12] only state change
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
The Spark session supports setting the time zone through `spark.sql.session.timeZone` and formatting according to the time zone, but `timestamp` does not use timezone, resulting in some incorrect results.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3958 from cxzl25/fix_session_timezone.
Closes#3958
3f2e375c [sychen] ut
e2fd90ac [sychen] session timezone format
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
#3847https://app.travis-ci.com/github/apache/incubator-kyuubi/jobs/590599391
```
SparkOperationSuite:
- get catalogs
- get schemas
- get tables
- get type info
- audit Kyuubi Hive JDBC connection common MetaData *** FAILED ***
"...i Project Hive JDBC []Client" did not equal "...i Project Hive JDBC [Shaded ]Client" (SparkMetadataTests.scala:407)
Analysis:
"...i Project Hive JDBC []Client" -> "...i Project Hive JDBC [Shaded ]Client"
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3955 from cxzl25/jdbc_shaded_followup.
Closes#3847
b229ebe5 [sychen] or
921b44bf [sychen] travis -Pjdbc-shaded
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3946 from turboFei/config_rest_pool_size.
Closes#3946
30641c14 [fwang12] save
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
#3847
Windows users need to generate `kyuubi-version-info.properties` to debug.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3903 from cxzl25/win_gen_version_info.
Closes#3903
089cf9eb [sychen] support windows generate kyuubi-version-info.properties
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Kyuubi supports GROUP engine share level, currently, it just simply delegates the group provider to Hadoop UserGroupInformation, which is not flexible enough for users who want to use other group mapping mechanisms, e.g. LDAP, JDBC.
This PR supplies a pluggable plugin interface `GroupProvider` and provides a built-in `HadoopGroupProvider` which has the same behavior w/ the current implementation.
W/ this change, users can easily implement `LDAPGroupProvider`, `JDBCGroupProvider`, `FileGroupProvider`, `CustomGroupProvider`, etc. then the GROUP engine share level will be more powerful and flexible.
The alternative option is to guide users to learn and extend the Hadoop group mapping system[1].
[1] https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/GroupsMapping.html
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3897 from pan3793/group.
Closes#3897
b100348e [Cheng Pan] nit
3cce6451 [Cheng Pan] Supplying plugable GroupProvider
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
If binary ssl enabled, we will bind the port to ssl server socket.
if we invoke the lazy val serverSocket, it will throw BindException.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3778 from turboFei/ssl_fix.
Closes#3658
5d6f71b7 [fwang12] Fix ssl port binded
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
to close#3839 .
The session signing feature is introduced with asymmetric encryption to prevent manipulation of session user identity or other key session elements, which could cause privilege leaking in scripts.
1. Server: Server creates and holds the singleton `keypair` for session signing if feature enabled
2. Server -> Engine: Server passes the `public key` to Engine when launching
3. Server -> Engine: Server generates session signing on `session user` when opening Kyuubi Session to Engine
4. Session -> Statement: Kyuubi session create Statement with context bringing `publickey` and `session user signature`
5. Engine: Engine verify `session user signature` with signature wherever necessary, e.g. in Authz for session name authentication.
ECDSA, with a shorter key length and better performance than RSA, is the supported asymmetric encryption in the initial implementation, which is widely supported on JDK7+ in HotSpot or OpenJDK. The session signature is generated with `SHA256withECDSA`.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3838 from bowenliang123/check-sessionuser.
Closes#3839
925eee47 [liangbowen] Revert "generalize setSparkLocalProperty"
d726d661 [liangbowen] generalize setSparkLocalProperty
470176db [liangbowen] allow setSparkLocalProperty to set value null
8f2a6e38 [liangbowen] nit
cb5891c9 [liangbowen] nit
edbe3c9c [liangbowen] only set to opensession conf when true, since kyuubi.session.user.sign.enabled is default to false
60546c82 [liangbowen] remove setting setSessionSigningPublicKey in KyuubiSessionManager
9a2a60dd [liangbowen] fix asserts
2e2a51b4 [liangbowen] nit
f37f4dd6 [Bowen Liang] Merge branch 'master' into check-sessionuser
9cd22003 [liangbowen] generalize illegalAccessWithUnverifiedUser in AuthZUtils
c1f27afa [liangbowen] generalize setSessionUserSign and clearSessionUserSign in SparkOperation
3683150f [liangbowen] update error message with `Invalid user identifier`
e143ea77 [liangbowen] fast fail for getting missing session configs
866fc821 [liangbowen] rename param name of base64 encoded pubkey with `Base64` suffix
280a95ef [liangbowen] refactor config key name to new class `ReservedKeys` in Authz
8592070e [liangbowen] nit
bf227e35 [liangbowen] remove redundant ut case
9ed14feb [liangbowen] remove redundant ut case
33a723d0 [liangbowen] add AuthzSessionSigningSuite with session user sign enabled
b77d53e3 [liangbowen] remove config setting in ut
2d56bd30 [liangbowen] remove config setting in ut
a25c1c0b [liangbowen] fail w/ AccessControlException, when kyuubi.session.user.sign.enabled is true and kyuubi.session.user is absent
c8a88fe9 [liangbowen] check not null of userPubKeyStr, userSign
c8bc590a [liangbowen] rename param to publicKeyBase64
b55beb60 [liangbowen] revert to use EC secp256k1 for compatibility
8acfd41d [liangbowen] nit
842b3698 [liangbowen] nit
983585bc [liangbowen] nit
d1003cd7 [liangbowen] nit
f9d6cfb6 [liangbowen] make generateKeyPair return (PrivateKey, PublicKey)
52eaaddc [liangbowen] add ut for SignUtils
b4a44687 [liangbowen] general keypair algorithm
7d40da49 [liangbowen] change to use secp192r1 curve for EC key pair for better performance
1ceed876 [liangbowen] sync settings.md with `false` default value
07d23602 [liangbowen] update ut
eec9d44a [liangbowen] move to _confIgnoreList
b7969446 [liangbowen] create SessionSigningSuite and add ut for 1. verifying user sign , 2. conf kyuubi.session.user.sign.enabled restriction
a50c71a3 [liangbowen] set kyuubi.session.user.sign.enabled to openEngineSessionConf
a2f1ed67 [liangbowen] set kyuubi.session.user.sign.enabled to _confRestrictList
5a12182e [liangbowen] make kyuubi.session.user.sign.enabled default to false, and removed from serverOnlyConfEntries
991a4569 [liangbowen] put SESSION_USER_SIGN_ENABLED in serverOnlyConfEntries
3e863af4 [liangbowen] nit
b232e5c0 [liangbowen] npe
13a046be [liangbowen] nit
2d2a6659 [liangbowen] nit
a0d4721d [liangbowen] move session signing keypair generation to KyuubiSessionImpl on server side. and rename config to KYUUBI_SESSION_SIGN_PUBLICKEY
ab430c39 [liangbowen] make generateKeyPair return Key pair and accept algorithm param
42ee2fe0 [liangbowen] nit
ce5f4af7 [liangbowen] refactoring session pubkey generation on server side (instead of engine side), and passing it to engine via OpenSession op
c2b9d897 [liangbowen] nit: rename to verifySignWithECDSA
d0b2cddb [liangbowen] nit
33a044d0 [liangbowen] add ut
2dc1f57b [liangbowen] change to use spark conf `kyuubi.session.user.sign.enabled` to decide whether verify kyuubi seesion user
b11ba5a5 [liangbowen] clear local prop `kyuubi.session.user.public.key` and `kyuubi.session.user.sign` after execution
2c8b4bf6 [liangbowen] move throwing AccessControlException inside verifyKyuubiSessionUser method
82f5c265 [liangbowen] update conf doc
7cf0d481 [liangbowen] rename config name to `kyuubi.session.user.sign.enabled`
cc2c3570 [liangbowen] typo: fix KYUUBI_SESSION_USER_PUBIC_KEY
bdec509d [liangbowen] update settings.md for config doc
2d00163a [liangbowen] nit
102561a8 [liangbowen] update
af99ea84 [liangbowen] move algorithmSpec to ecKeyPairGenerator
966a327e [liangbowen] update
41064712 [liangbowen] update
9d276799 [liangbowen] add config `kyuubi.session.user.verify.enabled` to control Whether to verify the integrity of session user name in Spark Authz
8a8840f6 [liangbowen] nit
bafd85e9 [liangbowen] replace RSA with ECDSA
2f0c87a5 [liangbowen] KeyPairGenerator init with new SecureRandom instance
7cb31204 [liangbowen] shorten the key size to 1024
5011cf49 [liangbowen] remove unused imports
045fd822 [liangbowen] add to SparkOperation for scala
ce6d394d [liangbowen] move generateRSAKeyPair and signWithRSA to SignUtils.scala
5f295792 [liangbowen] update
1d7f3191 [liangbowen] initial support for signing and verifying `kyuubi.session.user`
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
1. `mvn antrun:runbuild-info -pl kyuubi-common`
2. debug engine suite
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3847 from cxzl25/profile_jdbc_debug.
Closes#3847
e1c32067 [Cheng Pan] Merge branch 'master' into profile_jdbc_debug
f41d3cd9 [sychen] use jdbc-shaded
25b54b18 [sychen] use jdbc-shaded
dd23f864 [sychen] use kyuubi-hive-jdbc as default
79b0ffcb [sychen] add jdbc profile
Lead-authored-by: sychen <sychen@ctrip.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Expose `kyuubi.operation.result.codec` to KyuubiConf
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3866 from cfmcgrady/arrow.
Closes#3794
2047f364 [Fu Chen] update
6f3fe24d [Fu Chen] rename
20748ca7 [Fu Chen] expose `kyuubi.operation.result.codec` to KyuubiConf
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close#3848 .
Sort the config map returned in KyuubiConf.getAll to improve readability of engine launching configs in output.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3849 from bowenliang123/sort-configs.
Closes#3848
75e60c89 [liangbowen] sort the returned map of getAll in KyuubiConf (+4 squashed commits) Squashed commits: [f102b423] update method comment [465d7b25] use TreeMap for sorting [1a66e05b] add getAllSorted [2b3dbf91] sort the returned map of getAll in KyuubiConf
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Submit pyspark batch job by restful api
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3836 from leoluan2009/pyspark-1.
Closes#3836
550021ac [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KyuubiApplicationManager.scala
357691c2 [Xuedong Luan] fix comment
7dfdbe24 [Xuedong Luan] fix comment
31bda178 [Xuedong Luan] [WIP] Support pyspark batch job by restful api
Lead-authored-by: Xuedong Luan <luanxuedong2009@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Forward the server ip in `openSession`, and in the Spark engine we get the server ip to start the engine from the Environment tab.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3844 from cxzl25/forward_server_ip.
Closes#3844
7b2cf95b [sychen] fix conf npe
59e7a2fc [sychen] mysql and rest
f50c36d6 [sychen] forward server ip
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Introduce code style check support for Maven's pom.xml with sortPom in spotless maven plugin.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3843 from bowenliang123/spotless-pom.
Closes#3842
3c654597 [liangbowen] apply to pom.xml
fd1536f7 [liangbowen] set expandEmptyElements to true
e498423f [liangbowen] apply spotless:apply to all pom.xml
e46bcfec [liangbowen] add pom style check support in spotless
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close#3805 .
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3832 from bowenliang123/skip-pyspark-tests.
Closes#3805
02a59bfb [liangbowen] add maven.plugin.scalatest.exclude.tags=org.scalatest.tags.PySparkTest to pom instead
444b6f4e [liangbowen] introduce @PySparkTest annotation and skip PySparkTests for nightly builds
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
* [KYUUBI #3814] Make the duration of sessionConfCache in FileSessionConfAdvisor configurable instead of a fixed 10 minutes
* fix ut
* fix ut
* fix doc
* code review
* update doc
* update
* update
### _Why are the changes needed?_
When a kyuubi service connections to multiple engine clusters, we can only explicitly specify relevant parameters after the URL, for example:
`beeline -u 'jdbc:hive2://127.0.0.1:10009/?kyuubi.engineEnv.FLINK_HOME=/opt/flink;kyuubi.ha.namespace=kyuubi-ns-c;kyuubi.engine.type=FLINK_SQL'`
This method is extremely unfriendly to users.
I implemented a SessionConfAdvisor implementation class `org.apache.kyuubi.session.FileSessionConfAdvisor` based on the conf file that to manage session level configurations, so that we can access this cluster only by specifying a unique engine name, for example:
access cluster-a:
`beeline -u 'jdbc:hive2://127.0.0.1:10009/?kyuubi.session.conf.profile=cluster-a'`
access cluster-c:
`beeline -u 'jdbc:hive2://127.0.0.1:10009/?kyuubi.session.conf.profile=cluster-c'`
`kyuubi-session-cluster-a.conf` configuration is as follows:
```
kyuubi.ha.namespace kyuubi-ns-a
kyuubi.engine.type SPARK_SQL
kyuubi.engine.pool.balance.policy POLLING
kyuubi.engineEnv.SPARK_HOME /opt/spark30
kyuubi.engineEnv.HADOOP_CONF_DIR /opt/hadoop_conf_dir
```
`kyuubi-session-cluster-c.conf` configuration is as follows:
```
kyuubi.ha.namespace kyuubi-ns-a
kyuubi.engine.type SPARK_SQL
kyuubi.engineEnv.SPARK_HOME /opt/spark32
kyuubi.engineEnv.HADOOP_CONF_DIR /opt/hadoop_conf_dir
```
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3742 from ychris78/yuqi_1101.
Closes#3742
e1449867 [yuqi] Style
68bbfecf [yuqi] restructure
1c8daa86 [yuqi] Change the cache expiration time to 10 minutes and remove the manual cache
5b4a6d77 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/session/FileSessionConfAdvisor.scala
d332e2d8 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/session/FileSessionConfAdvisor.scala
cb644ab4 [Cheng Pan] Update docs/deployment/settings.md
1ee3c980 [Cheng Pan] Update kyuubi-server/src/test/resources/kyuubi-session-cluster-a.conf
30f8fde0 [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
5ef5841a [yuqi] Cache upgrade
5c2f7461 [yuqi] add cache and fix bug
1b1f9d67 [yuqi] scalastyle
a0507847 [yuqi] License info amendment
335250fc [yuqi] fix bug
9e8d313a [yuqi] Modify according to suggestions
5f57b637 [yuqi] Modify according to suggestions
5e504252 [yuqi] add org.apache.kyuubi.plugin.FileSessionConfAdvisor When kyuubi.session.conf.advisor=org.apache.kyuubi.plugin.FileSessionConfAdvisor,Specify a session level configuration file, which will be combined with default.conf to have an impact. The corresponding configuration file is conf/kyuubi-session-<profile>.conf
0cb71341 [yuqi] add org.apache.kyuubi.plugin.YamlSessionConfAdvisor
b524c221 [yuqi] add org.apache.kyuubi.plugin.YamlSessionConfAdvisor
51729d15 [yuqi] When a kyuubi service connections to multiple engine clusters, we can only explicitly specify relevant parameters after the URL, for example: beeline -u 'jdbc:hive2://127.0.0.1:10009/?kyuubi.engineEnv.FLINK_HOME=/opt/flink;kyuubi.ha.namespace=kyuubi-ns-c;kyuubi.engine.type=FLINK_SQL' This method is extremely unfriendly to users, and it is extremely unsafe to expose namespaces to users. So I added a yaml file(engine-cluster-env.yaml) to manage the engine clusters, so that we can use this cluster only by specifying a unique engine name, for example: use cluster-a: beeline -u 'jdbc:hive2://127.0.0.1:10009/?kyuubi.engine.cluster.name=cluster-a'
Lead-authored-by: yuqi <1450636878@qq.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Support real user for kyuubi session
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3766 from turboFei/real_user.
Closes#3766
49c996ee [Fei Wang] refactor
894b4720 [Fei Wang] owner -> user
477119bf [Fei Wang] address
c36511ae [Fei Wang] add ut
8efc310c [Fei Wang] save
8ca2d589 [Fei Wang] add ut
89264d90 [Fei Wang] get real user
a431b0ad [Fei Wang] real user
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Now we support multiple frontend services.
We need transfer the connection url into session to identify the connection type.
It can also fix#3575
### _How was this patch tested?_
Pass existing UT.
Closes#3577 from turboFei/connection_url.
Closes#3577
f1591ccd [Fei Wang] refactor the order
70d21ca1 [Fei Wang] add more ut
5eaba4f2 [Fei Wang] comments
a78470c5 [Fei Wang] add ut
72107da3 [Fei Wang] SessionData
37965486 [Fei Wang] style
832922d2 [Fei Wang] remove
7739be65 [Fei Wang] save
5078ec6c [Fei Wang] save
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Expose the engine id when registering the service.
So that we can get more details about the engine when using `list engine` command.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3684 from turboFei/extra_info.
Closes#3684
afbcfdef3 [Fei Wang] refactor
513dd7234 [Fei Wang] split 2
ad3a6ac4f [Fei Wang] comments
cc51c5954 [Fei Wang] fix ut
32e31618a [Fei Wang] check engine id
202bbdb68 [Fei Wang] refactor to details
c5c153b8d [Fei Wang] render extra info
d2642cc8a [Fei Wang] save
1da67c258 [Fei Wang] dto
724ac3af4 [Fei Wang] save
c16c15988 [Fei Wang] expose engine id
e91a61e5a [Fei Wang] save
e38bb0b93 [Fei Wang] save
b1c158ba8 [Fei Wang] parse extra info
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Closing existing seekable reader when adding extra log
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3745 from turboFei/fix_op_log_bug.
Closes#3745
5faf9a99 [Fei Wang] save
3bf78f3e [Fei Wang] recreate seek reader
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
fix#3727
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3729 from jiaoqingbo/kyuubi3727.
Closes#3727
cb494c82 [jiaoqingbo] [KYUUBI #3727] Extract the common variables of GetTables in flink and spark to ResultSetSchemaConstant
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close#3701 .
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3702 from bowenliang123/3701-jdbc-auth.
Closes#3701
7a04d091 [Bowen Liang] skip not empty checking for kyuubi.authentication.jdbc.user config
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Current kyuubi build kubernetes client for `KyuubiApplicationOperator` to find or kill spark engine on kubernetes.
This pr aims to extract the code for building kubernetes client into `KubernetesUtils` and to do auto build following the logic in Spark.
We prefer to building with user-configured parameters; When missing, the kubernetes client logic will build one from environment variables and kubeconfig.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3663 from zwangsheng/feature/perfer_build_from_pod_env.
Closes#3663
0a4c8c79 [zwangsheng] clear properties
ce408db0 [zwangsheng] fix test
fb28ea7a [zwangsheng] add ci test
943c68dd [zwangsheng] add unit test
ddc9a2b2 [zwangsheng] fix scala
75380d13 [zwangsheng] comment
e6826609 [zwangsheng] master
683cc260 [zwangsheng] context provided master
7b42133c [zwangsheng] add doc
b56829c5 [zwangsheng] fix helm
5884475d [zwangsheng] rebuild
a7f0b551 [zwangsheng] ad
e7f9ec05 [zwangsheng] add setting
b77bbd9b [zwangsheng] add doc
3f3253d3 [zwangsheng] init
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Support SSL for kyuubi thrift binary conenction.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
Will add screenshot for IT.
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3658 from turboFei/ssl_binary.
Closes#3658
ffd603ec [Fei Wang] comment
ce652b7e [Fei Wang] revert UT
52e24abb [Fei Wang] ut
f6cae357 [Fei Wang] save
c6b664f1 [Fei Wang] exclude passwords
d34a4ca2 [Fei Wang] comments
11bfa9ad [Fei Wang] refactor
e2ca77ea [Fei Wang] align
86aaf87c [Fei Wang] with alternative
d17e0d11 [Fei Wang] style
eb4ba52a [Fei Wang] Support SSL for Kyuubi thrift binary connection
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
As described in #2887, random policy may cause task-hot-issues or production accident, so a POLLING balance policy is added to avoid this problem, we can "set kyuubi.engine.pool.balance.policy = POLLING" to use it.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3662 from ychris78/epbp_1018.
Closes#2887
90834c1f [yuqi] fix bug
c0ce8a7e [yuqi] Fix typos
bec4ca13 [yuqi] Fix typos
dc4e3637 [yuqi] please add test case to cover the added methods
837c920b [yuqi] rename method getAndInc to getAndIncrement and modify the doc
02a8311f [yuqi] add getAndInc method and refer it
9436937d [yuqi] Improve configuration documentation
21b5d7d1 [yuqi] [KYUUBI #2887] Add a polling balance policy for spark engine pool Improve configuration documentation and delete useless comments
f0699b3c [yuqi] [KYUUBI #2887] Add a polling balance policy for spark engine pool
Authored-by: yuqi <1450636878@qq.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
to close#3671.
- fix assert message for '[SCHEMA_NOT_FOUND]' and '[TABLE_OR_VIEW_NOT_FOUND]'
- introduce SparkVersionUtil in kyuubi-common test for checking Spark version
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3676 from bowenliang123/3671-notfound.
Closes#3671
c505098f [Bowen Liang] update
a950abbf [Bowen Liang] fix typo
56464e84 [Bowen Liang] update SparkVersionUtil
2190a93e [Bowen Liang] fix assert message for '[SCHEMA_NOT_FOUND]' and '[TABLE_OR_VIEW_NOT_FOUND]', introduce SparkVersionUtil for checking spark version in kyuubi-common test
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
to close#3615
steps to reproduce
1. start kyuubi server
2. connect to the kyuubi server with client beeline and run query `select 1`
3. force kill the spark application
4. connect to the kyuubi server with another client **A** immediately
```
Error: org.apache.kyuubi.KyuubiSQLException: Error operating LaunchEngine: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at org.apache.kyuubi.client.KyuubiSyncThriftClient$.createTProtocol(KyuubiSyncThriftClient.scala:466)
at org.apache.kyuubi.client.KyuubiSyncThriftClient$.createClient(KyuubiSyncThriftClient.scala:482)
at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:128)
at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1$adapted(KyuubiSessionImpl.scala:113)
at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36)
at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:113)
at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$2(LaunchEngine.scala:60)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.aache.thrift.transport.TSocket.open(TSocket.java:221)
... 14 more (state=,code=0)
```
If the engine exits without calling the shutdown hook, the ZNODE with EPHEMERAL_SEQUENTIAL mode will be deleted upon the client's disconnect, and note that the engine will be marked as disconnected after `kyuubi.zookeeper.embedded.max.session.timeout` (the default value is 1 minutes)
This PR retries opening the engine when encountering a specialty exception (java.net.ConnectException). Before this PR, client **A** will throw an exception, after this PR, client **A** will work properly.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3618 from cfmcgrady/kyuubi-3615.
Closes#3615
96acd66b [Fu Chen] address comment
a1395a60 [Fu Chen] fix style
870216d1 [Fu Chen] Retry opening the engine when encountering a special error
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
fix#3640
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3641 from jiaoqingbo/kyuubi3640.
Closes#3640
c1b2d4a6 [jiaoqingbo] code review
a65bc132 [jiaoqingbo] code review
ab04f135 [jiaoqingbo] [KYUUBI #3640] Change the default event logger description of hive and trino to json instead of spark
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close#3635 .
- change test suits to to use Utils.createTempDir of kyuubi-common module to create temp dir with delete on exit to prevent disk usage leaking
- change signature Utils.createTempDir by making `prefix` param from second place to the first, to make it friendly to use without repeating param name itself. The name`prefix` is more closer to Java's style in `Files.createTempDirectory`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3660 from bowenliang123/3635-deletetmp.
Closes#3635
8947fb0f [Bowen Liang] remove unused imports
c06769d4 [Bowen Liang] nit
dac266c9 [Bowen Liang] - change test suits to to use Utils.createTempDir of kyuubi-common module to create temp dir with delete on exit to prevent disk usage leaking - change signature Utils.createTempDir by making prefix param from second place to the first, to make it friendly to use without repeating param name itself. The nameprefix is more closer to Java's style in Files.createTempDirectory
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Refer Spark PR: https://github.com/apache/spark/pull/18110
ConfigBuilder builds ConfigEntry which can only read value with one key, if we wanna change the config name but still keep the old one, it's hard to do.
This PR introduce ConfigBuilder.withAlternative, to support reading config value with alternative keys.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3659 from turboFei/conf_alternative.
Closes#3659
e268fef2 [Fei Wang] add ut
a2300b26 [Fei Wang] add ut
53eccf99 [Fei Wang] Support alternative keys in ConfigBuilder
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
to close#3635 .
- change test suits to to use Utils.createTempDir of kyuubi-common module to create temp dir with delete on exit to prevent disk usage leaking
- change signature Utils.createTempDir by making `prefix` param from second place to the first, to make it friendly to use without repeating param name itself. The name`prefix` is more closer to Java's style in `Files.createTempDirectory`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3636 from bowenliang123/3635-tempdir.
Closes#3635
5f84a16f [Bowen Liang] nit
b82a149f [Bowen Liang] rename `namePrefix` param to `prefix` of `Utils.createTempDir` method, and make it in first place
76d33143 [Bowen Liang] delete files on exit in test suits with Utils.createTempDir
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
When Kyuubi runs outside of K8s, and w/o enhanced DNS infrastructure, Kyuubi can not access the Pod by using the hostname of Pod, it blocks the user to run Spark on K8s w/ cluster mode out-of-box.
Kyuubi provided a configuration `kyuubi.frontend.connection.url.use.hostname`, turn it off could address this issue, but we can not change the default value globally because of https://github.com/apache/incubator-kyuubi/issues/2266
To improve user experience, we can detect if the Driver is running inside the Pod, and if yes,
change `kyuubi.frontend.connection.url.use.hostname` default value to `false`.
Close#3578
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3597 from pan3793/k8s.
Closes#3597
8b411781 [Cheng Pan] doc
0df15e79 [Cheng Pan] Engine should prefer to use ip for registing on K8s cluster mode
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3512
Currently, Kyuubi EventBus uses `scala.concurrent.ExecutionContext.Implicits.global` to execute async event handler. Generally, it's discouraged to use that global ec, instead, we should create dedicated ec for each workload, this pr aims to use dedicated ExecutionContext for EventBus async execution
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3524 from Yikf/ec.
Closes#3512
9757372d [yikf] Use dedicated ExecutionContext for EventBus async execution
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3510
Reading an uninitialized log should return empty rowSet , So we can set EmptyRowSet for TFetchResultsResp to avoid client side NPE if FetchResults fail on server side.
For example: Configure log4j for `ERROR` level, causing the engine side no log file output, so server side throw `java.nio.file. NoSuchFileException`, at this time the client get engine log, throws the NPE.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3511 from Yikf/issue-3510.
Closes#3510
3fee6591 [yikf] fix
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3490
Currently, when Kyuubi get or set conf by `ConfigEntry`, it does not check whether `ConfigEntry` have been registered, so this pr aims to two points as follow:
- Require ConfigEntry has been registered;
- Using copy-on-write for Kyuubi#kyuubiConfEntries to reduce contention in concurrent workloads.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3491 from Yikf/copy-on-write-conf.
Closes#3490
6d0d5ed2 [yikf] Use copy-on-write semantics for KyuubiConf registered configurations
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close https://github.com/apache/incubator-kyuubi/issues/3449, this pr aims to change `kyuubi.server.info.provider` default value to "ENGINE", and modify corresponding tests.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3476 from Yikf/default-server-info.
Closes#3449
bc474d3b [Cheng Pan] Update docs/deployment/migration-guide.md
23e7690e [yikf] Change default server info provider to ENGINE
Lead-authored-by: yikf <yikaifei1@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The default value of kyuubi.engine.event.loggers is SPARK, but it can't be used outside the spark engine, so we add the config for each engines.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3309 from zhaomin1423/add_engine_logger_conf.
Closes#3309
37b3c05f [Xiao Zhao] fix
b22356f2 [Min] fix typo
431bb02b [Min] update settings.md
44475f46 [Min] Add configs of engine event logger for each engine
Lead-authored-by: Min <zhaomin1423@163.com>
Co-authored-by: Xiao Zhao <zhaomin1423@163.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Workaround for #3032, close#3323
There are known some ODBC drivers e.g. [Databricks ODBC driver](https://www.databricks.com/spark/odbc-drivers-download) depending on `TGetInfoType.CLI_DBMS_VER` and `TGetInfoType.CLI_DBMS_NAME` to check server compatibilities and abilities.
This PR proposes to introduce a new configuration `kyuubi.server.info.provider=SERVER/ENGINE` to make GetInfo support return either server or engine information.
Since beeline will call GetInfo in the initialization phase, to make sure the beeline fast open experience, in async launch mode, when the engine is not ready, return server info regardless `kyuubi.server.info.provider`.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
Testing w/ PowerBI
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3122 from pan3793/info.
Closes#3122
742bdbe3 [Cheng Pan] nit
bb85d2ba [Cheng Pan] style
fd75238c [Cheng Pan] fix
9ddb2afd [Cheng Pan] nit
fd8f7979 [Cheng Pan] fix ut
840205e5 [Cheng Pan] nit
f9996d53 [Cheng Pan] GetInfo supports returning engine info
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Close#3064
Refer the comments by cxzl25
Spark uses the code
`URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory(hadoopConf))` supports the class path from HDFS.
But because the scala compiler only supports adding file schema urls to the class path, non-file schema urls will cause NPE.
```java
Error: Error operating ExecuteScala: java.lang.NullPointerException
at scala.tools.nsc.classpath.FileUtils$AbstractFileOps$.isJarOrZip$extension(FileUtils.scala:32)
at scala.tools.nsc.classpath.ClassPathFactory$.newClassPath(ClassPathFactory.scala:90)
at scala.tools.nsc.Global.$anonfun$extendCompilerClassPath$1(Global.scala:832)
```
scala.tools.nsc.Global#extendCompilerClassPath
```scala
AbstractFile.getURL(u)
```
scala.reflect.io.AbstractFile#getURL
```scala
def getURL(url: URL): AbstractFile =
if (url.getProtocol == "file") {
val f = new java.io.File(url.toURI)
if (f.isDirectory) getDirectory(f)
else getFile(f)
} else null
```
spark-shell supports --jars hdfs jar. At this time, submit will download the remote jar to the local and pass it to spark-shell through the `spark.repl.local.jars` configuration.
In this pr, I localize the remote jar url at first, and then add it into repl class path.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3292 from turboFei/scala_npe.
Closes#3064
095c40d8 [Fei Wang] filter
37637f76 [Fei Wang] add non-exist
ec87ea85 [Fei Wang] add ut
5a823d3b [Fei Wang] save
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3414
Currently, Kyuubi supports output of planOnly in different modes in different styles, As more and more modes and styles are supported, defining enumerations in kyuubiConf is a bit messy and not easy to extend, so this pr aims to tidy up these
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3415 from Yikf/tidy-up.
Closes#3414
e81142e5 [yikf] Tidy up plan only related modes and styles
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
close https://github.com/apache/incubator-kyuubi/issues/3311
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3324 from lsm1/fix/operation_language_incorrect.
Closes#3311
9192f61f [senmiaoliu] revert delete PlanOnlyOperationSuite.scala
9d0079b1 [senmiaoliu] move ut in SparkQueryTests
e8b160da [senmiaoliu] add unknown for operation language
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3376
This pr aims support planOnly output in different styles (Users can perform SQL scheduling based on the JSON description of TreeNode)
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3377 from Yikf/planOnly-support-json.
Closes#3376
b63ae718 [yikf] Support json_description for PlanOnlyStatement
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Support to get engine id, name, url in KyuubiConnection.
It is helpful and friendly for BI.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3354 from turboFei/lanch_engine_result.
Closes#3354
4966de28 [Fei Wang] comments
984b5d35 [Fei Wang] fix ut
963217af [Fei Wang] filter null
a2c0ce2c [Fei Wang] fix
4f2a8b18 [Fei Wang] get from engine side
61607c7c [Fei Wang] return
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3357
This pr aims to improve doc description for configuration kyuubi.operation.plan.only.mode
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3358 from Yikf/planOnly-doc.
Closes#3357
571f4e0c [yikf] Improve doc description for configuration `kyuubi.operation.plan.only.mode`
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Now the connection path is:
`client`-> `kyuubiServer` -> `kyuubiEngine`
Maybe the connection between `client -> kyuubiServer` is valid, but the connection between `kyuubiServer -> kyuubiEngine` is not.
So we need check the whole path.
BTW, before, if customer invoke KyuubiConnection::isValid, it can make the connection between client and kyuubiServer keep alive.
Now, it can make the whole connection path keep alive.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3318 from turboFei/is_valid.
Closes#3318
af4d6b7b [Fei Wang] Transfer the TGetInfoReq to kyuubi engine side to keep the kyuubi connection alive
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
close https://github.com/apache/incubator-kyuubi/issues/3128
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3304 from Kwafoor/master.
Closes#3128
88a5ebe9 [wangjunbo] [kyuubi-3128] Support CostMode for PlanOnlyStatement
Authored-by: wangjunbo <wangjunbo@qiyi.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
fix https://github.com/apache/incubator-kyuubi/issues/3298
Curently, a serval snippets use `getClass.getClassLoader` to get the classLoader in Apache Kyuubi, this pr aims to unify this behavior in `Utils.getContextOrKyuubiClassLoader`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3299 from yikf/unify-calssloader.
Closes#3298
ae081ad9 [yikf] Unify the approach of get classLoader in Kyuubi
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
To fix the config name and placeholder with `username` introduced in https://github.com/apache/incubator-kyuubi/pull/3235 violate this convention as in JDBC driver use `user` keyword used for connection user rather than `username`,
1. change config name from `kyuubi.authentication.jdbc.username` to `kyuubi.authentication.jdbc.user`
2. change placeholder from `${username}` to `${user}`
3. update docs and config description related to above changes, and sync the update in jdbc auth docs statement details to config docs.
4. fix error in throwing AuthenticationException with auth db password. ut added for the fix.
5. other minor update in docs of custom auth
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3288 from bowenliang123/jdbc-auth-config-update.
Closes#3222
502703fb [Bowen Liang] skip map for placeholder value lookup
3733be41 [liangbowen] nit
ab00525b [liangbowen] nit
2301c4f0 [liangbowen] fix ut of jdbc auth with wrong_password
06f0c1bb [liangbowen] remove redundant docs
ec4565b3 [liangbowen] remove redundant docs
ae1cce2e [liangbowen] fix compilation error of configLog
5d14103b [liangbowen] simplify configLog
6678e657 [liangbowen] reformat
52c1038e [liangbowen] simplify placeholder checking
21c2d5ea [liangbowen] check whether placeholders in supported list before conn establishment or authenticate
7db0adf5 [liangbowen] ut for unknown placeholder
657de6af [liangbowen] nit
736b3f22 [liangbowen] refactoring placeholder value lookup, for preventing setString multiple times with "i+1"
86c89125 [liangbowen] setMaxRows after prepare placeholder, to postpone operation on jdbc conn
115fae50 [liangbowen] increase test code coverage
b45b28cb [liangbowen] resultSet returned by executeQuery is never null
e1c07274 [liangbowen] update ut for redactPassword in JdbcUtils
b4a52e29 [liangbowen] fix typo in docs of custom auth
371c2c6e [liangbowen] move redactPassword method to JdbcUtils and add ut.
a4973c59 [liangbowen] reformat code
486e1503 [liangbowen] fix error in throwing AuthenticationException with auth db password. add ut for the fix.
efced90a [liangbowen] update settings.md
ef97e354 [liangbowen] add SELECT prefix hint for doc of kyuubi.authentication.jdbc.query
025f94c7 [liangbowen] fix username to user in JdbcAuthenticationProviderImpl by 1. use config name `kyuubi.authentication.jdbc.user`, 2. use ${user} placeholder instead of ${username}
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This is the followup of #3235, the main change is introdude `JdbcUtils` to simplify code, and allow empty password for Jdbc auth.
Jdbc connection pool has been removed because `JdbcAuthenticationProviderImpl` will be created on each connection, we can improve to use singleton in the future
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3278 from pan3793/jdbc-followup.
Closes#3222
2863cae7 [Cheng Pan] Update kyuubi-common/src/test/scala/org/apache/kyuubi/service/authentication/JdbcAuthenticationProviderImplSuite.scala
51a9c458 [Cheng Pan] Update kyuubi-common/src/test/scala/org/apache/kyuubi/service/authentication/JdbcAuthenticationProviderImplSuite.scala
eee3c55e [Cheng Pan] Update kyuubi-common/src/test/scala/org/apache/kyuubi/util/JdbcUtilsSuite.scala
d02bb999 [Cheng Pan] nit
e001b5b5 [Cheng Pan] nit
8cf5cd67 [Cheng Pan] nit
032f2df8 [Cheng Pan] nit
8a42f184 [Cheng Pan] nit
c7893fdf [Cheng Pan] JdbcUtilsSuite
f97f2d90 [Cheng Pan] remove pool
a8812d0f [Cheng Pan] move render result set to test
83d7d4cf [Cheng Pan] fix ut
db787a4b [Cheng Pan] nit
864f9ddc [Cheng Pan] nit
b60decf5 [Cheng Pan] nit
8c66e0be [Cheng Pan] nit
2063c438 [Cheng Pan] [KYUUBI #3222][FOLLOWUP] Introdude JdbcUtils to simplify code
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Followup of #3020
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3271 from turboFei/refine_ldap.
Closes#3020
caa35d84 [Fei Wang] comment
65597cb4 [Fei Wang] refine
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
fix https://github.com/apache/incubator-kyuubi/issues/3269
Add docs for JDBC auth provider from https://github.com/apache/incubator-kyuubi/issues/3222 shipping within 1.6.0, add jdbc.rst in /docs/security path
simplify the config docs for kyuubi.authentication.jdbc.query in /docs/deployment/settings.md by moving examples to the docs of jdbc auth provider.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3275 from bowenliang123/doc-jdbc-auth.
Closes#3275
e393f6db [liangbowen] fix config example mistakes
ab70cc8d [liangbowen] update doc for auth query sql
ebd5e194 [liangbowen] update doc description
0b15af59 [liangbowen] typo
953000fe [liangbowen] typo
ab944e73 [liangbowen] typo
e3c3d259 [liangbowen] regenerate conf docs. udpate jdbc.md.
45429ada [liangbowen] typo
3b05338e [liangbowen] add docs for jdbc authentication provider
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Among the kyuubi ldap configuration parameters, there is currently "kyuubi.authentication.ldap.base.dn=", but there is no place corresponding to the CN password setting. similar to hive
"hive.server2.authentication.ldap.binddn",
"hive.server2.authentication.ldap.bindpw".
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3213 from dev-lpq/branch-ldap-1.6.
Closes#3020
31817cc6 [pengqli] upgrade ldap to support ldap before 1.6.0
31094030 [pengqli] upgrade ldap to support ldap before 1.6.0
f62e4c96 [pengqli] upgrade ldap to support ldap before 1.6.0
e03e036e [pengqli] add AUTHENTICATION_LDAP_GUIDKEY deprecated
6f680dfc [pengqli] add ldap config AUTHENTICATION_LDAP_BINDDN
19f8e6ce [pengqli] add ldap config AUTHENTICATION_LDAP_BINDDN
07df56b8 [pengqli] upgrade LdapAuthenticationProviderImpl
1a48f418 [pengqli] upgrade settings.md
8e9b5721 [pengqli] upgrade ldap ldapBindpw
5793a811 [pengqli] upgrade ldap ldapBindpw
822d4c30 [pengqli] upgrade docs/deployment/settings.md
c2b20418 [pengqli] upgrade ldap WithLdapServer UTs
5d7a301c [pengqli] add AUTHENTICATION_LDAP_PASSWORD into serverOnlyConfEntries
10b42b1b [pengqli] upgrade AuthenticationProviderFactorySuite UTs
490bfae8 [pengqli] kyuubi ldap add new config property kyuubi.authentication.ldap.bindnpw and kyuubi.authentication.ldap.attrs
5e071252 [pengqli] upgrafe config kyuubi.authentication.ldap.attrs
4e771ec6 [pengqli] kyuubi ldap add new config property kyuubi.authentication.ldap.bindnpw and kyuubi.authentication.ldap.attrs
Authored-by: pengqli <pengqli@cisco.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Preparing v1.7.0-SNAPSHOT with branch-1.6 cut
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3264 from SteNicholas/prepare-1.7.0-snapshot.
Closes#3264
374d56bf [SteNicholas] preparing v1.7.0-SNAPSHOT with branch-1.6 cut
Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
master branch has supported 5 engine types. the docs of `kyuubi.engine.type` missed hive and jdbc.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3255 from ulysses-you/engine-type.
Closes#3255
a1f3c320 [ulysses-you] add config
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
close#3214
### manual tests
when we set `kyuubi.operation.plan.only.mode` an incorrect value,unset plan only mode
so we can set a correct value
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3216 from lsm1/fix/plan_only_mode.
Closes#3214
bb159ec6 [senmiaoliu] fix ut
f23dea1a [LSM] Merge branch 'apache:master' into fix/plan_only_mode
5204745a [senmiaoliu] implement in OperationModes
65898f40 [senmiaoliu] fix style, added ut
a1df52f8 [senmiaoliu] unset kyuubi.operation.plan.only.mode if mode is invalid
Lead-authored-by: senmiaoliu <senmiaoliu@trip.com>
Co-authored-by: LSM <senmiaoliu@trip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add JDBC authentication provider as implementation of PasswdAuthenticationProvider out of box.
Compared to currently support authentication methods like Kerberos and LDAP, JDBC source is a much easy and stable source for practical deployment.
The solution should provide:
easy to use and config jdbc connection details
handy to customize query statements for authentication
Adding JDBC to AuthMethods enum and JDBCPasswdAuthenticationProvider into package org.apache.kyuubi.service.authentication with listed features:
- specify JDBC driver name and load the driver class
- configs of JDBC url, username and password
- select query with placeholders for checking user and password , like `SELECT 1 from user_pass_hash where username = ${user} and password = MD5(${password})`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3235 from bowenliang123/feature-jdbc-auth-provider.
Closes#3222
17403b33 [liangbowen] cleanup docs
d5f43e0e [liangbowen] remove unuseful logs for unrecognized placeholder error
e9af0966 [liangbowen] use clone instead of repeatly generating configs
6fc42bf3 [liangbowen] code styling
a9404fa3 [liangbowen] use {} for intercept
77f5f86b [liangbowen] remove unuseful comment
6765affb [liangbowen] changed to use in-memory derby db for test
543c66cb [liangbowen] prefer scala style string usage
a4fe582f [liangbowen] refactor connection creation on using HikariDataSource in HikariCP. add HikariCP dependencies to kyuubi-common
3a4d5fe9 [liangbowen] update checkConfigs() signature
5a0ac495 [liangbowen] output password length only in checkConfigs
1c956dfb [liangbowen] update KyuubiAuthenticationFactorySuite
4ebe12e2 [liangbowen] add JDBC value to AuthTypes enum
9885f813 [liangbowen] add JDBC condition for getValidPasswordAuthMethod
b9ffac3c [liangbowen] Merge branch 'master' into feature-jdbc-auth-provider
aeb19ce5 [liangbowen] update doc
653bc126 [liangbowen] add more checks for query sql
cdec2066 [liangbowen] more test cases
36729197 [liangbowen] fix ddl statement and remove truncate statement in test
30974d16 [liangbowen] update format
575301ca [liangbowen] update options usage
1dc4187b [liangbowen] update settings.md config doc
cd2c7c2b [liangbowen] update settings.md config doc
15176b2b [liangbowen] fix import orders
46cc1dda [liangbowen] add config docs in docs/deployment/settings.md
7025330b [liangbowen] fix derby startup error in test
df4be568 [liangbowen] update code style
49c18c25 [liangbowen] update
996f7969 [liangbowen] add unit test in JdbcAuthenticationProviderImplSuite
0e7f0ad0 [liangbowen] refactor config and init process.remove unused import.
0dc75fee [liangbowen] implement JDBC Authentication Method
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
If we use scala enumeration, we need to use logging, and an error will be reported in an environment where log4j2 is not packaged.
This PR just moved the `initializeLogging` method of trait Logging to object Logging.
```scala
XX extends Enumeration with Logging
```
```
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetPublicMethods(Class.java:2902)
at java.lang.Class.privateGetPublicMethods(Class.java:2911)
at java.lang.Class.getMethods(Class.java:1615)
at scala.Enumeration.populateNameMap(Enumeration.scala:193)
at scala.Enumeration.$anonfun$nameOf$1(Enumeration.scala:212)
Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.Level
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
```
Scala enumeration uses reflection, resulting in class not found in some scenarios.
scala.Enumeration#populateNameMap
685017051a/src/library/scala/Enumeration.scala (L188-L193)
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3243 from cxzl25/logging_trait_object.
Closes#3243
330ad95f [sychen] move initializeLogging
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
To make sure flink sql engine build against flink-1.15 can run on flink-1.14.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3230 from pan3793/flink.
Closes#3230
5c3f859c [Cheng Pan] updated
e776ee63 [Cheng Pan] review
a5cfb0c8 [Cheng Pan] review
004c05d9 [Cheng Pan] nit
9c019554 [Cheng Pan] nit
7c3f553c [Cheng Pan] nit
8e1fb04b [Cheng Pan] nit
6adc7dbe [Cheng Pan] nit
bc10739c [Cheng Pan] fix
cf98111a [Cheng Pan] workflow
a8895e68 [Cheng Pan] nit
8cab5188 [Cheng Pan] Flink SQL engine supports run across versions
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3229 from lightning-L/kyuubi-3170.
Closes#3170
ccb14571 [Tianlin Liao] [KYUUBI #3170] Expose thrift binary connection metrics
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Spark SQL Engine failed to get issue date from some DelegationTokenIdentifiers, such as OzoneTokenIdentifier.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3225 from zhouyifan279/3219.
Closes#3219
0ffdcf98 [zhouyifan279] [KYUUBI #3219] Error renew delegation tokens: Unknown version of delegation token 8
c5a37a0c [zhouyifan279] [KYUUBI #3219] Error renew delegation tokens: Unknown version of delegation token 8
594c5a0f [zhouyifan279] [KYUUBI #3219] Error renew delegation tokens: Unknown version of delegation token 8
cbbafd51 [zhouyifan279] [KYUUBI #3219] Error renew delegation tokens: Unknown version of delegation token 8
394308e6 [zhouyifan279] [KYUUBI #3219] Error renew delegation tokens: Unknown version of delegation token 8
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
fix#3204
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3205 from jiaoqingbo/kyuubi-3204.
Closes#3204
185dd427 [jiaoqingbo] [KYUUBI #3204]In LdapAuthenticationProviderImplSuite afterAll() method，ldapServer.close() should be called only once
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add component version util class
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3181 from deadwind4/engine-version-util.
Closes#3180
12dcfbed [Luning Wang] Delete method in companion object
222a4ccc [Luning Wang] Change name to SemanticVersion
d254c0e5 [Luning Wang] Use case class and add method in companion class
f943ebd0 [Luning Wang] Add test case
e267a1fd [Luning Wang] Copy ComponentVersion class to authz module
343a2ded [Luning Wang] [KYUUBI #3180] Add component version util
Authored-by: Luning Wang <wang4luning@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
1. add a warning message when the user sets the deprecated configuration
```
10:49:58.558 WARN org.apache.kyuubi.config.KyuubiConf: The Kyuubi config 'kyuubi.frontend.bind.port' has been deprecated in Kyuubi v1.4.0 and may be removed in the future. Use kyuubi.frontend.thrift.binary.bind.port instead
```
2. make the annotation `deprecated` arguments only contain constants to fix the error when compiling for scala-2.13
```
[ERROR] /home/chenfu/workspace/incubator-kyuubi/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala:353: annotation argument needs to be a constant; found: "using ".+(KyuubiConf.this.FRONTEND_THRIFT_BINARY_BIND_PORT.key).+(" instead")
```
### _How was this patch tested?_
Pass CI.
Closes#3195 from cfmcgrady/log-deprecated-config.
Closes#3194
2c7567fc [Fu Chen] revert log4j2-test.xml
b97306d5 [Fu Chen] deprecated config
264d1276 [Fu Chen] log deprecated config
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The current JDBC Kerberos authentication and Hadoop UGI classes are strongly coupled, it's not friendly for downstream projects which do not have Hadoop dependencies, and users who are not familiar w/ Hadoop security mechanism.
This PR proposes to replace the UGI-based Kerberos authentication with JAAS-based.
The main logic is
```
if (param.auth == 'noSasl') {
// no-sasl
raw socket/http requrest
} else if (!param.principal) {
// sasl/plain
auth by param.username, param.password
} else if (param.principal && param.kerberosAuthType == 'fromSubject') {
[0]
} else if (param.principal && param.kyuubiClientPrincipal && param.kyuubiClientKeytab) {
[1]
} else if (param.principal && !param.kyuubiClientPrincipal && !param.kyuubiClientKeytab) {
[3]
}
- [0][JDK] subject from context
- [1][JDK] principle + keytab
- [2][UGI] principle + keytab (no required, use [1] instead)
- [3][JDK] principle + tgt cache
- [4][UGI] principle + tgt cache (no required, use [3] instead)
```
To achieve this, we need to introduce new JDBC parameters `kyuubiClientPrincipal` and `kyuubiClientKeytab`.
The above description is pure JDK-based and totally removed Hadoop dependencies from Kyuubi Hive JDBC driver.
To minimize the scope of this PR, I separate the support of delegation token to a follow-up PR #3096, and it still needs UGI.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3023 from pan3793/ugi.
Closes#3023
7b45f471 [Cheng Pan] Remove UGI in JDBC driver and use JAAS-based Kerberos authentication instead
3876c3d6 [Cheng Pan] Add util method isCommandAvailable
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
To fix the NoClassDefFoundError
```
Error: org.apache.kyuubi.KyuubiSQLException: org.apache.kyuubi.KyuubiSQLException: Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/bridge/SLF4JBridgeHandler
at org.apache.kyuubi.Logging.initializeLogging(Logging.scala:97)
at org.apache.kyuubi.Logging.initializeLoggerIfNecessary(Logging.scala:89)
at org.apache.kyuubi.Logging.initializeLoggerIfNecessary$(Logging.scala:85)
at org.apache.kyuubi.engine.flink.FlinkSQLEngine$.initializeLoggerIfNecessary(FlinkSQLEngine.scala:62)
at org.apache.kyuubi.Logging.logger(Logging.scala:43)
at org.apache.kyuubi.Logging.logger$(Logging.scala:41)
at org.apache.kyuubi.engine.flink.FlinkSQLEngine$.logger(FlinkSQLEngine.scala:62)
at org.apache.kyuubi.engine.flink.FlinkSQLEngine$.main(FlinkSQLEngine.scala:72)
at org.apache.kyuubi.engine.flink.FlinkSQLEngine.main(FlinkSQLEngine.scala)
Caused by: java.lang.ClassNotFoundException: org.slf4j.bridge.SLF4JBridgeHandler
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
See more: /Users/chengpan/app/kyuubi/work/anonymous/kyuubi-flink-sql-engine.log.0
at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69)
at org.apache.kyuubi.engine.ProcBuilder.$anonfun$start$1(ProcBuilder.scala:219)
at java.lang.Thread.run(Thread.java:750)
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3143 from pan3793/bridge.
Closes#3143
266c9bd1 [Cheng Pan] Check class loadable before applying SLF4JBridgeHandler
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
https://github.com/apache/iceberg/releases/tag/apache-iceberg-0.14.0
Notable changes:
- Support Spark 3.3
- Unified package name
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3050 from pan3793/iceberg.
Closes#3050
3e700a11 [Cheng Pan] Fix ut caused by SPARK-37931
039c4e2d [Cheng Pan] Enable Iceberg test w/ Spark 3.3
53cac396 [Cheng Pan] Bump Apache Iceberg 0.14.0
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Because the client can specify any ip in http header, to prevent CVE issue, we do not use it for authentication.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3078 from turboFei/http_real_ip_cve.
Closes#3052
e7c41ea9 [Fei Wang] prevent cve
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Close#3051
We deploy kyuubi server on k8s and using k8s lb and the connection path is:
client -> lb -> kyuubi server.
Now the client ip address is always the lb ip and is meaning less.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3048 from turboFei/session_ip.
Closes#3051
97f9b004 [Fei Wang] fix
524ddbb9 [Fei Wang] comments
ebf02113 [Fei Wang] comments
501d2177 [Fei Wang] comments
25da7e5e [Fei Wang] comments
f634321d [Fei Wang] remove unused new line
faed580a [Fei Wang] comments
403aafc6 [Fei Wang] reserve the kyuubi session ip to engine side
95bd4ec9 [Fei Wang] remove static
6c09fa80 [Fei Wang] refactor
20d269bd [Fei Wang] only in server side
dba85f36 [Fei Wang] get from session ip
591e7fb0 [Fei Wang] add key
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
To close#3052
If your server is behind a load balancer or other proxy, the server will see this load balancer or proxy IP address as the client IP address, to get around this common issue, most load balancers or proxies offer the ability to record the real remote IP address in an HTTP herader that will be added to the request for other devices to use.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3053 from turboFei/http_real_ip.
Closes#3052
b375d02a [Fei Wang] add ut
23ef1c9c [Fei Wang] refactor
907c61c0 [Fei Wang] also take affect for http
a958c50f [Fei Wang] add conf
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Handles configuring the JUL -> SLF4J bridge, otherwise, the components which use JUL like Jersey will be not controlled by log4j2 configurations
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3037 from pan3793/jul.
Closes#3037
fda05ff6 [Cheng Pan] update license
31f579fd [Cheng Pan] nit
7da0f9fb [Cheng Pan] Fix dependencyList
3179e6bd [Cheng Pan] Handles configuring the JUL -> SLF4J bridge
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
After #2948 and #2953, the OPERATION_THRIFT_CLIENT_REQUEST_MAX_ATTEMPTS is not used.
In this pr, I remove the attempts config entry and related ut that based on the request attempts.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3012 from turboFei/follow_up_engine_alive.
Closes#3012
3d15e44a [Fei Wang] remove deprecated docs
89c9f2cb [Fei Wang] remove docs
edb6ff2e [Fei Wang] remove ut
696bd853 [Fei Wang] saev
f91c61bf [Fei Wang] remove unused attempt
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2896 from hddong/trino-result-info.
Closes#2895
757019cf [hongdongdong] fix scala style
f40b1c53 [hongdongdong] Fix
3edbdfb3 [hongdongdong] [KYUUBI #2895] Show final info in trino engine
Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
### _Why are the changes needed?_
To close#2801
Customer might set some parameters to upload the local files, such as spark.files=/etc/security/keytabs/kyuubi.keytab. It is not security for kyuubi server.
In this pr, I introduce a local dir allow list and customer can only upload the files under these dirs.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2991 from turboFei/local_white_list.
Closes#2801
6315067d [Fei Wang] comments
deb9fa8a [Fei Wang] remove unused
3330fa98 [Fei Wang] refactor
6eca2f8b [Fei Wang] move to KyuubiApplicationManager and add ut
1290d9e2 [Fei Wang] add more
f14ec7d4 [Fei Wang] add more path configs and add comment
82357183 [Fei Wang] refactor
f201f577 [Fei Wang] disable relative path
f806e69a [Fei Wang] refactor
4f2faa6a [Fei Wang] save
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Remove `conf/log4j2.properties.template` as it has been replaced by `conf/log4j2.xml.template`
Remove `<ThresholdFilter level="FATAL"/>` in default stdout appender.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3000 from pan3793/log4j-followup.
Closes#2850
ccb9dd6e [Cheng Pan] [KYUUBI #2850][FOLLOWUP] Fix default log4j2 configuration
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix typo in KyuubiConf and mark more config entries server only.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2993 from turboFei/fix_typo.
Closes#2993
002e911f [Fei Wang] add more
53ef59a0 [Fei Wang] save
d5a55683 [Fei Wang] add into server only
fea3daa4 [Fei Wang] Fix typo
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
This is a follow up of #2850
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2971 from turboFei/remove_on_match_from_default.
Closes#2850
10e6067e [Fei Wang] refactor
929fde7d [Fei Wang] comments
4ad12f11 [Fei Wang] add log4j1
91168b6c [Fei Wang] fix
e5ac285e [Fei Wang] fix
efe789eb [Fei Wang] [KYUUBI #2850][FOLLOWUP] Use log4j2 xml to take replace of log4j2 properties
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Remove TProtocolVersion from SessionHandle/OperationHandle/Operation classes, and now it only kept in Session
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2966 from yaooqinn/proto.
Closes#2966
f698d7d3 [Kent Yao] [WIP] Remove TProtocolVersion from SessionHandle/OperationHandle
83f3676a [Kent Yao] [WIP] Remove TProtocolVersion from SessionHandle/OperationHandle
10024e9e [Kent Yao] [WIP] Remove TProtocolVersion from SessionHandle/OperationHandle
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
1. bugfix for issue #2960
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2961 from CavemanIV/kyuubi-2960.
Closes#2960
8aa38eb9 [zhangliang] TFrontendService always use newest hive cli TProtocolVersion
25e5110e [zhangliang] [KYUUBI #2960] TFrontendService.SERVER_VERSION shall be HIVE_CLI_SERVICE_PROTOCOL_V11
Authored-by: zhangliang <zhangliang@trip.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
We have to use canonical host name as kyuubi instance, otherwise, the ip is always change after pod restart, and the kyuubi instance can not get recovered.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2957 from turboFei/kyuubi_instance_url.
Closes#2957
ac8d9df4 [Fei Wang] respect FRONTEND_CONNECTION_URL_USE_HOSTNAME
c4dcd7d3 [Fei Wang] use getCanonicalHostName for EmbeddedZookeeper
c183b41f [Fei Wang] use getCanonicalHostName for TFrontendService
59f3c0a6 [Fei Wang] Use getCanonicalHostName
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
The Operation Type is only used for converting `TOperationType` through specific Thrift Operation APIs. So we can TOperationType in these APIs directly, such as, during calling TFrontendService.ExecuteStatement, we can set TOperationType.EXECUTE_STATEMENT at the end. There is no need to add it to OperationType.
This PR removes `OperationType` completely to simply `OperationHandle` which benefits other `kyuubi.frontend.protocols`.
The thrift protocol shall be able to be removed in following PRs.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2952 from yaooqinn/opType.
Closes#2952
af00c04a [Kent Yao] address comments
4e4f4341 [Kent Yao] address comments
54a2c4ff [Kent Yao] [WIP] Remove Operation Type
ba08ccaf [Kent Yao] [WIP] Remove Operation Type
1737176b [Kent Yao] [WIP] Remove Operation Type
8fe00820 [Kent Yao] [WIP] Remove Operation Type
f969d73d [Kent Yao] [WIP] Remove Operation Type
778f8f4f [Kent Yao] [WIP] Remove Operation Type
54f6573c [Kent Yao] [WIP] Remove Operation Type
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
to close#2949
Unquoted the function name in the error `SECOND_FUNCTION_ARGUMENT_NOT_INTEGER` since Spark-3.4.0, for more details, see https://github.com/apache/spark/pull/36693
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2950 from cfmcgrady/kyuubi-2949.
Closes#2949
4d4da0ba [Fu Chen] fix
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Remove thrift request timeout for KyuubiSyncThriftClient
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2948 from turboFei/remove_request_timeout.
Closes#2948
a9c5c084 [Fei Wang] remove request timeout
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
The cipher is not thread-safe.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2933 from turboFei/cipher_not_thread_safe.
Closes#2894
ea3f79c03 [Fei Wang] not thread safe
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
fix#2927
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2928 from iodone/dev-2.
Closes#2927
a5cb1908 [odone] [KYUUBI #2927] fixed
Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
