### _Why are the changes needed?_
- as pyspark is supported in #3780, exposing PYTHON option for `kyuubi.operation.language` as an experimental feature
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4256 from bowenliang123/doc-language-python.
Closes#4256
d10c0281 [liangbowen] update description
d05cedda [liangbowen] add PYTHON option in kyuubi.operation.language
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Prefer `https://` URLs in docs, and all changed URLs are validated.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4235 from bowenliang123/https-link.
Closes#4235
f114dde2 [liangbowen] update AllKyuubiConfiguration
ad8aaedf [liangbowen] style
e973be5a [liangbowen] update
2370f4bf [liangbowen] prefer https URLs in docs
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
This PR proposes to enhance the LDAP support, which mainly referring the code introduced in HIVE-14713.
Currently, Kyuubi has very limited LDAP support, and the implementation is from the early Hive codebase. Hive enhanced the LDAP support in later versions, considering the Hive ecosystem is quite mature, I think we'd better to porting this functionality and keep the same behavior w/ Hive first, and we can improve it if meet certain requirements/issues in the future.
Basically, this PR introduces the following configurations
```
kyuubi.authentication.ldap.url (since 1.0.0)
kyuubi.authentication.ldap.domain (since 1.0.0)
kyuubi.authentication.ldap.guidKey (since 1.2.0)
kyuubi.authentication.ldap.base.dn (since 1.0.0 deprecated)
kyuubi.authentication.ldap.baseDN
kyuubi.authentication.ldap.groupMembershipKey
kyuubi.authentication.ldap.userMembershipKey
kyuubi.authentication.ldap.groupClassKey
kyuubi.authentication.ldap.groupDNPattern
kyuubi.authentication.ldap.userDNPattern
kyuubi.authentication.ldap.groupFilter
kyuubi.authentication.ldap.userFilter
kyuubi.authentication.ldap.customLDAPQuery
kyuubi.authentication.ldap.binddn
kyuubi.authentication.ldap.bindpw
```
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
This PR ports all LDAP-related UT&IT from Hive codebase
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4152 from pan3793/ldap.
Closes#4152
d251c959 [Cheng Pan] nit
6d14f44b [Cheng Pan] nit
6b3d116c [Cheng Pan] nit
ab47d822 [Cheng Pan] nit
a56e8702 [Cheng Pan] nit
4624619a [Cheng Pan] nit
b82c0c05 [Cheng Pan] LDAP test password uses alphanumeric
86a01cca [Cheng Pan] Enhance LDAP authentication
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- fix word spelling typos in docs
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4226 from bowenliang123/doc-word-typo.
Closes#4226
393de90d [liangbowen] update
365cdc4b [liangbowen] fix word typos in docs
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- to consolidate styles in markdown files from manual written or auto-generated
- apply markdown formatting rules with flexmark from [spotless-maven-plugin](https://github.com/diffplug/spotless/tree/main/plugin-maven#markdown) to *.md files in `/docs`
- use `flexmark` to format markdown generation in `TestUtils` of common module used by `AllKyuubiConfiguration` and `KyuubiDefinedFunctionSuite`, as the same way in `FlexmarkFormatterFunc ` of `spotless-maven-plugin` using with `COMMONMARK` as `FORMATTER_EMULATION_PROFILE` (https://github.com/diffplug/spotless/blob/maven/2.30.0/lib/src/flexmark/java/com/diffplug/spotless/glue/markdown/FlexmarkFormatterFunc.java)
- using `flexmark` of` 0.62.2`, as the last version requiring Java 8+ (checked from pom file and bytecode version)
```
<markdown>
<includes>
<include>docs/**/*.md</include>
</includes>
<flexmark></flexmark>
</markdown>
```
- Changes applied to markdown doc files,
- no style change or breakings in built docs by `make html`
- removal all the first blank in licences and comments to conform markdown style rules
- tables regenerated by flexmark following as in [GitHub Flavored Markdown](https://help.github.com/articles/organizing-information-with-tables/) (https://github.com/vsch/flexmark-java/wiki/Extensions#tables)
### _How was this patch tested?_
- [x] regenerate docs using `make html` successfully and check all the markdown pages available
- [x] regenerate `settings.md` and `functions.md` by `AllKyuubiConfiguration` and `KyuubiDefinedFunctionSuite`, and pass the checks by both themselves and spotless check via `dev/reformat`
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4200 from bowenliang123/markdown-formatting.
Closes#4200
1eeafce4 [liangbowen] revert minor changes in AllKyuubiConfiguration
4f892857 [liangbowen] use flexmark in markdown doc generation
8c978abd [liangbowen] changes on markdown files
a9190556 [liangbowen] apply markdown formatting rules with `spotless-maven-plugin` to markdown files with in `/docs`
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Add catalog tree for `kyuubi.authentication` options
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
<img width="746" alt="image" src="https://user-images.githubusercontent.com/26535726/212236908-294b7fa9-e3db-4730-8c16-41c8fd409ae8.png">
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4159 from pan3793/auth-doc.
Closes#4159
8d65768d [Cheng Pan] [DOCS] Refine doc for kyuubi.authentication
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
As Kyuubi graduated as top level project, the setting page will be more often requested and should be increasingly reliable and readable with less grammar and spelling mistakes.
This PR is to
- correct mistakes in grammar, spelling, abbreviation and terminology
- with no config name or essential meanings changed
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4161 from bowenliang123/conf-grammar.
Closes#4161
038edfbea [liangbowen] nit
1ec073a4b [liangbowen] to JSON
4f5259a32 [liangbowen] to Prometheus
523855008 [liangbowen] to K8s
fc7a3a81e [liangbowen] to AUTO-GENERATED
da64f54fa [liangbowen] update
d54f9a528 [liangbowen] fix `comma separated` to `comma-separated`
f1d7cc1f1 [liangbowen] update
d84208844 [liangbowen] update
1b75f011c [liangbowen] correction of grammar and spelling mistakes
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Some links of the website are not available.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4165 from yuruguo/update_doc-url.
Closes#4165
f7a847c3 [Ruguo Yu] [DOCS] Update the docs link
Authored-by: Ruguo Yu <jiang7chengzitc@163.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
We should put the authorization info in request header, user,password,ip in SessionOpenRequest are unnecessary.
### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3739 from hddong/rm-unused-params.
Closes#3739
7fac61853 [hongdongdong] update
8f1f4ae01 [hongdongdong] added on migration guide
95962b472 [hongdongdong] fix
63606350b [hongdongdong] [REST] Remove unused parameters in SessionOpenRequest
Authored-by: hongdongdong <hongdd@apache.org>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
After some offline discussions, I propose to change the configuration key from`kyuubi.operation.result.codec` to `kyuubi.operation.result.format`.
### _How was this patch tested?_
Pass CI.
Closes#4075 from cfmcgrady/arrow-conf-rename.
Closes#4075
5ad45507 [Fu Chen] fix
214b43fd [Fu Chen] rename
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4069 from lightning-L/kyuubi-4020.
Closes#4020
97406ca0 [Tianlin Liao] [KYUUBI #4020] remove incubating from kyuubi source code
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Now, we support SQL, Scala and Python.
In this pr, refactor the session type: `SQL` => `INTERACTIVE`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4011 from turboFei/interactive_session_type.
Closes#4011
e4a4cae88 [fwang12] doc
106019d65 [fwang12] fix ut
7e8fa36c2 [fwang12] please
da4876449 [fwang12] retest
8cfe30b18 [fwang12] SQL => interactive
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Unlike `kyuubi.frontend.bind.port`, which is deprecated and replaced by `kyuubi.frontend.thrift.binary.bind.port`, `kyuubi.frontend.bind.host` is still encouraged to use, and it can be overwritten by `kyuubi.frontend.<protocol>.bind.host`, e.g. `kyuubi.frontend.thrift.binary.bind.host`, `kyuubi.frontend.thrift.http.bind.host`, `kyuubi.frontend.rest.bind.host`, `kyuubi.frontend.mysql.bind.host`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4005 from pan3793/doc.
Closes#4005
9c3b1277 [Cheng Pan] comments
0ef140e4 [Cheng Pan] Remove deprecated from description of kyuubi.frontend.bind.host
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
close#3905
### _Why are the changes needed?_
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3977 from iodone/kyuubi-3905.
Closes#3977Closes#3905
4e4a22b1 [odone] trino server init and run
Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
After some thought and some offline discussions, I propose to change the configuration key from `kyuubi.engine.pool.balance.policy` to `kyuubi.engine.pool.selectPolicy`.
And in the future if we introduce properties for specific policy, e.g. supposing we have a engine selection policy named `LOAD`, the properties would be
```
kyuubi.engine.pool.selectPolicy.load.abc=xxxx
kyuubi.engine.pool.selectPolicy.load.xyz=xxxx
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3966 from pan3793/rename.
Closes#2887
7c1eda70 [Cheng Pan] address comments
bf3a3863 [Cheng Pan] [KYUUBI #2887][FOLLOWUP] Rename configuration to kyuubi.engine.pool.selectPolicy
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add `Kyuubi Deploy On Kubernetes ServiceAccount-related` documentation, and explicitly assert serviceAccountName: default as the default configuration in the template file `${KYUUBI_HOME}/docker/kyuubi-pod(deployment).yaml` to make it easier for users to set.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3906 from zwangsheng/doc/kyuubi_on_k8s_service_account.
Closes#3902
e0ed06d6 [Cheng Pan] Update docs/deployment/kyuubi_on_kubernetes.md
6cd29dd0 [zwangsheng] build doc for serviceAccount
Lead-authored-by: zwangsheng <2213335496@qq.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Followup for #3957 comments
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3963 from turboFei/unlimited_user.
Closes#3957
987b71f1e [fwang12] refactor
e0c6ca515 [fwang12] [KYUUBI #3957][FOLLOWUP] Refactor the session connection unlimited user list config
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Prevent batch session & operation closed by kyuubi because of timeout
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3923 from turboFei/batch_idle.
Closes#3923
76b0fe16 [fwang12] use val
310119b0 [fwang12] refactor
61ce2036 [fwang12] use session timeout
52fb0cc4 [fwang12] 1.6.2
7819f2091 [fwang12] save
69ffa316 [fwang12] check batch session timeout
dd210329 [fwang12] add batch op timeout
e304ddb4 [fwang12] never idle batch session and never timeout batch op
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Support dedicated limiter for batch session.
Support user white list.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3957 from turboFei/batch_session_limiter.
Closes#3957
55db79e4 [fwang12] support white list
940b76bd [fwang12] add batch session limiter
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Refer the spark scheduler, warn that batch application maybe starved.
<img width="766" alt="image" src="https://user-images.githubusercontent.com/6757692/206888162-663a3ce0-597a-4818-8f7e-04e7be06cef4.png">
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3959 from turboFei/only_state_change.
Closes#3959
6ab59420 [fwang12] show the batch statvation timeout warn
74be3f48 [fwang12] only state change
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3946 from turboFei/config_rest_pool_size.
Closes#3946
30641c14 [fwang12] save
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Kyuubi supports GROUP engine share level, currently, it just simply delegates the group provider to Hadoop UserGroupInformation, which is not flexible enough for users who want to use other group mapping mechanisms, e.g. LDAP, JDBC.
This PR supplies a pluggable plugin interface `GroupProvider` and provides a built-in `HadoopGroupProvider` which has the same behavior w/ the current implementation.
W/ this change, users can easily implement `LDAPGroupProvider`, `JDBCGroupProvider`, `FileGroupProvider`, `CustomGroupProvider`, etc. then the GROUP engine share level will be more powerful and flexible.
The alternative option is to guide users to learn and extend the Hadoop group mapping system[1].
[1] https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/GroupsMapping.html
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3897 from pan3793/group.
Closes#3897
b100348e [Cheng Pan] nit
3cce6451 [Cheng Pan] Supplying plugable GroupProvider
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close#3839 .
The session signing feature is introduced with asymmetric encryption to prevent manipulation of session user identity or other key session elements, which could cause privilege leaking in scripts.
1. Server: Server creates and holds the singleton `keypair` for session signing if feature enabled
2. Server -> Engine: Server passes the `public key` to Engine when launching
3. Server -> Engine: Server generates session signing on `session user` when opening Kyuubi Session to Engine
4. Session -> Statement: Kyuubi session create Statement with context bringing `publickey` and `session user signature`
5. Engine: Engine verify `session user signature` with signature wherever necessary, e.g. in Authz for session name authentication.
ECDSA, with a shorter key length and better performance than RSA, is the supported asymmetric encryption in the initial implementation, which is widely supported on JDK7+ in HotSpot or OpenJDK. The session signature is generated with `SHA256withECDSA`.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3838 from bowenliang123/check-sessionuser.
Closes#3839
925eee47 [liangbowen] Revert "generalize setSparkLocalProperty"
d726d661 [liangbowen] generalize setSparkLocalProperty
470176db [liangbowen] allow setSparkLocalProperty to set value null
8f2a6e38 [liangbowen] nit
cb5891c9 [liangbowen] nit
edbe3c9c [liangbowen] only set to opensession conf when true, since kyuubi.session.user.sign.enabled is default to false
60546c82 [liangbowen] remove setting setSessionSigningPublicKey in KyuubiSessionManager
9a2a60dd [liangbowen] fix asserts
2e2a51b4 [liangbowen] nit
f37f4dd6 [Bowen Liang] Merge branch 'master' into check-sessionuser
9cd22003 [liangbowen] generalize illegalAccessWithUnverifiedUser in AuthZUtils
c1f27afa [liangbowen] generalize setSessionUserSign and clearSessionUserSign in SparkOperation
3683150f [liangbowen] update error message with `Invalid user identifier`
e143ea77 [liangbowen] fast fail for getting missing session configs
866fc821 [liangbowen] rename param name of base64 encoded pubkey with `Base64` suffix
280a95ef [liangbowen] refactor config key name to new class `ReservedKeys` in Authz
8592070e [liangbowen] nit
bf227e35 [liangbowen] remove redundant ut case
9ed14feb [liangbowen] remove redundant ut case
33a723d0 [liangbowen] add AuthzSessionSigningSuite with session user sign enabled
b77d53e3 [liangbowen] remove config setting in ut
2d56bd30 [liangbowen] remove config setting in ut
a25c1c0b [liangbowen] fail w/ AccessControlException, when kyuubi.session.user.sign.enabled is true and kyuubi.session.user is absent
c8a88fe9 [liangbowen] check not null of userPubKeyStr, userSign
c8bc590a [liangbowen] rename param to publicKeyBase64
b55beb60 [liangbowen] revert to use EC secp256k1 for compatibility
8acfd41d [liangbowen] nit
842b3698 [liangbowen] nit
983585bc [liangbowen] nit
d1003cd7 [liangbowen] nit
f9d6cfb6 [liangbowen] make generateKeyPair return (PrivateKey, PublicKey)
52eaaddc [liangbowen] add ut for SignUtils
b4a44687 [liangbowen] general keypair algorithm
7d40da49 [liangbowen] change to use secp192r1 curve for EC key pair for better performance
1ceed876 [liangbowen] sync settings.md with `false` default value
07d23602 [liangbowen] update ut
eec9d44a [liangbowen] move to _confIgnoreList
b7969446 [liangbowen] create SessionSigningSuite and add ut for 1. verifying user sign , 2. conf kyuubi.session.user.sign.enabled restriction
a50c71a3 [liangbowen] set kyuubi.session.user.sign.enabled to openEngineSessionConf
a2f1ed67 [liangbowen] set kyuubi.session.user.sign.enabled to _confRestrictList
5a12182e [liangbowen] make kyuubi.session.user.sign.enabled default to false, and removed from serverOnlyConfEntries
991a4569 [liangbowen] put SESSION_USER_SIGN_ENABLED in serverOnlyConfEntries
3e863af4 [liangbowen] nit
b232e5c0 [liangbowen] npe
13a046be [liangbowen] nit
2d2a6659 [liangbowen] nit
a0d4721d [liangbowen] move session signing keypair generation to KyuubiSessionImpl on server side. and rename config to KYUUBI_SESSION_SIGN_PUBLICKEY
ab430c39 [liangbowen] make generateKeyPair return Key pair and accept algorithm param
42ee2fe0 [liangbowen] nit
ce5f4af7 [liangbowen] refactoring session pubkey generation on server side (instead of engine side), and passing it to engine via OpenSession op
c2b9d897 [liangbowen] nit: rename to verifySignWithECDSA
d0b2cddb [liangbowen] nit
33a044d0 [liangbowen] add ut
2dc1f57b [liangbowen] change to use spark conf `kyuubi.session.user.sign.enabled` to decide whether verify kyuubi seesion user
b11ba5a5 [liangbowen] clear local prop `kyuubi.session.user.public.key` and `kyuubi.session.user.sign` after execution
2c8b4bf6 [liangbowen] move throwing AccessControlException inside verifyKyuubiSessionUser method
82f5c265 [liangbowen] update conf doc
7cf0d481 [liangbowen] rename config name to `kyuubi.session.user.sign.enabled`
cc2c3570 [liangbowen] typo: fix KYUUBI_SESSION_USER_PUBIC_KEY
bdec509d [liangbowen] update settings.md for config doc
2d00163a [liangbowen] nit
102561a8 [liangbowen] update
af99ea84 [liangbowen] move algorithmSpec to ecKeyPairGenerator
966a327e [liangbowen] update
41064712 [liangbowen] update
9d276799 [liangbowen] add config `kyuubi.session.user.verify.enabled` to control Whether to verify the integrity of session user name in Spark Authz
8a8840f6 [liangbowen] nit
bafd85e9 [liangbowen] replace RSA with ECDSA
2f0c87a5 [liangbowen] KeyPairGenerator init with new SecureRandom instance
7cb31204 [liangbowen] shorten the key size to 1024
5011cf49 [liangbowen] remove unused imports
045fd822 [liangbowen] add to SparkOperation for scala
ce6d394d [liangbowen] move generateRSAKeyPair and signWithRSA to SignUtils.scala
5f295792 [liangbowen] update
1d7f3191 [liangbowen] initial support for signing and verifying `kyuubi.session.user`
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Expose `kyuubi.operation.result.codec` to KyuubiConf
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3866 from cfmcgrady/arrow.
Closes#3794
2047f364 [Fu Chen] update
6f3fe24d [Fu Chen] rename
20748ca7 [Fu Chen] expose `kyuubi.operation.result.codec` to KyuubiConf
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
* [KYUUBI #3814] Make the duration of sessionConfCache in FileSessionConfAdvisor configurable instead of a fixed 10 minutes
* fix ut
* fix ut
* fix doc
* code review
* update doc
* update
* update
### _Why are the changes needed?_
When a kyuubi service connections to multiple engine clusters, we can only explicitly specify relevant parameters after the URL, for example:
`beeline -u 'jdbc:hive2://127.0.0.1:10009/?kyuubi.engineEnv.FLINK_HOME=/opt/flink;kyuubi.ha.namespace=kyuubi-ns-c;kyuubi.engine.type=FLINK_SQL'`
This method is extremely unfriendly to users.
I implemented a SessionConfAdvisor implementation class `org.apache.kyuubi.session.FileSessionConfAdvisor` based on the conf file that to manage session level configurations, so that we can access this cluster only by specifying a unique engine name, for example:
access cluster-a:
`beeline -u 'jdbc:hive2://127.0.0.1:10009/?kyuubi.session.conf.profile=cluster-a'`
access cluster-c:
`beeline -u 'jdbc:hive2://127.0.0.1:10009/?kyuubi.session.conf.profile=cluster-c'`
`kyuubi-session-cluster-a.conf` configuration is as follows:
```
kyuubi.ha.namespace kyuubi-ns-a
kyuubi.engine.type SPARK_SQL
kyuubi.engine.pool.balance.policy POLLING
kyuubi.engineEnv.SPARK_HOME /opt/spark30
kyuubi.engineEnv.HADOOP_CONF_DIR /opt/hadoop_conf_dir
```
`kyuubi-session-cluster-c.conf` configuration is as follows:
```
kyuubi.ha.namespace kyuubi-ns-a
kyuubi.engine.type SPARK_SQL
kyuubi.engineEnv.SPARK_HOME /opt/spark32
kyuubi.engineEnv.HADOOP_CONF_DIR /opt/hadoop_conf_dir
```
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3742 from ychris78/yuqi_1101.
Closes#3742
e1449867 [yuqi] Style
68bbfecf [yuqi] restructure
1c8daa86 [yuqi] Change the cache expiration time to 10 minutes and remove the manual cache
5b4a6d77 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/session/FileSessionConfAdvisor.scala
d332e2d8 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/session/FileSessionConfAdvisor.scala
cb644ab4 [Cheng Pan] Update docs/deployment/settings.md
1ee3c980 [Cheng Pan] Update kyuubi-server/src/test/resources/kyuubi-session-cluster-a.conf
30f8fde0 [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
5ef5841a [yuqi] Cache upgrade
5c2f7461 [yuqi] add cache and fix bug
1b1f9d67 [yuqi] scalastyle
a0507847 [yuqi] License info amendment
335250fc [yuqi] fix bug
9e8d313a [yuqi] Modify according to suggestions
5f57b637 [yuqi] Modify according to suggestions
5e504252 [yuqi] add org.apache.kyuubi.plugin.FileSessionConfAdvisor When kyuubi.session.conf.advisor=org.apache.kyuubi.plugin.FileSessionConfAdvisor,Specify a session level configuration file, which will be combined with default.conf to have an impact. The corresponding configuration file is conf/kyuubi-session-<profile>.conf
0cb71341 [yuqi] add org.apache.kyuubi.plugin.YamlSessionConfAdvisor
b524c221 [yuqi] add org.apache.kyuubi.plugin.YamlSessionConfAdvisor
51729d15 [yuqi] When a kyuubi service connections to multiple engine clusters, we can only explicitly specify relevant parameters after the URL, for example: beeline -u 'jdbc:hive2://127.0.0.1:10009/?kyuubi.engineEnv.FLINK_HOME=/opt/flink;kyuubi.ha.namespace=kyuubi-ns-c;kyuubi.engine.type=FLINK_SQL' This method is extremely unfriendly to users, and it is extremely unsafe to expose namespaces to users. So I added a yaml file(engine-cluster-env.yaml) to manage the engine clusters, so that we can use this cluster only by specifying a unique engine name, for example: use cluster-a: beeline -u 'jdbc:hive2://127.0.0.1:10009/?kyuubi.engine.cluster.name=cluster-a'
Lead-authored-by: yuqi <1450636878@qq.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix#3744
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3771 from zhouyifan279/3744.
Closes#3744
58764296 [zhouyifan279] [KYUUBI #3744][Bug] InvalidACL appears in engine after upgrade to Kyuubi 1.6.0
554e8898 [zhouyifan279] [KYUUBI #3744][Bug] InvalidACL appears in engine after upgrade to Kyuubi 1.6.0
a6bfa3d4 [zhouyifan279] [KYUUBI #3744][Bug] InvalidACL appears in engine after upgrade to Kyuubi 1.6.0
c90470f4 [zhouyifan279] [KYUUBI #3744][Bug] InvalidACL appears in engine after upgrade to Kyuubi 1.6.0
fe55f4ad [zhouyifan279] [KYUUBI #3744][Bug] InvalidACL appears in engine after upgrade to Kyuubi 1.6.0
e262872a [zhouyifan279] [KYUUBI #3744][Bug] InvalidACL appears in engine after upgrade to Kyuubi 1.6.0
ed5e8bda [zhouyifan279] [KYUUBI #3744][Bug] InvalidACL appears in engine after upgrade to Kyuubi 1.6.0
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Set the config version to 1.6.1.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3734 from turboFei/typo_version.
Closes#3720
4a552048 [Fei Wang] [KYUUBI #3720][FOLLOWUP] Using correct config version for CTL_BATCH_LOG_ON_FAILURE_TIMEOUT
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
If the batch failed, we shall try to show the remaining logs with timeout.
### _How was this patch tested?_
Existing UT.
Closes#3720 from turboFei/wait_log_fetched.
Closes#3720
a5d0b37f [Fei Wang] comments
220be4ee [Fei Wang] in same thread
90ada13c [Fei Wang] revert
e3ab5ec8 [Fei Wang] address comments
e94a348c [Fei Wang] refactor
c901e4b6 [Fei Wang] prevent NPE
9ed76685 [Fei Wang] save
1f3560bd [Fei Wang] save
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Current kyuubi build kubernetes client for `KyuubiApplicationOperator` to find or kill spark engine on kubernetes.
This pr aims to extract the code for building kubernetes client into `KubernetesUtils` and to do auto build following the logic in Spark.
We prefer to building with user-configured parameters; When missing, the kubernetes client logic will build one from environment variables and kubeconfig.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3663 from zwangsheng/feature/perfer_build_from_pod_env.
Closes#3663
0a4c8c79 [zwangsheng] clear properties
ce408db0 [zwangsheng] fix test
fb28ea7a [zwangsheng] add ci test
943c68dd [zwangsheng] add unit test
ddc9a2b2 [zwangsheng] fix scala
75380d13 [zwangsheng] comment
e6826609 [zwangsheng] master
683cc260 [zwangsheng] context provided master
7b42133c [zwangsheng] add doc
b56829c5 [zwangsheng] fix helm
5884475d [zwangsheng] rebuild
a7f0b551 [zwangsheng] ad
e7f9ec05 [zwangsheng] add setting
b77bbd9b [zwangsheng] add doc
3f3253d3 [zwangsheng] init
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Support SSL for kyuubi thrift binary conenction.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
Will add screenshot for IT.
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3658 from turboFei/ssl_binary.
Closes#3658
ffd603ec [Fei Wang] comment
ce652b7e [Fei Wang] revert UT
52e24abb [Fei Wang] ut
f6cae357 [Fei Wang] save
c6b664f1 [Fei Wang] exclude passwords
d34a4ca2 [Fei Wang] comments
11bfa9ad [Fei Wang] refactor
e2ca77ea [Fei Wang] align
86aaf87c [Fei Wang] with alternative
d17e0d11 [Fei Wang] style
eb4ba52a [Fei Wang] Support SSL for Kyuubi thrift binary connection
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
As described in #2887, random policy may cause task-hot-issues or production accident, so a POLLING balance policy is added to avoid this problem, we can "set kyuubi.engine.pool.balance.policy = POLLING" to use it.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3662 from ychris78/epbp_1018.
Closes#2887
90834c1f [yuqi] fix bug
c0ce8a7e [yuqi] Fix typos
bec4ca13 [yuqi] Fix typos
dc4e3637 [yuqi] please add test case to cover the added methods
837c920b [yuqi] rename method getAndInc to getAndIncrement and modify the doc
02a8311f [yuqi] add getAndInc method and refer it
9436937d [yuqi] Improve configuration documentation
21b5d7d1 [yuqi] [KYUUBI #2887] Add a polling balance policy for spark engine pool Improve configuration documentation and delete useless comments
f0699b3c [yuqi] [KYUUBI #2887] Add a polling balance policy for spark engine pool
Authored-by: yuqi <1450636878@qq.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
to close#3615
steps to reproduce
1. start kyuubi server
2. connect to the kyuubi server with client beeline and run query `select 1`
3. force kill the spark application
4. connect to the kyuubi server with another client **A** immediately
```
Error: org.apache.kyuubi.KyuubiSQLException: Error operating LaunchEngine: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at org.apache.kyuubi.client.KyuubiSyncThriftClient$.createTProtocol(KyuubiSyncThriftClient.scala:466)
at org.apache.kyuubi.client.KyuubiSyncThriftClient$.createClient(KyuubiSyncThriftClient.scala:482)
at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:128)
at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1$adapted(KyuubiSessionImpl.scala:113)
at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36)
at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:113)
at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$2(LaunchEngine.scala:60)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.aache.thrift.transport.TSocket.open(TSocket.java:221)
... 14 more (state=,code=0)
```
If the engine exits without calling the shutdown hook, the ZNODE with EPHEMERAL_SEQUENTIAL mode will be deleted upon the client's disconnect, and note that the engine will be marked as disconnected after `kyuubi.zookeeper.embedded.max.session.timeout` (the default value is 1 minutes)
This PR retries opening the engine when encountering a specialty exception (java.net.ConnectException). Before this PR, client **A** will throw an exception, after this PR, client **A** will work properly.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3618 from cfmcgrady/kyuubi-3615.
Closes#3615
96acd66b [Fu Chen] address comment
a1395a60 [Fu Chen] fix style
870216d1 [Fu Chen] Retry opening the engine when encountering a special error
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
fix#3640
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3641 from jiaoqingbo/kyuubi3640.
Closes#3640
c1b2d4a6 [jiaoqingbo] code review
a65bc132 [jiaoqingbo] code review
ab04f135 [jiaoqingbo] [KYUUBI #3640] Change the default event logger description of hive and trino to json instead of spark
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
When Kyuubi runs outside of K8s, and w/o enhanced DNS infrastructure, Kyuubi can not access the Pod by using the hostname of Pod, it blocks the user to run Spark on K8s w/ cluster mode out-of-box.
Kyuubi provided a configuration `kyuubi.frontend.connection.url.use.hostname`, turn it off could address this issue, but we can not change the default value globally because of https://github.com/apache/incubator-kyuubi/issues/2266
To improve user experience, we can detect if the Driver is running inside the Pod, and if yes,
change `kyuubi.frontend.connection.url.use.hostname` default value to `false`.
Close#3578
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3597 from pan3793/k8s.
Closes#3597
8b411781 [Cheng Pan] doc
0df15e79 [Cheng Pan] Engine should prefer to use ip for registing on K8s cluster mode
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3512
Currently, Kyuubi EventBus uses `scala.concurrent.ExecutionContext.Implicits.global` to execute async event handler. Generally, it's discouraged to use that global ec, instead, we should create dedicated ec for each workload, this pr aims to use dedicated ExecutionContext for EventBus async execution
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3524 from Yikf/ec.
Closes#3512
9757372d [yikf] Use dedicated ExecutionContext for EventBus async execution
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close https://github.com/apache/incubator-kyuubi/issues/3449, this pr aims to change `kyuubi.server.info.provider` default value to "ENGINE", and modify corresponding tests.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3476 from Yikf/default-server-info.
Closes#3449
bc474d3b [Cheng Pan] Update docs/deployment/migration-guide.md
23e7690e [yikf] Change default server info provider to ENGINE
Lead-authored-by: yikf <yikaifei1@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The default value of kyuubi.engine.event.loggers is SPARK, but it can't be used outside the spark engine, so we add the config for each engines.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3309 from zhaomin1423/add_engine_logger_conf.
Closes#3309
37b3c05f [Xiao Zhao] fix
b22356f2 [Min] fix typo
431bb02b [Min] update settings.md
44475f46 [Min] Add configs of engine event logger for each engine
Lead-authored-by: Min <zhaomin1423@163.com>
Co-authored-by: Xiao Zhao <zhaomin1423@163.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Workaround for #3032, close#3323
There are known some ODBC drivers e.g. [Databricks ODBC driver](https://www.databricks.com/spark/odbc-drivers-download) depending on `TGetInfoType.CLI_DBMS_VER` and `TGetInfoType.CLI_DBMS_NAME` to check server compatibilities and abilities.
This PR proposes to introduce a new configuration `kyuubi.server.info.provider=SERVER/ENGINE` to make GetInfo support return either server or engine information.
Since beeline will call GetInfo in the initialization phase, to make sure the beeline fast open experience, in async launch mode, when the engine is not ready, return server info regardless `kyuubi.server.info.provider`.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
Testing w/ PowerBI
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3122 from pan3793/info.
Closes#3122
742bdbe3 [Cheng Pan] nit
bb85d2ba [Cheng Pan] style
fd75238c [Cheng Pan] fix
9ddb2afd [Cheng Pan] nit
fd8f7979 [Cheng Pan] fix ut
840205e5 [Cheng Pan] nit
f9996d53 [Cheng Pan] GetInfo supports returning engine info
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3414
Currently, Kyuubi supports output of planOnly in different modes in different styles, As more and more modes and styles are supported, defining enumerations in kyuubiConf is a bit messy and not easy to extend, so this pr aims to tidy up these
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3415 from Yikf/tidy-up.
Closes#3414
e81142e5 [yikf] Tidy up plan only related modes and styles
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3376
This pr aims support planOnly output in different styles (Users can perform SQL scheduling based on the JSON description of TreeNode)
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3377 from Yikf/planOnly-support-json.
Closes#3376
b63ae718 [yikf] Support json_description for PlanOnlyStatement
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3357
This pr aims to improve doc description for configuration kyuubi.operation.plan.only.mode
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3358 from Yikf/planOnly-doc.
Closes#3357
571f4e0c [yikf] Improve doc description for configuration `kyuubi.operation.plan.only.mode`
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
close https://github.com/apache/incubator-kyuubi/issues/3128
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3304 from Kwafoor/master.
Closes#3128
88a5ebe9 [wangjunbo] [kyuubi-3128] Support CostMode for PlanOnlyStatement
Authored-by: wangjunbo <wangjunbo@qiyi.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
To fix the config name and placeholder with `username` introduced in https://github.com/apache/incubator-kyuubi/pull/3235 violate this convention as in JDBC driver use `user` keyword used for connection user rather than `username`,
1. change config name from `kyuubi.authentication.jdbc.username` to `kyuubi.authentication.jdbc.user`
2. change placeholder from `${username}` to `${user}`
3. update docs and config description related to above changes, and sync the update in jdbc auth docs statement details to config docs.
4. fix error in throwing AuthenticationException with auth db password. ut added for the fix.
5. other minor update in docs of custom auth
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3288 from bowenliang123/jdbc-auth-config-update.
Closes#3222
502703fb [Bowen Liang] skip map for placeholder value lookup
3733be41 [liangbowen] nit
ab00525b [liangbowen] nit
2301c4f0 [liangbowen] fix ut of jdbc auth with wrong_password
06f0c1bb [liangbowen] remove redundant docs
ec4565b3 [liangbowen] remove redundant docs
ae1cce2e [liangbowen] fix compilation error of configLog
5d14103b [liangbowen] simplify configLog
6678e657 [liangbowen] reformat
52c1038e [liangbowen] simplify placeholder checking
21c2d5ea [liangbowen] check whether placeholders in supported list before conn establishment or authenticate
7db0adf5 [liangbowen] ut for unknown placeholder
657de6af [liangbowen] nit
736b3f22 [liangbowen] refactoring placeholder value lookup, for preventing setString multiple times with "i+1"
86c89125 [liangbowen] setMaxRows after prepare placeholder, to postpone operation on jdbc conn
115fae50 [liangbowen] increase test code coverage
b45b28cb [liangbowen] resultSet returned by executeQuery is never null
e1c07274 [liangbowen] update ut for redactPassword in JdbcUtils
b4a52e29 [liangbowen] fix typo in docs of custom auth
371c2c6e [liangbowen] move redactPassword method to JdbcUtils and add ut.
a4973c59 [liangbowen] reformat code
486e1503 [liangbowen] fix error in throwing AuthenticationException with auth db password. add ut for the fix.
efced90a [liangbowen] update settings.md
ef97e354 [liangbowen] add SELECT prefix hint for doc of kyuubi.authentication.jdbc.query
025f94c7 [liangbowen] fix username to user in JdbcAuthenticationProviderImpl by 1. use config name `kyuubi.authentication.jdbc.user`, 2. use ${user} placeholder instead of ${username}
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
fix https://github.com/apache/incubator-kyuubi/issues/3269
Add docs for JDBC auth provider from https://github.com/apache/incubator-kyuubi/issues/3222 shipping within 1.6.0, add jdbc.rst in /docs/security path
simplify the config docs for kyuubi.authentication.jdbc.query in /docs/deployment/settings.md by moving examples to the docs of jdbc auth provider.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3275 from bowenliang123/doc-jdbc-auth.
Closes#3275
e393f6db [liangbowen] fix config example mistakes
ab70cc8d [liangbowen] update doc for auth query sql
ebd5e194 [liangbowen] update doc description
0b15af59 [liangbowen] typo
953000fe [liangbowen] typo
ab944e73 [liangbowen] typo
e3c3d259 [liangbowen] regenerate conf docs. udpate jdbc.md.
45429ada [liangbowen] typo
3b05338e [liangbowen] add docs for jdbc authentication provider
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
