### _Why are the changes needed?_
The changes are needed to make doc page clearer and a bit nicer, and reflect latest changes in the chart.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
This patch had conflicts when merged, resolved by
Committer: Cheng Pan <chengpan@apache.org>
Closes#4314 from dnskr/update_helm_chart_doc_page.
Closes#4314
d7c18fd53 [dnskr] [DOCS][HELM] Refine helm chart docs
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add a GA Job For Documentation Verification
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4269 from yaooqinn/docga.
Closes#4269
c9b39de6 [Cheng Pan] Update .github/workflows/docs.yml
db2d37f8 [Kent Yao] Update .github/workflows/docs.yml
c5bc8518 [Kent Yao] trigger ga
6340129b [Kent Yao] trigger ga
fafbf847 [Kent Yao] trigger ga
52f23240 [Kent Yao] trigger ga
1e0000a7 [Kent Yao] Add a GA Job For Documentaion Verification
Lead-authored-by: Kent Yao <yao@apache.org>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- as pyspark is supported in #3780, exposing PYTHON option for `kyuubi.operation.language` as an experimental feature
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4256 from bowenliang123/doc-language-python.
Closes#4256
d10c0281 [liangbowen] update description
d05cedda [liangbowen] add PYTHON option in kyuubi.operation.language
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Prefer `https://` URLs in docs, and all changed URLs are validated.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4235 from bowenliang123/https-link.
Closes#4235
f114dde2 [liangbowen] update AllKyuubiConfiguration
ad8aaedf [liangbowen] style
e973be5a [liangbowen] update
2370f4bf [liangbowen] prefer https URLs in docs
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
This PR proposes to enhance the LDAP support, which mainly referring the code introduced in HIVE-14713.
Currently, Kyuubi has very limited LDAP support, and the implementation is from the early Hive codebase. Hive enhanced the LDAP support in later versions, considering the Hive ecosystem is quite mature, I think we'd better to porting this functionality and keep the same behavior w/ Hive first, and we can improve it if meet certain requirements/issues in the future.
Basically, this PR introduces the following configurations
```
kyuubi.authentication.ldap.url (since 1.0.0)
kyuubi.authentication.ldap.domain (since 1.0.0)
kyuubi.authentication.ldap.guidKey (since 1.2.0)
kyuubi.authentication.ldap.base.dn (since 1.0.0 deprecated)
kyuubi.authentication.ldap.baseDN
kyuubi.authentication.ldap.groupMembershipKey
kyuubi.authentication.ldap.userMembershipKey
kyuubi.authentication.ldap.groupClassKey
kyuubi.authentication.ldap.groupDNPattern
kyuubi.authentication.ldap.userDNPattern
kyuubi.authentication.ldap.groupFilter
kyuubi.authentication.ldap.userFilter
kyuubi.authentication.ldap.customLDAPQuery
kyuubi.authentication.ldap.binddn
kyuubi.authentication.ldap.bindpw
```
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
This PR ports all LDAP-related UT&IT from Hive codebase
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4152 from pan3793/ldap.
Closes#4152
d251c959 [Cheng Pan] nit
6d14f44b [Cheng Pan] nit
6b3d116c [Cheng Pan] nit
ab47d822 [Cheng Pan] nit
a56e8702 [Cheng Pan] nit
4624619a [Cheng Pan] nit
b82c0c05 [Cheng Pan] LDAP test password uses alphanumeric
86a01cca [Cheng Pan] Enhance LDAP authentication
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- fix word spelling typos in docs
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4226 from bowenliang123/doc-word-typo.
Closes#4226
393de90d [liangbowen] update
365cdc4b [liangbowen] fix word typos in docs
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Add docs for the `fast` maven profile
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4225 from bowenliang123/doc-fast-profile.
Closes#4225
29aff164 [liangbowen] update
22ebef61 [liangbowen] typo
24a65920 [liangbowen] nit
803b86ad [liangbowen] doc for the `fast` maven profile
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Trace statement exec time histogram to provide more insights.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4214 from turboFei/sql_exec.
Closes#4214
12f080597 [fwang12] remove min > 0, incase it is 0
1edd11c8c [fwang12] scale out
68ebe52d8 [fwang12] reforamt
73eb652ca [fwang12] operation.exec_time
41e7da736 [fwang12] reformat
3d071c6c8 [fwang12] docs
6c3e4abac [fwang12] ut
0b0fbd465 [fwang12] save
11bca028a [fwang12] histo
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
- Show line number and assertion when verifying generated markdown files, e.g. `settings.md` from `AllKyuubiConfiguration`, for quicker locating the place and differences in files.
- updated hint message to `mvn clean test` instead of `mvn clean install` for faster verification or regeneration
Hints with line num as below if assertion fails in line comparison.
<img width="1121" alt="image" src="https://user-images.githubusercontent.com/1935105/215451115-813b90f0-9d9d-4ebd-974e-8a071424aa42.png">
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4212 from bowenliang123/markdown-verify-linenum.
Closes#4212
82791a88 [liangbowen] style
609c3b35 [liangbowen] update hints
ed889915 [liangbowen] nit
c676653b [liangbowen] updated hints of markdown generation for AllKyuubiConfiguration and KyuubiDefinedFunctionSuite
646832a0 [liangbowen] show line number and content of expected and got when verifying markdown files
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- Reformat markdown file `rest-api.md` with changes from (https://github.com/apache/kyuubi/pull/4199) in with ( https://github.com/apache/kyuubi/pull/4200)
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4210 from bowenliang123/4199-followup.
Closes#4199
f66cf8ee [liangbowen] reformat changes in markdown files
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- to consolidate styles in markdown files from manual written or auto-generated
- apply markdown formatting rules with flexmark from [spotless-maven-plugin](https://github.com/diffplug/spotless/tree/main/plugin-maven#markdown) to *.md files in `/docs`
- use `flexmark` to format markdown generation in `TestUtils` of common module used by `AllKyuubiConfiguration` and `KyuubiDefinedFunctionSuite`, as the same way in `FlexmarkFormatterFunc ` of `spotless-maven-plugin` using with `COMMONMARK` as `FORMATTER_EMULATION_PROFILE` (https://github.com/diffplug/spotless/blob/maven/2.30.0/lib/src/flexmark/java/com/diffplug/spotless/glue/markdown/FlexmarkFormatterFunc.java)
- using `flexmark` of` 0.62.2`, as the last version requiring Java 8+ (checked from pom file and bytecode version)
```
<markdown>
<includes>
<include>docs/**/*.md</include>
</includes>
<flexmark></flexmark>
</markdown>
```
- Changes applied to markdown doc files,
- no style change or breakings in built docs by `make html`
- removal all the first blank in licences and comments to conform markdown style rules
- tables regenerated by flexmark following as in [GitHub Flavored Markdown](https://help.github.com/articles/organizing-information-with-tables/) (https://github.com/vsch/flexmark-java/wiki/Extensions#tables)
### _How was this patch tested?_
- [x] regenerate docs using `make html` successfully and check all the markdown pages available
- [x] regenerate `settings.md` and `functions.md` by `AllKyuubiConfiguration` and `KyuubiDefinedFunctionSuite`, and pass the checks by both themselves and spotless check via `dev/reformat`
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4200 from bowenliang123/markdown-formatting.
Closes#4200
1eeafce4 [liangbowen] revert minor changes in AllKyuubiConfiguration
4f892857 [liangbowen] use flexmark in markdown doc generation
8c978abd [liangbowen] changes on markdown files
a9190556 [liangbowen] apply markdown formatting rules with `spotless-maven-plugin` to markdown files with in `/docs`
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
- add docs in REST API for POST /batches with uploading resource, which was introduced in #4106 .
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4199 from bowenliang123/rest-batchupload-doc.
Closes#4199
957249d4 [liangbowen] add doc for POST /batches (with uploading resource)
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
Trace connection metrics with session type.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4204 from turboFei/conn_type_open.
Closes#4204
67f45fd5d [fwang12] add ut
ef0bed665 [fwang12] save
ed4be0233 [fwang12] align
b29396306 [fwang12] order
94cdc5bfc [fwang12] order
91bee9082 [fwang12] docs
37128668a [fwang12] save
6cef34f10 [fwang12] session type
6a73760a8 [fwang12] reuse
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Add catalog tree for `kyuubi.authentication` options
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
<img width="746" alt="image" src="https://user-images.githubusercontent.com/26535726/212236908-294b7fa9-e3db-4730-8c16-41c8fd409ae8.png">
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4159 from pan3793/auth-doc.
Closes#4159
8d65768d [Cheng Pan] [DOCS] Refine doc for kyuubi.authentication
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4198 from a49a/fts-typo.
Closes#4197
306b16b2 [Luning Wang] [KYUUBI #4197][DOCS] Fix typo in a title of the Flink FTS Connector
Authored-by: Luning Wang <wang4luning@gmail.com>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
### _Why are the changes needed?_
As Kyuubi graduated as top level project, the setting page will be more often requested and should be increasingly reliable and readable with less grammar and spelling mistakes.
This PR is to
- correct mistakes in grammar, spelling, abbreviation and terminology
- with no config name or essential meanings changed
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4161 from bowenliang123/conf-grammar.
Closes#4161
038edfbea [liangbowen] nit
1ec073a4b [liangbowen] to JSON
4f5259a32 [liangbowen] to Prometheus
523855008 [liangbowen] to K8s
fc7a3a81e [liangbowen] to AUTO-GENERATED
da64f54fa [liangbowen] update
d54f9a528 [liangbowen] fix `comma separated` to `comma-separated`
f1d7cc1f1 [liangbowen] update
d84208844 [liangbowen] update
1b75f011c [liangbowen] correction of grammar and spelling mistakes
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Some links of the website are not available.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4165 from yuruguo/update_doc-url.
Closes#4165
f7a847c3 [Ruguo Yu] [DOCS] Update the docs link
Authored-by: Ruguo Yu <jiang7chengzitc@163.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
We should put the authorization info in request header, user,password,ip in SessionOpenRequest are unnecessary.
### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3739 from hddong/rm-unused-params.
Closes#3739
7fac61853 [hongdongdong] update
8f1f4ae01 [hongdongdong] added on migration guide
95962b472 [hongdongdong] fix
63606350b [hongdongdong] [REST] Remove unused parameters in SessionOpenRequest
Authored-by: hongdongdong <hongdd@apache.org>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Remove improper docs saving dataframe for pyspark, as hive-like JDBC driver not supporting `addBatch` method which is required by Spark JDBC datasource in `JDBCUtils`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4133 from bowenliang123/pyspark-remove-doc.
Closes#4133
45c9aa7f [liangbowen] remove improper docs saving dataframe for pyspark,as hive like JDBC driver not supporting `addBatch` which is reuqired by Spark JDBC datasource in `JDBCUtils`
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The changelogs are exported from the git command, which is quite duplicated and does not help much, after removing, users still can get them from GitHub or git history easily.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4094 from pan3793/changelog.
Closes#4094
2d0bfe07 [Cheng Pan] [INFRA] Remove changelogs and generate_changelog.sh
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This is part of https://github.com/apache/kyuubi/issues/4020, and should finalize the update for repo https://github.com/apache/kyuubi
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4089 from pan3793/graduate.
Closes#4089
60ece0c1 [Cheng Pan] [INFRA] Update release scripts and templates because of graduation
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- DISCLAIMER file on top-level folder of project was already removed after graduation in (https://github.com/apache/kyuubi/issues/4020)
- fix build/dist for `cp` non-existed DISCLAIMER file
- remove DISCLAIMER file reference in dev/dist script, doc
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4086 from bowenliang123/desclaimer-ref.
Closes#4086
18be7489 [liangbowen] remove DISCLAIMER file reference in dev/dist script, doc, docker , labeler.yml
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
After some offline discussions, I propose to change the configuration key from`kyuubi.operation.result.codec` to `kyuubi.operation.result.format`.
### _How was this patch tested?_
Pass CI.
Closes#4075 from cfmcgrady/arrow-conf-rename.
Closes#4075
5ad45507 [Fu Chen] fix
214b43fd [Fu Chen] rename
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
fix#4070 ，all commands in alphabetical order
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4072 from jiaoqingbo/kyuubi4070.
Closes#4070
abb62aeb [jiaoqingbo] [KYUUBI #4070] Add missing spark commands to lineage.md
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- add doc for`POST /admin/refresh/user_defaults_conf` REST Api , which introduced in #3982
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4077 from bowenliang123/3982-followup.
Closes#3982
a6acb389 [liangbowen] add REST api doc of `POST /admin/refresh/user_defaults_conf`
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4069 from lightning-L/kyuubi-4020.
Closes#4020
97406ca0 [Tianlin Liao] [KYUUBI #4020] remove incubating from kyuubi source code
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Detect and inject a tag if plan is for writing, then skip doing final stage isolation at query preparation phase.
To make final stage config more flexible with complex Spark application.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3988 from ulysses-you/final-stage.
Closes#3988
d0f2b622 [ulysses-you] fix
e5351fd5 [ulysses-you] nit
39082b20 [ulysses-you] Final stage config isolation support write only
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
The changes are needed to be compliant with Helm recommendations:
1. https://helm.sh/docs/topics/charts/#the-chart-file-structure
```
The directory name is the name of the chart (without versioning information).
Thus, a chart describing WordPress would be stored in a wordpress/ directory.
```
2. https://helm.sh/docs/howto/chart_releaser_action/#repository-changes
```
The charts should be placed under /charts directory at the top-level of the directory tree.
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4034 from dnskr/move_helm_chart_to_dedicated_directory.
Closes#4034
b6f4b661 [dnskr] [K8S][HELM] Move helm chart to dedicated directory
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Update outdated docs.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4024 from QianyongY/features/update-outdated-docs.
Closes#4024
0ad7173f [yongqian] [DOCS] Update the rules documentation for Kyuubi Spark SQL extension
Authored-by: yongqian <yongqian@trip.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Fix spelling(steaming -> streaming) in kyuubi_ecosystem figure
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3992 from yaooqinn/eco.
Closes#3992
d2d95147a [Kent Yao] Fix spelling in kyuubi_ecosystem pic
c1db38a9e [Kent Yao] Fix spelling in kyuubi_ecosystem pic
390d175fe [Kent Yao] Fix spelling in kyuubi_ecosystem pic
d9538ba38 [Kent Yao] Fix spelling in kyuubi_ecosystem pic
77cac723e [Kent Yao] Fix spelling in kyuubi_ecosystem pic
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Now, we support SQL, Scala and Python.
In this pr, refactor the session type: `SQL` => `INTERACTIVE`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4011 from turboFei/interactive_session_type.
Closes#4011
e4a4cae88 [fwang12] doc
106019d65 [fwang12] fix ut
7e8fa36c2 [fwang12] please
da4876449 [fwang12] retest
8cfe30b18 [fwang12] SQL => interactive
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Unlike `kyuubi.frontend.bind.port`, which is deprecated and replaced by `kyuubi.frontend.thrift.binary.bind.port`, `kyuubi.frontend.bind.host` is still encouraged to use, and it can be overwritten by `kyuubi.frontend.<protocol>.bind.host`, e.g. `kyuubi.frontend.thrift.binary.bind.host`, `kyuubi.frontend.thrift.http.bind.host`, `kyuubi.frontend.rest.bind.host`, `kyuubi.frontend.mysql.bind.host`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4005 from pan3793/doc.
Closes#4005
9c3b1277 [Cheng Pan] comments
0ef140e4 [Cheng Pan] Remove deprecated from description of kyuubi.frontend.bind.host
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
close#3905
### _Why are the changes needed?_
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3977 from iodone/kyuubi-3905.
Closes#3977Closes#3905
4e4a22b1 [odone] trino server init and run
Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
After some thought and some offline discussions, I propose to change the configuration key from `kyuubi.engine.pool.balance.policy` to `kyuubi.engine.pool.selectPolicy`.
And in the future if we introduce properties for specific policy, e.g. supposing we have a engine selection policy named `LOAD`, the properties would be
```
kyuubi.engine.pool.selectPolicy.load.abc=xxxx
kyuubi.engine.pool.selectPolicy.load.xyz=xxxx
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3966 from pan3793/rename.
Closes#2887
7c1eda70 [Cheng Pan] address comments
bf3a3863 [Cheng Pan] [KYUUBI #2887][FOLLOWUP] Rename configuration to kyuubi.engine.pool.selectPolicy
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
add two conditions to decide if we should add shuffle.
1. make sure AQE is enabled, otherwise it is no meaning to add a shuffle
2. try to reduce the performance regression if add a shuffle
for condition 2: we do not add shuffle if the original plan does not have shuffle
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3962 from ulysses-you/no-shuffle.
Closes#3962
a084cccc [ulysses-you] address comment
9d0aab1b [ulysses-you] address comment
09fc9b21 [ulysses-you] fix ut
06f249a2 [ulysses-you] Reduce the performance regression
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Add `Kyuubi Deploy On Kubernetes ServiceAccount-related` documentation, and explicitly assert serviceAccountName: default as the default configuration in the template file `${KYUUBI_HOME}/docker/kyuubi-pod(deployment).yaml` to make it easier for users to set.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3906 from zwangsheng/doc/kyuubi_on_k8s_service_account.
Closes#3902
e0ed06d6 [Cheng Pan] Update docs/deployment/kyuubi_on_kubernetes.md
6cd29dd0 [zwangsheng] build doc for serviceAccount
Lead-authored-by: zwangsheng <2213335496@qq.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Followup for #3957 comments
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3963 from turboFei/unlimited_user.
Closes#3957
987b71f1e [fwang12] refactor
e0c6ca515 [fwang12] [KYUUBI #3957][FOLLOWUP] Refactor the session connection unlimited user list config
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Close#3887
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3933 from lightning-L/kyuubi-3887.
Closes#3887
65566232 [Tianlin Liao] [KYUUBI #3887] Provide kyuubiServerPrincipal as alias for principal in JDBC parameter
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Prevent batch session & operation closed by kyuubi because of timeout
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3923 from turboFei/batch_idle.
Closes#3923
76b0fe16 [fwang12] use val
310119b0 [fwang12] refactor
61ce2036 [fwang12] use session timeout
52fb0cc4 [fwang12] 1.6.2
7819f2091 [fwang12] save
69ffa316 [fwang12] check batch session timeout
dd210329 [fwang12] add batch op timeout
e304ddb4 [fwang12] never idle batch session and never timeout batch op
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Support dedicated limiter for batch session.
Support user white list.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3957 from turboFei/batch_session_limiter.
Closes#3957
55db79e4 [fwang12] support white list
940b76bd [fwang12] add batch session limiter
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Refer the spark scheduler, warn that batch application maybe starved.
<img width="766" alt="image" src="https://user-images.githubusercontent.com/6757692/206888162-663a3ce0-597a-4818-8f7e-04e7be06cef4.png">
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3959 from turboFei/only_state_change.
Closes#3959
6ab59420 [fwang12] show the batch statvation timeout warn
74be3f48 [fwang12] only state change
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3946 from turboFei/config_rest_pool_size.
Closes#3946
30641c14 [fwang12] save
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Kyuubi supports GROUP engine share level, currently, it just simply delegates the group provider to Hadoop UserGroupInformation, which is not flexible enough for users who want to use other group mapping mechanisms, e.g. LDAP, JDBC.
This PR supplies a pluggable plugin interface `GroupProvider` and provides a built-in `HadoopGroupProvider` which has the same behavior w/ the current implementation.
W/ this change, users can easily implement `LDAPGroupProvider`, `JDBCGroupProvider`, `FileGroupProvider`, `CustomGroupProvider`, etc. then the GROUP engine share level will be more powerful and flexible.
The alternative option is to guide users to learn and extend the Hadoop group mapping system[1].
[1] https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/GroupsMapping.html
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3897 from pan3793/group.
Closes#3897
b100348e [Cheng Pan] nit
3cce6451 [Cheng Pan] Supplying plugable GroupProvider
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close#3839 .
The session signing feature is introduced with asymmetric encryption to prevent manipulation of session user identity or other key session elements, which could cause privilege leaking in scripts.
1. Server: Server creates and holds the singleton `keypair` for session signing if feature enabled
2. Server -> Engine: Server passes the `public key` to Engine when launching
3. Server -> Engine: Server generates session signing on `session user` when opening Kyuubi Session to Engine
4. Session -> Statement: Kyuubi session create Statement with context bringing `publickey` and `session user signature`
5. Engine: Engine verify `session user signature` with signature wherever necessary, e.g. in Authz for session name authentication.
ECDSA, with a shorter key length and better performance than RSA, is the supported asymmetric encryption in the initial implementation, which is widely supported on JDK7+ in HotSpot or OpenJDK. The session signature is generated with `SHA256withECDSA`.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3838 from bowenliang123/check-sessionuser.
Closes#3839
925eee47 [liangbowen] Revert "generalize setSparkLocalProperty"
d726d661 [liangbowen] generalize setSparkLocalProperty
470176db [liangbowen] allow setSparkLocalProperty to set value null
8f2a6e38 [liangbowen] nit
cb5891c9 [liangbowen] nit
edbe3c9c [liangbowen] only set to opensession conf when true, since kyuubi.session.user.sign.enabled is default to false
60546c82 [liangbowen] remove setting setSessionSigningPublicKey in KyuubiSessionManager
9a2a60dd [liangbowen] fix asserts
2e2a51b4 [liangbowen] nit
f37f4dd6 [Bowen Liang] Merge branch 'master' into check-sessionuser
9cd22003 [liangbowen] generalize illegalAccessWithUnverifiedUser in AuthZUtils
c1f27afa [liangbowen] generalize setSessionUserSign and clearSessionUserSign in SparkOperation
3683150f [liangbowen] update error message with `Invalid user identifier`
e143ea77 [liangbowen] fast fail for getting missing session configs
866fc821 [liangbowen] rename param name of base64 encoded pubkey with `Base64` suffix
280a95ef [liangbowen] refactor config key name to new class `ReservedKeys` in Authz
8592070e [liangbowen] nit
bf227e35 [liangbowen] remove redundant ut case
9ed14feb [liangbowen] remove redundant ut case
33a723d0 [liangbowen] add AuthzSessionSigningSuite with session user sign enabled
b77d53e3 [liangbowen] remove config setting in ut
2d56bd30 [liangbowen] remove config setting in ut
a25c1c0b [liangbowen] fail w/ AccessControlException, when kyuubi.session.user.sign.enabled is true and kyuubi.session.user is absent
c8a88fe9 [liangbowen] check not null of userPubKeyStr, userSign
c8bc590a [liangbowen] rename param to publicKeyBase64
b55beb60 [liangbowen] revert to use EC secp256k1 for compatibility
8acfd41d [liangbowen] nit
842b3698 [liangbowen] nit
983585bc [liangbowen] nit
d1003cd7 [liangbowen] nit
f9d6cfb6 [liangbowen] make generateKeyPair return (PrivateKey, PublicKey)
52eaaddc [liangbowen] add ut for SignUtils
b4a44687 [liangbowen] general keypair algorithm
7d40da49 [liangbowen] change to use secp192r1 curve for EC key pair for better performance
1ceed876 [liangbowen] sync settings.md with `false` default value
07d23602 [liangbowen] update ut
eec9d44a [liangbowen] move to _confIgnoreList
b7969446 [liangbowen] create SessionSigningSuite and add ut for 1. verifying user sign , 2. conf kyuubi.session.user.sign.enabled restriction
a50c71a3 [liangbowen] set kyuubi.session.user.sign.enabled to openEngineSessionConf
a2f1ed67 [liangbowen] set kyuubi.session.user.sign.enabled to _confRestrictList
5a12182e [liangbowen] make kyuubi.session.user.sign.enabled default to false, and removed from serverOnlyConfEntries
991a4569 [liangbowen] put SESSION_USER_SIGN_ENABLED in serverOnlyConfEntries
3e863af4 [liangbowen] nit
b232e5c0 [liangbowen] npe
13a046be [liangbowen] nit
2d2a6659 [liangbowen] nit
a0d4721d [liangbowen] move session signing keypair generation to KyuubiSessionImpl on server side. and rename config to KYUUBI_SESSION_SIGN_PUBLICKEY
ab430c39 [liangbowen] make generateKeyPair return Key pair and accept algorithm param
42ee2fe0 [liangbowen] nit
ce5f4af7 [liangbowen] refactoring session pubkey generation on server side (instead of engine side), and passing it to engine via OpenSession op
c2b9d897 [liangbowen] nit: rename to verifySignWithECDSA
d0b2cddb [liangbowen] nit
33a044d0 [liangbowen] add ut
2dc1f57b [liangbowen] change to use spark conf `kyuubi.session.user.sign.enabled` to decide whether verify kyuubi seesion user
b11ba5a5 [liangbowen] clear local prop `kyuubi.session.user.public.key` and `kyuubi.session.user.sign` after execution
2c8b4bf6 [liangbowen] move throwing AccessControlException inside verifyKyuubiSessionUser method
82f5c265 [liangbowen] update conf doc
7cf0d481 [liangbowen] rename config name to `kyuubi.session.user.sign.enabled`
cc2c3570 [liangbowen] typo: fix KYUUBI_SESSION_USER_PUBIC_KEY
bdec509d [liangbowen] update settings.md for config doc
2d00163a [liangbowen] nit
102561a8 [liangbowen] update
af99ea84 [liangbowen] move algorithmSpec to ecKeyPairGenerator
966a327e [liangbowen] update
41064712 [liangbowen] update
9d276799 [liangbowen] add config `kyuubi.session.user.verify.enabled` to control Whether to verify the integrity of session user name in Spark Authz
8a8840f6 [liangbowen] nit
bafd85e9 [liangbowen] replace RSA with ECDSA
2f0c87a5 [liangbowen] KeyPairGenerator init with new SecureRandom instance
7cb31204 [liangbowen] shorten the key size to 1024
5011cf49 [liangbowen] remove unused imports
045fd822 [liangbowen] add to SparkOperation for scala
ce6d394d [liangbowen] move generateRSAKeyPair and signWithRSA to SignUtils.scala
5f295792 [liangbowen] update
1d7f3191 [liangbowen] initial support for signing and verifying `kyuubi.session.user`
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Expose `kyuubi.operation.result.codec` to KyuubiConf
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3866 from cfmcgrady/arrow.
Closes#3794
2047f364 [Fu Chen] update
6f3fe24d [Fu Chen] rename
20748ca7 [Fu Chen] expose `kyuubi.operation.result.codec` to KyuubiConf
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This PR changes the doc authority from Kent to Apache Kyuubi Community. Meanwhile, the footnotes for copyright/license are also simplified.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3824 from yaooqinn/autority.
Closes#3824
fc25156a [Kent Yao] [DOCS] Authority of the documentation belongs to the community
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: fwang12 <fwang12@ebay.com>
