### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3357
This pr aims to improve doc description for configuration kyuubi.operation.plan.only.mode
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3358 from Yikf/planOnly-doc.
Closes#3357
571f4e0c [yikf] Improve doc description for configuration `kyuubi.operation.plan.only.mode`
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3217#issuecomment-1219271537
This pr aims to bump ranger version from 2.2.0 to 2.3.0
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3333 from Yikf/pr/3282.
Closes#3333
07b35b9f [Cheng Pan] Update log4j2-test.xml
45a403b4 [yikf] [KYUUBI #3217][FOLLOWUP] Bump ranger version to 2.3.0
Lead-authored-by: yikf <yikaifei1@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add v1.6.0-incubating changelog.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3342 from SteNicholas/changelog-1.6.0-incubating.
Closes#3342
76bf8c6e [SteNicholas] Add v1.6.0-incubating changelog
Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close https://github.com/apache/incubator-kyuubi/issues/3128
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3304 from Kwafoor/master.
Closes#3128
88a5ebe9 [wangjunbo] [kyuubi-3128] Support CostMode for PlanOnlyStatement
Authored-by: wangjunbo <wangjunbo@qiyi.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Currently, Apache Kyuubi support Spark 3.0.0 and above, but Kyuubi-AuthZ support Spark 2.4, Spark 3.0 and above, to avoid subsequent maintenance costs, we plan to move out support for Spark 2.4.
This pr aims to modify the documentation to indicate that version 2.4 is not supported
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3305 from Yikf/authz-spark2.4-notsupport.
Closes#3226
5c756f51 [yikf] authz not support spark2.4
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
To fix the config name and placeholder with `username` introduced in https://github.com/apache/incubator-kyuubi/pull/3235 violate this convention as in JDBC driver use `user` keyword used for connection user rather than `username`,
1. change config name from `kyuubi.authentication.jdbc.username` to `kyuubi.authentication.jdbc.user`
2. change placeholder from `${username}` to `${user}`
3. update docs and config description related to above changes, and sync the update in jdbc auth docs statement details to config docs.
4. fix error in throwing AuthenticationException with auth db password. ut added for the fix.
5. other minor update in docs of custom auth
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3288 from bowenliang123/jdbc-auth-config-update.
Closes#3222
502703fb [Bowen Liang] skip map for placeholder value lookup
3733be41 [liangbowen] nit
ab00525b [liangbowen] nit
2301c4f0 [liangbowen] fix ut of jdbc auth with wrong_password
06f0c1bb [liangbowen] remove redundant docs
ec4565b3 [liangbowen] remove redundant docs
ae1cce2e [liangbowen] fix compilation error of configLog
5d14103b [liangbowen] simplify configLog
6678e657 [liangbowen] reformat
52c1038e [liangbowen] simplify placeholder checking
21c2d5ea [liangbowen] check whether placeholders in supported list before conn establishment or authenticate
7db0adf5 [liangbowen] ut for unknown placeholder
657de6af [liangbowen] nit
736b3f22 [liangbowen] refactoring placeholder value lookup, for preventing setString multiple times with "i+1"
86c89125 [liangbowen] setMaxRows after prepare placeholder, to postpone operation on jdbc conn
115fae50 [liangbowen] increase test code coverage
b45b28cb [liangbowen] resultSet returned by executeQuery is never null
e1c07274 [liangbowen] update ut for redactPassword in JdbcUtils
b4a52e29 [liangbowen] fix typo in docs of custom auth
371c2c6e [liangbowen] move redactPassword method to JdbcUtils and add ut.
a4973c59 [liangbowen] reformat code
486e1503 [liangbowen] fix error in throwing AuthenticationException with auth db password. add ut for the fix.
efced90a [liangbowen] update settings.md
ef97e354 [liangbowen] add SELECT prefix hint for doc of kyuubi.authentication.jdbc.query
025f94c7 [liangbowen] fix username to user in JdbcAuthenticationProviderImpl by 1. use config name `kyuubi.authentication.jdbc.user`, 2. use ${user} placeholder instead of ${username}
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
fix https://github.com/apache/incubator-kyuubi/issues/3269
Add docs for JDBC auth provider from https://github.com/apache/incubator-kyuubi/issues/3222 shipping within 1.6.0, add jdbc.rst in /docs/security path
simplify the config docs for kyuubi.authentication.jdbc.query in /docs/deployment/settings.md by moving examples to the docs of jdbc auth provider.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3275 from bowenliang123/doc-jdbc-auth.
Closes#3275
e393f6db [liangbowen] fix config example mistakes
ab70cc8d [liangbowen] update doc for auth query sql
ebd5e194 [liangbowen] update doc description
0b15af59 [liangbowen] typo
953000fe [liangbowen] typo
ab944e73 [liangbowen] typo
e3c3d259 [liangbowen] regenerate conf docs. udpate jdbc.md.
45429ada [liangbowen] typo
3b05338e [liangbowen] add docs for jdbc authentication provider
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Support macros in Row-filter condition expression, introduced in Ranger 2.3 ([release notes](https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+2.3.0+-+Release+Notes)), is an major feature to significantly simplify the row-filter condition expression in practice by replacing explicit condition query by using user/group's attributes.
- [RANGER-3605](https://issues.apache.org/jira/browse/RANGER-3605) : Support macros in row-filter/condition expressions
- [RANGER-3550](https://issues.apache.org/jira/browse/RANGER-3550) : support for using user/tag attributes in row-filter expressions and conditions
Consider user liangtiancheng with attribute born_city = guangzhou, we can define the row filter condition with city='${{USER.born_city}}' with the macro feature.
However, This feature implicit relies on an config named `ranger.plugin.spark.enable.implicit.userstore.enricher` and the default value false will prevent RangerUserStoreEnricher fetching user/group and their attributes. Macros in row-filter condition will fallback to null value (as lack of user attributes value in UserStore of auth context) in script transformation unexpectedly and imperceptibly.
Improving doc of ranger-spark-security.xml to aware of this feature and related config.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3267 from bowenliang123/doc-ranger-macros.
Closes#3217
aee4b201 [liangbowen] plain text
a7ec3bc9 [liangbowen] update docs and clearify difference between ranger.plugin.hive.policy.cache.dir and ranger.plugin.spark.policy.cache.dir configs
4887bd1f [liangbowen] simplify with "row filter expressions"
fa62402f [liangbowen] skip list items
9dd1cd4c [liangbowen] h5. Using Marcos in Row Level Filters
849bed5b [liangbowen] add docs for Additional configs for using Marcos in row-level filter
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Among the kyuubi ldap configuration parameters, there is currently "kyuubi.authentication.ldap.base.dn=", but there is no place corresponding to the CN password setting. similar to hive
"hive.server2.authentication.ldap.binddn",
"hive.server2.authentication.ldap.bindpw".
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3213 from dev-lpq/branch-ldap-1.6.
Closes#3020
31817cc6 [pengqli] upgrade ldap to support ldap before 1.6.0
31094030 [pengqli] upgrade ldap to support ldap before 1.6.0
f62e4c96 [pengqli] upgrade ldap to support ldap before 1.6.0
e03e036e [pengqli] add AUTHENTICATION_LDAP_GUIDKEY deprecated
6f680dfc [pengqli] add ldap config AUTHENTICATION_LDAP_BINDDN
19f8e6ce [pengqli] add ldap config AUTHENTICATION_LDAP_BINDDN
07df56b8 [pengqli] upgrade LdapAuthenticationProviderImpl
1a48f418 [pengqli] upgrade settings.md
8e9b5721 [pengqli] upgrade ldap ldapBindpw
5793a811 [pengqli] upgrade ldap ldapBindpw
822d4c30 [pengqli] upgrade docs/deployment/settings.md
c2b20418 [pengqli] upgrade ldap WithLdapServer UTs
5d7a301c [pengqli] add AUTHENTICATION_LDAP_PASSWORD into serverOnlyConfEntries
10b42b1b [pengqli] upgrade AuthenticationProviderFactorySuite UTs
490bfae8 [pengqli] kyuubi ldap add new config property kyuubi.authentication.ldap.bindnpw and kyuubi.authentication.ldap.attrs
5e071252 [pengqli] upgrafe config kyuubi.authentication.ldap.attrs
4e771ec6 [pengqli] kyuubi ldap add new config property kyuubi.authentication.ldap.bindnpw and kyuubi.authentication.ldap.attrs
Authored-by: pengqli <pengqli@cisco.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
We should add news in https://incubator.apache.org/projects/kyuubi.html if we release a new version.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3268 from ulysses-you/docs.
Closes#3268
527c98ec [ulysses-you] status file
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
master branch has supported 5 engine types. the docs of `kyuubi.engine.type` missed hive and jdbc.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3255 from ulysses-you/engine-type.
Closes#3255
a1f3c320 [ulysses-you] add config
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Add JDBC authentication provider as implementation of PasswdAuthenticationProvider out of box.
Compared to currently support authentication methods like Kerberos and LDAP, JDBC source is a much easy and stable source for practical deployment.
The solution should provide:
easy to use and config jdbc connection details
handy to customize query statements for authentication
Adding JDBC to AuthMethods enum and JDBCPasswdAuthenticationProvider into package org.apache.kyuubi.service.authentication with listed features:
- specify JDBC driver name and load the driver class
- configs of JDBC url, username and password
- select query with placeholders for checking user and password , like `SELECT 1 from user_pass_hash where username = ${user} and password = MD5(${password})`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3235 from bowenliang123/feature-jdbc-auth-provider.
Closes#3222
17403b33 [liangbowen] cleanup docs
d5f43e0e [liangbowen] remove unuseful logs for unrecognized placeholder error
e9af0966 [liangbowen] use clone instead of repeatly generating configs
6fc42bf3 [liangbowen] code styling
a9404fa3 [liangbowen] use {} for intercept
77f5f86b [liangbowen] remove unuseful comment
6765affb [liangbowen] changed to use in-memory derby db for test
543c66cb [liangbowen] prefer scala style string usage
a4fe582f [liangbowen] refactor connection creation on using HikariDataSource in HikariCP. add HikariCP dependencies to kyuubi-common
3a4d5fe9 [liangbowen] update checkConfigs() signature
5a0ac495 [liangbowen] output password length only in checkConfigs
1c956dfb [liangbowen] update KyuubiAuthenticationFactorySuite
4ebe12e2 [liangbowen] add JDBC value to AuthTypes enum
9885f813 [liangbowen] add JDBC condition for getValidPasswordAuthMethod
b9ffac3c [liangbowen] Merge branch 'master' into feature-jdbc-auth-provider
aeb19ce5 [liangbowen] update doc
653bc126 [liangbowen] add more checks for query sql
cdec2066 [liangbowen] more test cases
36729197 [liangbowen] fix ddl statement and remove truncate statement in test
30974d16 [liangbowen] update format
575301ca [liangbowen] update options usage
1dc4187b [liangbowen] update settings.md config doc
cd2c7c2b [liangbowen] update settings.md config doc
15176b2b [liangbowen] fix import orders
46cc1dda [liangbowen] add config docs in docs/deployment/settings.md
7025330b [liangbowen] fix derby startup error in test
df4be568 [liangbowen] update code style
49c18c25 [liangbowen] update
996f7969 [liangbowen] add unit test in JdbcAuthenticationProviderImplSuite
0e7f0ad0 [liangbowen] refactor config and init process.remove unused import.
0dc75fee [liangbowen] implement JDBC Authentication Method
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This pr aims to add spark-3.3 profile in building.md document, it's a followup of the https://github.com/apache/incubator-kyuubi/issues/2619
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3245 from yikf/build-doc.
Closes#3245
02838517 [yikf] Add spark-3.3 profile in building.md
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
To close#3157
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3224 from zwangsheng/issuse/3157.
Closes#3157
33225deb [zwangsheng] fix
b338a4a7 [zwangsheng] fix
d7ba40ad [zwangsheng] fix
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Document Kyuubi Spark TPC-DS connector
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
<img width="819" alt="image" src="https://user-images.githubusercontent.com/26535726/184519770-f2abcc0a-70af-4dd6-b592-3510db9c1bec.png">
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3228 from pan3793/tpcds-doc.
Closes#3228
0cafdf88 [Cheng Pan] [Subtask] Connectors for Spark SQL Query Engine -> TPC-DS
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3234 from deadwind4/fts-flink-doc.
Closes#3072
f389629c [Luning Wang] [KYUUBI #3072][DOC] Add a doc of Flink Table Store for Flink SQL engine
Authored-by: Luning Wang <wang4luning@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
SparkConfParser supports parse bytes and time for better user experience.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3227 from pan3793/tpc.
Closes#3227
a2d88470 [Cheng Pan] style
87e54f58 [Cheng Pan] review
381316eb [Cheng Pan] nit
5a23f0e4 [Cheng Pan] fix
48dfe335 [Cheng Pan] nit
c7c85f75 [Cheng Pan] SparkConfParser support parse bytes and time
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add a doc of the Flink Table Store for the Trino SQL Engine
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3215 from deadwind4/fts-trino-doc.
Closes#3080
fdf1770b [Luning Wang] Fix url and add config path prefix
6cce215a [Luning Wang] [KYUUBI #3080][DOC] Add a doc of the Flink Table Store for the Trino SQL Engine
Authored-by: Luning Wang <wang4luning@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3207 from deadwind4/flink-default-version.
Closes#3206
4c9b5d6c [Luning Wang] Fix test case to fit flink 1.14 and 1.15
09c854e2 [Luning Wang] change sql-client.jar name to fit flink 1.15
eef06d1e [Luning Wang] change the suffix to fit 1.15
93fb44f4 [Luning Wang] Comment out cross validation
8ea10218 [Luning Wang] Fix cross validation
a5e5cdb9 [Luning Wang] Fix github CI Flink version
1ee8a5d9 [Luning Wang] [KYUUBI #3206] Change Flink default version to 1.15
Authored-by: Luning Wang <wang4luning@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Update outdated docs.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3210 from pan3793/spark-extension.
Closes#3210
5e5ebd35 [Cheng Pan] Mention Kyuubi Spark SQL extension supports Spark 3.3
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Minor doc fix.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3191 from pan3793/doc-jar.
Closes#3191
6a9c6dc5 [Cheng Pan] [DOCS] Add missing binary scala version in engine jar name
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3198 from deadwind4/flink-hudi-doc-index.
Closes#3198
70484b0c [Luning Wang] [hotfix] Fix doc index of Hudi Flink connector
Authored-by: Luning Wang <wang4luning@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add a docs for session conf advisor which includes:
1. the step of how to use
2. an example
### _How was this patch tested?_
Closes#3175 from ulysses-you/session-advisor.
Closes#3175
0743644a [ulysses-you] address comment
43f2cda3 [ulysses-you] Add session conf advisor docs
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
close#3150
Expose the metadata store requests metrics:
```
kyuubi.metadata_request.total
kyuubi.metadata_request.failed
kyuubi.metadata_request.retrying
```
`total` and `failed` are cumulative, `retrying` is not.
If there are still some retrying metadata requests, we shall not restart the kyuubi server, otherwise, some metadata requests will lose.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3149 from turboFei/metadata_store_metrics.
Closes#3150
1c63a64b [Fei Wang] comments
c9f55362 [Fei Wang] use meter for retrying
b68dd8e7 [Fei Wang] save
2ae533ba [Fei Wang] save
89f30ed4 [Fei Wang] save
5b491cf0 [Fei Wang] save
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Add a doc of the Hudi connector for the Flink SQL Engine
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3140 from deadwind4/hudi-flink-doc.
Closes#3070
69a2ac4d [Luning Wang] Rename dependencies
d1a01fad [Luning Wang] [KYUUBI #3070][DOC] Add a doc of the Hudi connector for the Flink SQL Engine
Authored-by: Luning Wang <wang4luning@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Support SSL for Etcd
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3105 from hddong/support-etcd-ssl.
Closes#3104
49aadb9c [hongdongdong] change enable to enabled
87fa6269 [hongdongdong] [KYUUBI #3104] Support SSL for Etcd
Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Close#3154
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3155 from zhouyifan279/3154.
Closes#3154
682aaf58 [zhouyifan279] [KYUUBI #3154][Subtask] Connectors for Spark SQL Query Engine -> TiDB
4301ca44 [zhouyifan279] [KYUUBI #3154][Subtask] Connectors for Spark SQL Query Engine -> TiDB
65acabe6 [zhouyifan279] [KYUUBI #3154][Subtask] Connectors for Spark SQL Query Engine -> TiSpark
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Add iceberg connector doc for Trino SQL Engine.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3130 from deadwind4/iceberg-trino-doc.
Closes#3082
2f11524c [Luning Wang] [KYUUBI #3082] Add iceberg connector doc for Trino SQL Engine
Authored-by: Luning Wang <wang4luning@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Add iceberg connector for Flink SQL Engine
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3118 from deadwind4/iceberg-flink-doc.
Closes#3071
bf9d158b [Luning Wang] [KYUUBI #3071][DOC] Add iceberg connector for Flink SQL Engine
Authored-by: Luning Wang <wang4luning@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Add Flink Table Store connector doc for Spark SQL Engine
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3151 from huage1994/3067.
Closes#3067
b2bc67d1 [guanhua.lgh] [KYUUBI #3067][DOC] Add Flink Table Store connector doc for Spark SQL Engine
Authored-by: guanhua.lgh <guanhua.lgh@alibaba-inc.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Close#3126
The error:
```
Running Sphinx v4.5.0
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 231 source files that are out of date
updating environment: [new config] 231 added, 0 changed, 0 removed
reading sources... [ 0%] appendix/index
Extension error (sphinx_markdown_tables):
Handler <function process_tables at 0x102589090> for event 'source-read' threw an exception (exception: TableProcessor.__init__() missing 1 required positional argument: 'config')
make: *** [html] Error 2
```
I checked the lib difference with before, seems the only change is Markdown version.
Before, it was 3.3.7 and now it is 3.4.1.
### _How was this patch tested?_
Re-create the kyuubi virtual env and build docs locally.
Closes#3127 from turboFei/kyuubi_docs.
Closes#3126
ac428ce3 [Fei Wang] nit
38b56bc3 [Fei Wang] Use markdown 3.3.7 for kyuubi documents build
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Add Iceberg connector doc for Spark SQL Engine
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3115 from deadwind4/iceberg-spark-doc.
Closes#3069
4c9adeb0 [Luning Wang] Add merge into
119be819 [Luning Wang] update mulit engine support
eb4180d6 [Luning Wang] [KYUUBI #3069][DOC] Add Iceberg connector doc for Spark SQL Engine
Authored-by: Luning Wang <wang4luning@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add the Hudi connector doc for Spark SQL Query Engine
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3099 from deadwind4/hudi-spark-doc.
Closes#3068
fcd2cf6a [Luning Wang] update doc
0ee870d9 [Luning Wang] [KYUUBI #3068][DOC] Add the Hudi connector doc for Spark SQL Query Engine
Authored-by: Luning Wang <wang4luning@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Fix path errors in the build document
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3109 from deadwind4/K3108.
Closes#3108
23830c6e [Luning Wang] [KYUUBI #3108][DOC] Fix path errors in the build document
Authored-by: Luning Wang <wang4luning@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Build the content for extension points documentation, pre-work for #3100
<img width="1767" alt="image" src="https://user-images.githubusercontent.com/8326978/179930987-1accbbb7-e804-4230-871f-6c4b1152f4a1.png">
1. the extensions are divided into 2: server side and engine side extensions. (Do we have client side extension support?)
2. the server side authentication page is cross-referenced by the security section, see 1 in the picture.
3. the engine side ones are grouped by different compute frameworks.
4. connector is one type of extension, so we cross-reference the connector pages directly, see 2 & 3 in the picture.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3103 from yaooqinn/3101.
Closes#3101
a9ae3e32 [Kent Yao] [KYUUBI #3101] [Subtask][#3100] Build content for extension points documentation
3b7367e9 [Kent Yao] [KYUUBI #3101] [Subtask][#3100] Build content for extension points documentation
b5eda13e [Kent Yao] [KYUUBI #3101] [Subtask][#3100] Build content for extension points documentation
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Because the client can specify any ip in http header, to prevent CVE issue, we do not use it for authentication.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3078 from turboFei/http_real_ip_cve.
Closes#3052
e7c41ea9 [Fei Wang] prevent cve
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Close#3046
Expose the metrics likes:
```
kyuubi.operation.state.BatchJobSubmission.pending
kyuubi.operation.state.BatchJobSubmission.running
kyuubi.operation.state.BatchJobSubmission.finished
kyuubi.operation.state.BatchJobSubmission.error
kyuubi.operation.state.BatchJobSubmission.canceled
```
So that the kyuubi service admin can know that how many BatchJobSubmission operations are pending, running and so on.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3063 from turboFei/batch_metrics.
Closes#3046
89ee21db [Fei Wang] remove statement overwrite
25ae41e9 [Fei Wang] update docs
7287527c [Fei Wang] use meter
a2dde891 [Fei Wang] save
a0c6eade [Fei Wang] remove another bug fix
c250a089 [Fei Wang] add docs
87f27e89 [Fei Wang] comments
6bdc4ca9 [Fei Wang] fix flaky test
b000525d [Fei Wang] fix flaky test
f748093e [Fei Wang] expose more metrics
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
1. The version options should be in zkOptions.
2. Add docs for kyuubi-adminctl.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3066 from turboFei/admin_ctl_docs.
Closes#3045
983d133f [Fei Wang] comments
65675ad7 [Fei Wang] add more
b5b772d5 [Fei Wang] comments
222b409c [Fei Wang] add docs
20c50c00 [Fei Wang] kyuubi-adminctl -> kyuubi-admin
078044e0 [Fei Wang] refactor
4e89b52d [Fei Wang] remove unused
c2d1dc70 [Fei Wang] add admin ctl docs
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Add the docs for batch resource rest api.
Close#3076
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3075 from turboFei/rest_api.
Closes#3076
970498e6 [Fei Wang] Cap
445fde21 [Fei Wang] nit
4b0306df [Fei Wang] refactor
b4a5a74f [Fei Wang] add rest api docs
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Add description of the discovery client in the conf doc.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3054 from hddong/conf-to-etcd.
Closes#3054
d364d099 [hongdongdong] Add description of the discovery client in the conf doc
Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Build content of the connector document section
#### Main changes proposed in this PR
- the `integration` section is renamed to `connector` and lifted up
- connectors are separated by engines
- the usage guide is broken into 2 section: Admin Guide and User Guide
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3061 from yaooqinn/3060.
Closes#3060
6932cbb8 [Kent Yao] [KYUUBI #3060] [Subtask][#3059] Build the content of connector document section
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
To close#3052
If your server is behind a load balancer or other proxy, the server will see this load balancer or proxy IP address as the client IP address, to get around this common issue, most load balancers or proxies offer the ability to record the real remote IP address in an HTTP herader that will be added to the request for other devices to use.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3053 from turboFei/http_real_ip.
Closes#3052
b375d02a [Fei Wang] add ut
23ef1c9c [Fei Wang] refactor
907c61c0 [Fei Wang] also take affect for http
a958c50f [Fei Wang] add conf
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
