### _Why are the changes needed?_
to close issue #4058 and help to close pr #3852.
When execute test `KyuubiOnKubernetesWithClientSparkTestsSuite` with `execute statement - select with variable substitution`, the initial method
`org.apache.kyuubi.operation.HiveJDBCTestHelper#withThriftClient` will overlook the connectionConf in `org.apache.kyuubi.operation.JDBCTestHelper#jdbcUrl`:
```
def withThriftClient[T](user: Option[String] = None)(f: TCLIService.Iface => T): T = {
TClientTestUtils.withThriftClient(
jdbcUrl.stripPrefix(URL_PREFIX).split("/;").head,
user)(f)`
}
```
Here we just aborted the tail of `jdbcUrl.stripPrefix(URL_PREFIX).split("/;")`, in which the connectionConfs are set.
In that case the `org.apache.kyuubi.kubernetes.test.deployment.KyuubiOnKubernetesWithClientSparkTestsSuite#connectionConf` was invalid actually.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4059 from xuefeimiaoao/branch-fix-connectionConf.
Closes#4058
a6395631 [xuefeimiaoao] [KYUUBI #4058] [IT][Test][K8S] Fix the missing of connectionConf of SparkQueryTests
Authored-by: xuefeimiaoao <1255072085@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- DISCLAIMER file on top-level folder of project was already removed after graduation in (https://github.com/apache/kyuubi/issues/4020)
- fix build/dist for `cp` non-existed DISCLAIMER file
- remove DISCLAIMER file reference in dev/dist script, doc
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4086 from bowenliang123/desclaimer-ref.
Closes#4086
18be7489 [liangbowen] remove DISCLAIMER file reference in dev/dist script, doc, docker , labeler.yml
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
After some offline discussions, I propose to change the configuration key from`kyuubi.operation.result.codec` to `kyuubi.operation.result.format`.
### _How was this patch tested?_
Pass CI.
Closes#4075 from cfmcgrady/arrow-conf-rename.
Closes#4075
5ad45507 [Fu Chen] fix
214b43fd [Fu Chen] rename
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
It was reported in https://github.com/apache/kyuubi/discussions/4063
When run `SHOW DATABASES;` on Databricks Runtime, the error occurs
```
com.databricks.backend.common.rpc.SparkDriverExceptions$SQLExecutionException: java.lang.UnsupportedOperationException
at org.apache.spark.sql.catalyst.plans.logical.LeafNode.computeStats(LogicalPlan.scala:183)
at org.apache.spark.sql.catalyst.plans.logical.LeafNode.computeStats$(LogicalPlan.scala:183)
at org.apache.kyuubi.plugin.spark.authz.util.ObjectFilterPlaceHolder.computeStats(ObjectFilterPlaceHolder.scala:23)
at org.apache.spark.sql.catalyst.plans.QueryPlanStats.$anonfun$stats$1(QueryPlanStats.scala:39)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.catalyst.plans.QueryPlanStats.stats(QueryPlanStats.scala:38)
at org.apache.spark.sql.catalyst.plans.QueryPlanStats.stats$(QueryPlanStats.scala:38)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.stats(LogicalPlan.scala:31)
```
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4076 from pan3793/stats.
Closes#4076
8238f317 [Cheng Pan] ut
a3c5f7d3 [Cheng Pan] [AUTHZ] Modified query plan should correctly report stats
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
fix#4070 ，all commands in alphabetical order
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4072 from jiaoqingbo/kyuubi4070.
Closes#4070
abb62aeb [jiaoqingbo] [KYUUBI #4070] Add missing spark commands to lineage.md
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add serde layer for queries, which extracts table information from a scan relation, such as HiveTableRelation, logical relation, etc.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4044 from yaooqinn/scan.
Closes#4044
d2d830c50 [Kent Yao] addr comments
3cac8dd54 [Kent Yao] [Authz] Serde for queries
fd5425679 [Kent Yao] [Authz] Serde for queries
d42778999 [Kent Yao] Merge branch 'master' of github.com:apache/incubator-kyuubi into scan2
e19282e1f [Kent Yao] [Authz] Serde for queries
50ee2335d [Kent Yao] [Authz] Serde for queries
78ce87855 [Kent Yao] [Authz] Serde for queries
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
- add doc for`POST /admin/refresh/user_defaults_conf` REST Api , which introduced in #3982
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4077 from bowenliang123/3982-followup.
Closes#3982
a6acb389 [liangbowen] add REST api doc of `POST /admin/refresh/user_defaults_conf`
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
to close#3982 .
Introduce feature of refresh user defaults config (as `___${user}___.*` which starts with three continuous underscores "___") from config file via `kyuubi-admin` cli and `refresh/user_defaults_conf` Rest API.
1. add `refreshUserDefaultsConf` methond in KyuubiServer to read user defautls configs from property file and apply config changes to server's KyuubiConf
3. add `refresh/user_defaults_conf` api to AdminRestApi calling `refreshUserDefaultsConf` of KyuubiServer
3. add config type `userDefautls` in kyuubi-admin cli refresh command
This feature will
- help to apply user defaults conf without restarting server or losing connections
- load latest config for engine launch, e.g. spark related config `spark.*`
It won't
- affect the components already started and using the clone of server conf
- affect configs for launched engine instance
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3983 from bowenliang123/3982-reload-server-conf.
Closes#3983
a8fb0bf2 [liangbowen] fix typo
b9b80f56 [liangbowen] update
b47a1541 [liangbowen] minor
786cb2a0 [liangbowen] add logging for statistics
0860e3a7 [liangbowen] fix: loadFileDefaults in refreshUserDefaultsConf
8dbbbcb8 [liangbowen] fix typo
619acd2e [liangbowen] import
e405dc8e [liangbowen] fix user defaults key filtering by adding `getAllUserDefaults` to `KyuubiConf`
ac407bd5 [liangbowen] rename config refresh option to `userDefaultsConf` and extracted to RefreshConfigCommandConfigType
c65398b2 [liangbowen] fix redundant loadFileDefaults in refreshUserDefaultsConf
1b046feb [liangbowen] update comments
e5dd5dbb [liangbowen] typo
c0a358ba [liangbowen] change to refresh users' config. rename cli command to `refresh config userDefaults`
bf5448e3 [liangbowen] support reload server config from config file 1. add reloadServerConf in KyuubiServer to read config and put all to config, 2. add "refresh/server_conf" api to AdminRestApi, 3. add config type "serverConf" in kyuubi-admin cli
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4069 from lightning-L/kyuubi-4020.
Closes#4020
97406ca0 [Tianlin Liao] [KYUUBI #4020] remove incubating from kyuubi source code
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
9342a29284/kyuubi-ctl/src/main/scala/org/apache/kyuubi/ctl/cmd/log/LogBatchCommand.scala (L134-L141)
We need to reset the operation log before fetching new operation log, otherwise, we might meet corner case.
- The last operation log is not empty
- we meet log not found exception
- we can not terminate the loop because the `shouldFinishLog` is always false
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4067 from turboFei/check_status.
Closes#4067
b61e14f6 [fwang12] style
5787ad97 [fwang12] do not fetch again
6dc479d1 [fwang12] refactor
37a91909 [fwang12] reset the log before fetch log
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Close https://github.com/apache/kyuubi/issues/4065, This pr aims to support `GetTypeInfo` for Trino Fe
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4066 from Yikf/master.
Closes#4065
bafe4ba4 [Yikf] Support GetTypeInfo for Trino Fe
Authored-by: Yikf <yikaifei1@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
We intend to introduce Trino FE at https://github.com/apache/kyuubi/issues/3901, This issue aims to support `GetTableTypes` as subtask.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4054 from Yikf/getTableTypes.
Closes#4053
d702cf85 [Yikf] Support GetTableTypes for trino fe
Authored-by: Yikf <yikaifei1@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
It is now 2023. Happy new year!
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4062 from AdithyaPadmashali/Update-NOTICE.
Closes#4061
4abb2bc3 [AdithyaPadmashali] Update NOTICE-binary to mention 2023
5ba49f7a [AdithyaPadmashali] Update NOTICE to mention 2023
Lead-authored-by: AdithyaPadmashali <adithya.p2001@gmail.com>
Co-authored-by: AdithyaPadmashali <57041819+AdithyaPadmashali@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The changes allow to add initContainers and containers to Kyuubi server pod.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4060 from dnskr/add_initcontainers.
Closes#4060
355d8d0e [dnskr] [K8S][HELM] Add initContainers and containers support to helm chart
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This pr aims to suppress warnning of the maven-antrun-plugin echo;
before:
```log
[WARNING] [echo] unpacking netty jar
[WARNING] [echo] renaming netty native libraries
[WARNING] [echo] deleting META-INF/native-image folder
[WARNING] [echo] repackaging netty jar
```
after:
```log
[INFO] [echo] unpacking netty jar
[INFO] [echo] renaming netty native libraries
[INFO] [echo] deleting META-INF/native-image folder
[INFO] [echo] repackaging netty jar
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4050 from Yikf/echo.
Closes#4050
2729f7988 [Yikf] Suppress warnning of the maven-antrun-plugin echo
Authored-by: Yikf <yikaifei1@gmail.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Followup for #4035
Add `eventEnabled` method for KyuubiOperation and SparkOperation.
For `KyuubiOperation`, enable below operations as before
- ExecuteStatement
- BatchJobSubmission
For `SparkOperation`, disable
- GetTypeInfo
- because it is used by `KyuubiConnection::isValid` and might be called with interval to check session alive
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4046 from turboFei/post_event.
Closes#4035
d558d7706 [fwang12] event enabled
4926c8b56 [fwang12] event enabled
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4045 from turboFei/with_release.
Closes#4045
1382f96ab [fwang12] withAcquireRelease
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
close#3978
### _Why are the changes needed?_
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4033 from iodone/kyuubi-3978.
Closes#4033
2a3ae68b [odone] add datasource v2 support
72feb958 [odone] add datasourcev2 support
Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Close https://github.com/apache/kyuubi/issues/3990
This pr aims to support `GetCatalogs` for trino fe.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4048 from Yikf/catalogs.
Closes#3990
6ace4a6f [Yikf] Support GetCatalogs for trino fe
Authored-by: Yikf <yikaifei1@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
add unit test of row filter for MergeIntoIcebergTable of Iceberg catalog.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4047 from bowenliang123/iceberg-rowfilter-ut.
Closes#4047
c6b7d5e17 [liangbowen] update ut name
c26c61b0d [liangbowen] add e2e ut for row filter of iceberg
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
subtask of #3586 .
1. passing catalog field from `Table` and `Database` to PrivilegeObject for privilege checks. As for`Function` related commands, no catalog supported other than spark_catalog or v1 catalog in Spark 3.1-3.3.
2. extract catalog from `DataSourceRelationV2` in buildQuery with added `CatalogPluginOptionCatalogExtractor`
3. add ut for v2commands SetNamespaceProperties/CreateNamespace/SetNamespaceLocation/DescribeNamespace/DropNameSpace to verify related PrivilegeObject extraction with catalog in spark 3.3
4. in future PR, SparkRangerAdminPlugin will use catalog-related instances for privilege checks using catalog info from PrivilegeObject
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4032 from bowenliang123/catalog-priviobj.
Closes#4032
f66e2f40f [liangbowen] typo
24abdb1dc [liangbowen] add ut for v2commands SetNamespaceProperties/CreateNamespace/SetNamespaceLocation/DescribeNamespace/DropNameSpace
01af55f5f [liangbowen] change assertion to sparkCatalogVal to sparkSessionCatalogName of string type
72dd64e0e [liangbowen] fix catalog assertion for some spark version variation
25160b120 [liangbowen] fix catalog assertion for some spark version variation
4a89b87f4 [liangbowen] fix catalog assertion for some spark version variation
333d47b9e [liangbowen] add catalog assertion to PrivilegesBuilderSuite and V2CommandsPrivilegesSuite
a02b6803e [liangbowen] pass catalog from Database in databasePrivileges
b68a0dc8a [liangbowen] fix
30c7e7de9 [Bowen Liang] Merge branch 'master' into catalog-priviobj
598c6bfef [liangbowen] implement CatalogPluginOptionCatalogExtractor
003c97a58 [liangbowen] add catalog support in method tablePrivileges/v2TablePrivileges of PrivilegesBuilder and pass catalog to PrivilegeObject
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
The changes allow to add volumes to Kyuubi server pod and volumeMounts to Kyuubi server container.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4043 from dnskr/add_volumes_and_volumeMounts.
Closes#4043
a332b92a [dnskr] [K8S][HELM] Add volumes and volumeMounts support to helm chart
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This pr aims to abstract and rename parser module related classes.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4040 from Yikf/trino-fe-parser-rename.
Closes#4040
a13887a7 [Yikf] Abstract parser related classes
Authored-by: Yikf <yikaifei1@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Fixes a typo in README.md from architect to architecture. This is a simple typo fix.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Simple typo fix, and I checked it manually via the web browser.
Closes#4042 from HyukjinKwon/minor-typo.
Closes#4042
dec2faae [Hyukjin Kwon] Fix a typo from architect to architecture in README.md
Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This PR aims to bump [antlr4](https://github.com/antlr/antlr4) from 4.8 to 4.9.3.
Version 4.8 was released on Jan 21, 2020, It's been nearly three years, and there have been a lot of bug fixes during that time, see more at [change list](https://github.com/antlr/antlr4/compare/4.8...4.9.3).
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4039 from Yikf/bump-antlr4.
Closes#4039
24aed3a7 [Yikf] Bump Antlr4 from 4.8 to 4.9.3
Authored-by: Yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Use `globalSparkContext` to simplify the code.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4041 from zwangsheng/improve/diagnostics_deploy_mode_for_k8s_cluster.
Closes#4038
9426944c [zwangsheng] [FOLLOW UP]Improve print diagnostics info in kubernetes cluster deploy-mode
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Detect and inject a tag if plan is for writing, then skip doing final stage isolation at query preparation phase.
To make final stage config more flexible with complex Spark application.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3988 from ulysses-you/final-stage.
Closes#3988
d0f2b622 [ulysses-you] fix
e5351fd5 [ulysses-you] nit
39082b20 [ulysses-you] Final stage config isolation support write only
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
For Spark On Kubernetes Cluster Deploy-Mode, spark will set `spark.submit.deployMode=client` and `spark.kubernetes.submitInDriver=true`.
Kyuubi print diagnostics about deploy-mode for spark sql engine should not only dependence on `spark.submit.deployMode`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4038 from zwangsheng/improve/diagnostics_deploy_mode_for_k8s_cluster.
Closes#4038
5149a0be [zwangsheng] Improve Diagnostics Info in Kubernetes Cluster Mode Case
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Support Trino get schemas statement. We convert statement to operation at Trino frontend.
Note, this pr does not support the whole code path from Trino frontend. This pr aims to add an example how to make Trino frontend work with existed Kyuubi server framework.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4030 from ulysses-you/trino-get-schemas.
Closes#4029
f43408c2 [ulysses-you] address comments
a29bcdb8 [ulysses-you] fix
6fddbe6e [ulysses-you] support trino get schemas
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
1. make DatabaseDesc return added `Database` type instance
2. add `catalogDesc` to DatabaseDesc and `catalog` filed in Database
3. extract catalog for ResolvedNamespaceDatabaseExtractor and ResolvedDBObjectName in databaseExtractors of DSV2
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4015 from bowenliang123/catalog-v2database.
Closes#4015
2d925f046 [liangbowen] remove ut for SetCatalogAndNamespace
cd56ca6a7 [liangbowen] move ut to V2CommandsPrivilegesSuite
36e44415b [liangbowen] update
8ca4f7f9d [liangbowen] add ut for SetCatalogAndNamespace
a29068d21 [liangbowen] move ut for SetNamespaceCommand to V2CommandsPrivilegesSuite
bce9471f2 [liangbowen] revert workaround for blank fieldName of CatalogDesc
63c341797 [Bowen Liang] Merge branch 'master' into catalog-v2database
7d043d7db [liangbowen] remove CurrentCatalogExtractor
49e52ff14 [liangbowen] fix None catalog of `USE NAMESPACE` in spark 3.1/3.2 by filling current catalog name into it
ae1f84030 [liangbowen] implement CurrentCatalogExtractor to fetch current catalog from session state, and fill catalog for SetNamespaceCommand
d1654b936 [liangbowen] implement StringOptionCatalogExtractor for using with StringSeqOptionDatabaseExtractor in SetCatalogAndNamespace
ff464828f [liangbowen] change var order moving catalogSpec before isInput
f924bbc88 [liangbowen] regenerate db spec
a6c58accf [liangbowen] add CatalogPluginCatalogExtractor to catalogDesc for CreateNamespace following StringSeqDatabaseExtractor
696e38c51 [liangbowen] update
08c1c7edd [liangbowen] add ut for ResolvedNamespaceDatabaseExtractor
4101a7294 [liangbowen] use CatalogPluginCatalogExtractor in ResolvedNamespaceDatabaseExtractor
e97f1dd2b [liangbowen] use CatalogPluginCatalogExtractor in ResolvedDBObjectNameDatabaseExtractor
d1b68c25d [liangbowen] support catalog in Database and databaseExtractors
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
fix#4036
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4037 from jiaoqingbo/kyuubi4036.
Closes#4037
f3e550d2 [jiaoqingbo] [KYUUBY #4036] Remove redundant Casting session to KyuubiSession
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This pr aims to tidy up the `Are you willing to submit PR` option in the issue template.
Currently, The `Are you willing to submit PR` option of some Apache kyuubi issue templates has three options as follow:
- [ ] Yes. I can submit a PR independently to improve.
- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
- [ ] No. I cannot submit a PR at this time.
The intent of `Are you willing to submit PR` option is to confirm whether the questioner can submit PR at now, So two options are enough.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4025 from Yikf/issue-template.
Closes#4025
11b08ee8 [Yikf] Tidy up the Are you willing to submit PR option in the issue template
Authored-by: Yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The changes are needed to be compliant with Helm recommendations:
1. https://helm.sh/docs/topics/charts/#the-chart-file-structure
```
The directory name is the name of the chart (without versioning information).
Thus, a chart describing WordPress would be stored in a wordpress/ directory.
```
2. https://helm.sh/docs/howto/chart_releaser_action/#repository-changes
```
The charts should be placed under /charts directory at the top-level of the directory tree.
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4034 from dnskr/move_helm_chart_to_dedicated_directory.
Closes#4034
b6f4b661 [dnskr] [K8S][HELM] Move helm chart to dedicated directory
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Before, if the pyspark environment is not set up correctly，the python response was always `None`.
In this pr, fail if the session python worker process has been exited.
BTW: Filter the empty log.
<img width="1422" alt="image" src="https://user-images.githubusercontent.com/6757692/209502683-49aa9088-8686-4a54-b88c-85881a3fb089.png">
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4026 from turboFei/python_exec.
Closes#4026
499e19b54 [fwang12] more insights
17cefc02e [fwang12] Fail if the session python worker has been exited
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
1. wrap the code with correct delimiter
Before:
```
{"code":"spark.sparkContext.setJobGroup(07753dd9-804e-478f-b84f-bf0735732334, ..., true)","cmd":"run_code"}
```
After:
```
{"code":"spark.sparkContext.setJobGroup('07753dd9-804e-478f-b84f-bf0735732334', '...', True)","cmd":"run_code"}
```
2. using cancelJobGroup for pyspark
Before:
```
'SparkContext' object has no attribute 'clearJobGroup'
```
After:
Using SparkContext.cancelJobGroup
3. Simplify the internal python code and throw exception on failure
We can not trust the user code is formatted correctly and we shall ensure the internal python code is simple and correct to prevent code correctness and even cause result out of sequence.
Such as, the user code might be below(maybe user invoke executePython api)
```
spark.sql('123\'\n\b\t'
```
It is difficult to escape the user code and set the job description as the statement as.
So, in this pr, I simplify the job description, just record its statementId, user can check the original code from log or on UI I think.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4028 from turboFei/python_async_debug.
Closes#4028
51a4c5ea5 [fwang12] typo
6da88d1b3 [fwang12] code style
83f5a48f7 [fwang12] fail the internal python code
5f2db042c [fwang12] remove debug code
3a798cf6c [fwang12] Simplify the statement
c3b4640ca [fwang12] do not lock for close
009f66aaa [fwang12] add ReentrantLock for SessionPythonWorker run python code
39bd861a1 [fwang12] fix
4116dabbc [fwang12] job desc
f16c656fc [fwang12] escape
81db20ccb [fwang12] fix 'SparkContext' object has no attribute 'clearJobGroup'
985118e92 [fwang12] escape for python
f7250c114 [fwang12] revert withLocalProperties
13228f964 [fwang12] debug
e318c698a [fwang12] Revert "prevent timeout"
f81c605e0 [fwang12] prevent timeout
2ca5339e3 [fwang12] test
1390b0f21 [fwang12] remove not needed
26ee60275 [fwang12] remove not needed
93c08ff08 [fwang12] debug
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
extract catalog for ResolvedTable and DataSourceV2Relation in tableExtractors
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3989 from bowenliang123/catalog-v2table.
Closes#3989
88392051c [liangbowen] update
e32dbd54a [liangbowen] revert catalogExtractor and remove spark session for input
0e36e2fb3 [liangbowen] style
90a671640 [liangbowen] remove CurrentCatalogExtractor
77e0f864a [liangbowen] add package name in comments for extractor
5c2642b38 [liangbowen] update CurrentCatalogExtractor
8bdb7d767 [liangbowen] fix ut error and cancel fetching catalog inside IdentifierTableExtractor
6be3496f3 [liangbowen] check catalog for TableIdentifierTableExtractor with DescribeColumnCommand
5ee473de3 [liangbowen] fill catalog in IdentifierTableExtractor/CatalogTableTableExtractor/TableIdentifierTableExtractor
802ea065d [liangbowen] 1. fill catalog for table with current catalog name if None catalog return after catalogdesc extracted 2. implement CurrentCatalogExtractor
273875ae1 [liangbowen] use CatalogPluginCatalogExtractor in ResolvedDbObjectNameTableExtractor
0368ac513 [liangbowen] remove unnecessary seq in ut
0b0ca7802 [liangbowen] revert relevant changes in ResolvedDbObjectNameTableExtractor
7ebcc92f0 [liangbowen] revert relevant changes in ResolvedDbObjectNameTableExtractor
bd13a1ef8 [liangbowen] lint
551153d0d [liangbowen] add ut for DataSourceV2RelationTableExtractor
e13987f86 [liangbowen] extract catalog for DataSourceV2Relation
cdbc705d8 [liangbowen] add ut for ResolvedTableTableExtractor
85f26947a [liangbowen] extract catalog for ResolvedTable
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Bind python and SQL spark session, then the variables we set on the python side can be visited on the SQL side
After this PR, we can change the execution mode from python to sql by running
```python
spark.sql("SET kyuubi.operation.language=SQL").show()
```
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4019 from cfmcgrady/binding-sql.
Closes#4019
2fd16a8e2 [Fu Chen] address comment
2136dfd64 [Fu Chen] fix style
cf8a612ee [Fu Chen] fix ut
57c592ed6 [Fu Chen] fix ut
fed7614dd [Fu Chen] binding python/sql spark session
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Fu Chen <cfmcgrady@gmail.com>
### _Why are the changes needed?_
Update outdated docs.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4024 from QianyongY/features/update-outdated-docs.
Closes#4024
0ad7173f [yongqian] [DOCS] Update the rules documentation for Kyuubi Spark SQL extension
Authored-by: yongqian <yongqian@trip.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4021 from cxzl25/op_metric.
Closes#4021
a8f54ea5 [sychen] operation metrics
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
To fix https://github.com/apache/incubator-kyuubi/issues/3973, we found similar problem during upgrade from 1.5 to 1.6
### _How was this patch tested?_
Manual test on internal cluster, UT will be added later on
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3997 from CavemanIV/kyuubi-3973.
Closes#3973
7dae77cf [John Caveman] fix validation for ut
96adf586 [John Caveman] use conf.get
0b0df721 [John Caveman] style fix
f0f167fb [John Caveman] replace for-conf-key-lambda to option chain for a more consistent coding style
6a308d5c [John Caveman] revert ut, command only read conf from kyuubi-defaults directly
ff999221 [John Caveman] fix kyuubi-ctl list fall back to HA_ZK_QUORUM in kyuubi-default.conf when HA_ADDRESSES is not set
Authored-by: John Caveman <selnteer@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4018 from turboFei/python_async.
Closes#4018
2afe3979 [fwang12] move ut
0d9d2f10 [fwang12] only OK
46d14f4c [fwang12] add ut
0e3a0399 [fwang12] add ut
0f2eba5e [fwang12] add ut
e2718ab2 [fwang12] async python
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The changes allow to:
1. set Kyuubi configuration files:
- `kyuubi-env.sh`
- `kyuubi-defaults.conf`
- `log4j2.xml`
2. restart (recreate) Kyuubi server pods if configuration files have changed
- this probably should be revisited after https://github.com/apache/incubator-kyuubi/pull/3983
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4004 from dnskr/helm_conf_files_support.
Closes#4004
83678254 [dnskr] [K8S][HELM] Add configuration files support to helm chart
Authored-by: dnskr <dnskrv88@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix spelling(steaming -> streaming) in kyuubi_ecosystem figure
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3992 from yaooqinn/eco.
Closes#3992
d2d95147a [Kent Yao] Fix spelling in kyuubi_ecosystem pic
c1db38a9e [Kent Yao] Fix spelling in kyuubi_ecosystem pic
390d175fe [Kent Yao] Fix spelling in kyuubi_ecosystem pic
d9538ba38 [Kent Yao] Fix spelling in kyuubi_ecosystem pic
77cac723e [Kent Yao] Fix spelling in kyuubi_ecosystem pic
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Now, we support SQL, Scala and Python.
In this pr, refactor the session type: `SQL` => `INTERACTIVE`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4011 from turboFei/interactive_session_type.
Closes#4011
e4a4cae88 [fwang12] doc
106019d65 [fwang12] fix ut
7e8fa36c2 [fwang12] please
da4876449 [fwang12] retest
8cfe30b18 [fwang12] SQL => interactive
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Client will retry to open the engine when encountering a special error in [KYUUBI #3615](https://github.com/apache/incubator-kyuubi/pull/3618), but it will lead to `TProtocol` leaks when the attempt failed.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4017 from mattshma/close_client_when_retry_fail.
Closes#3615
769e4ca50 [maming] reformat code style
6bf81dff1 [maming] optimize warn log
088105e90 [maming] optimize code
fc98810e3 [maming] Client close session when retry to open failed
Authored-by: maming <maming@ctrip.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Current
```
2022-12-20 13:00:34.038 INFO org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient:
Get service instance:XXX:54876 and version:Some(1.7.0-SNAPSHOT) under /kyuubi_1.7.0-SNAPSHOT_USER_SPARK_SQL/anonymous/default
```
Fix
```
2022-12-20 13:02:01.622 INFO org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient:
Get service instance:XXX:55039 engine id:local-1671512509106 and version:1.7.0-SNAPSHOT under /kyuubi_1.7.0-SNAPSHOT_USER_SPARK_SQL/anonymous/default
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4016 from cxzl25/print_engine_id.
Closes#4016
5eabfe7fa [sychen] print engine id when get service node
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
