### _Why are the changes needed?_
to close#3592
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3593 from packyan/branch-spark-sql-authz-only-check-permanent-functions.
Closes#3592
fb59e3d8 [Deng An] isPersistentFunction use current db when db name is missing.
4e708976 [Deng An] use Option other than Some as info.getDb may return null.
5aefc66d [Deng An] fix unit test failed
ac38e0c2 [Deng An] generalizing isPersistentFunction method.
fcef751b [Deng An] Update extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala
532fd73d [Deng An] Update extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/PrivilegesBuilder.scala
7877d916 [Deng An] optimize the code
c7005b36 [packyan] Spark SQL authz only consider permanent functions
Lead-authored-by: Deng An <packyande@gmail.com>
Co-authored-by: Deng An <36296995+packyan@users.noreply.github.com>
Co-authored-by: packyan <packyande@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
In Spark, if Kubernetes executor pod is not conforming to Kubernetes naming conventions, a user gets the next error:
```
org.apache.kyuubi.KyuubiSQLException: org.apache.kyuubi.KyuubiSQLException: java.lang.IllegalArgumentException: 'kyuubi-connection-spark-sql-USER-8fc2847d-b65d-4a46-bb41-4a86385cb61c-988995839f29e15f' in spark.kubernetes.executor.podNamePrefix is invalid.
```
It can be fixed by manually setting 'spark.kubernetes.executor.podNamePrefix' in configurations, but since you cannot dynamically set a value for every user to contain their username, it will miss this essential information that helps to distinguish executors one from another. I suggest Kyuubi server adds this config automatically if it's not coming from user.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3590 from hanna-liashchuk/master.
Closes#3385
94613145 [Hanna Liashchuk] Fixed style check
4b2281ce [Hanna Liashchuk] Merged if clauses
2c52988d [hanna-liashchuk] Merge branch 'apache:master' into master
f397cdc7 [Hanna Liashchuk] Added if clause
8b91109c [Hanna Liashchuk] Added EpochMilli to the podNamePrefix
13e8fb53 [hanna-liashchuk] Merge branch 'apache:master' into master
dcef3e10 [Hanna Liashchuk] Set executor.podNamePrefix for k8s if missing
Lead-authored-by: Hanna Liashchuk <g.liashchuk@temabit.com>
Co-authored-by: hanna-liashchuk <47921651+hanna-liashchuk@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Subtask of #3607
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3665 from zhouyifan279/3608.
Closes#3608
bcea68d8 [zhouyifan279] DataSourceV2Relation
a50f0a72 [zhouyifan279] Empty-Commit
5c8e2e7b [zhouyifan279] Fix typo
a2c3ab70 [zhouyifan279] Fix typo
31e730ea [zhouyifan279] Move test from RangerSparkExtensionSuite to HiveCatalogRangerSparkExtensionSuite
35c627ec [zhouyifan279] Add `hasTableOwner` field in `SparkSessionProvider`
49a08e0d [zhouyifan279] Improve code style
2d7e6be1 [zhouyifan279] Run test "[KYUUBI #3607] Support {OWNER} variable defined in Ranger Policy" if owner can be resolved
e975f36a [zhouyifan279] [Subtask] Get table owner from CatalogTable #3608
da9f52ce [zhouyifan279] [Subtask] Get table owner from CatalogTable #3608
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close#3615
steps to reproduce
1. start kyuubi server
2. connect to the kyuubi server with client beeline and run query `select 1`
3. force kill the spark application
4. connect to the kyuubi server with another client **A** immediately
```
Error: org.apache.kyuubi.KyuubiSQLException: Error operating LaunchEngine: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at org.apache.kyuubi.client.KyuubiSyncThriftClient$.createTProtocol(KyuubiSyncThriftClient.scala:466)
at org.apache.kyuubi.client.KyuubiSyncThriftClient$.createClient(KyuubiSyncThriftClient.scala:482)
at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:128)
at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1$adapted(KyuubiSessionImpl.scala:113)
at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36)
at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:113)
at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$2(LaunchEngine.scala:60)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.aache.thrift.transport.TSocket.open(TSocket.java:221)
... 14 more (state=,code=0)
```
If the engine exits without calling the shutdown hook, the ZNODE with EPHEMERAL_SEQUENTIAL mode will be deleted upon the client's disconnect, and note that the engine will be marked as disconnected after `kyuubi.zookeeper.embedded.max.session.timeout` (the default value is 1 minutes)
This PR retries opening the engine when encountering a specialty exception (java.net.ConnectException). Before this PR, client **A** will throw an exception, after this PR, client **A** will work properly.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3618 from cfmcgrady/kyuubi-3615.
Closes#3615
96acd66b [Fu Chen] address comment
a1395a60 [Fu Chen] fix style
870216d1 [Fu Chen] Retry opening the engine when encountering a special error
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Batch job should also submit with `SPARK_USER_NAME`.
See more in pre pr #3527 .
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3664 from zwangsheng/feature/batch_with_user_name.
Closes#3664
76dee3e3 [zwangsheng] style
0d405082 [zwangsheng] fix
a1fcf7fa [zwangsheng] fix
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix kyuubi application manager tag spark kubernetes application with yarn tag.
The reason for this is the need to determine whether resourceManager start with K8S instead of being equal to K8S.
Remove duplicate code from spark batch process builder.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3670 from zwangsheng/bugfix/kyuubi_application_manager_tag_application.
Closes#3670
d4e8aa30 [zwangsheng] fix
7b5419eb [zwangsheng] back
a976f994 [zwangsheng] fix
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
fix#3640
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3641 from jiaoqingbo/kyuubi3640.
Closes#3640
c1b2d4a6 [jiaoqingbo] code review
a65bc132 [jiaoqingbo] code review
ab04f135 [jiaoqingbo] [KYUUBI #3640] Change the default event logger description of hive and trino to json instead of spark
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add a hint in creating batch REST API.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3622 from deadwind4/create-batch-rest-doc.
Closes#3621
0d1e6764 [Luning Wang] [KYUUBI #3621][DOCS] Add a hint in create batch REST api
Authored-by: Luning Wang <wang4luning@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add session resource REST docs in `rest_api.md`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3642 from deadwind4/session-rest-doc.
Closes#3611
3d8e5e6b [Luning Wang] Fix wrong description
8a1372a5 [Luning Wang] [KYUUBI #3611][DOCS] Add session resource REST docs
Authored-by: Luning Wang <wang4luning@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close#3635 .
- change test suits to to use Utils.createTempDir of kyuubi-common module to create temp dir with delete on exit to prevent disk usage leaking
- change signature Utils.createTempDir by making `prefix` param from second place to the first, to make it friendly to use without repeating param name itself. The name`prefix` is more closer to Java's style in `Files.createTempDirectory`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3660 from bowenliang123/3635-deletetmp.
Closes#3635
8947fb0f [Bowen Liang] remove unused imports
c06769d4 [Bowen Liang] nit
dac266c9 [Bowen Liang] - change test suits to to use Utils.createTempDir of kyuubi-common module to create temp dir with delete on exit to prevent disk usage leaking - change signature Utils.createTempDir by making prefix param from second place to the first, to make it friendly to use without repeating param name itself. The nameprefix is more closer to Java's style in Files.createTempDirectory
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3629 from lightning-L/list-engine.
Closes#3626
f63faa53 [Tianlin Liao] fix test case
3d9af3a1 [Tianlin Liao] do not need LIstEnginesResponse
718c115f [Tianlin Liao] refactor
defc9f52 [Tianlin Liao] refactor getUserName and getEngineSpace method
e5015508 [Tianlin Liao] typo and grammer fix
84256f68 [Tianlin Liao] [KYUUBI #3626] AdminResources rest api support to list current engines
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Refer Spark PR: https://github.com/apache/spark/pull/18110
ConfigBuilder builds ConfigEntry which can only read value with one key, if we wanna change the config name but still keep the old one, it's hard to do.
This PR introduce ConfigBuilder.withAlternative, to support reading config value with alternative keys.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3659 from turboFei/conf_alternative.
Closes#3659
e268fef2 [Fei Wang] add ut
a2300b26 [Fei Wang] add ut
53eccf99 [Fei Wang] Support alternative keys in ConfigBuilder
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Improve the rebalance before writing rule.
The rebalance before writing rule adds a rebalance at the top of query for data writing command, however the default partitioning of rebalance uses RoundRobinPartitioning which would break the original partitioning of data. It may cause the the output data size bigger than before.
This pr supports infer the columns from join and aggregate for rebalance and sort to improve the compression ratio.
Note that, this improvement only works for static partition writing.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3601 from ulysses-you/smart-order.
Closes#3601
c190dc1a [ulysses-you] docs
995969b5 [ulysses-you] view
ea23c417 [ulysses-you] Support infer columns for rebalance and sort
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Fix `getPrimaryKeys` and `getCrossReference` descriptions typo of session REST open API
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3625 from deadwind4/cross-ref-rest-typo.
Closes#3624
c2e586d9 [Luning Wang] [KYUUBI #3624][REST] Fix description typo of session REST open api
Authored-by: Luning Wang <wang4luning@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix#3627.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
manual test, and the class not found error is gone.
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3630 from pan3793/jetty.
Closes#3627
bf9e4f9b [Cheng Pan] private
f6c496f6 [Cheng Pan] nit
25a7c8cf [Cheng Pan] [KYUUBI #3627] Support vanilla Jetty for Spark packaged by sbt
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close#3635 .
- change test suits to to use Utils.createTempDir of kyuubi-common module to create temp dir with delete on exit to prevent disk usage leaking
- change signature Utils.createTempDir by making `prefix` param from second place to the first, to make it friendly to use without repeating param name itself. The name`prefix` is more closer to Java's style in `Files.createTempDirectory`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3636 from bowenliang123/3635-tempdir.
Closes#3635
5f84a16f [Bowen Liang] nit
b82a149f [Bowen Liang] rename `namePrefix` param to `prefix` of `Utils.createTempDir` method, and make it in first place
76d33143 [Bowen Liang] delete files on exit in test suits with Utils.createTempDir
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
- remove duplicate and shadowed v2Commands `DropNamespace` and `MergeIntoTable` from `PrivilegesBuilder`, since they are already implemented in #3424 .
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3639 from bowenliang123/3424-followup2.
Closes#3424
232dd751 [Bowen Liang] remove `DropNamespace` and `MergeIntoTable` in `PrivilegesBuilder` which shadowed and implemented in v2Commands.scala
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3529
The intent of this PR is the following:
- Add tests related to catalog, including the listTables, loadTable, and listNamespaces methods;
- Initialize the DDL test framework.
- Add CreateNamespaceSuite, DropNamespaceSuite and ShowTablesSuite to check for consistency with V1 in hive connector.
- Rectify the fault that namespaces are deleted in cascades. During cascades, ignore the exception that the table exists in the namespace.
- Fix the tableName problem of HiveTable, which should contain namespace name.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3530 from Yikf/hivev2-test.
Closes#3529
d0af0760 [yikf] Add tests to check for consistency with V1
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
To close#3410
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3609 from lightning-L/kyuubi-3410.
Closes#3410
83acd702 [Tianlin Liao] refactor
31773937 [Tianlin Liao] minor fix
d5b50c1b [Tianlin Liao] remove unnecessary parameters; get username from authentication
590e993a [Tianlin Liao] Support to delete the engine node from engine discovery space with AdminResource
Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
In #1785, the `stop()` call of `org.apache.kyuubi.ha.client.ServiceDiscovery#stopGracefully` was removed, causing the znode to be recreated after deletion.
Currently in `DeRegisterWatcher`, the `NodeDeleted` event is received, and the znode is not immediately closed, which causes `org.apache.curator.framework.recipes.nodes.PersistentNode#watcher` to receive the `NodeDeleted` event and asynchronously creates the same znode, which makes the service node unable to gracefully shutdown.
In addition, because the watcher is one-time, after receiving the `NodeDataChanged` event, we should re-register the watcher to avoid missing the next `NodeDeleted` event.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3543 from cxzl25/ha_zk_watcher.
Closes#3543
b3a109e4 [sychen] volatile
eb16380f [sychen] Merge branch 'master' into ha_zk_watcher
8621aee0 [sychen] add test
c4ad5309 [sychen] handle zk watch events
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
By default, on CentOS 7 w/ openJDK 8 installed via YUM/RPM, the `/usr/bin/java` links to `/usr/lib/jvm/java-1.8.0-openjdk/jre/bin/java`, the current detection logic will think the `JAVA_HOME` is `/usr/lib/jvm/java-1.8.0-openjdk/jre`, and then cause the issue.
```
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.6.1:compile (scala-compile-first) on project kyuubi-common_2.12: wrap: java.io.IOException: Cannot run program "/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64/jre/bin/javac" (in directory "/home/pancheng/apache-kyuubi"): error=2, No such file or directory -> [Help 1]
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
w/ this patch, the `build/dist` works expected on CentOS w/ openJDK 8 installed via `yum`, the debug info shows it detected the right `JAVA_HOME`
```
+ [[ -z '' ]]
++ command -v rpm
+ '[' /usr/bin/rpm ']'
++ rpm -E %java_home
+ RPM_JAVA_HOME=/usr/lib/jvm/java
+ '[' /usr/lib/jvm/java '!=' %java_home ']'
+ JAVA_HOME=/usr/lib/jvm/java
+ echo 'No JAVA_HOME set, proceeding with '\''/usr/lib/jvm/java'\'' learned from rpm'
No JAVA_HOME set, proceeding with '/usr/lib/jvm/java' learned from rpm
+ '[' -z /usr/lib/jvm/java ']'
+ [[ -z /usr/lib/jvm/java ]]
+ echo 'JAVA_HOME is set to /usr/lib/jvm/java'
JAVA_HOME is set to /usr/lib/jvm/java
```
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3619 from pan3793/java-home.
Closes#3619
b6e10a78 [Cheng Pan] nit
2a59d299 [Cheng Pan] [BUILD] Detect JAVA_HOME from rpm
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Ignore the VSCode setting folder.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3614 from deadwind4/git-ignore-vscode.
Closes#3613
f377f9b8 [Luning Wang] [KYUUBI #3613] Git ignore VSCode setting folder
Authored-by: Luning Wang <wang4luning@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3604 from dongkelun/flink-proxy-user.
Closes#3604
e56daea9 [dongkelun] Move HADOOP_PROXY_USER to env of FlinkProcessBuilder
c6127d48 [dongkelun] Move HADOOP_PROXY_USER to conf of FlinkProcessBuilder
f5731b2e [dongkelun] Enable Flink to proxy other kerberos users
Authored-by: dongkelun <dongkelun01@inspur.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close#3515.
By replacing mapChildren in `RuleApplyRowFilterAndDataMasking`to skip head of children query as insterted by iceberg in `IcebergSparkSessionExtensions` .
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3520 from bowenliang123/3515-authz-iceberg.
Closes#3515
a10fe43a [Bowen Liang] improve mapPlanChildren
5eb7f845 [Bowen Liang] fix problem after merging from master
bb1eefbd [Bowen Liang] merge from master
6c6b071c [Bowen Liang] Merge commit 'ae2df990a32c695de6f0345ea7e73e51103e87e5' into 3515-authz-iceberg
dbcfe6d8 [Bowen Liang] restrict skipMapchiled to Iceberg command privilege builders and add skipMappedChildren method to IcebergCommands to handling them
ae2df990 [Bowen Liang] nit
0c691798 [liangbowen] update mapPlanChildren and passSparkVersionCheck
161215d0 [liangbowen] nit
0006dee3 [liangbowen] generalize passSparkVersionCheck method to AuthZUtils
44163638 [liangbowen] refactor getFieldValOpt to getFieldValOption
5b8aa40b [Bowen Liang] improvements on skippedMapChildren of mapPlanChildren
11c2e637 [liangbowen] nit
5a971945 [liangbowen] unifiying general way for ensure skipped table is in plan's children
b161d70f [Bowen Liang] nit
952c1e1c [Bowen Liang] reuse MergeIntoTable of v2Commands to MergeIntoIcebergTable of IcebergCommands
0b25bd1b [Bowen Liang] generalize mapPlanChildren for iceberg commands
04fb651f [Bowen Liang] nit
a1f33bc3 [Bowen Liang] update DELETE FROM TABLE ut
34d65e5a [Bowen Liang] introduce IcebergCommands for access checking iceberg table. skip head child for iceberg commands in RuleApplyRowFilterAndDataMasking to prevent marking output tables.
085bfa31 [liangbowen] repalce mapChildren in RuleApplyRowFilterAndDataMasking with mapPlanChildren method, to skip head child for iceberg UpdateIcebergTable/MergeIntoIcebergTable/DeleteFromIcebergTable
6a245019 [liangbowen] init iceberg ut
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Now kyuubi will help set web ui port to 0 while scalaing up spark sql engine, which is good for the general situation, help avoid port conflicts.
But on k8s case, kyuubi should respect the user's default settings, here means web ui should be `4040`.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3600 from zwangsheng/improve/fix_spark_sql_engine_web_ui_on_k8s.
Closes#3600
0174e22f [zwangsheng] comment
ac2dbedc [zwangsheng] fix
1a2bff70 [zwangsheng] fix
bd4c35c2 [zwangsheng] fix
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close#3602 .
Bump log4j from 2.18.0 to 2.19.0.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3603 from bowenliang123/3602-log4j2-218.
Closes#3602
fef6865c [Bowen Liang] Bump log4j from 2.18.0 to 2.19.0
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
When Kyuubi runs outside of K8s, and w/o enhanced DNS infrastructure, Kyuubi can not access the Pod by using the hostname of Pod, it blocks the user to run Spark on K8s w/ cluster mode out-of-box.
Kyuubi provided a configuration `kyuubi.frontend.connection.url.use.hostname`, turn it off could address this issue, but we can not change the default value globally because of https://github.com/apache/incubator-kyuubi/issues/2266
To improve user experience, we can detect if the Driver is running inside the Pod, and if yes,
change `kyuubi.frontend.connection.url.use.hostname` default value to `false`.
Close#3578
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3597 from pan3793/k8s.
Closes#3597
8b411781 [Cheng Pan] doc
0df15e79 [Cheng Pan] Engine should prefer to use ip for registing on K8s cluster mode
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
On kubernetes case, we may need global env.
We add examples in `docker/helm/values.yaml`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3555 from zwangsheng/feature/on_kubernetes_env.
Closes#3555
b3553dc3 [zwangsheng] add
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix#3598
Hive Beeline has the same issue: https://issues.apache.org/jira/browse/HIVE-6758 and was fixed by append jline option "-Djline.termial=jline.UnsupportedTerminal".
We do the same in this PR.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
<img width="1043" alt="image" src="https://user-images.githubusercontent.com/88070094/194760564-04258a46-646f-4f76-ab19-b3e305b0ef69.png">
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3599 from zhouyifan279/3598.
Closes#3598
3c11d27c [zhouyifan279] improve indent
6d39b3cc [zhouyifan279] Beeline doesn't work with -e option when started in background
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
fix#3504
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3516 from jiaoqingbo/kyuubi3504.
Closes#3504
93fba696 [jiaoqingbo] remove outdated comment
af3e6a16 [jiaoqingbo] code review
894877b7 [jiaoqingbo] code review
856b28fc [jiaoqingbo] code review
60876a95 [jiaoqingbo] [KYUUBI #3504] Extend JDBC URL to support catalog
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix bug in #3527
Modify `spark.kubernetes.executorEnv` to `spark.executorEnv`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
- [x] Unit Test
Closes#3591 from zwangsheng/follow/SPARK_USER_NAME.
Closes#3527
7254fe91 [zwangsheng] fix executorEnv
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This pr aims to fix malforme format in hive connector doc.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3587 from Yikf/doc-follow.
Closes#3531
e6238f6f [Yikf] fix
Authored-by: Yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Keep apache pom update-to-date.
The whole release notes:
https://github.com/apache/maven-apache-parent/releases
This PR also disables source-release-checksum since we already disabled source-release-assembly
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3589 from pan3793/pom.
Closes#3589
e0231312 [Cheng Pan] Disable checksum-maven-plugin
05119f5a [Cheng Pan] [BUILD] Bump apache pom from 23 to 27
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
ban end-user from security settings
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3564 from cfmcgrady/kyuubi-3563.
Closes#3545Closes#3563
9d912b11 [Fu Chen] rename
ee44f7df [Fu Chen] fix ci
158f1552 [Fu Chen] address comment
3cbf4794 [Fu Chen] doc
0125a862 [Fu Chen] address comment
e4e554f5 [Kent Yao] Update extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AuthzConfigurationCheckExtension.scala
a8d35efa [Fu Chen] ban end-user from security settings
Lead-authored-by: Fu Chen <cfmcgrady@gmail.com>
Co-authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
to close#3581 .
support row filter and data masking on permanent views,
1. matching permanent views with `PermanentViewMarker` injected by `RuleApplyPermanentViewMarker`
2. reusing attributes's exprId in new Alias for permanent views for data masking
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3576 from bowenliang123/authz-permview-rowfilter.
Closes#3581
84d79a0d [Bowen Liang] restrict ut to spark 3.1+, as permanent view
eb3c9fa9 [liangbowen] update id for ranger policy
634eff0a [liangbowen] nit
48ec536f [liangbowen] add ut for row filter on perm view
892fe207 [liangbowen] only reusing attr.exprId in Alias for permanent view
ae796343 [liangbowen] simplify getting view identifier from PermanentViewMarker
b907f6e4 [Bowen Liang] apply Alias with exprId from original atttribute to solve atttribute confliction problem
5a44cddb [Bowen Liang] update
f08cf694 [liangbowen] add ut and try to adapt conversion
3059c71e [liangbowen] support row filter and data masking on permanent views
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
close https://github.com/apache/incubator-kyuubi/issues/3568#issue-1388744898
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3569 from zhaomin1423/fix_doris_type.
Closes#3568
f83cc481 [Min Zhao] fix style
d0447473 [Min Zhao] [KYUUBI #3568] [Bug] [Doris Engine] Fix return decimal, date, timestamp using string
Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Add a doc of Iceberg for Hive SQL engine
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3580 from deadwind4/iceberg-hive-doc.
Closes#3083
505b84ea [Luning Wang] [KYUUBI #3083][DOC] Add a doc of Iceberg for Hive SQL engine
Authored-by: Luning Wang <wang4luning@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Fix NPE if batch request/args/config is not specified.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3571 from turboFei/npe_batch.
Closes#3571
682de4f4 [Fei Wang] fix npe if value is null
f5b94d98 [Fei Wang] ut
68ee3101 [Fei Wang] fix npe
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Close https://github.com/apache/incubator-kyuubi/issues/3566.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3570 from Yikf/shade-spark-engine.
Closes#3566
0290cbc9 [Yikf] shade zookeeper in spark-engine
Authored-by: Yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
#1322#2129
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3519 from cxzl25/flink_get_columns.
Closes#3519
ab81776f [sychen] add column size
efdf1b90 [sychen] indent
630e907a [sychen] refactor
48a79d5b [sychen] add ut
69763bd3 [sychen] GetColumns
8e5e6c51 [sychen] GetColumns
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3512
Currently, Kyuubi EventBus uses `scala.concurrent.ExecutionContext.Implicits.global` to execute async event handler. Generally, it's discouraged to use that global ec, instead, we should create dedicated ec for each workload, this pr aims to use dedicated ExecutionContext for EventBus async execution
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3524 from Yikf/ec.
Closes#3512
9757372d [yikf] Use dedicated ExecutionContext for EventBus async execution
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### _Why are the changes needed?_
Documents for Kyuubi client advanced features plan_only.md
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3557 from CauchyHu/plan-only-doc.
Closes#3557
084ce5dd [胡安忠] [DOCS]update documents for Kyuubi client advanced features plan_only.md
75ec33d3 [胡安忠] [DOCS] Add documents for Kyuubi client advanced features plan_only.md
Authored-by: 胡安忠 <huanzhong@xiaomi.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Add `white-list` help run the specified queries.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
`--white-list q4,q15,q19,q25,q42,q49,q60,q66,q68,q72,q76,q78,q79,q89,q97`
Closes#3539 from zwangsheng/feature/tpcds_white_list.
Closes#3539
74657957 [zwangsheng] fix
26ade916 [Binjie Yang] Update dev/kyuubi-tpcds/src/main/scala/org/apache/kyuubi/tpcds/benchmark/RunBenchmark.scala
11a5b5c9 [Binjie Yang] Update dev/kyuubi-tpcds/src/main/scala/org/apache/kyuubi/tpcds/benchmark/RunBenchmark.scala
780410a3 [zwangsheng] fix
19018932 [zwangsheng] naming
1a4d1a25 [zwangsheng] naming
e1bb0069 [zwangsheng] fix
1be2bcb9 [zwangsheng] white
Lead-authored-by: zwangsheng <2213335496@qq.com>
Co-authored-by: Binjie Yang <52876270+zwangsheng@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
change kerberized_kyuubi to kerberos since the docs file name is kerberos.md
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3562 from ulysses-you/fix-docs.
Closes#3562
615c7fcf [ulysses-you] docs
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Followup #3230
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3560 from df-Liu/flink_ddl.
Closes#3560
0dbdfb3f [df_liu] flink ddl
Authored-by: df_liu <df_liu@trip.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
