936a532d2e
1264 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
simon
|
2ae6acda65
|
[KYUUBI #1039] Kill yarn job when engine initialize timeout and yarnApplicationState is ACCEPTED
kill yarn job when engine initialize timeout and yarnApplicationState is ACCEPTED #1039 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1119 from simon824/killyarnjob. Closes #1039 7face4db [simon] get KYUUBI_HOME da8c3ec4 [Simon] Merge branch 'apache:master' into killyarnjob 95ae8f59 [simon] add appId 8a6ddcd1 [simon] kill application by script e0cd2af8 [simon] kill application by script d15d38c5 [simon] kill application by script 1fd1373b [simon] mv sparklauncher to sparkProcessBuilder 5e806426 [simon] fix codestyle a9cc4505 [simon] fix option No value Exception 9c7ca2f8 [simon] fix option No value Exception bbfe8e25 [simon] add set sparkHome 04f23c8a [simon] fix codestyle 75599233 [simon] add spark-launcher dep ef4b2706 [simon] implement by sparkLauncher 59c25b7c [simon] kill yarn application by restful api 49921a48 [simon] fix ut a31d8f6a [simon] fix ut 1abc6665 [simon] rename killApplication 43a95c1d [simon] Merge branch 'master' into killyarnjob 3d9c12d3 [simon] Merge remote-tracking branch 'upstream/master' 9eaeb16d [simon] fix ut 64ee1b11 [simon] code style 5b905dbd [simon] Merge remote-tracking branch 'upstream/master' 139f3b79 [Simon] Merge branch 'apache:master' into killyarnjob 1a52401d [simon] #1039 Lead-authored-by: simon <zhangshiming@cvte.com> Co-authored-by: Simon <3656562@qq.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
sychen
|
6420a7eb04
|
[KYUUBI #1174] Service discovery supports kerberos
### _Why are the changes needed?_ https://github.com/apache/incubator-kyuubi/issues/1174 Now kyuubi servcie uses kerberos authentication and registers with the zookeeper to the node information, but the client cannot connect to the service. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1175 from cxzl25/KYUUBI-1174. Closes #1174 386c7d57 [sychen] add comment dc37a158 [sychen] respect user's setting 97401a96 [sychen] update settings.md fd8cceda [sychen] Service discovery supports kerberos Authored-by: sychen <sychen@trip.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
ac3e6c36ac
|
[KYUUBI #1149] [SUB-TASK][BUILD] Create shaded jar for kyuubi-hive-jdbc module
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Create a shaded jar for `kyuubi-hive-jdbc` module, which relocate every classes except `org.apache.hadoop`, `org.slf4j`. Test passed on DataGrip w/o Kerberos. ``` ➜ kyuubi-hive-jdbc ll kyuubi-hive-jdbc-1.4.0-SNAPSHOT.jar -rw-r--r-- 1 chengpan staff 12M Sep 24 16:46 kyuubi-hive-jdbc-1.4.0-SNAPSHOT.jar ``` ``` ➜ kyuubi-hive-jdbc tree -dL 5 . ├── META-INF │ └── services └── org └── apache ├── hadoop │ ├── fs │ ├── mapred │ └── security │ └── token └── kyuubi ├── jdbc │ └── hive └── shade ├── com └── org ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate   - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request - [x] Pass #1152 Closes #1149 from pan3793/jdbc-shade. Closes #1149 448eaa6a [Cheng Pan] [BUILD] Create shaded jar for kyuubi-hive-jdbc module Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
zhouyifan279
|
d44ccc3644
|
[KYUUBI #1008][FOLLOWUP] No need to send private token from Kyuubi server to engine
### _Why are the changes needed?_ When communicating with NameNode HA, private tokens are created from original token:  In above image, "ha-hdfs:mycluster" is the original token. "ha-hdfs://192.168.56.101:8020" and "ha-hdfs://192.168.56.102:8020" are private tokens. `KyuubiHadoopUtils.getCredentialsInternal` was supposed to extract these private tokens at Kyuubi server side and send them to SQL engine. But in fact, SQL engine side private tokens are automatically created when adding original token, inside `org.apache.hadoop.security.Credentials#addToken`. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate SequenceNumber of private tokens always equal to original token:   - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1173 from zhouyifan279/#1008. Closes #1008 891dc2b6 [zhouyifan279] [KYUUBI #1008][FOLLOWUP] No need to send private token from Kyuubi server to engine Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
sychen
|
d332534325
|
[KYUUBI #1176] InvalidACL appears in the engine when zookeeper acl is turned on
### _Why are the changes needed?_ https://github.com/apache/incubator-kyuubi/issues/1176 When `kyuubi.ha.zookeeper.acl.enabled=true`, both service and engine will use zookeeper acl to create znode, but engine has no keytab information and cannot write information to zookeeper, throwing an exception. ```java Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /kyuubi_USER/XXXX at org.apache.zookeeper.KeeperException.create(KeeperException.java:124) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:792) at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:740) ``` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1177 from cxzl25/KYUUBI-1176. Closes #1176 ecc08fa7 [sychen] fix engine acl 0b7cc2ec [sychen] fix InvalidACL Authored-by: sychen <sychen@trip.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
d1e7f0cc20
|
[KYUUBI #1143][FOLLOWUP] Correct KyuubiThriftBinaryFrontendService name
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Correct typo. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1186 from pan3793/typo. Closes #1143 fd53e94a [Cheng Pan] [KYUUBI #1143][FOLLOWUP] Correct KyuubiThriftBinaryFrontendService name Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
hongdongdong
|
0e68678f20
|
[KYUUBI #1184] Support ending with .* to restrict and ingore session config
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Now, restricted and ingored keys must be complete. We can support eding with '.*' to restricted a list of keys. ### _How was this patch tested?_ - [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1185 from hddong/support-perfix-ignore. Closes #1184 dade3301 [hongdongdong] reset c58b0aa6 [hongdongdong] [KYUUBI#1184] Support ending with .* to restrict and ingore session config Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
hzxiongyinke
|
0ecf8fbc7e
|
[KYUUBI #939] z-order performance_test
### What is the purpose of the pull request pr for KYUUBI #939:Add Z-Order extensions to optimize table with zorder.Z-order is a technique that allows you to map multidimensional data to a single dimension. We did a performance test for this test ,we used aliyun Databricks Delta test case https://help.aliyun.com/document_detail/168137.html?spm=a2c4g.11186623.6.563.10d758ccclYtVb Prepare data for the three scenarios: 1. 10 billion data and 2 hundred files（parquet files）: for big file(1G) 2. 10 billion data and 1 thousand files（parquet files）: for medium file(200m) 3. one billion data and 10 hundred files（parquet files）: for smaller file(200k) test env： spark-3.1.2 hadoop-2.7.2 kyubbi-1.4.0 test step： Step1: create hive tables ```scala spark.sql(s"drop database if exists $dbName cascade") spark.sql(s"create database if not exists $dbName") spark.sql(s"use $dbName") spark.sql(s"create table $connRandomParquet (src_ip string, src_port int, dst_ip string, dst_port int) stored as parquet") spark.sql(s"create table $connZorderOnlyIp (src_ip string, src_port int, dst_ip string, dst_port int) stored as parquet") spark.sql(s"create table $connZorder (src_ip string, src_port int, dst_ip string, dst_port int) stored as parquet") spark.sql(s"show tables").show(false) ``` Step2： prepare data for parquet table with three scenarios we use the following code ```scala def randomIPv4(r: Random) = Seq.fill(4)(r.nextInt(256)).mkString(".") def randomPort(r: Random) = r.nextInt(65536) def randomConnRecord(r: Random) = ConnRecord( src_ip = randomIPv4(r), src_port = randomPort(r), dst_ip = randomIPv4(r), dst_port = randomPort(r)) ``` Step3： do optimize with z-order only ip， sort column： src_ip, dst_ip and shuffle partition just as file numbers . execute 'OPTIMIZE conn_zorder_only_ip ZORDER BY src_ip, dst_ip;' by kyuubi. Step4： do optimize with z-order only ip， sort column： src_ip, dst_ip and shuffle partition just as file numbers . execute 'OPTIMIZE conn_zorder ZORDER BY src_ip, src_port, dst_ip, dst_port;' by kyuubi. --------------------- # benchmark result by querying the tables before and after optimization, we find that **10 billion data and 200 files and Query resource:200 core 600G memory** | Table | Average File Size | Scan row count | Average query time | row count Skipping ratio | | ------------------- | ----------------- | -------------- | ------------------ | ------------------------ | | conn_random_parquet | 1.2 G | 10,000,000,000 | 27.554 s | 0.0% | | conn_zorder_only_ip | 890 M | 43,170,600 | 2.459 s | 99.568% | | conn_zorder | 890 M | 54,841,302 | 3.185 s | 99.451% | **10 billion data and 2000 files and Query resource:200 core 600G memory** | Table | Average File Size | Scan row count | Average query time | row count Skipping ratio | | ------------------- | ----------------- | -------------- | ------------------ | ------------------------ | | conn_random_parquet | 234.8 M | 10,000,000,000 | 27.031 s | 0.0% | | conn_zorder_only_ip | 173.9 M | 43,170,600 | 2.668 s | 99.568% | | conn_zorder | 174.0 M | 54,841,302 | 3.207 s | 99.451% | **1 billion data and 10000 files and Query resource:10 core 40G memory** | Table | Average File Size | Scan row count | Average query time | row count Skipping ratio | | ------------------- | ----------------- | -------------- | ------------------ | ------------------------ | | conn_random_parquet | 2.7 M | 1,000,000,000 | 76.772 s | 0.0% | | conn_zorder_only_ip | 2.1 M | 406,572 | 3.963 s | 99.959% | | conn_zorder | 2.2 M | 387,942 | 3.621s | 99.961% | Closes #1178 from hzxiongyinke/zorder_performance_test. Closes #939 369a9b41 [hzxiongyinke] remove set spark.sql.extensions=org.apache.kyuubi.sql.KyuubiSparkSQLExtension; 8c8ae458 [hzxiongyinke] add index z-order-benchmark 66bd20fd [hzxiongyinke] change tables to three scenarios cc80f4e7 [hzxiongyinke] add License 70c29daa [hzxiongyinke] z-order performance_test 6f1892be [hzxiongyinke] Merge pull request #1 from apache/master Lead-authored-by: hzxiongyinke <1062376716@qq.com> Co-authored-by: hzxiongyinke <75288351+hzxiongyinke@users.noreply.github.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
fwang12
|
4f74aba50e |
[KYUUBI #1160] Support to config operation log root dir both for kyuubi server and engine sides
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Support to config operation log root dir with kyuubi configuration both for server and engine sides. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1160 from turboFei/support_config_operationlog_dir. Closes #1160 ed424f46 [fwang12] fix ut 5edfd4ae [fwang12] refactor ut b7026b2f [fwang12] add ut 6ff4d891 [fwang12] save 30cd5952 [fwang12] fix ut f058619a [fwang12] fix ut 96eb020d [fwang12] refactor ae0ded96 [fwang12] set config value for testing 83d932f4 [fwang12] exclude 33e3e214 [fwang12] update 6f030cba [fwang12] fix 77ec9ca2 [fwang12] save 711ececf [fwang12] Fix code style cfa3bfac [fwang12] refactor b299f8cf [fwang12] complete ut 26de3712 [fwang12] Support to config operation log root dir for kyuubi server and engine side Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
Kent Yao
|
be0b9c1e69
|
[KYUUBI #1169] Decouple EngineLoggingService From SparkSQLEngine
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. Make EngineLoggingService tolerant for creating if any error occurs, and then the main code goes. 2. Separate EngineLoggingService From SparkSQLEngine, and we will try to start EngineLoggingService before SparkSQLEngine to capture all the lifecycle statuses. On the other hand, add a shutdown hook that executed after SparkSQLEngine.stop for EngineLoggingService.stop 3. try-catch for every step of SparkSQLEngine bootstrapping for creating better diagnosis messages. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1171 from yaooqinn/1169. Closes #1169 3a661555 [Kent Yao] ci 733bd1ed [Kent Yao] address comments 36ba1661 [Kent Yao] [KYUUBI #1169] Decouple EngineLoggingService From SparkSQLEngine Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
191d30e2ac
|
[KYUUBI #1167][FOLLOWUP] Refine ZorderCoreBenchmark doc
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Minor change, update `ZorderCoreBenchmark` doc. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1172 from pan3793/t. Closes #1167 8372a3a7 [Cheng Pan] nit 1b6ee9da [Cheng Pan] typo bd1817a7 [Cheng Pan] [KYUUBI #1167][FOLLOWUP] Refine ZorderCoreBenchmark doc Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
zhouyifan279
|
413ba2eb24
|
[KYUUBI #1009][FOLLOWUP] Hive delegation token sent from Kyuubi server is ignore by SQL engine
### _Why are the changes needed?_ Hive delegation token sent from Kyuubi server is ignore by SQL engine as token alias does not match `hive.metastore.uris`. ``` 21/09/27 14:22:39 INFO operation.ExecuteStatement: Processing morty's query[5711be2b-1d2f-412a-a2e0-b52a5b0e0445]: RUNNING_STATE -> FINISHED_STATE, statement: show databases, time taken: 0.056 seconds 21/09/27 14:22:39 INFO spark.SparkThriftBinaryFrontendService: Ignore Hive token as engine has no Hive token ever before 21/09/27 14:22:39 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user. ``` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate ``` 21/09/27 20:51:20 INFO operation.ExecuteStatement: Processing morty's query[652a9f74-4813-406c-a842-6fc8b61baaf1]: RUNNING_STATE -> FINISHED_STATE, statement: show databases, time taken: 0.05 seconds 21/09/27 20:51:20 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user. ``` - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1168 from zhouyifan279/#1009. Closes #1009 fd326347 [zhouyifan279] [KYUUBI #1009][FOLLOWUP] Hive delegation token is ignore by SQL engine 3302ee4b [zhouyifan279] [KYUUBI #1009][FOLLOWUP] Hive delegation token is ignore by SQL engine Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
zhouyifan279
|
d45421b37e
|
[KYUUBI #1008][FOLLOWUP] Can not obtain hadoop file system delegation token
### _Why are the changes needed?_
Kyuubi server throws exception when obtaining hadoop file system delegation token
```
21/09/27 19:53:07 WARN credentials.HadoopCredentialsManager: Failed to update tokens for morty, try again in 3000 ms
java.io.IOException: DestHost:destPort kdc.hadoop.com:9000 , LocalHost:localPort client.hadoop.com/192.168.56.1:0. Failed on local exception: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal n
ame
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:836)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:811)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1566)
at org.apache.hadoop.ipc.Client.call(Client.java:1508)
at org.apache.hadoop.ipc.Client.call(Client.java:1405)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy10.getDelegationToken(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDelegationToken(ClientNamenodeProtocolTranslatorPB.java:1081)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy11.getDelegationToken(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:703)
at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:1845)
at org.apache.hadoop.security.token.DelegationTokenIssuer.collectDelegationTokens(DelegationTokenIssuer.java:95)
at org.apache.hadoop.security.token.DelegationTokenIssuer.addDelegationTokens(DelegationTokenIssuer.java:76)
at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.$anonfun$obtainDelegationTokens$2(HadoopFsDelegationTokenProvider.scala:66)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:141)
at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.$anonfun$obtainDelegationTokens$1(HadoopFsDelegationTokenProvider.scala:64)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider$$anon$1.run(HadoopFsDelegationTokenProvider.scala:106)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider$.doAsProxyUser(HadoopFsDelegationTokenProvider.scala:105)
at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.obtainDelegationTokens(HadoopFsDelegationTokenProvider.scala:57)
at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.$anonfun$run$1(HadoopCredentialsManager.scala:210)
at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.$anonfun$run$1$adapted(HadoopCredentialsManager.scala:210)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:214)
at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.run(HadoopCredentialsManager.scala:210)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:884)
at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:413)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1636)
at org.apache.hadoop.ipc.Client.call(Client.java:1452)
... 43 more
Caused by: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name
at org.apache.hadoop.security.SaslRpcClient.getServerPrincipal(SaslRpcClient.java:327)
at org.apache.hadoop.security.SaslRpcClient.createSaslClient(SaslRpcClient.java:234)
at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:160)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:622)
at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:413)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:822)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:818)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:818)
... 46 more
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes #1166 from zhouyifan279/#1008.
Closes #1008
43dbf605 [zhouyifan279] [KYUUBI #1008][FOLLOWUP] Can not obtain hadoop fs delegation token
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
|
||
|
ulysses-you
|
627d271fc6
|
[KYUUBI #1167] Add zorder core benchmark
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add zorder core benchmark, so we can get the perf number of zorder. This PR adds two column type `int` and `long` benchmark: - 2 int cols - 3 int cols - 4 int cols - 2 long cols - 3 long cols - 4 long cols ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1167 from ulysses-you/benchmark. Closes #1167 27ecf51f [ulysses-you] Add zorder core benchmark Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
hongdongdong
|
55d5b109c0
|
[KYUUBI #1164] Grep WARN in dependency.sh
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> When MVN dependency has WARN log, ./build/dependency.sh --replace will exit with execption: ``` awk: trying to access out of range field -1 input record number 1, file source line number 2 ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1165 from hddong/fix-build-dependency. Closes #1164 1f2904dc [hongdongdong] [KYUUBI-1164] Grep WARN in dependency.sh Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
fb6ce8fcc6
|
[KYUUBI #1131][TEST] Move KyuubiHiveDriverSuite to kyuubi-spark-sql-engine module
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> It's pre-step of #1152, otherwise will cause circular dependence. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1151 from pan3793/move-jdbc-test. Closes #1131 e65f9979 [Cheng Pan] nit f2e66535 [Cheng Pan] [KYUUBI #1131][TEST] Move KyuubiHiveDriverSuite to kyuubi-spark-sql-engine module Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
zhouyifan279
|
114ec9a046
|
[KYUUBI #1009][FOLLOWUP] Kyuubi server failed to start when connecting to secured HMS
### _Why are the changes needed?_ Fix #1161 [Bug] Kyuubi server failed to start when connecting to a secured Hive metastore ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate  - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1162 from zhouyifan279/KYUUBI#1161. Closes #1009 f5b54818 [zhouyifan279] [KYUUBI #1161][Bug] Kyuubi server failed to start when connecting to a secured Hive metastore Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
85a7c63418
|
[KYUUBI #1163] [BUILD] Disable parallelization in dist
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Seems that `maven-shade-plugin` has some issue with parallelization, #1149 stuck w/o this change. For refererence: https://github.com/apache/spark/pull/23061 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1163 from pan3793/dist. Closes #1163 7b322e5c [Cheng Pan] [BUILD] Disable parallelization in dist Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Min Zhao
|
d829bae526
|
[KYUUBI #939][FOLLOWUP][TEST] Add string column test for Z-Order
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Added string column test for zorder. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1157 from zhaomin1423/zorder_test. Closes #939 59543776 [Min Zhao] add string column test for zorder Authored-by: Min Zhao <zhaomin1423@163.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
c8d9d8be21
|
[KYUUBI #1155] [BUILD] Support passthrough env var CI into build container
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Continuous integration(aka. CI) services like GitHub Actions, Travis always provide an environment variable `CI` in runners, and we detect this variable to run some specific actions. In [KYUUBI #742], we add a `CI` env var detection in `build/mvn` to suppress noisy maven logs in CI scenes. This PR introduces a new build arg `CI` in `docker/Dockerfile`, which is intent to passthrough env var CI into the build container to suppress noisy maven logs in GitHub Action and Travis. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1155 from pan3793/ci. Closes #1155 0f66b37e [Cheng Pan] typo e1b9d35e [Cheng Pan] nit d78bb563 [Cheng Pan] update workflow 4209ae67 [Cheng Pan] Update comments ffc3a1e1 [Cheng Pan] nit 761c9e6c [Cheng Pan] Update usage 2f74b977 [Cheng Pan] Add build arg CI 509da2aa [Cheng Pan] [BUILD] Suppress maven log in Minikube Integration Test Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Brian Yue
|
683675be92
|
[KYUUBI #1146] Move KinitAuxiliaryService to kyuubi-server module
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Close #1146. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1156 from byyue/feature-1146. Closes #1146 6b677723 [Brian Yue] Move KinitAuxiliaryService to kyuubi-server module Authored-by: Brian Yue <code.byyue@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
21744a99bd
|
[KYUUBI #1158] [INFRA] Update ANNOUNCE mail template
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Update ANNOUNCE mail template to add more detailed descriptions of Kyuubi. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1158 from pan3793/ann. Closes #1158 46c19975 [Cheng Pan] typo a87321f5 [Cheng Pan] [INFRA] Update ANNOUNCE mail template Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
timothy65535
|
22e6432e4d
|
[KYUUBI #1022] Add basic EngineStatusStore for events
### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> For more detail, please go to https://github.com/apache/incubator-kyuubi/issues/981 `EngineStatusStore` helps to push events to listener bus `EngineStatusStore` is a memory store that tracking the number of statements and sessions, it provides: - stores all elements, and sorted by startTimestamp. - cleanup the last elements when reach a certain threshold. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1023 from timothy65535/ky-1022. Closes #1022 b9f355c4 [timothy65535] [KYUUBI #1022] Add basic EngineStatusStore for events Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
zhouyifan279
|
3a1da695e6
|
[KYUUBI #1145] Suppress MiniDFSCluster/MiniYARNCluster/Kerberos output in tests
### _Why are the changes needed?_ - Kerberos login module outputs lots of debug level log records to stdout, which is annoying at most of the time. - MiniYARNCluster already outputs most of the log records to unit-tests.log. But some referred classes, such as `GuiceComponentProviderFactory` use java.util.logging.Logging and output log records to stdout. In this PR, we turn off kerberos logging by default and allow users to turn on it by setting java system property `sun.security.krb5.debug`. java.util.logging.Logging is redirected to slf4j using `SLF4JBridgeHandler`. **Note:** MiniDFSCluster outputs a few log records using `System.stdout`/`System.stderr`. ``` Formatting using clusterid: testClusterID isSaslEnabled:false Opened streaming server at /0.0.0.0:1025 Successfully obtained privileged resources (streaming port = ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=1025] ) (http listener port = 9864) Opened info server at /0.0.0.0:9864 ``` It should be acceptable. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1150 from zhouyifan279/KYUUBI#1145. Closes #1145 5c91816d [zhouyifan279] [KYUUBI #1145] Suppress MiniDFSCluster/MiniYARNCluster/Kerberos output in tests 1a9f6345 [zhouyifan279] [KYUUBI #1145] Suppress MiniDFS/YARNCluster/ Kerberos output in tests 9877085a [zhouyifan279] [KYUUBI #1145] Suppress MiniDFS/YARNCluster/ Kerberos output in tests 8b05cb02 [zhouyifan279] [KYUUBI #1145] Suppress MiniDFS/YARNCluster/ Kerberos output in tests 435a4715 [zhouyifan279] [KYUUBI #1145] Suppress MiniDFS/YARNCluster/ Kerberos output in tests Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
张宇翔
|
103969a042
|
[KYUUBI #1067] Catch the real exception when the log directory is not exist
BUGFIX: #1067 fix: 2021-09-23 16:03:43.248 INFO HiveMetaStore.audit: ugi=zhangyu ip=unknown-ip-addr cmd=get_databases: * 2021-09-23 16:03:43.319 ERROR events.JsonEventLogger: Get log directory /Users/zhangyuxiang/Downloads/kyuubi_test failed java.io.FileNotFoundException: File /Users/zhangyuxiang/Downloads/kyuubi_test does not exist at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:666) at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:987) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:656) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454) at org.apache.kyuubi.events.JsonEventLogger.requireLogRootWritable(JsonEventLogger.scala:79) at org.apache.kyuubi.events.JsonEventLogger.initialize(JsonEventLogger.scala:93) at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1(CompositeService.scala:40) at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1$adapted(CompositeService.scala:40) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1136 from zhang1002/branch-1.3.0_checkRightException. Closes #1067 de16c63d [张宇翔] delete some ut 5a29bb80 [张宇翔] Merge branch 'master' into branch-1.3.0_checkRightException 21e856a3 [张宇翔] Merge branch 'master' into branch-1.3.0_checkRightException 873cc9c4 [张宇翔] Merge remote-tracking branch 'upstream/master' d1e7a019 [张宇翔] catch IOException when log directory is not exist bc442d8c [张宇翔] catch IOException when log directory is not exist 0d7418e9 [张宇翔] Merge branch 'master' into branch-1.3.0_checkRightException 41cff19a [张宇翔] Merge remote-tracking branch 'upstream/master' 74b36efb [张宇翔] catch IOException when log directory is not exist def8d1f6 [张宇翔] Merge remote-tracking branch 'upstream/master' 8384a6e2 [张宇翔] Merge remote-tracking branch 'upstream/master' 714ab1b8 [张宇翔] Merge remote-tracking branch 'upstream/master' af28520b [张宇翔] Merge remote-tracking branch 'upstream/master' c328c884 [张宇翔] Merge remote-tracking branch 'upstream/master' abcaae1f [张宇翔] Merge remote-tracking branch 'upstream/master' f0a7e566 [张宇翔] Merge remote-tracking branch 'upstream/master' 112aa6b6 [张宇翔] Merge remote-tracking branch 'upstream/master' c55652fd [张宇翔] Merge remote-tracking branch 'upstream/master' ef5e8c55 [张宇翔] Merge remote-tracking branch 'upstream/master' 5a75384c [张宇翔] Merge remote-tracking branch 'upstream/master' 8da4f7ed [张宇翔] Merge remote-tracking branch 'upstream/master' a7ba1bfc [张宇翔] Merge remote-tracking branch 'upstream/master' b662a989 [张宇翔] Merge remote-tracking branch 'upstream/master' 4c8f3b87 [张宇翔] Merge remote-tracking branch 'upstream/master' 8b686767 [张宇翔] Merge remote-tracking branch 'upstream/master' cf99e309 [张宇翔] Merge remote-tracking branch 'upstream/master' 0afaa578 [张宇翔] Merge remote-tracking branch 'upstream/master' b24fea07 [张宇翔] Merge remote-tracking branch 'upstream/master' e517cfc5 [张宇翔] Merge remote-tracking branch 'upstream/master' 18aebe76 [张宇翔] Merge remote-tracking branch 'upstream/master' f248bef7 [张宇翔] Merge remote-tracking branch 'upstream/master' 5ffb54f3 [张宇翔] Add kyuubi-spark-monitor module for nightly.yml Authored-by: 张宇翔 <zhang1002@126.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
eeb35f31dd
|
[KYUUBI #1131] Rename KyuubiDriver to KyuubiHiveDriver
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This PR is based on discussion https://www.mail-archive.com/devkyuubi.apache.org/msg00977.html ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1147 from pan3793/jdbc-rename. Closes #1131 500edae6 [Cheng Pan] Add javadoc bd9f94c2 [Cheng Pan] Add deprecated KyuubiDriver test d9688d8b [Cheng Pan] [KYUUBI #1131] Rename KyuubiDriver to KyuubiHiveDriver Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
0ee8eff5a0
|
[KYUUBI #1143] Refact Serverable/FrontendService/ServiceDiscovery hierarchy
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> After this PR, the service named service discovery will be a child service of frontend service and it's optional. A servalable will contain a backend and a list of frontends, the former one stay AS-IS and the latter is changed from single to multiple, which means a serverable can have multiple interfaces to interact with its different clients ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1144 from yaooqinn/1143. Closes #1143 f2beb721 [Kent Yao] fix test 128cf282 [Kent Yao] nit 37aaff9c [Kent Yao] lift FrontendService trait add335c4 [Kent Yao] nit 3a111cba [Kent Yao] [KYUUBI #1143] Refact Serverable/FrontendService/ServiceDiscovery hierarchy bfa63b81 [Kent Yao] [KYUUBI #1143] Refact Serverable/FrontendService/ServiceDiscovery hierarchy 7e5a5cbb [Kent Yao] Merge branch 'master' into 1143 09263bc0 [Kent Yao] [KYUUBI #1143] Refact Serverable/FrontendService/ServiceDiscovery hierarchy Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
1489d1dbf5
|
[KYUUBI #1107] [INFRA] Add release mail templates
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1107 from pan3793/template. Closes #1107 659a2120 [Cheng Pan] Update ANNOUNCE template 971ce74a [Cheng Pan] Update ANNOUNCE template 13465519 [Cheng Pan] Update Links 6a5faf5c [Cheng Pan] typo 0f037c3f [Cheng Pan] Update rat excludes d8e4cbb9 [Cheng Pan] Add license headers and remove incorrect words 8872a34a [Cheng Pan] Update template d7f01643 [Cheng Pan] Add mail template 2a170071 [Cheng Pan] Add mail template Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
ulysses-you
|
7a3545e748
|
[KYUUBI #1002][FOLLOWUP] Refine sql classification rule
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Follow up [KYUUBI #1002] [KYUUBI #1035] [KYUUBI #1037] - correct the config key; we should use `spark.` as the prefix - change the config to false by default - correct the class loader; thread context is not safe, spark and hive often change it - update the log and exception following is the failed error msg: ``` Caused by: java.io.FileNotFoundException: file:/Users/cathy/Desktop/tmp/spark-3.1.2-bin-hadoop2.7/jars/kyuubi-extension-spark-3-1_2.12-1.4.0-SNAPSHOT.jar!/sql-classification-default.json (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:916) at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2745) at org.apache.kyuubi.sql.sqlclassification.KyuubiGetSqlClassification$.<init>(KyuubiGetSqlClassification.scala:51) at org.apache.kyuubi.sql.sqlclassification.KyuubiGetSqlClassification$.<clinit>(KyuubiGetSqlClassification.scala) ... 59 more ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1140 from ulysses-you/refine. Closes #1002 f39ef2cf [ulysses-you] address comment 5cb94459 [ulysses-you] nit de9fd932 [ulysses-you] test 78e8c147 [ulysses-you] default 6f23c63e [ulysses-you] refine Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
997d72a1cd
|
[KYUUBI #884][FOLLOWUP] Fix catalog in KyuubiDatabaseMetaData#getTables
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
The behavior of `KyuubiDatabaseMetaData#getTables` changed by accident in #884[1], this PR restores the method's previous behavior and adds multi-catalog test case to verify it works as expected.
[1] [This](
|
||
|
yanghua
|
ee8e5a12f1
|
[KYUUBI #1137] KyuubiFrontendServices oom hook should call the stop on serverable level
… <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1142 from yanghua/KYUUBI-1137. Closes #1137 418f3dff [yanghua] [KYUUBI #1137] KyuubiFrontendServices oom hook should call the stop on serverable level Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
timothy65535
|
d93a6c8b5b
|
[KYUUBI #1141] Remove redundant unit
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Remove redundant unit. `MB` ``` 2021-09-23 08:41:08.889 INFO spark.SparkSQLEngine: Spark application name: kyuubi_USER_root_6f67c12c-f362-4a07-87d0-7aa7e5f43171 application ID: spark-461363e9cc934d1986f7ae4191dc0410 application web UI: http://spark-740a727c11ce3da7-driver-svc.default.svc:44692 master: k8s://https://10.10.9.35:6443 version: 3.1.2 driver: [cpu: 1, mem: 1G] executor: [cpu: 2, mem: 2G MB, maxNum: 18] Start time: Thu Sep 23 08:40:39 UTC 2021 User: root (shared mode: USER) State: LATENT ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1141 from timothy65535/remove. Closes #1141 af1827e9 [timothy65535] Remove redundant unit Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Fu Chen
|
0be523f6fe |
[KYUUBI #1135] [SUB-TASK][KPIP-1] Register default ObjectMapper instead of manually serialize/deserialize JSON input argument.
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Register default `ObjectMapper` instead of manually serialize/deserialize JSON input argument. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1135 from cfmcgrady/objectmapper. Closes #1135 580707cb [Fu Chen] fix style 48da4fe4 [Fu Chen] refactor unit test 067f11be [Fu Chen] fix style 9462d936 [Fu Chen] object mapper Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
ulysses-you
|
b26563126f
|
[KYUUBI #1138] Revert "[KYUUBI #937] Push Kyuubi docker image"
This reverts commit
|
||
|
ulysses-you
|
0cd41e1eac
|
[KYUUBI #937] Push Kyuubi docker image
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Push docker image if we commit to master ### _How was this patch tested?_ test in forked repo. Closes #1117 from ulysses-you/docker-image. Closes #937 9d948806 [ulysses-you] push docker image Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
787ba9efe2
|
[KYUUBI #1134] [TEST] DataLake suites should clean up withKyuubiConf instead of extraConfigs
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Keep consistent with ``` withKyuubiConf.foreach { case (k, v) => System.setProperty(k, v) kyuubiConf.set(k, v) } ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1134 from pan3793/t. Closes #1134 b8e0aa65 [Cheng Pan] [TEST] DataLake Suites should clean up withKyuubiConf instead of extraConfigs Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Brian Yue
|
bd0989487d
|
[KYUUBI #1115] Add Affects Version/s area in bug report issue template
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Close #1115. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1116 from byyue/feature-1115. Closes #1115 d4513b22 [Brian Yue] Add Affects Version/s area in bug report issue template Authored-by: Brian Yue <code.byyue@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Simon
|
7b7d228fb4
|
[KYUUBI #1123][TEST] Split KyuubiExtensionSuite according to different extensions
split KyuubiExtensionSuite according to different extensions #1123 drop： `KyuubiSparkExtensionTest` add： - `FinalStageConfigIsolationSuite` - `InsertShuffleNodeBeforeJoinSuite` - `WatchDogSuite` - `RepartitionBeforeWriteSuite` - `SqlClassificationSuite` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1124 from simon824/extension-ut. Closes #1123 4b1ae4db [Simon] Update FinalStageConfigIsolationSuite.scala 2068a74d [Simon] Update SparkProcessBuilder.scala cd6aa7d2 [Simon] Merge branch 'apache:master' into extension-ut 1401d69c [simon] rename test to suite 98095823 [Simon] Rename WatchDogSuit.scala to WatchDogSuite.scala 0392d5b5 [Simon] Rename SqlClassificationSuit.scala to SqlClassificationSuite.scala a8776df9 [Simon] Rename RepartitionBeforeWriteSuit.scala to RepartitionBeforeWriteSuite.scala 4d06966c [Simon] Rename InsertShuffleNodeBeforeJoinSuit.scala to InsertShuffleNodeBeforeJoinSuite.scala 60b2932e [Simon] Rename FinalStageConfigIsolationSuit.scala to FinalStageConfigIsolationSuite.scala 175fb47f [Simon] Rename WatchDogTest.scala to WatchDogSuit.scala 755ad22c [Simon] Rename SqlClassificationTest.scala to SqlClassificationSuit.scala b36171b8 [Simon] Rename RepartitionBeforeWriteTest.scala to RepartitionBeforeWriteSuit.scala 74c110da [Simon] Rename InsertShuffleNodeBeforeJoinTest.scala to InsertShuffleNodeBeforeJoinSuit.scala 405d8b72 [Simon] Rename FinalStageConfigIsolationTest.scala to FinalStageConfigIsolationSuit.scala 7a158057 [simon] fix codestyle 86148f3e [simon] fix ut 291c93aa [simon] split KyuubiExtensionSuite to different extension ut class 2098ce89 [simon] split KyuubiExtensionSuite to different extension ut class 2aaed1c8 [simon] Merge remote-tracking branch 'upstream/master' into extension-ut Lead-authored-by: Simon <3656562@qq.com> Co-authored-by: simon <zhangshiming@cvte.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
126ba514dd
|
[KYUUBI #1130] [DOCS] Restore Kyuubi logo image size
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1130 from pan3793/logo. Closes #1130 e79da016 [Cheng Pan] [DOCS] Restore Kyuubi logo image size Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
e0f880433f
|
[KYUUBI #1125][FOLLOWUP] Compress and remove duplicated new logo and keep legacy logo
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1128 from pan3793/logo. Closes #1125 affa2b1a [Cheng Pan] [KYUUBI #1125][FOLLOWUP] Compress and remove duplicated new logo and keep legacy logo Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
happytofly
|
a5e9c83040
|
[KYUUBI #1125] A new version of logo flie, apache information included.
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1125 from happytofly/master. Closes #1125 9e3ffc42 [happytofly] Add files via upload f91316d7 [happytofly] Add files via upload Authored-by: happytofly <systemvv@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Fu Chen
|
208c74d197
|
[KYUUBI #1105][FOLLOWUP] Minor improve ZorderBytesUtils.defaultValue
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> fix [comment](https://github.com/apache/incubator-kyuubi/pull/1109#discussion_r711007709) ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1126 from cfmcgrady/zorder. Closes #1105 48e4d73a [Fu Chen] review a58c5e52 [Fu Chen] minor improve ZorderBytesUtils.defaultValue Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
h
|
cc3ca56ab2
|
[KYUUBI #1085] Add forcedMaxOutputRows rule for limitation to avoid huge output unexpectly
Add MaxOutputRows rule for output rows limitation to avoid huge output unexpectedly <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add <img width="1440" alt="截屏2021-09-12 下午12 19 28" src="https://user-images.githubusercontent.com/635169/132972063-b12937bb-807a-47bd-8d21-835d83031191.png"> '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> We support the PR feature with limitation that avoid huge output rows in user ad-hoc query unexpected，Generally ad-hoc query seems needle in a Haystack, user pick few computed result data in huge data from warehouse, we mainly used in below cases: - CASE 1: ``` SELECT [c1, c2, ...] ``` - CASE 2: ``` WITH CTE AS (...) SELECT [c1, c2, ...] FROM Express(CTE) ... ``` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate <img width="1440" alt="截屏2021-09-12 下午12 19 28" src="https://user-images.githubusercontent.com/635169/132972078-c4821135-0520-420d-9ab8-24e124f6c6c9.png"> - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1085 from i7xh/watchdogMaxOutputRows. Closes #1085 77aa6ff3 [h] Resolve issue: unify maxOutputRows and using unittest setupData 939d2556 [h] Resolve issue: well-format code 5b0383dd [h] Fix issue: Final Limit according to spark.sql.watchdog.forcedMaxOutputRows 7dcbb0a4 [h] Resolve issue: remove origin match from rule f4dff4cf [h] Resolve issue: update forcedMaxOutputRows config doc ae21c1ac [h] Resovled issue: Support Aggregate force limitation and Remove InsertIntoDataSourceDirCommand process a9d3640b [h] Resolved code review issue 01c87fd2 [h] Add MaxOutputRows rule for output rows limitation to avoid huge output rows of query unexpectedly Authored-by: h <h@zhihu.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Brian Yue
|
cbe5bee6ec
|
[KYUUBI #1112] Upgrade scala to 2.12.15
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Close #1112. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1120 from byyue/feature-1112. Closes #1112 9bb10858 [Brian Yue] [FEATURE] [Dependency] Upgrade scala to 2.14.15 Authored-by: Brian Yue <code.byyue@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
timothy65535
|
6983139487
|
[KYUUBI #1111] Remove duplicate keys
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Remove duplicate keys. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1111 from timothy65535/remove. Closes #1111 75b001e3 [timothy65535] Remove duplicate keys Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Fu Chen
|
7da30bd992
|
[KYUUBI #1105][FOLLOWUP][TEST] Refine KyuubiSparkSQLExtension suites
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> fix [comment](https://github.com/apache/incubator-kyuubi/pull/1109#discussion_r709838217) ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1118 from cfmcgrady/refine-extension-test. Closes #1105 a3f31e5d [Fu Chen] minor 853bf8f2 [Fu Chen] fix style 89f2338a [Fu Chen] minor d4d57ebd [Fu Chen] refine KyuubiSparkSQLExtension suites Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
861bf6af31
|
[KYUUBI #1108] Register KyuubiDriver in ServiceLoader
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1108 from pan3793/jdbc. Closes #1108 0f14d12a [Cheng Pan] Add License header 18e19c5f [Cheng Pan] Exclude META-INF/services from rat 5048c471 [Cheng Pan] Register KyuubiDriver in ServiceLoader Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Fu Chen
|
5602c29b36
|
[KYUUBI #1105] Zorder codegen support
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. zorder codegen support 2. move all of the zorder tests to a separate file. 3. add more zorder unit tests. generate.java ```java /* 001 */ public java.lang.Object generate(Object[] references) { /* 002 */ return new SpecificUnsafeProjection(references); /* 003 */ } /* 004 */ /* 005 */ class SpecificUnsafeProjection extends org.apache.spark.sql.catalyst.expressions.UnsafeProjection { /* 006 */ /* 007 */ private Object[] references; /* 008 */ private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[] mutableStateArray_0 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[1]; /* 009 */ /* 010 */ public SpecificUnsafeProjection(Object[] references) { /* 011 */ this.references = references; /* 012 */ mutableStateArray_0[0] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(1, 32); /* 013 */ /* 014 */ } /* 015 */ /* 016 */ public void initialize(int partitionIndex) { /* 017 */ /* 018 */ } /* 019 */ /* 020 */ // Scala.Function1 need this /* 021 */ public java.lang.Object apply(java.lang.Object row) { /* 022 */ return apply((InternalRow) row); /* 023 */ } /* 024 */ /* 025 */ public UnsafeRow apply(InternalRow i) { /* 026 */ mutableStateArray_0[0].reset(); /* 027 */ /* 028 */ /* 029 */ /* 030 */ /* 031 */ byte[] value_0 = null; /* 032 */ byte[][] binaryArray_0 = new byte[2][]; /* 033 */ /* 034 */ boolean isNull_1 = i.isNullAt(0); /* 035 */ long value_1 = isNull_1 ? /* 036 */ -1L : (i.getLong(0)); /* 037 */ if (isNull_1) { /* 038 */ binaryArray_0[0] = (byte[]) ((byte[][]) references[0] /* defaultValues */)[0]; /* 039 */ } else { /* 040 */ binaryArray_0[0] = org.apache.kyuubi.sql.zorder.ZorderBytesUtils.toByte(value_1); /* 041 */ } /* 042 */ /* 043 */ /* 044 */ boolean isNull_2 = i.isNullAt(1); /* 045 */ long value_2 = isNull_2 ? /* 046 */ -1L : (i.getLong(1)); /* 047 */ if (isNull_2) { /* 048 */ binaryArray_0[1] = (byte[]) ((byte[][]) references[0] /* defaultValues */)[1]; /* 049 */ } else { /* 050 */ binaryArray_0[1] = org.apache.kyuubi.sql.zorder.ZorderBytesUtils.toByte(value_2); /* 051 */ } /* 052 */ /* 053 */ value_0 = org.apache.kyuubi.sql.zorder.ZorderBytesUtils.interleaveMultiByteArray(binaryArray_0); /* 054 */ mutableStateArray_0[0].write(0, value_0); /* 055 */ return (mutableStateArray_0[0].getRow()); /* 056 */ } /* 057 */ /* 058 */ /* 059 */ } ``` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1109 from cfmcgrady/zorder-codegen. Closes #1105 f06da25a [Fu Chen] update dependency 8e04dc4e [Fu Chen] JMap -> Array da00f86c [Fu Chen] remove Logging trait 266b65b3 [Fu Chen] fix style and revert Expression e15165f3 [Fu Chen] ev.isNull = FalseLiteral 0d77d9de [Fu Chen] review 0d8467f4 [Fu Chen] fix style 5ede5df9 [Fu Chen] refactor test 56e12172 [Fu Chen] clean up 88573102 [Fu Chen] update 5f3d1c02 [Fu Chen] zorder codegen support Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
timothy65535
|
b6de06c5cc
|
[KYUUBI #1113] Fix error fetching results after run PlanOnlyStatement
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> **RUN QUERY** ``` set kyuubi.operation.plan.only.mode=ANALYZE; select 1; ``` **ERROR MSG** ``` 2021-09-16 15:19:57.278 INFO operation.PlanOnlyStatement: Processing anonymous's query[015cbe32-c872-407c-b6a8-d3be6b43d745]: INITIALIZED_STATE -> RUNNING_STATE, statement: set kyuubi.operation.plan.only.mode=ANALYZE 2021-09-16 15:19:57.293 INFO operation.PlanOnlyStatement: Processing anonymous's query[015cbe32-c872-407c-b6a8-d3be6b43d745]: RUNNING_STATE -> FINISHED_STATE, statement: set kyuubi.operation.plan.only.mode=ANALYZE, time taken: 0.015 seconds 2021-09-16 15:19:57.309 WARN spark.SparkThriftFrontendService: Error fetching results: org.apache.kyuubi.KyuubiSQLException: OperationHandle [type=EXECUTE_STATEMENT, identifier: 015cbe32-c872-407c-b6a8-d3be6b43d745] failed to generate operation log at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) at org.apache.kyuubi.operation.OperationManager.$anonfun$getOperationLogRowSet$2(OperationManager.scala:125) at scala.Option.getOrElse(Option.scala:189) at org.apache.kyuubi.operation.OperationManager.getOperationLogRowSet(OperationManager.scala:125) at org.apache.kyuubi.session.AbstractSession.fetchResults(AbstractSession.scala:193) at org.apache.kyuubi.service.AbstractBackendService.fetchResults(AbstractBackendService.scala:164) at org.apache.kyuubi.service.ThriftFrontendService.FetchResults(ThriftFrontendService.scala:479) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2021-09-16 15:19:57.313 WARN spark.SparkThriftFrontendService: Error fetching results: org.apache.kyuubi.KyuubiSQLException: OperationHandle [type=EXECUTE_STATEMENT, identifier: 015cbe32-c872-407c-b6a8-d3be6b43d745] failed to generate operation log at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) at org.apache.kyuubi.operation.OperationManager.$anonfun$getOperationLogRowSet$2(OperationManager.scala:125) at scala.Option.getOrElse(Option.scala:189) at org.apache.kyuubi.operation.OperationManager.getOperationLogRowSet(OperationManager.scala:125) at org.apache.kyuubi.session.AbstractSession.fetchResults(AbstractSession.scala:193) at org.apache.kyuubi.service.AbstractBackendService.fetchResults(AbstractBackendService.scala:164) at org.apache.kyuubi.service.ThriftFrontendService.FetchResults(ThriftFrontendService.scala:479) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1114 from timothy65535/ky-1113. Closes #1113 e8409d12 [timothy65535] [KYUUBI #1113] Fix error fetching results after run PlanOnlyStatement Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
4f11b51673
|
[KYUUBI #1059] Add Plan Only Operations
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Introduce a session-level configuration `operation.plan.only.mode` to change ExecuteStatement to PlanOnlyStatement, PlanOnlyStatement will only parse, analyze or optimize the statements instead of executing them. This feature could run all Spark SQLs to verify the semantic compatibility without having side effects, which is very useful for testing, migrating without query rewrite. This is similar to the `explain` command ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1060 from yaooqinn/1059. Closes #1059 e0aa1060 [Kent Yao] [KYUUBI #1059] Add Plan Only Operations 4104aa42 [Kent Yao] [KYUUBI #1059] Add Plan Only Operations aae52f26 [Kent Yao] [KYUUBI #1059] Add Plan Only Operations 575dbf8f [Kent Yao] [KYUUBI #1059] Add Plan Only Operations Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
