7a77b7da40
231 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
simon
|
7a77b7da40
|
[KYUUBI #1039][FOLLOWUP] Application stop log redirect append to engineLog
### _Why are the changes needed?_ Now application stop log will overwrite log file `engineLog`, which cause engine logs are lost.  This pr change `overwrite` to `append`. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1210 from simon824/redirect. Closes #1039 ae3e031d [simon] Redirect append to engineLog 2d399883 [simon] Merge remote-tracking branch 'upstream/master' 3d9c12d3 [simon] Merge remote-tracking branch 'upstream/master' 5b905dbd [simon] Merge remote-tracking branch 'upstream/master' Authored-by: simon <zhangshiming@cvte.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
simon
|
a55d5fbf79
|
[KYUUBI #1039] [TEST] Add kill application test
### Why are the changes needed? Add kill application test for #1119 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1194 from simon824/killTest. Closes #1039 36386726 [Simon] Merge branch 'apache:master' into killTest acb55492 [simon] init 3d9c12d3 [simon] Merge remote-tracking branch 'upstream/master' 5b905dbd [simon] Merge remote-tracking branch 'upstream/master' Lead-authored-by: simon <zhangshiming@cvte.com> Co-authored-by: Simon <3656562@qq.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Kent Yao
|
bf629291bf
|
[KYUUBI #1206][FEATURE] Support GROUP for engine.share.level
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> The detail locates #1206 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1207 from yaooqinn/1206. Closes #1206 99e324a9 [Kent Yao] [KYUUBI #1206][FEATURE] Support GROUP for engine.share.level f41fd944 [Kent Yao] Merge branch 'master' into 1206 a7cb80da [Kent Yao] [KYUUBI #1206][FEATURE] Support GROUP for engine.share.level 6ecef063 [Kent Yao] [KYUUBI #1206][FEATURE] Support GROUP for engine.share.level d7ff7583 [Kent Yao] [KYUUBI #1206][FEATURE] Support GROUP for engine.share.level 7b3b2f53 [Kent Yao] [KYUUBI #1206][FEATURE] Support GROUP for engine.share.level Authored-by: Kent Yao <yao@apache.org> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
fwang12
|
95c74cdefc |
[KYUUBI #1205] Support to show more engine submission failure log for issue tracking
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Customers might meet exception when launching a spark engine, especially for beginner, now kyuubi only show the last line of engine log and it is difficult to find the root cause in customer side. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1205 from turboFei/show_more_engine_log. Closes #1205 72002dd6 [fwang12] address comments a74fa1de [fwang12] update docs fd967371 [fwang12] refactor 434a093d [fwang12] update docs 880c8cc2 [fwang12] show more engine failure log for debug Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
Cheng Pan
|
1b1b898123
|
[KYUUBI #1191] [SUB-TASK] KyuubiHiveDriver should implements Driver directly
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Before this change, `KyuubiHiveDriver extends HiveDriver`, even we don't expose `HiveDriver` to service loader, when `KyuubiHiveDriver` is loaded, the parent class `HiveDriver` will be loaded and initialized first, which is not expected. ``` static { try { DriverManager.registerDriver(new HiveDriver()); } catch (SQLException var1) { var1.printStackTrace(); } manifestAttributes = null; } ``` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1191 from pan3793/kyuubi-hive-jdbc. Closes #1191 e5276667 [Cheng Pan] KyuubiHiveDriver should implements Driver directly Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
9e6eb086b1
|
[KYUUBI #1203] [SUB-TASK][BUILD] Remove Scala stuff from kyuubi-hive-jdbc pom
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> #1131 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1203 from pan3793/pom. Closes #1203 8e6b8f9b [Cheng Pan] fix d31fd198 [Cheng Pan] fix 9da82843 [Cheng Pan] fix 6d150de9 [Cheng Pan] fix 9faf77e5 [Cheng Pan] Remove Scala stuff from kyuubi-hive-jdbc pom Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
simon
|
4f8f4d9c6b
|
[KYUUBI #1190] Enhance error log detection
### Why are the changes needed? When line2 is read, it does not contain `at` or `cause by`, so all of the following messages are discarded. #1190 ``` 2021-10-09 09:35:14.528 ERROR spark.SparkSQLEngine: Failed to instantiate SparkSession: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient; org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient; at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:113) at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:225) at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:137) ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1200 from simon824/fixut. Closes #1190 d239df0e [simon] codestyle 83cb7d83 [simon] fixut 2d399883 [simon] Merge remote-tracking branch 'upstream/master' 3d9c12d3 [simon] Merge remote-tracking branch 'upstream/master' 5b905dbd [simon] Merge remote-tracking branch 'upstream/master' Authored-by: simon <zhangshiming@cvte.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
zhang1002
|
ca9ecd6e34
|
[KYUUBI #1066] Automatically create engine event log root dir when it does not exist
bugfix: #1066 ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1183 from zhang1002/auto-create-log-root. Closes #1066 d686e33c [zhang1002] 1. change logRoot to engineLogRoot 2. add some judge for mkDir fb6987f5 [zhang1002] create log root automatically c319127e [张宇翔] test Lead-authored-by: zhang1002 <zhang1002@126.com> Co-authored-by: 张宇翔 <zhang1002@126.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
simon
|
2ae6acda65
|
[KYUUBI #1039] Kill yarn job when engine initialize timeout and yarnApplicationState is ACCEPTED
kill yarn job when engine initialize timeout and yarnApplicationState is ACCEPTED #1039 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1119 from simon824/killyarnjob. Closes #1039 7face4db [simon] get KYUUBI_HOME da8c3ec4 [Simon] Merge branch 'apache:master' into killyarnjob 95ae8f59 [simon] add appId 8a6ddcd1 [simon] kill application by script e0cd2af8 [simon] kill application by script d15d38c5 [simon] kill application by script 1fd1373b [simon] mv sparklauncher to sparkProcessBuilder 5e806426 [simon] fix codestyle a9cc4505 [simon] fix option No value Exception 9c7ca2f8 [simon] fix option No value Exception bbfe8e25 [simon] add set sparkHome 04f23c8a [simon] fix codestyle 75599233 [simon] add spark-launcher dep ef4b2706 [simon] implement by sparkLauncher 59c25b7c [simon] kill yarn application by restful api 49921a48 [simon] fix ut a31d8f6a [simon] fix ut 1abc6665 [simon] rename killApplication 43a95c1d [simon] Merge branch 'master' into killyarnjob 3d9c12d3 [simon] Merge remote-tracking branch 'upstream/master' 9eaeb16d [simon] fix ut 64ee1b11 [simon] code style 5b905dbd [simon] Merge remote-tracking branch 'upstream/master' 139f3b79 [Simon] Merge branch 'apache:master' into killyarnjob 1a52401d [simon] #1039 Lead-authored-by: simon <zhangshiming@cvte.com> Co-authored-by: Simon <3656562@qq.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
d1e7f0cc20
|
[KYUUBI #1143][FOLLOWUP] Correct KyuubiThriftBinaryFrontendService name
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Correct typo. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1186 from pan3793/typo. Closes #1143 fd53e94a [Cheng Pan] [KYUUBI #1143][FOLLOWUP] Correct KyuubiThriftBinaryFrontendService name Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
fwang12
|
4f74aba50e |
[KYUUBI #1160] Support to config operation log root dir both for kyuubi server and engine sides
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Support to config operation log root dir with kyuubi configuration both for server and engine sides. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1160 from turboFei/support_config_operationlog_dir. Closes #1160 ed424f46 [fwang12] fix ut 5edfd4ae [fwang12] refactor ut b7026b2f [fwang12] add ut 6ff4d891 [fwang12] save 30cd5952 [fwang12] fix ut f058619a [fwang12] fix ut 96eb020d [fwang12] refactor ae0ded96 [fwang12] set config value for testing 83d932f4 [fwang12] exclude 33e3e214 [fwang12] update 6f030cba [fwang12] fix 77ec9ca2 [fwang12] save 711ececf [fwang12] Fix code style cfa3bfac [fwang12] refactor b299f8cf [fwang12] complete ut 26de3712 [fwang12] Support to config operation log root dir for kyuubi server and engine side Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
zhouyifan279
|
413ba2eb24
|
[KYUUBI #1009][FOLLOWUP] Hive delegation token sent from Kyuubi server is ignore by SQL engine
### _Why are the changes needed?_ Hive delegation token sent from Kyuubi server is ignore by SQL engine as token alias does not match `hive.metastore.uris`. ``` 21/09/27 14:22:39 INFO operation.ExecuteStatement: Processing morty's query[5711be2b-1d2f-412a-a2e0-b52a5b0e0445]: RUNNING_STATE -> FINISHED_STATE, statement: show databases, time taken: 0.056 seconds 21/09/27 14:22:39 INFO spark.SparkThriftBinaryFrontendService: Ignore Hive token as engine has no Hive token ever before 21/09/27 14:22:39 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user. ``` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate ``` 21/09/27 20:51:20 INFO operation.ExecuteStatement: Processing morty's query[652a9f74-4813-406c-a842-6fc8b61baaf1]: RUNNING_STATE -> FINISHED_STATE, statement: show databases, time taken: 0.05 seconds 21/09/27 20:51:20 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user. ``` - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1168 from zhouyifan279/#1009. Closes #1009 fd326347 [zhouyifan279] [KYUUBI #1009][FOLLOWUP] Hive delegation token is ignore by SQL engine 3302ee4b [zhouyifan279] [KYUUBI #1009][FOLLOWUP] Hive delegation token is ignore by SQL engine Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
zhouyifan279
|
d45421b37e
|
[KYUUBI #1008][FOLLOWUP] Can not obtain hadoop file system delegation token
### _Why are the changes needed?_
Kyuubi server throws exception when obtaining hadoop file system delegation token
```
21/09/27 19:53:07 WARN credentials.HadoopCredentialsManager: Failed to update tokens for morty, try again in 3000 ms
java.io.IOException: DestHost:destPort kdc.hadoop.com:9000 , LocalHost:localPort client.hadoop.com/192.168.56.1:0. Failed on local exception: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal n
ame
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:836)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:811)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1566)
at org.apache.hadoop.ipc.Client.call(Client.java:1508)
at org.apache.hadoop.ipc.Client.call(Client.java:1405)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy10.getDelegationToken(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDelegationToken(ClientNamenodeProtocolTranslatorPB.java:1081)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy11.getDelegationToken(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:703)
at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:1845)
at org.apache.hadoop.security.token.DelegationTokenIssuer.collectDelegationTokens(DelegationTokenIssuer.java:95)
at org.apache.hadoop.security.token.DelegationTokenIssuer.addDelegationTokens(DelegationTokenIssuer.java:76)
at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.$anonfun$obtainDelegationTokens$2(HadoopFsDelegationTokenProvider.scala:66)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:141)
at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.$anonfun$obtainDelegationTokens$1(HadoopFsDelegationTokenProvider.scala:64)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider$$anon$1.run(HadoopFsDelegationTokenProvider.scala:106)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider$.doAsProxyUser(HadoopFsDelegationTokenProvider.scala:105)
at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.obtainDelegationTokens(HadoopFsDelegationTokenProvider.scala:57)
at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.$anonfun$run$1(HadoopCredentialsManager.scala:210)
at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.$anonfun$run$1$adapted(HadoopCredentialsManager.scala:210)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:214)
at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.run(HadoopCredentialsManager.scala:210)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:884)
at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:413)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1636)
at org.apache.hadoop.ipc.Client.call(Client.java:1452)
... 43 more
Caused by: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name
at org.apache.hadoop.security.SaslRpcClient.getServerPrincipal(SaslRpcClient.java:327)
at org.apache.hadoop.security.SaslRpcClient.createSaslClient(SaslRpcClient.java:234)
at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:160)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:622)
at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:413)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:822)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:818)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:818)
... 46 more
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes #1166 from zhouyifan279/#1008.
Closes #1008
43dbf605 [zhouyifan279] [KYUUBI #1008][FOLLOWUP] Can not obtain hadoop fs delegation token
Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
|
||
|
zhouyifan279
|
114ec9a046
|
[KYUUBI #1009][FOLLOWUP] Kyuubi server failed to start when connecting to secured HMS
### _Why are the changes needed?_ Fix #1161 [Bug] Kyuubi server failed to start when connecting to a secured Hive metastore ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate  - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1162 from zhouyifan279/KYUUBI#1161. Closes #1009 f5b54818 [zhouyifan279] [KYUUBI #1161][Bug] Kyuubi server failed to start when connecting to a secured Hive metastore Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Brian Yue
|
683675be92
|
[KYUUBI #1146] Move KinitAuxiliaryService to kyuubi-server module
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Close #1146. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1156 from byyue/feature-1146. Closes #1146 6b677723 [Brian Yue] Move KinitAuxiliaryService to kyuubi-server module Authored-by: Brian Yue <code.byyue@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Kent Yao
|
0ee8eff5a0
|
[KYUUBI #1143] Refact Serverable/FrontendService/ServiceDiscovery hierarchy
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> After this PR, the service named service discovery will be a child service of frontend service and it's optional. A servalable will contain a backend and a list of frontends, the former one stay AS-IS and the latter is changed from single to multiple, which means a serverable can have multiple interfaces to interact with its different clients ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1144 from yaooqinn/1143. Closes #1143 f2beb721 [Kent Yao] fix test 128cf282 [Kent Yao] nit 37aaff9c [Kent Yao] lift FrontendService trait add335c4 [Kent Yao] nit 3a111cba [Kent Yao] [KYUUBI #1143] Refact Serverable/FrontendService/ServiceDiscovery hierarchy bfa63b81 [Kent Yao] [KYUUBI #1143] Refact Serverable/FrontendService/ServiceDiscovery hierarchy 7e5a5cbb [Kent Yao] Merge branch 'master' into 1143 09263bc0 [Kent Yao] [KYUUBI #1143] Refact Serverable/FrontendService/ServiceDiscovery hierarchy Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
yanghua
|
ee8e5a12f1
|
[KYUUBI #1137] KyuubiFrontendServices oom hook should call the stop on serverable level
… <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1142 from yanghua/KYUUBI-1137. Closes #1137 418f3dff [yanghua] [KYUUBI #1137] KyuubiFrontendServices oom hook should call the stop on serverable level Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Fu Chen
|
0be523f6fe |
[KYUUBI #1135] [SUB-TASK][KPIP-1] Register default ObjectMapper instead of manually serialize/deserialize JSON input argument.
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Register default `ObjectMapper` instead of manually serialize/deserialize JSON input argument. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1135 from cfmcgrady/objectmapper. Closes #1135 580707cb [Fu Chen] fix style 48da4fe4 [Fu Chen] refactor unit test 067f11be [Fu Chen] fix style 9462d936 [Fu Chen] object mapper Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Kent Yao
|
4f11b51673
|
[KYUUBI #1059] Add Plan Only Operations
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Introduce a session-level configuration `operation.plan.only.mode` to change ExecuteStatement to PlanOnlyStatement, PlanOnlyStatement will only parse, analyze or optimize the statements instead of executing them. This feature could run all Spark SQLs to verify the semantic compatibility without having side effects, which is very useful for testing, migrating without query rewrite. This is similar to the `explain` command ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1060 from yaooqinn/1059. Closes #1059 e0aa1060 [Kent Yao] [KYUUBI #1059] Add Plan Only Operations 4104aa42 [Kent Yao] [KYUUBI #1059] Add Plan Only Operations aae52f26 [Kent Yao] [KYUUBI #1059] Add Plan Only Operations 575dbf8f [Kent Yao] [KYUUBI #1059] Add Plan Only Operations Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
yanghua
|
d098f9eb21
|
[KYUUBI #1102] Add test cases for error and exception request
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1103 from yanghua/KYUUBI-1102. Closes #1102 fdd0d57c [yanghua] [KYUUBI #1102] Add test cases for error and exception request Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
timothy65535
|
aa1a7d63be
|
[KYUUBI #1097] Rename HadoopDelegationProvider
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Rename `HadoopDelegationProvider` to `HadoopDelegationTokenProvider`. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1097 from timothy65535/renametokenfile. Closes #1097 cb93ed89 [timothy65535] Rename HadoopDelegationProvider Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
zhouyifan279
|
6a01f65d3e
|
[KYUUBI #916] Send DelegationTokens to SQL engines
### _Why are the changes needed?_ This PR finishes issue #916 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1074 from zhouyifan279/KYUUBI#916. Closes #916 28176c7e [zhouyifan279] [KYUUBI #916] Send DelegationTokens to SQL engines b6c931bd [zhouyifan279] [KYUUBI #916] Send DelegationTokens to SQL engines Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
yanghua
|
9da8e4642b
|
[KYUUBI #1092] Implement api: /sessions/${sessionhandle} (DELETE)
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1093 from yanghua/KYUUBI-1092. Closes #1092 f4e863a8 [yanghua] [KYUUBI #1092] Implement api: /sessions/${sessionhandle} (DELETE) Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
张宇翔
|
354e3c0b5a
|
[KYUUBI #1068] event log can't upload to hdfs
bug: #1068 This version, event log don't support upload file to hdfs ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1075 from zhang1002/branch-1.3_supportLogUploadHdfs. Closes #1068 7b1483c5 [张宇翔] Modify some UT 27aa4596 [张宇翔] Modify some UT c8223914 [张宇翔] Modify some UT 8b3ef67b [张宇翔] Modify some UT fa06ead9 [张宇翔] Merge branch 'master' into branch-1.3_supportLogUploadHdfs 8384a6e2 [张宇翔] Merge remote-tracking branch 'upstream/master' 8801ece7 [张宇翔] Remove "toString" from ut 3288724b [张宇翔] Modify some UT 0bd7ecbd [张宇翔] Support the eventLog upload to hdfs 714ab1b8 [张宇翔] Merge remote-tracking branch 'upstream/master' af28520b [张宇翔] Merge remote-tracking branch 'upstream/master' c328c884 [张宇翔] Merge remote-tracking branch 'upstream/master' abcaae1f [张宇翔] Merge remote-tracking branch 'upstream/master' f0a7e566 [张宇翔] Merge remote-tracking branch 'upstream/master' 112aa6b6 [张宇翔] Merge remote-tracking branch 'upstream/master' c55652fd [张宇翔] Merge remote-tracking branch 'upstream/master' ef5e8c55 [张宇翔] Merge remote-tracking branch 'upstream/master' 5a75384c [张宇翔] Merge remote-tracking branch 'upstream/master' 8da4f7ed [张宇翔] Merge remote-tracking branch 'upstream/master' a7ba1bfc [张宇翔] Merge remote-tracking branch 'upstream/master' b662a989 [张宇翔] Merge remote-tracking branch 'upstream/master' 4c8f3b87 [张宇翔] Merge remote-tracking branch 'upstream/master' 8b686767 [张宇翔] Merge remote-tracking branch 'upstream/master' cf99e309 [张宇翔] Merge remote-tracking branch 'upstream/master' 0afaa578 [张宇翔] Merge remote-tracking branch 'upstream/master' b24fea07 [张宇翔] Merge remote-tracking branch 'upstream/master' e517cfc5 [张宇翔] Merge remote-tracking branch 'upstream/master' 18aebe76 [张宇翔] Merge remote-tracking branch 'upstream/master' f248bef7 [张宇翔] Merge remote-tracking branch 'upstream/master' 5ffb54f3 [张宇翔] Add kyuubi-spark-monitor module for nightly.yml Authored-by: 张宇翔 <zhang1002@126.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
yanghua
|
61752d90de
|
[KYUUBI #1088] Refactor java version dto to scala version
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1089 from yanghua/KYUUBI-1088. Closes #1088 c905bde3 [yanghua] [KYUUBI #1088] Refactor java version dto to scala version Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
timothy65535
|
c080283671
|
[KYUUBI #1080] Move stringifyException to Utils
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Move stringifyException from KyuubiSQLException to Utils, make it more general. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1081 from timothy65535/ky-1080. Closes #1080 ce3015d5 [timothy65535] [KYUUBI #1080] Move stringifyException to Utils Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
yanghua
|
fc3d28295d |
[KYUUBI #1052] Implement api: /sessions (POST)
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1073 from yanghua/KYUUBI-1052. Closes #1052 926a6a62 [yanghua] Addressed review suggestion f227c042 [yanghua] [KYUUBI #1052] Implement api: /sessions (POST) Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
zhouyifan279
|
1b89272749
|
[KYUUBI #1009] Implement Hive metastore server Delegation Token Provider
### _Why are the changes needed?_ This PR finishes the work of issue #1009 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1053 from zhouyifan279/KYUUBI#1009. Closes #1009 342fca91 [zhouyifan279] [KYUUBI #1009] Implement Hive metastore server Delegation Token Provider a9e1b34d [zhouyifan279] [KYUUBI #1009] Implement Hive metastore server Delegation Token Provider 82da9007 [zhouyifan279] [KYUUBI #1009] Implement Hive metastore server Delegation Token Provider 2c00dd3a [zhouyifan279] [KYUUBI #1009] Implement Hive metastore server Delegation Token Provider Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
qiuliang
|
d83674248e
|
[KYUUBI #1034] Engine may deadlock when close operationLog
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1062 from qiuliang988/1034. Closes #1034 41d510f5 [qiuliang] add comments dbd7c770 [qiuliang] [KYUUBI #1034]Engine may deadlock when close operationLog Authored-by: qiuliang <qlcumt@163.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
zhouyifan279
|
455bdac2bf |
[KYUUBI #1008] Implement Hadoop File System Delegation Token Provider
### _Why are the changes needed?_ This PR finishes the work of task #1008 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1029 from zhouyifan279/KYUUBI#1008. Closes #1008 695cd955 [zhouyifan279] [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider da50aaef [zhouyifan279] [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider bf7c628f [zhouyifan279] [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider ed9739ab [zhouyifan279] [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider 2512abf9 [zhouyifan279] [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider 9c88f721 [zhouyifan279] [KYUUBI #1008] Implement Hadoop File System Delegation Token Provider Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
yanghua
|
10d8ab90dd |
[KYUUBI #1031] Implement api: /sessions/count
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1038 from yanghua/KYUUBI-1031. Closes #1031 36ad5b97 [yanghua] Addressed the review concerns 66d8b0fe [yanghua] [KYUUBI #1031] Implement api: /sessions/count Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
ulysses-you
|
347c932768 |
[KYUUBI #1032] Keep server session id and engien session id consistent
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Closes https://github.com/apache/incubator-kyuubi/issues/1032 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1033 from ulysses-you/engine-session-id. Closes #1032 5aa5f5f8 [ulysses-you] indention a97ae1b5 [ulysses-you] nit aef754b4 [ulysses-you] engine ref id 62399c40 [ulysses-you] cleanup 2bf13a63 [ulysses-you] nit 11e925dd [ulysses-you] engine side generate da3f51be [ulysses-you] nit 6136f66b [ulysses-you] keep server session id and engien session id consistent Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Cheng Pan
|
f584953f7f
|
[KYUUBI #703][FOLLOWUP][DEPS][TEST] Reduce Hudi dependencies
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> HBase deps is optional for creating hudi table and query, but required for writing data to hudi table, as currently we only test create table and query for data lake formats, it's reasonable to remove the hbase deps to reduce the deps pulled from hudi. This PR also change all data lake test suites in spark engine from `hive` catalog to `in-memory` catalog. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1028 from pan3793/hudi. Closes #703 56c41a30 [Cheng Pan] Use in-memory catalog for DataLake test b582a865 [Cheng Pan] Revert "Default use in-memory in Spark engine test" e62c403a [Cheng Pan] Default use in-memory in Spark engine test e2b8e0fd [Cheng Pan] Remove unused hudi spark conf 069fbacc [Cheng Pan] Revert "HFileBootstrapIndex" ca1b20e1 [Cheng Pan] HFileBootstrapIndex da37b0fc [Cheng Pan] Remove log4j 41ce8526 [Cheng Pan] Align jackson version f56f5e89 [Cheng Pan] Remove HBase dependencies c2b76b54 [Cheng Pan] Exclude hudi-timeline-service 56641762 [Cheng Pan] nit e588c12f [Cheng Pan] [DEPS][TEST] Reduce Hudi dependencies Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
simon
|
098e660a3d
|
[KYUUBI #703] support hudi-0.9.0
### _Why are the changes needed?_ Hudi has add Spark Sql Support in version 0.9.0. This PR add Hudi-0.9.0 dependency and Unit Testing #703 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #994 from simon824/master. Closes #703 a04b2e42 [simon] exclude jdk.tools dep 50e1ab01 [simon] fix jdk11 dep f33ebdf1 [simon] fix hudi dep conflicts ff2585c2 [simon] fix yarn dep conflicts 32dd1ea8 [simon] fix scala version conflicts b8a37401 [simon] add spark.sql.catalogImplementation 82f5422f [simon] fix spark3 dependency 2a6c497d [simon] fix spark3 dependency 4dbec8be [Simon] Merge branch 'apache:master' into master 3f180157 [simon] fix maven 73e48d0a [simon] add spark3 support maven dependency 3def658b [simon] fix missing spark-sql-engine hudi maven dependency 524132d3 [simon] bugfix e98998a6 [simon] fix pom error 59fc6669 [Simon] Merge branch 'apache:master' into master d9e17ebf [simon] fix conflict 2466ece3 [simon] add sparkSQL test on hudi-0.9.0 Lead-authored-by: simon <zhangshiming@cvte.com> Co-authored-by: Simon <3656562@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
yanghua
|
32111a3014
|
[KYUUBI #999] Build the basic framework for rest frontend service
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1003 from yanghua/KYUUBI-999. Closes #999 78fdf542 [yanghua] Changed rest frontend service port from 10009 to 10099 73929d5d [yanghua] [KYUUBI #999] Build the basic framework for rest frontend service Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
zhouyifan279
|
f8056f458b
|
[KYUUBI #1007] Implement delegation token renewal framework
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This PR finishes sub-task #1007 under umbrella issue #913. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1015 from zhouyifan279/KYUUBI#1007. Closes #1007 94263804 [zhouyifan279] [KYUUBI #1007] Implement delegation token renewal framework 6cea96d0 [zhouyifan279] [KYUUBI #1007] Implement delegation token renewal framework dbffcf15 [zhouyifan279] [KYUUBI #1007] Implement delegation token renewal framework 57100271 [zhouyifan279] [KYUUBI #1007] Implement delegation token renewal framework Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
yanghua
|
01bebe36e8 |
[KYUUBI #1011] Fallback some frontend service config option about thrift
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1016 from yanghua/KYUUBI-1011. Closes #1011 78107b76 [yanghua] Addressed review suggestion 651b5989 [yanghua] [KYUUBI #1011] Fallback some frontend service config option about thrift Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Kent Yao
|
d375eb0cfc
|
[KYUUBI #1010] Rename engine.share.level.sub.domain to engine.share.level.subdomain
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> `subdomain` is a widely used term in computer science ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1010 from yaooqinn/subdomain. Closes #1010 72ed19a3 [Kent Yao] address comments 4dcae414 [Kent Yao] fix test c3bb788d [Kent Yao] Rename engine.share.level.sub.domain to engine.share.level.subdomain b7762357 [Kent Yao] Rename engine.share.level.sub.domain to engine.share.level.subdomain Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
ulysses-you
|
b382ed0dd8 |
[KYUUBI #988][FOLLOW] Fix test failed since event type changed
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Fix test ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1001 from ulysses-you/kyuubi-988. Closes #988 378a75bb [ulysses-you] fix Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
hongdongdong
|
94d60ea505 |
[KYUUBI #1000] Use underscore instead hyphen in path
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Use underscore instead hyphen in path, friendly to sql. ### _How was this patch tested?_ - [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1000 from hddong/fix-underscore. Closes #1000 8b3459a5 [hongdongdong] Use underscore instead hyphen in path Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
ulysses-you
|
d49f8bf06d |
[KYUUBI #988] Add Kyuubi session event
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> closes https://github.com/apache/incubator-kyuubi/issues/988 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #989 from ulysses-you/kyuubi-session-event. Closes #988 baebf2eb [ulysses-you] address comment 283d574c [ulysses-you] fix c5d1d5f7 [ulysses-you] adddress comment 44cfd06f [ulysses-you] docs 43639993 [ulysses-you] simplify session 2618b764 [ulysses-you] config name a2d4608b [ulysses-you] docs 5adbe5c8 [ulysses-you] config ea5c70e1 [ulysses-you] nit eea73ed5 [ulysses-you] config cda6d38b [ulysses-you] test a754c76f [ulysses-you] init Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
yanghua
|
4ed5704590
|
[KYUUBI #983] Refactor the basic abstraction of frontend service
… <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #984 from yanghua/KYUUBI-983. Closes #983 db953b8a [yanghua] Rename KyuubiFrontendService to be KyuubiFrontendServices 8c3d7f2a [yanghua] [KYUUBI #983] Refactor the basic abstraction of frontend service Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Fu Chen
|
cd426798bf |
[KYUUBI #991][FOLLOWUP] Improve error msg and add SPARK_HOME in Examples
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> According [comments](https://github.com/apache/incubator-kyuubi/pull/991#issuecomment-906914618) 1. Improve error message. 2. add SPARK_HOME in Examples. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #993 from cfmcgrady/ky-991-followup. Closes #991 e7c25be6 [Fu Chen] update docs 945eeb73 [Fu Chen] review 98555196 [Fu Chen] improve error msg and add SPARK_HOME in Examples Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
timothy65535
|
acb4f29906
|
[KYUUBI #962] Support engine pool feature
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> For detail, please go to _Originally posted by yaooqinn in https://github.com/apache/incubator-kyuubi/issues/962#issuecomment-905116261_ ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request ``` ./bin/beeline -u 'jdbc:hive2://localhost:10009/;?kyuubi.engine.pool.enabled=true;kyuubi.engine.pool.size=3;' ``` Closes #986 from timothy65535/ky-962-v2. Closes #962 b79f8778 [timothy65535] fix test name 9a435607 [timothy65535] fix style abc746b8 [timothy65535] update tests 71b6933d [timothy65535] revert changes 018dee8b [timothy65535] revert changes e3492934 [timothy65535] revert changes 67fc8fd9 [timothy65535] improve threshold fa52acb2 [timothy65535] update pool threshold d8b5977d [timothy65535] remove dup get 7f39b8d7 [timothy65535] fix test error 4fdf8d76 [timothy65535] update thread pool size to 1 -> threshold 2477348a [timothy65535] rich engine pool tests 8eee427f [timothy65535] fix update error msg 104be5ea [timothy65535] fix error tests 8c2d1441 [timothy65535] fix doc ddd0eda9 [timothy65535] improve doc & tests 83d7c5cf [timothy65535] update doc 4b6c7ee3 [timothy65535] improve subdomain 169d9d06 [timothy65535] update todo 1424e565 [timothy65535] increase pool size range 3523cd6c [timothy65535] update todo 2cbeafb2 [timothy65535] remove enable option 424b5364 [timothy65535] revert ServiceDiscovery 6fc1c510 [timothy65535] remove random policy 9feef8f3 [timothy65535] [KYUUBI #962] Support engine pool feature Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Fu Chen
|
9695f75889 |
[KYUUBI #991] [MINOR] improve the error message of SPARK package not found.
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> As `File.listFiles` may return null when spark package not found in folder `${PROJECT_ROOT_DIR}/kyuubi-server/externals/kyuubi-download/target`, the NPE will throw. Improve the error message. Before thie pr: ``` Caused by: java.lang.RuntimeException: org.apache.kyuubi.KyuubiSQLException:Error opening session SessionHandle [9749221f-5b30-457b-b8e5-25affba25061] for fchen due to null at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:68) at org.apache.kyuubi.session.KyuubiSessionManager.openSession(KyuubiSessionManager.scala:74) at org.apache.kyuubi.service.AbstractBackendService.openSession(AbstractBackendService.scala:45) at org.apache.kyuubi.service.ThriftFrontendService.getSessionHandle(ThriftFrontendService.scala:190) at org.apache.kyuubi.service.ThriftFrontendService.OpenSession(ThriftFrontendService.scala:199) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1377) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1362) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NullPointerException: null at scala.collection.mutable.ArrayOps$ofRef$.length$extension(ArrayOps.scala:204) at scala.collection.mutable.ArrayOps$ofRef.length(ArrayOps.scala:204) at scala.collection.IndexedSeqOptimized.isEmpty(IndexedSeqOptimized.scala:30) at scala.collection.IndexedSeqOptimized.isEmpty$(IndexedSeqOptimized.scala:30) at scala.collection.mutable.ArrayOps$ofRef.isEmpty(ArrayOps.scala:198) at scala.collection.TraversableLike.headOption(TraversableLike.scala:608) at scala.collection.TraversableLike.headOption$(TraversableLike.scala:608) at scala.collection.mutable.ArrayOps$ofRef.headOption(ArrayOps.scala:198) at org.apache.kyuubi.engine.spark.SparkProcessBuilder.$anonfun$executable$1(SparkProcessBuilder.scala:50) at scala.Option.orElse(Option.scala:447) at org.apache.kyuubi.engine.spark.SparkProcessBuilder.<init>(SparkProcessBuilder.scala:41) at org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:140) at org.apache.kyuubi.engine.EngineRef.tryWithLock(EngineRef.scala:116) at org.apache.kyuubi.engine.EngineRef.create(EngineRef.scala:128) at org.apache.kyuubi.engine.EngineRef.$anonfun$getOrCreate$1(EngineRef.scala:182) at scala.Option.getOrElse(Option.scala:189) at org.apache.kyuubi.engine.EngineRef.getOrCreate(EngineRef.scala:182) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$open$2(KyuubiSessionImpl.scala:63) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$open$2$adapted(KyuubiSessionImpl.scala:62) at org.apache.kyuubi.ha.client.ZooKeeperClientProvider$.withZkClient(ZooKeeperClientProvider.scala:74) at org.apache.kyuubi.session.KyuubiSessionImpl.open(KyuubiSessionImpl.scala:62) at org.apache.kyuubi.session.KyuubiSessionManager.openSession(KyuubiSessionManager.scala:58) ... 12 more ``` After this pr: ``` Caused by: java.lang.RuntimeException: org.apache.kyuubi.KyuubiSQLException:Error opening session SessionHandle [5ea6a8b3-1727-4e0d-b542-992f84afcde8] for fchen due to SPARK_HOME is not set! at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:68) at org.apache.kyuubi.session.KyuubiSessionManager.openSession(KyuubiSessionManager.scala:74) at org.apache.kyuubi.service.AbstractBackendService.openSession(AbstractBackendService.scala:45) at org.apache.kyuubi.service.ThriftFrontendService.getSessionHandle(ThriftFrontendService.scala:190) at org.apache.kyuubi.service.ThriftFrontendService.OpenSession(ThriftFrontendService.scala:199) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1377) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1362) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: org.apache.kyuubi.KyuubiSQLException:SPARK_HOME is not set! at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:68) at org.apache.kyuubi.engine.spark.SparkProcessBuilder.$anonfun$executable$5(SparkProcessBuilder.scala:61) at scala.Option.getOrElse(Option.scala:189) at org.apache.kyuubi.engine.spark.SparkProcessBuilder.<init>(SparkProcessBuilder.scala:61) at org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:140) at org.apache.kyuubi.engine.EngineRef.tryWithLock(EngineRef.scala:116) at org.apache.kyuubi.engine.EngineRef.create(EngineRef.scala:128) at org.apache.kyuubi.engine.EngineRef.$anonfun$getOrCreate$1(EngineRef.scala:182) at scala.Option.getOrElse(Option.scala:189) at org.apache.kyuubi.engine.EngineRef.getOrCreate(EngineRef.scala:182) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$open$2(KyuubiSessionImpl.scala:63) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$open$2$adapted(KyuubiSessionImpl.scala:62) at org.apache.kyuubi.ha.client.ZooKeeperClientProvider$.withZkClient(ZooKeeperClientProvider.scala:74) at org.apache.kyuubi.session.KyuubiSessionImpl.open(KyuubiSessionImpl.scala:62) at org.apache.kyuubi.session.KyuubiSessionManager.openSession(KyuubiSessionManager.scala:58) ... 12 more ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #991 from cfmcgrady/improve-error-msg. Closes #991 1e8da683 [Fu Chen] imporove the error message Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
ulysses-you
|
3afd20e03a
|
[KYUUBI #985] Add logCaptureThreadReleased flag
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> `isInterrupted` is always false if some `InterruptedException` are throwed. So we should use a new flag to check if log capture thread is released. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #985 from ulysses-you/flaky-test. Closes #985 1fcc0a99 [ulysses-you] empty 463a996f [ulysses-you] empty d1f2ea54 [ulysses-you] eventually f9f37343 [ulysses-you] fix Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
hongdongdong
|
d704e4d49d |
[KYUUBI #922] Add OperationEvent for server to track operation state
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Track operation state change and support output to custom system. ### _How was this patch tested?_ - [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #968 from hddong/add-server-events. Closes #922 8be8f268 [hongdongdong] Add prefix for StatementEvent 9fdcc504 [hongdongdong] fix comment a0cdb35d [hongdongdong] fix test 4d340c67 [hongdongdong] add dependency 00920751 [hongdongdong] [KYUUBI#922]Add OperationEvent for server to track operation state Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
yanghua
|
cb70e062c3
|
[KYUUBI #979] Introduce AbstractFrontendService and KyuubiFrontendService
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #980 from yanghua/KPIP-1. Closes #979 e67e37a4 [yanghua] [KYUUBI 979] Introduce AbstractFrontendService and KyuubiFrontendService Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
yanghua
|
4aa93d3cbd
|
[KYUUBI #957] Reopen KyuubiServerSuite
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #960 from yanghua/KYUUBI-957. Closes #957 94cdddda [yanghua] [KYUUBI #957] Reopen KyuubiServerSuite Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
timothy65535
|
32b18ba19c
|
[KYUUBI #933] Enhance the detection mechanism for engine startup
### 1. Description EngineRef use exists znode to determine whether engine started successfully or not(use the last znode), there are inconsistencies in this. So, take an idea to improve the detection mechanism by adding the started sessionid to the znone. `EngineRef` will check the sessionid in a loop after a engine started. ### 2. Here are two options **Option I, add sesssionid to the data of znode**  **Option II, add sessionid to the path of znode**  ### 3. Solution In order to be consistent with the design of server znode, prefer to choose _Option II_ . **Demo** ``` /kyuubi - serviceUri=bigdata:10009;version=1.3.0-SNAPSHOT;sequence=0000000000 /kyuubi_USER/test - serviceUri=bigdata:39869;version=1.3.0-SNAPSHOT;session=8178069f-0b22-4d06-b1c0-908094769397;sequence=0000000000] ``` Closes #935 from timothy65535/ky-933. Closes #933 ab5d6d80 [timothy65535] add issue id d21cb990 [timothy65535] compatible with back 568acf21 [timothy65535] fix error 3db8ba7f [timothy65535] move EngineRef get to Discovery 827f3780 [timothy65535] rename get to getEngineBySessionId 88229314 [timothy65535] update session id using internal ab06adf8 [timothy65535] improve EngineRef get logic 6a1cc39a [timothy65535] update session version c94b6e55 [timothy65535] rename sessionId to createSessionId 0cc6aae2 [timothy65535] update ha conf d24a152f [timothy65535] update ha conf 1a67c864 [timothy65535] fix initialize sql suite b3b502de [timothy65535] [KYUUBI #933] Enhance the detection mechanism for engine startup Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
