 [](https://github.com/yaooqinn/kyuubi/pull/400)      [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Hive compatibility is essential! This PR tries to restore the previous behavior.
1. Not use Spark's internal class HiveResult to convert spark catalyst type-values to JDBC results, the interval APIs are not APIs and changed unexpectedly. It involves a lot of overhead. (For this reason, pan3793 and I propose a Casting way to solve it.)
2. Unfortunately, the casting way breaks things like #397. So this is a reverting PR.
3. I checked the spark `HiveResult` carefully, I noticed that the `nested` logic is not needed in Kyuubi as we already matched primitives ahead.
4. Also the date time formatter used in `HiveResult` is excessively packaged with massively complicated logic inside spark. It's not clear and simple enough to call.
5. (tiny breaking change)the trailing spaces of a timestamp in milliseconds may not be trimmed. But this is a correctness issue and acceptable.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#400 from yaooqinn/397.
388c86d [Kent Yao] nit
bbf458b [Kent Yao] time stamp formatter
e5a7436 [Kent Yao] more tests
83f3142 [Kent Yao] timezone
38f5a76 [Kent Yao] Merge branch 'master' into 397
563d69c [Kent Yao] mute flacky test
9d834b0 [Kent Yao] Merge branch 'master' into 397
e8e05b5 [Kent Yao] nit
2a2d47b [Kent Yao] Implement a simpler toHiveString to convert spark row
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
 [](https://github.com/yaooqinn/kyuubi/pull/399)    [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
close#398
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
manual test release workflow
```
➜ kyuubi git:(KYUUBI-398) ll | grep tar.gz
-rw-r--r-- 1 chengpan staff 265M Mar 4 21:38 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0-hadoop2.7.tar.gz
-rw-r--r-- 1 chengpan staff 269M Mar 4 21:40 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0-hadoop3.2.tar.gz
-rw-r--r-- 1 chengpan staff 269M Mar 4 21:46 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.1-hadoop2.7.tar.gz
-rw-r--r-- 1 chengpan staff 273M Mar 4 21:44 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.1-hadoop3.2.tar.gz
```
Closes#399 from pan3793/KYUUBI-398.
a9294a1 [Cheng Pan] fix ut
d1d816d [Cheng Pan] fix dist script
2e3bc20 [Cheng Pan] update release workflow and dist script
0428b1b [Cheng Pan] update travis.yml
4a9bc1b [Cheng Pan] [KYUUBI #398] move to hadoop shaded client
Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
 [](https://github.com/yaooqinn/kyuubi/pull/366)     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Spark 3.1.1 released, it looks more stable than the first minor release of Spark 3.x, and a lot of great features were involved.
In this PR, I add a profile for `spark-3.1.1` support, and temporary we only support `hadoop2.7`
TODOS:
- Add Hadoop 3.2 support with some dependency issues
- Enable Iceberg tests for spark-3.1.1 after Apache iceberg‘s next release
- Using spark-3.1.1 as default
- Restore HiveResult string?
fix#298
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#366 from yaooqinn/spark3.1.
28ddf93 [Kent Yao] ga
37cb49e [Kent Yao] Merge branch 'master' into spark3.1
7b332ec [Kent Yao] typo
b6b33d5 [Kent Yao] set up ga
00005c0 [Kent Yao] set up travis
2c09fe4 [Kent Yao] test fix
68c7edb [Kent Yao] deps
2c65e98 [Kent Yao] deps
236c7e9 [Kent Yao] nit
8c8a783 [Kent Yao] Merge branch 'master' into spark3.1
46c318a [Kent Yao] nit
32f6511 [Kent Yao] Merge branch 'master' into spark3.1
27c836b [Kent Yao] Merge branch 'master' into spark3.1
5574a8e [Kent Yao] ga
d2ec2d5 [Kent Yao] [WIP]Spark3.1
49313cc [Kent Yao] [WIP]Travis
ccb1a97 [Kent Yao] [WIP]Spark3.1
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
 [](https://github.com/yaooqinn/kyuubi/pull/387)     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
In connection level, we will release engine if the session connection is closed but we don't clean up the discovery service (i.e. the namespace in zookeeper). That may cause the disk stress after running a long time.
### _How was this patch tested?_
Add some new suites in spark engine moudle with zk emmbedded.
Closes#387 from ulysses-you/issue-386.
ccb1112 [ulysses-you] move method
1558dd9 [ulysses-you] test
6cd7e57 [ulysses-you] nit
f8559d2 [ulysses-you] check level at engine discovery
eedfaee [ulysses-you] split ServiceDiscovery to server and engine
8b20e6c [ulysses-you] timeout
0cf524c [ulysses-you] remove unnecessary test
41d36f9 [ulysses-you] move exception
eaaa12d [ulysses-you] avoid stop twice
b801c14 [ulysses-you] fix test
db39960 [ulysses-you] add start check
2d1f6dd [ulysses-you] add state check
e7374aa [ulysses-you] nit
ddf383c [ulysses-you] fi
ee088be [ulysses-you] nit
3c2013b [ulysses-you] nit
73b386f [ulysses-you] improve conf of test
2185f49 [ulysses-you] init
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
 [](https://github.com/yaooqinn/kyuubi/pull/382)     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
In CONNECTION level, it's better to release session if the connection is closed.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#382 from ulysses-you/KYUUBI-381.
cddba6c [ulysses-you] type
a3fd691 [ulysses-you] increase engine init timeout
5030553 [ulysses-you] reload system properties
7a2e72f [ulysses-you] clean code
1034409 [ulysses-you] stopped
d2a001a [ulysses-you] address comment
4eb62de [ulysses-you] 60s
64d0f16 [ulysses-you] fix test
4f10d34 [ulysses-you] sleep
62dc9f9 [ulysses-you] add new stop method
77be261 [ulysses-you] remove config
fd9b73c [ulysses-you] add time sleep
209a59e [ulysses-you] fix test
88ed744 [ulysses-you] import
90015e9 [ulysses-you] simply
311fce3 [ulysses-you] fix test
7af3f72 [ulysses-you] fix test
e91377c [ulysses-you] Merge branch 'master' of https://github.com/yaooqinn/kyuubi into KYUUBI-381
f77ddbe [ulysses-you] nit
2173222 [ulysses-you] init
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
 [](https://github.com/yaooqinn/kyuubi/pull/367)      [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Currently, the engine starts to check its lifecycle during the session manager's startup without any initial delay. Before the engine gets ready finally, it might trigger self-terminating.
In this PR, we move the checker to the end where the engine is exposed to the servers.
Avoiding errors like:
```
org.apache.hive.service.cli.HiveSQLException: Error operating GET_CATALOGS: java.net.SocketException: Connection reset
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:267)
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:258)
at org.apache.hive.jdbc.HiveDatabaseMetaData.getCatalogs(HiveDatabaseMetaData.java:142)
at org.apache.kyuubi.operation.BasicIcebergJDBCTests.$anonfun$$init$$2(BasicIcebergJDBCTests.scala:50)
at org.apache.kyuubi.operation.BasicIcebergJDBCTests.$anonfun$$init$$2$adapted(BasicIcebergJDBCTests.scala:48)
at org.apache.kyuubi.operation.JDBCTestUtils.$anonfun$withMultipleConnectionJdbcStatement$3(JDBCTestUtils.scala:43)
at org.apache.kyuubi.operation.JDBCTestUtils.$anonfun$withMultipleConnectionJdbcStatement$3$adapted(JDBCTestUtils.scala:43)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#367 from yaooqinn/engine.
f6a979e [Kent Yao] fix test
9cc561c [Kent Yao] address comment
3d15ab2 [Kent Yao] nit
a85d2e0 [Kent Yao] Engine terminating checker should start after exposed
Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
 [](https://github.com/yaooqinn/kyuubi/pull/345)       [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
We might get this error when running spark sql，so it's more reasonable to catch `Throwable`.
```
java.lang.NoClassDefFoundError: org/apache/hadoop/hive/serde2/SerDe
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Closes#345 from ulysses-you/throwable.
b6ffacd [ulysses-you] throw
Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
fixes#271
Squashed commit of the following:
commit fe0f469b2068865398d6e5be1957fdf6c6f5eb87
Author: Cheng Pan <379377944@qq.com>
Date: Sat Jan 9 22:28:43 2021 +0800
followup
commit 527e32419ecb2ae584d1251cb345e3fc870965a9
Author: Cheng Pan <379377944@qq.com>
Date: Sat Jan 9 22:21:29 2021 +0800
revert some stuffs
commit 82f188972b8491c5d381ba076a53f84a2fbc1898
Author: Cheng Pan <379377944@qq.com>
Date: Sat Jan 9 16:49:14 2021 +0800
remove unused dependencies
commit c7eb27228d2749ab9100102467e94d3a6a98cf44
Author: Cheng Pan <379377944@qq.com>
Date: Sat Jan 9 15:41:59 2021 +0800
minimize dependencies in kyuubi-spark-sql-engine
commit da902b1b1e02c7e93810ca832f6c9e7109cf95b9
Author: Cheng Pan <379377944@qq.com>
Date: Sat Jan 9 13:26:34 2021 +0800
exclude hive dependencies
commit 55ae7f025a5ffcec402919e28b7b3af5987dfe16
Author: Cheng Pan <379377944@qq.com>
Date: Sat Jan 9 01:08:09 2021 +0800
pin versions and excludes in <dependencyManagement>
commit 9ef84a5f62afd66fb4b817d593ba1b13ff8348ed
Author: Cheng Pan <379377944@qq.com>
Date: Sat Jan 9 14:12:04 2021 +0800
trigger kyuubi-spark-sql-engine package
commit d492e3fdce64e6554fe0db1d810c6f4bea198baa
Author: Cheng Pan <379377944@qq.com>
Date: Sat Jan 9 13:44:13 2021 +0800
trigger kyuubi-download
commit 050281b8fc400f3ae5fa1d43d56da49846664ea4
Author: Cheng Pan <379377944@qq.com>
Date: Sat Jan 9 15:30:58 2021 +0800
pin modules version in dependencyManagement
commit 1655fc453ded4626ae5e12e7b17af6f10c30a0dd
Author: Cheng Pan <379377944@qq.com>
Date: Fri Jan 8 19:45:27 2021 +0800
tune modules in alphabetical order
commit 7f3f4d987336d71d73eab059e93f3d008cfcd1a6
Author: Cheng Pan <379377944@qq.com>
Date: Sat Jan 9 00:09:49 2021 +0800
change dependencies scope to test
fix#258
Squashed commit of the following:
commit 5b0a4d1f77b32cef00f55b80b8562752c1f462d7
Author: zen <xinjingziranchan@gmail.com>
Date: Wed Dec 16 10:15:16 2020 +0800
fix closing session log
commit c0fe24d06638f9be18d91a4ad05943a023967c62
Author: zen <xinjingziranchan@gmail.com>
Date: Sat Dec 12 22:08:25 2020 +0800
move this check task to SparkSQLSessionManager.
commit 1968913702872f0ba433679596596f114f2e16d1
Author: zen <xinjingziranchan@gmail.com>
Date: Thu Dec 10 15:55:32 2020 +0800
add log when close session
commit 26a944787c63666813a0d64e67a36b671640d493
Author: zen <xinjingziranchan@gmail.com>
Date: Thu Dec 10 11:32:45 2020 +0800
delete the engine's timeoutChecker scheduler and merge it to sessionManager's scheduler.
2020-11-10 10:15:42,001 [61208289] - ERROR [FrontendServiceHandler-Pool: Thread-449:TThreadPoolServer$WorkerProcess@315] - Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client?
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:234)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
