Commit Graph

119 Commits

Author SHA1 Message Date
Romani
2a21de7eed
[KYUUBI #497]upgrade iceberg-spark3-runtime to 0.11.1
![chaosmin](https://badgen.net/badge/Hello/chaosmin/green) [![Closes #509](https://badgen.net/badge/Preview/Closes%20%23509/blue)](https://github.com/yaooqinn/kyuubi/pull/509) ![1](https://badgen.net/badge/%2B/1/red) ![1](https://badgen.net/badge/-/1/green) ![1](https://badgen.net/badge/commits/1/yellow) [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

close #497

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #509 from chaosmin/master.

Closes #497

c1631a4 [Romani] upgrade `iceberg-spark3-runtime` to 0.11.1

Authored-by: Romani <chaosmin@hotmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-04-11 23:56:22 +08:00
fwang12
9a8b2860f6 [KYUUBI #506] Shade curator dependencies to avoid impaction by external jars
![turboFei](https://badgen.net/badge/Hello/turboFei/green) [![Closes #506](https://badgen.net/badge/Preview/Closes%20%23506/blue)](https://github.com/yaooqinn/kyuubi/pull/506) ![26](https://badgen.net/badge/%2B/26/red) ![3](https://badgen.net/badge/-/3/green) ![1](https://badgen.net/badge/commits/1/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) ![Bug](https://badgen.net/badge/Label/Bug/) [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
Shade curator dependencies to avoid impaction by external jars.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #506 from turboFei/shade_curator_new.

Closes #506

4c6fdeaa [fwang12] Shade curator dependencies to avoid impaction by external jars

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Cheng Pan <379377944@qq.com>
2021-04-10 11:07:13 +08:00
fwang12
f2ba4523f8 [KYUUBI #502][SPARK-35002][INFRA] Fix the java.net.BindException when testing with Github Action
![turboFei](https://badgen.net/badge/Hello/turboFei/green) [![Closes #503](https://badgen.net/badge/Preview/Closes%20%23503/blue)](https://github.com/yaooqinn/kyuubi/pull/503) ![6](https://badgen.net/badge/%2B/6/red) ![1](https://badgen.net/badge/-/1/green) ![10](https://badgen.net/badge/commits/10/yellow) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
Refer https://github.com/apache/spark/pull/32096, https://github.com/apache/spark/pull/32102,  this PR tries to fix the java.net.BindException when testing with Github Action.

```
SparkOperationSuite:
*** RUN ABORTED ***
  java.net.BindException: Cannot assign requested address: Service 'sparkDriver' failed after 16 retries (on a random free port)! Consider explicitly setting the appropriate binding address for the service 'sparkDriver' (for example spark.driver.bindAddress for SparkDriver) to the correct binding address.
  at sun.nio.ch.Net.bind0(Native Method)
  at sun.nio.ch.Net.bind(Net.java:461)
  at sun.nio.ch.Net.bind(Net.java:453)
  at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222)
  at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:134)
  at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:550)
  at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1334)
  at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:506)
  at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:491)
  at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973)
```

Also transfer FRONTEND_BIND_HOST by connection string to fix similar issue.

```
Cause: java.lang.RuntimeException: org.apache.kyuubi.KyuubiSQLException:org.apache.kyuubi.KyuubiException: Failed to initialize frontend service on fv-az207-19/10.1.1.0:0.
	at org.apache.kyuubi.service.FrontendService.initialize(FrontendService.scala:102)
	at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1(CompositeService.scala:40)
	at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1$adapted(CompositeService.scala:40)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at org.apache.kyuubi.service.CompositeService.initialize(CompositeService.scala:40)
	at org.apache.kyuubi.service.Serverable.initialize(Serverable.scala:44)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine.initialize(SparkSQLEngine.scala:49)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine$.startEngine(SparkSQLEngine.scala:105)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:118)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #503 from turboFei/KYUUBI-502.

Closes #502

1b10253a [fwang12] use localhost instead of 127.0.0.1
c104ce3f [fwang12] address comments
1e549c16 [fwang12] revert install shade
457ce2f5 [fwang12] try set frontend bind host in connection string
8bcd5a0d [fwang12] revert env KYUUBI_FRONTEND_BIND_HOST and set kyuubi.frontend.bind.host to 127.0.0.1 in scalatest-maven-plugin
717a992f [fwang12] update doc
d5ba05a4 [fwang12] add install shaded jars in release.yml
e8b23725 [fwang12] involve KYUUBI_FRONTEND_BIND_HOST
5eb7cdb2 [fwang12] also set KYUUBI_FRONTEND_BIND_HOST env to 127.0.0.1
7d708198 [fwang12] [KYUUBI #502][SPARK-35002][INFRA] Fix the java.net.BindException when testing with Github Action

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Cheng Pan <379377944@qq.com>
2021-04-10 02:45:45 +08:00
Yikun Jiang
9b3c22db54 [KYUUBI #491] Add Spark nightly build CI
![Yikun](https://badgen.net/badge/Hello/Yikun/green) [![Closes #495](https://badgen.net/badge/Preview/Closes%20%23495/blue)](https://github.com/yaooqinn/kyuubi/pull/495) ![24](https://badgen.net/badge/%2B/24/red) ![0](https://badgen.net/badge/-/0/green) ![2](https://badgen.net/badge/commits/2/yellow) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

### _Why are the changes needed?_
This patch add Spark nightly build CI

### _How was this patch tested?_
github action CI passed

Closes #495 from Yikun/kyuubi-491.

Closes #491

c65e1f5f [Yikun Jiang] Skip dependency check in spark-3.2-snapshot
d57ea16e [Yikun Jiang] Add Spark nightly build CI

Authored-by: Yikun Jiang <yikunkero@gmail.com>
Signed-off-by: Cheng Pan <379377944@qq.com>
2021-04-10 00:29:07 +08:00
Cheng Pan
b7ac3cb48c Revert "[KYUUBI #488] Add kyuubi-shaded module to avoid conflicts with external jars"
This reverts commit e058b87dd2.
2021-04-10 00:16:04 +08:00
fwang12
e058b87dd2
[KYUUBI #488] Add kyuubi-shaded module to avoid conflicts with external jars
![turboFei](https://badgen.net/badge/Hello/turboFei/green) [![Closes #488](https://badgen.net/badge/Preview/Closes%20%23488/blue)](https://github.com/yaooqinn/kyuubi/pull/488) ![454](https://badgen.net/badge/%2B/454/red) ![289](https://badgen.net/badge/-/289/green) ![10](https://badgen.net/badge/commits/10/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) ![Feature](https://badgen.net/badge/Label/Feature/) [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
Shade curator dependency in kyuubi-ha module to avoid impaction by external jars.

### _How was this patch tested?_
Existing UT.

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #488 from turboFei/shade_curator.

Closes #488

c88414c [fwang12] Use wildcards when selecting more than 6 elements
a2dfbf6 [fwang12] add doc for install shaded jars
224cc38 [fwang12] remove transactions
c7b9df3 [fwang12] address comments
81fc123 [fwang12] scala style
408c978 [fwang12] create kyuubi-shaded module to shade dependencies
6b15b45 [fwang12] shade hive service rpc
b4b3b1a [fwang12] include curator-client
65086ed [fwang12] address comments
a46f00f [fwang12] Shade curator dependencies to avoid impaction by external jars

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-04-09 11:29:24 +08:00
Yikun Jiang
1e23e47b47
[KYUUBI #494] [MINOR] Fix the properties on spark-hadoop-3.2
![Yikun](https://badgen.net/badge/Hello/Yikun/green) [![Closes #494](https://badgen.net/badge/Preview/Closes%20%23494/blue)](https://github.com/yaooqinn/kyuubi/pull/494) ![1](https://badgen.net/badge/%2B/1/red) ![1](https://badgen.net/badge/-/1/green) ![1](https://badgen.net/badge/commits/1/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) ![Bug](https://badgen.net/badge/Label/Bug/) [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

### _Why are the changes needed?_
Change the `spark.hadoop.binary.version` to `hadoop.binary.version`

### _How was this patch tested?_
spark-hadoop-3.2 CI passed

Closes #494 from Yikun/patch-1.

Closes #494

57b7876 [Yikun Jiang] Fix the properties on spark-hadoop-3.2

Authored-by: Yikun Jiang <yikunkero@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-04-09 00:03:03 +08:00
Kent Yao
c6585b4708
[KYUUBI #471] Separate Embedded Zookeeper server from ha
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #471](https://badgen.net/badge/Preview/Closes%20%23471/blue)](https://github.com/yaooqinn/kyuubi/pull/471) ![386](https://badgen.net/badge/%2B/386/red) ![152](https://badgen.net/badge/-/152/green) ![8](https://badgen.net/badge/commits/8/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

1. Separated embedded zk from ha module, as it's only used for setup a zk from Kyuubi server for now if there is not an external one
2. Support more configuration for the embedded zookeeper, for example, HOSTNAME, then it's more close to prod usage.
3. TODO: Make the embedded zookeeper of our own, which should support as quorum peers

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #471 from yaooqinn/qp.

Closes #471

1c24d02 [Kent Yao] address comments
9666a40 [Kent Yao] refine
f069fc0 [Kent Yao] refine
8bb2f3a [Kent Yao] refine
c20593a [Kent Yao] refine
8fa9a40 [Kent Yao] Merge branch 'master' into qp
1e2e1a3 [Kent Yao] tmp
7d66d05 [Kent Yao] tmp

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-30 18:34:01 +08:00
fwang12
0aafc55bba
[KYUUBI #304] Add Kyuubi Ctl arguments parser
![turboFei](https://badgen.net/badge/Hello/turboFei/green) [![Closes #465](https://badgen.net/badge/Preview/Closes%20%23465/blue)](https://github.com/yaooqinn/kyuubi/pull/465) ![919](https://badgen.net/badge/%2B/919/red) ![0](https://badgen.net/badge/-/0/green) ![22](https://badgen.net/badge/commits/22/yellow) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
```
bin/kyuubi-service <create|get|delete|list>  <server|engine> --zkAddress ... --namespace ... --user ... --host ... --port ... --version

Operations:
    - create  -  expose a service to a namespace, this case is rare but sometimes we may want one server to be reached in 2 or more namespaces by different user groups
    - get - get the service node info
    - delete - delete the specified serviceNode
    - list - list all the service nodes for a particular domain
Role:
    - server default
    - engine

Args:

   --zkAddress  - one of the zk ensemble address, using kyuubi-defaults/conf if absent
   --namespace  -  the namespace, using kyuubi-defaults/conf if absent
   --user -
   --host
   --port
   --version
```

### _How was this patch tested?_
UT

Closes #465 from turboFei/KYUUBI_304_CMD.

Closes #304

4bab34b [fwang12] retest pleaes
8083a12 [fwang12] to increase code converage
c7e51a2 [fwang12] complete
7249cd6 [fwang12] add ut
c809c27 [fwang12] enable set verbose at first
bb3cbb6 [fwang12] fix ut
604820d [fwang12] validate for each action
a01ac1f [fwang12] fix scala style issue
76c9b4c [fwang12] save
7139dd5 [fwang12] increase test converage
318ebce [fwang12] add more ut
72978a6 [fwang12] save
2931f93 [fwang12] address comments
10b855d [fwang12] save
420912a [fwang12] treat help as an action
b27d0a6 [fwang12] treat help as action
896e20d [fwang12] add ctl module into codecov
ea43d69 [fwang12] rename kyuubi-service to kyuubi-ctl
65a0e30 [fwang12] save
db718b0 [fwang12] refactor
41b503e [fwang12] Add kyuubi-ctl arguments parser
cb3f6a8 [fwang12] with log appender

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-29 20:43:13 +08:00
fwang12
5684407ce0
[KYUUBI #467] Set operation log root dir under target when running unit tests
![turboFei](https://badgen.net/badge/Hello/turboFei/green) [![Closes #468](https://badgen.net/badge/Preview/Closes%20%23468/blue)](https://github.com/yaooqinn/kyuubi/pull/468) ![46](https://badgen.net/badge/%2B/46/red) ![2](https://badgen.net/badge/-/2/green) ![4](https://badgen.net/badge/commits/4/yellow) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

### _Why are the changes needed?_
When running unit tests, the operation logs should be placed under target.

### _How was this patch tested?_

Closes #468 from turboFei/KYUUBI_467_lic.

Closes #467

90fc2b5 [fwang12] save
19a8e55 [fwang12] Merge branch 'IS_TESTING' into KYUUBI_467_lic
48fd4ee [fwang12] Use config to check whether is testing
124dc28 [fwang12] [KYUUBI #467] Exclude spark sql engine operation log file

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-28 10:16:51 +08:00
Kent Yao
9e7649296d
[KYUUBI #449] Add Basic Metrics Service framework
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #450](https://badgen.net/badge/Preview/Closes%20%23450/blue)](https://github.com/yaooqinn/kyuubi/pull/450) ![847](https://badgen.net/badge/%2B/847/red) ![73](https://badgen.net/badge/-/73/green) ![8](https://badgen.net/badge/commits/8/yellow) [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

This PR adds a future-proofing Metrics Service framework close #499

add some reporters, like json file, jmx, etc..
add some basic metrics, like statement.error

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

WIP
- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #450 from yaooqinn/449.

ea21a43 [Kent Yao] add tests
6513d07 [Kent Yao] update codecov
7cfe318 [Kent Yao] update docs
a8454d0 [Kent Yao] update docs
91c33f4 [Kent Yao] update deps
05b13fb [Kent Yao] fix dep issues
babc739 [Kent Yao] Add Basic Metrics Service Platform
acc716a [Kent Yao] Add Basic Metrics Service Platform

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-24 18:56:56 +08:00
ulysses-you
f67b8ef1f3 [KYUUBI #447] Enhance maven clean
![ulysses-you](https://badgen.net/badge/Hello/ulysses-you/green) [![Closes #447](https://badgen.net/badge/Preview/Closes%20%23447/blue)](https://github.com/yaooqinn/kyuubi/pull/447) ![17](https://badgen.net/badge/%2B/17/red) ![0](https://badgen.net/badge/-/0/green) ![1](https://badgen.net/badge/commits/1/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Cleanup the automatically create directory

### _How was this patch tested?_
manual test `./build/mvn clean`

Closes #447 from ulysses-you/enhance-maven-clean.

d48c8dc [ulysses-you] enhance

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-03-24 09:49:23 +08:00
Cheng Pan
794a75395f
[KYUUBI #441] [BUILD] Downgrade scala-maven-plugin to 4.3.0
![pan3793](https://badgen.net/badge/Hello/pan3793/green) [![Closes #441](https://badgen.net/badge/Preview/Closes%20%23441/blue)](https://github.com/yaooqinn/kyuubi/pull/441) ![2](https://badgen.net/badge/%2B/2/red) ![1](https://badgen.net/badge/-/1/green) ![2](https://badgen.net/badge/commits/2/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
There are several known issues with `scala-maven-plugin:4.4.0` about `zinc`.

On CentOS 7, with Oracle JDK 8 installed via RPM, `/bin/java` links to `jdk/jre/bin/java` but `/bin/javac` links to `jdk/bin/javac`
```
[opsbd-ck-uat-30007-vm ~]$ ll -l /bin/java
lrwxrwxrwx 1 root root 22 Mar 12 14:03 /bin/java -> /etc/alternatives/java
[opsbd-ck-uat-30007-vm ~]$ ll -l /etc/alternatives/java
lrwxrwxrwx 1 root root 41 Mar 12 14:03 /etc/alternatives/java -> /usr/java/jdk1.8.0_251-amd64/jre/bin/java
[opsbd-ck-uat-30007-vm ~]$ ll -l /bin/javac
lrwxrwxrwx 1 root root 23 Mar 12 14:03 /bin/javac -> /etc/alternatives/javac
[opsbd-ck-uat-30007-vm ~]$ ll -l /etc/alternatives/javac
lrwxrwxrwx 1 root root 38 Mar 12 14:03 /etc/alternatives/javac -> /usr/java/jdk1.8.0_251-amd64/bin/javac
```
when run `build/dist --tgz` cause
```
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.4.0:testCompile (scala-test-compile-first) on project kyuubi-common: wrap: java.io.IOException: Cannot run program "/usr/java/jdk1.8.0_221-amd64/jre/bin/javac" (in directory "/data/ansible/ci/kyuubi"): error=2, No such file or directory -> [Help 1]
```
And there is another issue reported by Apache Spark: https://github.com/apache/spark/pull/31031

Please also merge it into branch-1.1

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #441 from pan3793/build.

6f6c34d [Cheng Pan] add comments
312b2c5 [Cheng Pan] [BUILD] Downgrade scala-maven-plugin to 4.3.0

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-16 22:57:59 +08:00
谢波
9c13f73bd9 [KYUUBI #417] [MINOR] Remove all unused import
![MyLanPangzi](https://badgen.net/badge/Hello/MyLanPangzi/green) [![Closes #417](https://badgen.net/badge/Preview/Closes%20%23417/blue)](https://github.com/yaooqinn/kyuubi/pull/417) ![29](https://badgen.net/badge/%2B/29/red) ![27](https://badgen.net/badge/-/27/green) ![7](https://badgen.net/badge/commits/7/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

### _Why are the changes needed?_
remove all unused import and replace deprecated class PersistentEphemeralNode with PersistentNode in ServiceDiscovery

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #417 from MyLanPangzi/master.

821e0d2 [hiscat] Merge branch 'master' into master
db9cd3c [谢波] [MINOR]: revert ServiceDiscovery change and add scala maven plugin compiler args
c56cd2b [谢波] [MINOR]: revert ProcBuilder indent
ad5e24a [谢波] [MINOR]: revert maven.plugin.scalatest.exclude.tags properties
7c23e37 [谢波] [MINOR]: scala maven plugin config move to plugin management
6029b99 [谢波] [MINOR]: remove all unused import
a305567 [谢波] [MINOR]: remove all unused import

Lead-authored-by: 谢波 <xiebo1@yonghui.cn>
Co-authored-by: hiscat <46845236+MyLanPangzi@users.noreply.github.com>
Signed-off-by: Cheng Pan <379377944@qq.com>
2021-03-15 20:05:08 +08:00
Cheng Pan
63817634fb
[KYUUBI #432] [LICENSE] Introduce Apache Rat
![pan3793](https://badgen.net/badge/Hello/pan3793/green) [![Closes #432](https://badgen.net/badge/Preview/Closes%20%23432/blue)](https://github.com/yaooqinn/kyuubi/pull/432) ![84](https://badgen.net/badge/%2B/84/red) ![9](https://badgen.net/badge/-/9/green) ![4](https://badgen.net/badge/commits/4/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #432 from pan3793/rat.

c7cad27 [Cheng Pan] simplify rat rule
24f0688 [Cheng Pan] use rat maven plugin and improve workflow
891ccc1 [Cheng Pan] enable check-license in GH workflow
0bb1c09 [Cheng Pan] [LICENSE] enable apache rat

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-14 00:57:25 +08:00
Kent Yao
5e2a1290c3
Prepare Kyuubi v1.2.0 2021-03-08 19:50:12 +08:00
Kent Yao
183e18f1bc
[KYUUBI #404] How To Use Spark Dynamic Resource Allocation (DRA) in Kyuubi
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #404](https://badgen.net/badge/Preview/Closes%20%23404/blue)](https://github.com/yaooqinn/kyuubi/pull/404) ![518](https://badgen.net/badge/%2B/518/red) ![443](https://badgen.net/badge/-/443/green) ![3](https://badgen.net/badge/commits/3/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #404 from yaooqinn/dra.

b8b755c [Kent Yao] How To Use Spark Dynamic Resource Allocation (DRA) in Kyuubi
9bb47b1 [Kent Yao] How To Use Spark Dynamic Resource Allocation (DRA) in Kyuubi
e786787 [Kent Yao] How To Use Spark Dynamic Resource Allocation (DRA) in Kyuubi

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-08 18:27:49 +08:00
Cheng Pan
562c45865f
[KYUUBI #398][FOLLOWUP] exclude unused jars
![pan3793](https://badgen.net/badge/Hello/pan3793/green) [![Closes #401](https://badgen.net/badge/Preview/Closes%20%23401/blue)](https://github.com/yaooqinn/kyuubi/pull/401) ![47](https://badgen.net/badge/%2B/47/red) ![155](https://badgen.net/badge/-/155/green) ![3](https://badgen.net/badge/commits/3/yellow) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Current kyuubi server dist jars
```
➜  kyuubi git:(KYUUBI-398-followup) ll dist/jars
total 122064
-rw-r--r--  1 chengpan  staff   346K Mar  5 01:00 commons-codec-1.15.jar
-rw-r--r--  1 chengpan  staff   511K Mar  5 01:00 commons-lang3-3.10.jar
-rw-r--r--  1 chengpan  staff   2.3M Mar  5 01:00 curator-client-2.12.0.jar
-rw-r--r--  1 chengpan  staff   197K Mar  5 01:00 curator-framework-2.12.0.jar
-rw-r--r--  1 chengpan  staff   277K Mar  5 01:00 curator-recipes-2.12.0.jar
-rw-r--r--  1 chengpan  staff   2.3M Mar  5 01:00 curator-test-2.12.0.jar
-rw-r--r--  1 chengpan  staff   4.5K Mar  5 01:00 failureaccess-1.0.1.jar
-rw-r--r--  1 chengpan  staff   2.7M Mar  5 01:00 guava-30.1-jre.jar
-rw-r--r--  1 chengpan  staff    17M Mar  5 01:00 hadoop-client-api-3.2.2.jar
-rw-r--r--  1 chengpan  staff    22M Mar  5 01:00 hadoop-client-runtime-3.2.2.jar
-rw-r--r--  1 chengpan  staff   1.5M Mar  5 01:00 hive-service-rpc-2.3.7.jar
-rw-r--r--  1 chengpan  staff   1.4M Mar  5 01:00 htrace-core4-4.1.0-incubating.jar
-rw-r--r--  1 chengpan  staff   697K Mar  5 01:00 javassist-3.18.1-GA.jar
-rw-r--r--  1 chengpan  staff    16K Mar  5 01:00 jcl-over-slf4j-1.7.30.jar
-rw-r--r--  1 chengpan  staff   325K Mar  5 01:00 kyuubi-common-1.1.0-SNAPSHOT.jar
-rw-r--r--  1 chengpan  staff    34K Mar  5 01:00 kyuubi-ha-1.1.0-SNAPSHOT.jar
-rw-r--r--  1 chengpan  staff    81K Mar  5 01:00 kyuubi-main-1.1.0-SNAPSHOT.jar
-rw-r--r--  1 chengpan  staff   306K Mar  5 01:00 libfb303-0.9.3.jar
-rw-r--r--  1 chengpan  staff   229K Mar  5 01:00 libthrift-0.9.3.jar
-rw-r--r--  1 chengpan  staff   478K Mar  5 01:00 log4j-1.2.17.jar
-rw-r--r--  1 chengpan  staff   5.0M Mar  5 01:00 scala-library-2.12.10.jar
-rw-r--r--  1 chengpan  staff    41K Mar  5 01:00 slf4j-api-1.7.30.jar
-rw-r--r--  1 chengpan  staff    12K Mar  5 01:00 slf4j-log4j12-1.7.30.jar
-rw-r--r--  1 chengpan  staff   890K Mar  5 01:00 zookeeper-3.4.14.jar
```

Closes #401 from pan3793/KYUUBI-398-followup.

c96586b [Cheng Pan] reorder deps
207f0b1 [Cheng Pan] restore docs
0f5f2e7 [Cheng Pan] [KYUUBI #398][FOLLOWUP] exclude unused jars

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-05 08:57:23 +08:00
Cheng Pan
4abc3ff6a0
[KYUUBI #398] Move to hadoop shaded client
![pan3793](https://badgen.net/badge/Hello/pan3793/green) [![Closes #399](https://badgen.net/badge/Preview/Closes%20%23399/blue)](https://github.com/yaooqinn/kyuubi/pull/399) ![101](https://badgen.net/badge/%2B/101/red) ![41](https://badgen.net/badge/-/41/green) ![5](https://badgen.net/badge/commits/5/yellow) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
close #398

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

manual test release workflow
```
➜  kyuubi git:(KYUUBI-398) ll | grep tar.gz
-rw-r--r--   1 chengpan  staff   265M Mar  4 21:38 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0-hadoop2.7.tar.gz
-rw-r--r--   1 chengpan  staff   269M Mar  4 21:40 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0-hadoop3.2.tar.gz
-rw-r--r--   1 chengpan  staff   269M Mar  4 21:46 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.1-hadoop2.7.tar.gz
-rw-r--r--   1 chengpan  staff   273M Mar  4 21:44 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.1-hadoop3.2.tar.gz
```

Closes #399 from pan3793/KYUUBI-398.

a9294a1 [Cheng Pan] fix ut
d1d816d [Cheng Pan] fix dist script
2e3bc20 [Cheng Pan] update release workflow and dist script
0428b1b [Cheng Pan] update travis.yml
4a9bc1b [Cheng Pan] [KYUUBI #398] move to hadoop shaded client

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-04 23:33:31 +08:00
Kent Yao
25d2d587b0
[KYUUBI #393] Add Support for Hadoop 3.2
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #393](https://badgen.net/badge/Preview/Closes%20%23393/blue)](https://github.com/yaooqinn/kyuubi/pull/393) ![42](https://badgen.net/badge/%2B/42/red) ![41](https://badgen.net/badge/-/41/green) ![3](https://badgen.net/badge/commits/3/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

This PR Add Support for Hadoop 3.2 Profile to work with spark3.1.1

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #393 from yaooqinn/guava2.

1f56082 [Kent Yao] nit
c8c576c [Kent Yao] nit
ccdfd03 [Kent Yao] Add Support for Hadoop 3.2

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-04 18:25:11 +08:00
Kent Yao
2e3315867b
[KYUUBI #298]Add Support for Apache Spark 3.1.1
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #366](https://badgen.net/badge/Preview/Closes%20%23366/blue)](https://github.com/yaooqinn/kyuubi/pull/366) ![359](https://badgen.net/badge/%2B/359/red) ![58](https://badgen.net/badge/-/58/green) ![17](https://badgen.net/badge/commits/17/yellow) ![Feature](https://badgen.net/badge/Label/Feature/) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Spark 3.1.1 released, it looks more stable than the first minor release of Spark 3.x, and a lot of great features were involved.

In this PR, I add a profile for `spark-3.1.1` support, and temporary we only support `hadoop2.7`

TODOS:
- Add Hadoop 3.2 support with some dependency issues
- Enable Iceberg tests for spark-3.1.1 after Apache iceberg‘s next release
- Using spark-3.1.1 as default
- Restore HiveResult string?

fix #298
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #366 from yaooqinn/spark3.1.

28ddf93 [Kent Yao] ga
37cb49e [Kent Yao] Merge branch 'master' into spark3.1
7b332ec [Kent Yao] typo
b6b33d5 [Kent Yao] set up ga
00005c0 [Kent Yao] set up travis
2c09fe4 [Kent Yao] test fix
68c7edb [Kent Yao] deps
2c65e98 [Kent Yao] deps
236c7e9 [Kent Yao] nit
8c8a783 [Kent Yao] Merge branch 'master' into spark3.1
46c318a [Kent Yao] nit
32f6511 [Kent Yao] Merge branch 'master' into spark3.1
27c836b [Kent Yao] Merge branch 'master' into spark3.1
5574a8e [Kent Yao] ga
d2ec2d5 [Kent Yao] [WIP]Spark3.1
49313cc [Kent Yao] [WIP]Travis
ccb1a97 [Kent Yao] [WIP]Spark3.1

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-04 13:30:05 +08:00
Kent Yao
8e8679e92b
[KYUUBI #392] Remove netty dependency
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #392](https://badgen.net/badge/Preview/Closes%20%23392/blue)](https://github.com/yaooqinn/kyuubi/pull/392) ![0](https://badgen.net/badge/%2B/0/red) ![23](https://badgen.net/badge/-/23/green) ![2](https://badgen.net/badge/commits/2/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

netty is unnecessary

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #392 from yaooqinn/netty.

824d84e [Kent Yao] nit
402317d [Kent Yao] Remove netty dependency

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-03 22:23:38 +08:00
Kent Yao
d10b17d87f
[KYUUBI #391] Bump up guava to 14.0.1
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #391](https://badgen.net/badge/Preview/Closes%20%23391/blue)](https://github.com/yaooqinn/kyuubi/pull/391) ![1](https://badgen.net/badge/%2B/1/red) ![1](https://badgen.net/badge/-/1/green) ![1](https://badgen.net/badge/commits/1/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

align with spark master

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #391 from yaooqinn/guava.

164501b [Kent Yao] Bump up guava to 14.0.1

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-03 22:05:05 +08:00
Kent Yao
f83faa8356 [KYUUBI #389] Bump up curator to 2.12.0
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #389](https://badgen.net/badge/Preview/Closes%20%23389/blue)](https://github.com/yaooqinn/kyuubi/pull/389) ![1](https://badgen.net/badge/%2B/1/red) ![28](https://badgen.net/badge/-/28/green) ![1](https://badgen.net/badge/commits/1/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

2.12.0 is the last update for curator-test to support zk v3.4.x

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #389 from yaooqinn/curator.

0b12323 [Kent Yao] Bump up curator to 2.12.0

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Cheng Pan <379377944@qq.com>
2021-03-03 19:37:59 +08:00
Cheng Pan
2f8df2355d [KYUUBI #377] build/dist support --spark-provided
![pan3793](https://badgen.net/badge/Hello/pan3793/green) [![Closes #378](https://badgen.net/badge/Preview/Closes%20%23378/blue)](https://github.com/yaooqinn/kyuubi/pull/378) ![55](https://badgen.net/badge/%2B/55/red) ![13](https://badgen.net/badge/-/13/green) ![3](https://badgen.net/badge/commits/3/yellow) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
close #377

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

- [x] Manual test

`build/dist --spark-provided`

```
(kyuubi) ➜  kyuubi git:(master) ✗ tree dist
dist
├── LICENSE
├── RELEASE
├── bin
│   ├── kyuubi
│   ├── kyuubi-logo
│   └── load-kyuubi-env.sh
├── conf
│   ├── kyuubi-defaults.conf.template
│   ├── kyuubi-env.sh.template
│   └── log4j.properties.template
├── externals
│   └── engines
│       └── spark
│           └── kyuubi-spark-sql-engine-1.1.0-SNAPSHOT.jar
├── jars
│   ├── apacheds-i18n-2.0.0-M15.jar
│   ├── apacheds-kerberos-codec-2.0.0-M15.jar
│   ├── api-asn1-api-1.0.0-M20.jar
│   ├── api-util-1.0.0-M20.jar
│   ├── audience-annotations-0.5.0.jar
│   ├── commons-beanutils-1.7.0.jar
│   ├── commons-beanutils-core-1.8.0.jar
│   ├── commons-codec-1.4.jar
│   ├── commons-collections-3.2.2.jar
│   ├── commons-configuration-1.6.jar
│   ├── commons-digester-1.8.jar
│   ├── commons-io-2.4.jar
│   ├── commons-lang-2.6.jar
│   ├── commons-lang3-3.10.jar
│   ├── commons-math-2.2.jar
│   ├── curator-client-2.7.1.jar
│   ├── curator-framework-2.7.1.jar
│   ├── curator-recipes-2.7.1.jar
│   ├── curator-test-2.7.1.jar
│   ├── guava-11.0.2.jar
│   ├── hadoop-annotations-2.7.4.jar
│   ├── hadoop-auth-2.7.4.jar
│   ├── hadoop-common-2.7.4.jar
│   ├── hive-service-rpc-2.3.7.jar
│   ├── httpclient-4.5.6.jar
│   ├── httpcore-4.4.12.jar
│   ├── javassist-3.18.1-GA.jar
│   ├── jcl-over-slf4j-1.7.30.jar
│   ├── jsr305-3.0.2.jar
│   ├── kyuubi-common-1.1.0-SNAPSHOT.jar
│   ├── kyuubi-ha-1.1.0-SNAPSHOT.jar
│   ├── kyuubi-main-1.1.0-SNAPSHOT.jar
│   ├── libfb303-0.9.3.jar
│   ├── libthrift-0.9.3.jar
│   ├── log4j-1.2.17.jar
│   ├── netty-3.7.0.Final.jar
│   ├── scala-library-2.12.10.jar
│   ├── slf4j-api-1.7.30.jar
│   ├── slf4j-log4j12-1.7.30.jar
│   ├── spotbugs-annotations-3.1.9.jar
│   └── zookeeper-3.4.14.jar
├── logs
├── pid
└── work
```

Bundle size
```
(kyuubi) ➜  kyuubi git:(KYUUBI-377) ll | grep tar.gz
-rw-r--r--   1 chengpan  staff   230M Feb 26 12:59 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0.2.tar.gz
-rw-r--r--   1 chengpan  staff    20M Feb 26 12:56 kyuubi-1.1.0-SNAPSHOT-bin-without-spark.tar.gz
```

Closes #378 from pan3793/KYUUBI-377.

442569e [Cheng Pan] add profile spark-provided for skip download spark binary
0d8da24 [Cheng Pan] update doc
5b7c41f [Cheng Pan] [KYUUBI #377] build/dist support --spark-provided

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-02-26 15:00:42 +08:00
Kent Yao
968ed4f079
[KYUUBI #376] Use spark-hive instead of spark-hive-thriftserver in engine test
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #376](https://badgen.net/badge/Preview/Closes%20%23376/blue)](https://github.com/yaooqinn/kyuubi/pull/376) ![23](https://badgen.net/badge/%2B/23/red) ![6](https://badgen.net/badge/-/6/green) ![1](https://badgen.net/badge/commits/1/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

get rid of compile error when upgrading spark versions. instead, we shall add some e2e tests for it

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #376 from yaooqinn/hive.

26da066 [Kent Yao] Use spark-hive instead of spark-hive-thriftserver in engine test

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-26 00:28:17 +08:00
Kent Yao
d94b1c4519
[KYUUBI #369] Using a lightweight zk client for kyuubi session
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #369](https://badgen.net/badge/Preview/Closes%20%23369/blue)](https://github.com/yaooqinn/kyuubi/pull/369) ![60](https://badgen.net/badge/%2B/60/red) ![34](https://badgen.net/badge/-/34/green) ![3](https://badgen.net/badge/commits/3/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

1. limit zkClient to method wide
2. a more lightweight retry policy
3. add some trace log for Kyuubi session
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #369 from yaooqinn/zk.

5cbe14f [Kent Yao] typo
731bae8 [Kent Yao] typo
51365da [Kent Yao] Using a lightweight zk client for kyuubi session

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-25 16:57:55 +08:00
Cheng Pan
399b30bb95
[KYUUBI #363] exclude hive-shims unused dependencies
![pan3793](https://badgen.net/badge/Hello/pan3793/green) [![Closes #364](https://badgen.net/badge/Preview/Closes%20%23364/blue)](https://github.com/yaooqinn/kyuubi/pull/364) ![48](https://badgen.net/badge/%2B/48/red) ![0](https://badgen.net/badge/-/0/green) ![1](https://badgen.net/badge/commits/1/yellow) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Fix #363

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #364 from pan3793/KYUUBI-363.

9b2c8c7 [Cheng Pan] [KYUUBI #363] exclude hive-shims unused dependencies

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-24 11:38:31 +08:00
ulysses-you
bacff6c59a [KYUUBI #356] Update built-in Spark version to 3.0.2
![ulysses-you](https://badgen.net/badge/Hello/ulysses-you/green) [![Closes #356](https://badgen.net/badge/Preview/Closes%20%23356/blue)](https://github.com/yaooqinn/kyuubi/pull/356) ![11](https://badgen.net/badge/%2B/11/red) ![3](https://badgen.net/badge/-/3/green) ![2](https://badgen.net/badge/commits/2/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Spark 3.0.2 has been released now.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #356 from ulysses-you/spark-3.0.2.

c791e19 [ulysses-you] fix
ad6c564 [ulysses-you] 3.0.2

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-02-20 18:26:26 +08:00
Cheng Pan
aed27d7a65
[KYUUBI #342] Exclude unused hive-jdbc transitive dependencies
fix #343

![pan3793](https://badgen.net/badge/Hello/pan3793/green) [![Closes #343](https://badgen.net/badge/Preview/Closes%20%23343/blue)](https://github.com/yaooqinn/kyuubi/pull/343) ![278](https://badgen.net/badge/%2B/278/red) ![17](https://badgen.net/badge/-/17/green) ![5](https://badgen.net/badge/commits/5/yellow) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
close #342

- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
Squashed commit of the following:

commit c1b2eeae02dbca56ddf44d140ccc5f558927b618
Author: Cheng Pan <379377944@qq.com>
Date:   Wed Feb 3 10:54:27 2021 +0800

    remove unused hive-cli, hive-beeline

commit 9f1a9de05ade4d155b44d817f6a36edcf4f27f03
Author: Cheng Pan <379377944@qq.com>
Date:   Wed Feb 3 10:43:07 2021 +0800

    tune pom

commit 5d63596abc91412b8f60589deb46e0f92e13cc00
Author: Cheng Pan <379377944@qq.com>
Date:   Wed Feb 3 00:25:40 2021 +0800

    update pom.xml

commit 70cb27a352db38578eb2b4857c2d96afd9fc4204
Author: Cheng Pan <379377944@qq.com>
Date:   Tue Feb 2 22:12:29 2021 +0800

    remove unused hive deps in kyuubi-common

commit ba8038c8981ae4dd9a597259925a38276fdaa567
Author: Cheng Pan <379377944@qq.com>
Date:   Tue Feb 2 21:52:47 2021 +0800

    [KYUUBI #342] Exclude unused hive-jdbc transitive dependencies
2021-02-03 12:50:16 +08:00
Cheng Pan
2dc580456a
[KYUUBI #340] Swtich to commons-lang3
![pan3793](https://badgen.net/badge/Hello/pan3793/green) [![Closes #341](https://badgen.net/badge/Preview/Closes%20%23341/blue)](https://github.com/yaooqinn/kyuubi/pull/341) ![15](https://badgen.net/badge/%2B/15/red) ![3](https://badgen.net/badge/-/3/green) ![1](https://badgen.net/badge/commits/1/yellow) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
close #340

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #341 from pan3793/KYUUBI-340.

e014fe9 [Cheng Pan] swtich to commons-lang3

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-02 20:47:45 +08:00
Kent Yao
efdd9fafc8
[KYUUBI #307]GetCatalogs supports DSv2 and keeps its backward compatibility (#307)
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![PR 307](https://badgen.net/badge/Preview/PR%20307/blue)](https://github.com/yaooqinn/kyuubi/pull/307) ![Feature](https://badgen.net/badge/Label/Feature/) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!  Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
-->

### Please add issue ID here?
<!-- replace ${issue ID} with the actual issue id -->
Fixes #307

### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the user case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

GetCatalogs supports DSv2 and keeps its backward compatibility

### Test Plan:
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
![image](https://user-images.githubusercontent.com/8326978/106161043-7259c400-61c1-11eb-9beb-3326f6093284.png)

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

* GetCatalogs supports DSv2

* pr template

* nit

* nit

* shim

* add iceberg tests

* nit
2021-01-29 10:27:48 +08:00
Kent Yao
eadca7589c
[KYUUBI #296] Add spark 2.4 profile and shade hive-sevice-rpc to engine jar to fix thrift build err (#296)
* add spark 2.4.7 profile

* nit

* nit

* scala versions for profile spark-2.4
2021-01-18 14:23:31 +08:00
Cheng Pan
50f86b3659
[KYUUBI#282] Downgrade libthrift:0.9.3 due to THRIFT-4805
fixes #282
Squashed commit of the following:

commit 6b6f75ce5d0171fc58472d21876ad5fd2a571aa0
Author: Cheng Pan <379377944@qq.com>
Date:   Thu Jan 14 00:25:38 2021 +0800

    downgrade libthrift:0.9.3 due to THRIFT-4805
2021-01-14 16:56:44 +08:00
fwang12
bf793f4b7a
Remove unused setting in pom
fixes #281
Squashed commit of the following:

commit 209729719e0f18b94377de8c126cc94cb14d1839
Author: fwang12 <fwang12@ebay.com>
Date:   Wed Jan 13 20:56:39 2021 +0800

    remove also

commit afdd72b7d39acfb286769a0f991d3437786f4341
Author: fwang12 <fwang12@ebay.com>
Date:   Wed Jan 13 20:31:47 2021 +0800

    Remove unused setting in pom
2021-01-13 21:11:34 +08:00
Cheng Pan
24b8b22e33
exclude hbase-* from hive-jdbc
fixes #275
Squashed commit of the following:

commit 6429ef117150087e8232ea73eb5b4a3668a1f017
Author: Cheng Pan <379377944@qq.com>
Date:   Sun Jan 10 00:53:41 2021 +0800

    exclude hbase-* from hive-jdbc
2021-01-10 01:21:41 +08:00
Kent Yao
61a2da4943
Merge pull request #272 from yaooqinn/dependabot/maven/org.apache.zookeeper-zookeeper-3.4.14
Bump zookeeper from 3.4.6 to 3.4.14
2021-01-10 00:23:17 +08:00
dependabot[bot]
4d1d3a49f8
Bump libthrift from 0.9.3 to 0.12.0
fixes #273

Squashed commit of the following:

commit b6a8d787ede228246af7b9d142f2edbd08e07eac
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Sat Jan 9 15:34:57 2021 +0000

    Bump libthrift from 0.9.3 to 0.12.0

    Bumps [libthrift](https://github.com/apache/thrift) from 0.9.3 to 0.12.0.
    - [Release notes](https://github.com/apache/thrift/releases)
    - [Changelog](https://github.com/apache/thrift/blob/master/CHANGES.md)
    - [Commits](https://github.com/apache/thrift/compare/0.9.3...v0.12.0)

    Signed-off-by: dependabot[bot] <support@github.com>
2021-01-10 00:15:08 +08:00
dependabot[bot]
da2b22eedb
Bump zookeeper from 3.4.6 to 3.4.14
Bumps zookeeper from 3.4.6 to 3.4.14.

Signed-off-by: dependabot[bot] <support@github.com>
2021-01-09 15:34:54 +00:00
Cheng Pan
0eeb99774f
Tune pom
fixes #271
Squashed commit of the following:

commit fe0f469b2068865398d6e5be1957fdf6c6f5eb87
Author: Cheng Pan <379377944@qq.com>
Date:   Sat Jan 9 22:28:43 2021 +0800

    followup

commit 527e32419ecb2ae584d1251cb345e3fc870965a9
Author: Cheng Pan <379377944@qq.com>
Date:   Sat Jan 9 22:21:29 2021 +0800

    revert some stuffs

commit 82f188972b8491c5d381ba076a53f84a2fbc1898
Author: Cheng Pan <379377944@qq.com>
Date:   Sat Jan 9 16:49:14 2021 +0800

    remove unused dependencies

commit c7eb27228d2749ab9100102467e94d3a6a98cf44
Author: Cheng Pan <379377944@qq.com>
Date:   Sat Jan 9 15:41:59 2021 +0800

    minimize dependencies in kyuubi-spark-sql-engine

commit da902b1b1e02c7e93810ca832f6c9e7109cf95b9
Author: Cheng Pan <379377944@qq.com>
Date:   Sat Jan 9 13:26:34 2021 +0800

    exclude hive dependencies

commit 55ae7f025a5ffcec402919e28b7b3af5987dfe16
Author: Cheng Pan <379377944@qq.com>
Date:   Sat Jan 9 01:08:09 2021 +0800

    pin versions and excludes in <dependencyManagement>

commit 9ef84a5f62afd66fb4b817d593ba1b13ff8348ed
Author: Cheng Pan <379377944@qq.com>
Date:   Sat Jan 9 14:12:04 2021 +0800

    trigger kyuubi-spark-sql-engine package

commit d492e3fdce64e6554fe0db1d810c6f4bea198baa
Author: Cheng Pan <379377944@qq.com>
Date:   Sat Jan 9 13:44:13 2021 +0800

    trigger kyuubi-download

commit 050281b8fc400f3ae5fa1d43d56da49846664ea4
Author: Cheng Pan <379377944@qq.com>
Date:   Sat Jan 9 15:30:58 2021 +0800

    pin modules version in dependencyManagement

commit 1655fc453ded4626ae5e12e7b17af6f10c30a0dd
Author: Cheng Pan <379377944@qq.com>
Date:   Fri Jan 8 19:45:27 2021 +0800

    tune modules in alphabetical order

commit 7f3f4d987336d71d73eab059e93f3d008cfcd1a6
Author: Cheng Pan <379377944@qq.com>
Date:   Sat Jan 9 00:09:49 2021 +0800

    change dependencies scope to test
2021-01-09 23:30:46 +08:00
Cheng Pan
90c120759e
Improve pom.xml
fixes #268
Squashed commit of the following:

commit d40d0831d578af23559ea2a2544ceebc47f6df50
Author: Cheng Pan <379377944@qq.com>
Date:   Fri Jan 8 18:30:51 2021 +0800

    remove unused mockito

commit 386e7e4eb1c93006e302c41ffbc04d0acfaf9a9b
Author: Cheng Pan <379377944@qq.com>
Date:   Fri Jan 8 17:35:44 2021 +0800

    extract versions to properties
2021-01-08 19:04:48 +08:00
Kent Yao
ca1b452e9a
Prepare Kyuubi v1.1.0 2020-12-31 15:51:12 +08:00
Kent Yao
bb2b3038e1
Improve Session Share Level 2020-12-27 02:09:57 +08:00
Kent Yao
a8b70d0841
Add a one-click tpcds generating tool (#262)
* Add a one click tpcds generating tool

* nit
2020-12-12 03:04:42 +08:00
Kent Yao
87e243c596 support query log for async exec 2020-11-27 16:40:07 +08:00
Kent Yao
9bd4139453 Remove Kyuubi useless deps 2020-11-16 20:52:45 +08:00
Kent Yao
91f622f94f Remove Kyuubi 0.8 and before 2020-11-16 20:12:08 +08:00
Kent Yao
7157370ef6 Add Building Doc 2020-11-05 19:05:26 +08:00
Kent Yao
53205a43cc Revert "Revert "fix""
This reverts commit 9f4ccdd70c.
2020-09-23 05:08:26 +08:00
Kent Yao
9f4ccdd70c Revert "fix"
This reverts commit d97f71456b.
2020-09-22 18:33:11 +08:00