Commit Graph

34 Commits

Author SHA1 Message Date
fwang12
119e7f2c2c [KYUUBI #1441] [BUILD] Bump 1.5.0-SNAPSHOT
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1441 from turboFei/bump_1.5.0.

Closes #1441

0c03c75d [fwang12] [BUILD] Bump 1.5.0-SNAPSHOT

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
2021-11-23 22:22:26 +08:00
sunfangbin
e54fabab68
[KYUUBI #1416] Refine jline dependency
### _Why are the changes needed?_
Follows on #1413.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1416 from murong00/branch-1415.

Closes #1416

05335c45 [sunfangbin] Refine jline dependency

Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-11-19 10:40:40 +08:00
sunfangbin
fb8bf49dbc
[KYUUBI #1413] Improve usage of zookeeper cli
### _Why are the changes needed?_
This change enabled the JLine support so that the `kyuubi-zookeeper-shell` command can be easier to use with history, prompt and completion, etc. Note that the `jline` version `0.9.94` is dependent on `zookeeper` version `3.4.14` which is a lower version. BTW, users can consider upgrading the `zookeeper` version for the sake of performance improvements and new features.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1413 from murong00/branch-1413.

Closes #1413

2d1ec084 [sunfangbin] Rename shell script to zk-cli
d09f64dc [sunfangbin] Fixes failed ci
ce88357a [sunfangbin] Improve usage of zookeeper cli

Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-11-18 19:17:11 +08:00
Cheng Pan
98f38ba517
[BUILD] Bump 1.4.0-SNAPSHOT 2021-08-17 01:39:06 +08:00
timothy65535
5526898f7b [KYUUBI #894] Enhance kyuubi-zookeeper module
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### 1. Describe the feature

Kyuubi use `TestingServer` as the EmbeddedZookeeper service in codebase, and require many args to init the service. These `electionPort`, `quorumPort`, `serverId` parameters only work in cluster mode, these ports are not bound to the host.

From hadoop and bookkeeper project, they use `ZooKeeperServer` as a built-in zk service, especially the bookeeper project.
- https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/server/services/MicroZookeeperService.java
- https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/shims/zk/ZooKeeperServerShimImpl.java

### 2. Describe the solution

Improve
- Remove `electionPort`, `quorumPort`, `serverId` which are nerver be used, hadoop and bookkeer do the same
- Replace `TestingServer` with `ZooKeeperServer`
- Remove `curator-test` dependency

### 3. `deleteDataDirectoryOnClose`

The name style for `deleteDataDirectoryOnClose` option, refer to hadoop `core-default.xml`
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/core-default.xml
```
hadoop.security.dns.log-slow-lookups.enabled
hadoop.security.dns.log-slow-lookups.threshold.ms
dfs.client.block.write.replace-datanode-on-failure.policy
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #907 from timothy65535/ky-894.

Closes #894

7d7fde9f [timothy65535] remove useless config
ce06a0fd [timothy65535] limit travis build only master branch
ff825c3f [timothy65535] revert deleteDataDirectoryOnClose config
a336544a [timothy65535] remove javassist
3da0bd55 [timothy65535] remove javassist
0abb44df [timothy65535] [KYUUBI #894] Enhance kyuubi-zookeeper module

Authored-by: timothy65535 <timothy65535@163.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-08-11 10:06:44 +08:00
Cheng Pan
4fecf13902 [KYUUBI #888] Add scala version suffix on scala modules
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
- Add scala version suffix on scala modules to match the name pattern.
- Enable `kyuubi-extension-spark-3-1` in daily publish snapshot

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #888 from pan3793/scala.

Closes #888

571eb9c3 [Cheng Pan] comments
16cb263a [Cheng Pan] remove hardcode scala version in pom
6090c798 [Cheng Pan] Fix shade
5178847f [Cheng Pan] Fixup
a1547753 [Cheng Pan] Fix engine jar name
2cf4793e [Cheng Pan] fix
cac96531 [Cheng Pan] Enable kyuubi-extension-spark-3-1 in daily publish snapshot
1ca31de5 [Cheng Pan] Add scala version suffix on scala modules

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-08-04 09:31:19 +08:00
Cheng Pan
962d721842
[KYUUBI #887] [GA] Daily publish snapshot
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #887 from pan3793/publish-snapshot.

Closes #887

5b2044a1 [Cheng Pan] Pin ubuntu-20.04 in GA
98945a12 [Cheng Pan] Remove Kyuubi local cache before deploy
f29cb6e4 [Cheng Pan] [GA] Daily publish snapshot

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-08-03 13:08:42 +08:00
ulysses-you
f4fb822743
[KYUUBI #871] Change module name kyuubi-main to kyuubi-server
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
As we will release soon, it's better to format the module name and artifactId.

The name of kyuubi-main sounds like a bootstrap of Kyuubi but not a server.

### _How was this patch tested?_
Pass CI

Closes #872 from ulysses-you/kyuubi-server.

Closes #871

1437f9b5 [ulysses-you] address comment
117f2c7c [ulysses-you] server
86986baa [ulysses-you] master
eba660a2 [ulysses-you] Merge branch 'master' of https://github.com/NetEase/kyuubi into kyuubi-server
e649012b [ulysses-you] change module name

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-07-29 13:41:43 +08:00
Cheng Pan
ce5fb97036
[KYUUBI #849] ASF Nexus SNAPSHOT publish
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
https://infra.apache.org/publishing-maven-artifacts.html
https://issues.apache.org/jira/browse/INFRA-22114

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #849 from pan3793/asf.

Closes #849

c609f6ba [Cheng Pan] update pom
31f1f2e1 [Cheng Pan] rename root project name to kyuubi-parent
9c5616d8 [Cheng Pan] [INFRA] Update POM for Apache release

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-07-23 13:15:05 +08:00
Cheng Pan
98e44636fd
[KYUUBI #823] [BUILD] [DEPS] Leverage maven to resolve transitive dependencies except hive
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #823 from pan3793/deps.

Closes #823

51cdcb5d [Cheng Pan] restore thrift deps from spark-hive
148038a3 [Cheng Pan] [BUILD] [DEPS] Leverage maven to resolve transitive dependencies except hive

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-07-18 21:00:38 +08:00
ulysses-you
7f040fb2c2
[KYUUBI #634] Move kyuubi extension to its own dir when building
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Make Kyuubi jars clear.

### _How was this patch tested?_
Manual test.  After this PR, the packaged dir looks like:
![image](https://user-images.githubusercontent.com/12025282/118102872-e3895400-b40b-11eb-994c-26d28b45e9a8.png)

Closes #634 from ulysses-you/extension-dir.

Closes #634

2c5009a [ulysses-you] comment
35063ea [ulysses-you] Merge branch 'master' of https://github.com/NetEase/kyuubi into extension-dir
f2b4fff [ulysses-you] init

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-05-13 18:32:55 +08:00
ulysses-you
43f40dcb8a
[KYUUBI #631] Add kyuubi-spark-extensions module
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
The added sql module structure looks like:
```
kyuubi
  |
  - dev
      |
      - kyuubi-extension-spark_3.1
```

This PR mainly add 3 feature:
* merging small files automatically (include dynamic partition insertion case)
* insert shuffle node before Join to make AQE `OptimizeSkewedJoin` work
* stage level config isolation in AQE

Note that,  the sql rule  depend on the Apache Spark interface so we need make the sql module verion independence. Currently, this PR only supports the Spark 3.1.1.

Due to the version issue, we need to check and deploy this extension manually currently.

### _How was this patch tested?_
Add new test.

Closes #631 from ulysses-you/add-sql-module.

Closes #631

2cf12f1 [ulysses-you] version
cfbf72c [ulysses-you] address comment
7740ca6 [ulysses-you] module name
0f723eb [ulysses-you] workflow
45c23d8 [ulysses-you] line
80378f5 [ulysses-you] assembly
95528aa [ulysses-you] move module
5fe5d87 [ulysses-you] license
6578440 [ulysses-you] init work

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-05-13 16:03:32 +08:00
Kent Yao
bb9e8a50e5
Prepare Kyuubi v1.3.0 2021-05-13 13:58:20 +08:00
Cheng Pan
b7ac3cb48c Revert "[KYUUBI #488] Add kyuubi-shaded module to avoid conflicts with external jars"
This reverts commit e058b87dd2.
2021-04-10 00:16:04 +08:00
fwang12
e058b87dd2
[KYUUBI #488] Add kyuubi-shaded module to avoid conflicts with external jars
![turboFei](https://badgen.net/badge/Hello/turboFei/green) [![Closes #488](https://badgen.net/badge/Preview/Closes%20%23488/blue)](https://github.com/yaooqinn/kyuubi/pull/488) ![454](https://badgen.net/badge/%2B/454/red) ![289](https://badgen.net/badge/-/289/green) ![10](https://badgen.net/badge/commits/10/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) ![Feature](https://badgen.net/badge/Label/Feature/) [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
Shade curator dependency in kyuubi-ha module to avoid impaction by external jars.

### _How was this patch tested?_
Existing UT.

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #488 from turboFei/shade_curator.

Closes #488

c88414c [fwang12] Use wildcards when selecting more than 6 elements
a2dfbf6 [fwang12] add doc for install shaded jars
224cc38 [fwang12] remove transactions
c7b9df3 [fwang12] address comments
81fc123 [fwang12] scala style
408c978 [fwang12] create kyuubi-shaded module to shade dependencies
6b15b45 [fwang12] shade hive service rpc
b4b3b1a [fwang12] include curator-client
65086ed [fwang12] address comments
a46f00f [fwang12] Shade curator dependencies to avoid impaction by external jars

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-04-09 11:29:24 +08:00
fwang12
0aafc55bba
[KYUUBI #304] Add Kyuubi Ctl arguments parser
![turboFei](https://badgen.net/badge/Hello/turboFei/green) [![Closes #465](https://badgen.net/badge/Preview/Closes%20%23465/blue)](https://github.com/yaooqinn/kyuubi/pull/465) ![919](https://badgen.net/badge/%2B/919/red) ![0](https://badgen.net/badge/-/0/green) ![22](https://badgen.net/badge/commits/22/yellow) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
```
bin/kyuubi-service <create|get|delete|list>  <server|engine> --zkAddress ... --namespace ... --user ... --host ... --port ... --version

Operations:
    - create  -  expose a service to a namespace, this case is rare but sometimes we may want one server to be reached in 2 or more namespaces by different user groups
    - get - get the service node info
    - delete - delete the specified serviceNode
    - list - list all the service nodes for a particular domain
Role:
    - server default
    - engine

Args:

   --zkAddress  - one of the zk ensemble address, using kyuubi-defaults/conf if absent
   --namespace  -  the namespace, using kyuubi-defaults/conf if absent
   --user -
   --host
   --port
   --version
```

### _How was this patch tested?_
UT

Closes #465 from turboFei/KYUUBI_304_CMD.

Closes #304

4bab34b [fwang12] retest pleaes
8083a12 [fwang12] to increase code converage
c7e51a2 [fwang12] complete
7249cd6 [fwang12] add ut
c809c27 [fwang12] enable set verbose at first
bb3cbb6 [fwang12] fix ut
604820d [fwang12] validate for each action
a01ac1f [fwang12] fix scala style issue
76c9b4c [fwang12] save
7139dd5 [fwang12] increase test converage
318ebce [fwang12] add more ut
72978a6 [fwang12] save
2931f93 [fwang12] address comments
10b855d [fwang12] save
420912a [fwang12] treat help as an action
b27d0a6 [fwang12] treat help as action
896e20d [fwang12] add ctl module into codecov
ea43d69 [fwang12] rename kyuubi-service to kyuubi-ctl
65a0e30 [fwang12] save
db718b0 [fwang12] refactor
41b503e [fwang12] Add kyuubi-ctl arguments parser
cb3f6a8 [fwang12] with log appender

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-29 20:43:13 +08:00
Kent Yao
9e7649296d
[KYUUBI #449] Add Basic Metrics Service framework
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #450](https://badgen.net/badge/Preview/Closes%20%23450/blue)](https://github.com/yaooqinn/kyuubi/pull/450) ![847](https://badgen.net/badge/%2B/847/red) ![73](https://badgen.net/badge/-/73/green) ![8](https://badgen.net/badge/commits/8/yellow) [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

This PR adds a future-proofing Metrics Service framework close #499

add some reporters, like json file, jmx, etc..
add some basic metrics, like statement.error

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

WIP
- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #450 from yaooqinn/449.

ea21a43 [Kent Yao] add tests
6513d07 [Kent Yao] update codecov
7cfe318 [Kent Yao] update docs
a8454d0 [Kent Yao] update docs
91c33f4 [Kent Yao] update deps
05b13fb [Kent Yao] fix dep issues
babc739 [Kent Yao] Add Basic Metrics Service Platform
acc716a [Kent Yao] Add Basic Metrics Service Platform

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-24 18:56:56 +08:00
Kent Yao
5e2a1290c3
Prepare Kyuubi v1.2.0 2021-03-08 19:50:12 +08:00
Cheng Pan
562c45865f
[KYUUBI #398][FOLLOWUP] exclude unused jars
![pan3793](https://badgen.net/badge/Hello/pan3793/green) [![Closes #401](https://badgen.net/badge/Preview/Closes%20%23401/blue)](https://github.com/yaooqinn/kyuubi/pull/401) ![47](https://badgen.net/badge/%2B/47/red) ![155](https://badgen.net/badge/-/155/green) ![3](https://badgen.net/badge/commits/3/yellow) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Current kyuubi server dist jars
```
➜  kyuubi git:(KYUUBI-398-followup) ll dist/jars
total 122064
-rw-r--r--  1 chengpan  staff   346K Mar  5 01:00 commons-codec-1.15.jar
-rw-r--r--  1 chengpan  staff   511K Mar  5 01:00 commons-lang3-3.10.jar
-rw-r--r--  1 chengpan  staff   2.3M Mar  5 01:00 curator-client-2.12.0.jar
-rw-r--r--  1 chengpan  staff   197K Mar  5 01:00 curator-framework-2.12.0.jar
-rw-r--r--  1 chengpan  staff   277K Mar  5 01:00 curator-recipes-2.12.0.jar
-rw-r--r--  1 chengpan  staff   2.3M Mar  5 01:00 curator-test-2.12.0.jar
-rw-r--r--  1 chengpan  staff   4.5K Mar  5 01:00 failureaccess-1.0.1.jar
-rw-r--r--  1 chengpan  staff   2.7M Mar  5 01:00 guava-30.1-jre.jar
-rw-r--r--  1 chengpan  staff    17M Mar  5 01:00 hadoop-client-api-3.2.2.jar
-rw-r--r--  1 chengpan  staff    22M Mar  5 01:00 hadoop-client-runtime-3.2.2.jar
-rw-r--r--  1 chengpan  staff   1.5M Mar  5 01:00 hive-service-rpc-2.3.7.jar
-rw-r--r--  1 chengpan  staff   1.4M Mar  5 01:00 htrace-core4-4.1.0-incubating.jar
-rw-r--r--  1 chengpan  staff   697K Mar  5 01:00 javassist-3.18.1-GA.jar
-rw-r--r--  1 chengpan  staff    16K Mar  5 01:00 jcl-over-slf4j-1.7.30.jar
-rw-r--r--  1 chengpan  staff   325K Mar  5 01:00 kyuubi-common-1.1.0-SNAPSHOT.jar
-rw-r--r--  1 chengpan  staff    34K Mar  5 01:00 kyuubi-ha-1.1.0-SNAPSHOT.jar
-rw-r--r--  1 chengpan  staff    81K Mar  5 01:00 kyuubi-main-1.1.0-SNAPSHOT.jar
-rw-r--r--  1 chengpan  staff   306K Mar  5 01:00 libfb303-0.9.3.jar
-rw-r--r--  1 chengpan  staff   229K Mar  5 01:00 libthrift-0.9.3.jar
-rw-r--r--  1 chengpan  staff   478K Mar  5 01:00 log4j-1.2.17.jar
-rw-r--r--  1 chengpan  staff   5.0M Mar  5 01:00 scala-library-2.12.10.jar
-rw-r--r--  1 chengpan  staff    41K Mar  5 01:00 slf4j-api-1.7.30.jar
-rw-r--r--  1 chengpan  staff    12K Mar  5 01:00 slf4j-log4j12-1.7.30.jar
-rw-r--r--  1 chengpan  staff   890K Mar  5 01:00 zookeeper-3.4.14.jar
```

Closes #401 from pan3793/KYUUBI-398-followup.

c96586b [Cheng Pan] reorder deps
207f0b1 [Cheng Pan] restore docs
0f5f2e7 [Cheng Pan] [KYUUBI #398][FOLLOWUP] exclude unused jars

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-05 08:57:23 +08:00
Cheng Pan
4abc3ff6a0
[KYUUBI #398] Move to hadoop shaded client
![pan3793](https://badgen.net/badge/Hello/pan3793/green) [![Closes #399](https://badgen.net/badge/Preview/Closes%20%23399/blue)](https://github.com/yaooqinn/kyuubi/pull/399) ![101](https://badgen.net/badge/%2B/101/red) ![41](https://badgen.net/badge/-/41/green) ![5](https://badgen.net/badge/commits/5/yellow) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
close #398

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

manual test release workflow
```
➜  kyuubi git:(KYUUBI-398) ll | grep tar.gz
-rw-r--r--   1 chengpan  staff   265M Mar  4 21:38 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0-hadoop2.7.tar.gz
-rw-r--r--   1 chengpan  staff   269M Mar  4 21:40 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.0-hadoop3.2.tar.gz
-rw-r--r--   1 chengpan  staff   269M Mar  4 21:46 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.1-hadoop2.7.tar.gz
-rw-r--r--   1 chengpan  staff   273M Mar  4 21:44 kyuubi-1.1.0-SNAPSHOT-bin-spark-3.1-hadoop3.2.tar.gz
```

Closes #399 from pan3793/KYUUBI-398.

a9294a1 [Cheng Pan] fix ut
d1d816d [Cheng Pan] fix dist script
2e3bc20 [Cheng Pan] update release workflow and dist script
0428b1b [Cheng Pan] update travis.yml
4a9bc1b [Cheng Pan] [KYUUBI #398] move to hadoop shaded client

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-04 23:33:31 +08:00
Kent Yao
8e8679e92b
[KYUUBI #392] Remove netty dependency
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #392](https://badgen.net/badge/Preview/Closes%20%23392/blue)](https://github.com/yaooqinn/kyuubi/pull/392) ![0](https://badgen.net/badge/%2B/0/red) ![23](https://badgen.net/badge/-/23/green) ![2](https://badgen.net/badge/commits/2/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

netty is unnecessary

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #392 from yaooqinn/netty.

824d84e [Kent Yao] nit
402317d [Kent Yao] Remove netty dependency

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-03 22:23:38 +08:00
Cheng Pan
0eeb99774f
Tune pom
fixes #271
Squashed commit of the following:

commit fe0f469b2068865398d6e5be1957fdf6c6f5eb87
Author: Cheng Pan <379377944@qq.com>
Date:   Sat Jan 9 22:28:43 2021 +0800

    followup

commit 527e32419ecb2ae584d1251cb345e3fc870965a9
Author: Cheng Pan <379377944@qq.com>
Date:   Sat Jan 9 22:21:29 2021 +0800

    revert some stuffs

commit 82f188972b8491c5d381ba076a53f84a2fbc1898
Author: Cheng Pan <379377944@qq.com>
Date:   Sat Jan 9 16:49:14 2021 +0800

    remove unused dependencies

commit c7eb27228d2749ab9100102467e94d3a6a98cf44
Author: Cheng Pan <379377944@qq.com>
Date:   Sat Jan 9 15:41:59 2021 +0800

    minimize dependencies in kyuubi-spark-sql-engine

commit da902b1b1e02c7e93810ca832f6c9e7109cf95b9
Author: Cheng Pan <379377944@qq.com>
Date:   Sat Jan 9 13:26:34 2021 +0800

    exclude hive dependencies

commit 55ae7f025a5ffcec402919e28b7b3af5987dfe16
Author: Cheng Pan <379377944@qq.com>
Date:   Sat Jan 9 01:08:09 2021 +0800

    pin versions and excludes in <dependencyManagement>

commit 9ef84a5f62afd66fb4b817d593ba1b13ff8348ed
Author: Cheng Pan <379377944@qq.com>
Date:   Sat Jan 9 14:12:04 2021 +0800

    trigger kyuubi-spark-sql-engine package

commit d492e3fdce64e6554fe0db1d810c6f4bea198baa
Author: Cheng Pan <379377944@qq.com>
Date:   Sat Jan 9 13:44:13 2021 +0800

    trigger kyuubi-download

commit 050281b8fc400f3ae5fa1d43d56da49846664ea4
Author: Cheng Pan <379377944@qq.com>
Date:   Sat Jan 9 15:30:58 2021 +0800

    pin modules version in dependencyManagement

commit 1655fc453ded4626ae5e12e7b17af6f10c30a0dd
Author: Cheng Pan <379377944@qq.com>
Date:   Fri Jan 8 19:45:27 2021 +0800

    tune modules in alphabetical order

commit 7f3f4d987336d71d73eab059e93f3d008cfcd1a6
Author: Cheng Pan <379377944@qq.com>
Date:   Sat Jan 9 00:09:49 2021 +0800

    change dependencies scope to test
2021-01-09 23:30:46 +08:00
Cheng Pan
90c120759e
Improve pom.xml
fixes #268
Squashed commit of the following:

commit d40d0831d578af23559ea2a2544ceebc47f6df50
Author: Cheng Pan <379377944@qq.com>
Date:   Fri Jan 8 18:30:51 2021 +0800

    remove unused mockito

commit 386e7e4eb1c93006e302c41ffbc04d0acfaf9a9b
Author: Cheng Pan <379377944@qq.com>
Date:   Fri Jan 8 17:35:44 2021 +0800

    extract versions to properties
2021-01-08 19:04:48 +08:00
Kent Yao
ca1b452e9a
Prepare Kyuubi v1.1.0 2020-12-31 15:51:12 +08:00
Kent Yao
17fe83778e
ServiceDiscovery for kyuubi servver 2020-12-28 13:05:35 +08:00
Kent Yao
9bd4139453 Remove Kyuubi useless deps 2020-11-16 20:52:45 +08:00
Kent Yao
06178e52fe KyuubiServer Part 1 2020-09-08 14:45:40 +08:00
Kent Yao
1a3e886627 drop module kyuubi-hive-thrift 2020-06-23 17:32:43 +08:00
Kent Yao
c2271ac05b Bumpup Spark to 3.0.0 from 3.0.0-preview2 2020-06-18 17:41:43 +08:00
Kent Yao
7fb0b1a32f Enable EmbeddedZkServer for KyuubiServer 2020-06-17 22:42:19 +08:00
Kent Yao
5100d8a519 Support EmbeddedZkServer 2020-06-12 17:18:46 +08:00
Kent Yao
15d179507f Create Kyuubi Project Spark SQL Engine 2020-06-09 10:34:47 +08:00
Kent Yao
651db99866
HIVE_CLI_SERVICE_PROTOCOL_V11 (#235)
* HIVE_CLI_SERVICE_PROTOCOL_V11

* pom

* style

* tests

* travis

* code cover
2020-05-26 16:07:41 +08:00
Kent Yao
b9a50d62ee new scripts for new framework 2020-05-22 19:16:04 +08:00