Commit Graph

186 Commits

Author SHA1 Message Date
ulysses-you
3afd20e03a
[KYUUBI #985] Add logCaptureThreadReleased flag
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
`isInterrupted` is always false if some `InterruptedException` are throwed. So we should use a new flag to check if log capture thread is released.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #985 from ulysses-you/flaky-test.

Closes #985

1fcc0a99 [ulysses-you] empty
463a996f [ulysses-you] empty
d1f2ea54 [ulysses-you] eventually
f9f37343 [ulysses-you] fix

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-08-25 19:21:35 +08:00
hongdongdong
d704e4d49d [KYUUBI #922] Add OperationEvent for server to track operation state
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Track operation state change and support output to custom system.

### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #968 from hddong/add-server-events.

Closes #922

8be8f268 [hongdongdong] Add prefix for StatementEvent
9fdcc504 [hongdongdong] fix comment
a0cdb35d [hongdongdong] fix test
4d340c67 [hongdongdong] add dependency
00920751 [hongdongdong] [KYUUBI#922]Add OperationEvent for server to track operation state

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-08-25 14:28:48 +08:00
yanghua
cb70e062c3
[KYUUBI #979] Introduce AbstractFrontendService and KyuubiFrontendService
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #980 from yanghua/KPIP-1.

Closes #979

e67e37a4 [yanghua] [KYUUBI 979] Introduce AbstractFrontendService and KyuubiFrontendService

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-08-24 21:36:30 +08:00
yanghua
4aa93d3cbd
[KYUUBI #957] Reopen KyuubiServerSuite
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #960 from yanghua/KYUUBI-957.

Closes #957

94cdddda [yanghua] [KYUUBI #957] Reopen KyuubiServerSuite

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-08-19 15:05:40 +08:00
timothy65535
32b18ba19c
[KYUUBI #933] Enhance the detection mechanism for engine startup
### 1. Description

EngineRef use exists znode to determine whether engine started successfully or not(use the last znode), there are inconsistencies in this. So, take an idea to improve the detection mechanism by adding the started sessionid to the znone. `EngineRef` will check the sessionid in a loop after a engine started.

### 2. Here are two options
**Option I, add sesssionid to the data of znode**
![image](https://user-images.githubusercontent.com/86483005/128994276-5c9c7839-0a21-4253-8989-110b88ac0e74.png)

**Option II, add sessionid to the path of znode**
![image](https://user-images.githubusercontent.com/86483005/129527282-46f7f726-e110-4d87-9272-dda9b9f325bf.png)

### 3. Solution

In order to be consistent with the design of server znode, prefer to choose _Option II_ .

**Demo**
```
/kyuubi
- serviceUri=bigdata:10009;version=1.3.0-SNAPSHOT;sequence=0000000000
/kyuubi_USER/test
- serviceUri=bigdata:39869;version=1.3.0-SNAPSHOT;session=8178069f-0b22-4d06-b1c0-908094769397;sequence=0000000000]
```

Closes #935 from timothy65535/ky-933.

Closes #933

ab5d6d80 [timothy65535] add issue id
d21cb990 [timothy65535] compatible with back
568acf21 [timothy65535] fix error
3db8ba7f [timothy65535] move EngineRef get to Discovery
827f3780 [timothy65535] rename get to getEngineBySessionId
88229314 [timothy65535] update session id using internal
ab06adf8 [timothy65535] improve EngineRef get logic
6a1cc39a [timothy65535] update session version
c94b6e55 [timothy65535] rename sessionId to createSessionId
0cc6aae2 [timothy65535] update ha conf
d24a152f [timothy65535] update ha conf
1a67c864 [timothy65535] fix initialize sql suite
b3b502de [timothy65535] [KYUUBI #933] Enhance the detection mechanism for engine startup

Authored-by: timothy65535 <timothy65535@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-08-19 11:10:21 +08:00
Kent Yao
636d60aa71
[KYUUBI #958] [TEST] Ensure two connections in user mode share the same engine
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Ensure changes like #935 do not break the rule for the semantics of engine sharing

If and only if we want engines to be pooling,  this test can be changed

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #958 from yaooqinn/same.

Closes #958

e172cebf [Kent Yao] Ensure two connections in user mode share the same engine

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-08-19 10:46:24 +08:00
Cheng Pan
a76c344042
[KYUUBI #951] [LICENSE] Add license header on all docs
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #951 from pan3793/license.

Closes #951

4629eecd [Cheng Pan] Fix
c45a0784 [Cheng Pan] nit
b9a46b42 [Cheng Pan] pin license header at first line
80d1a71b [Cheng Pan] nit
b2a46e4c [Cheng Pan] Update
f6acaaf8 [Cheng Pan] minor
ef99183f [Cheng Pan] Add license header on all docs

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-08-19 09:53:52 +08:00
timothy65535
11d1de2a0e
[KYUUBI #942] Support for adding internal attributes to ConfigBuilder
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### Description

`ConfigBuilder` currently only supports public configuration by default. In practice, there are also some configurations, which are internal configurations and do not need to be reserved for users.

refer to https://github.com/apache/incubator-kyuubi/pull/935#issuecomment-900117832

### Modification

- Add `internal` to ConfigBuilder
- Update `AllKyuubiConfiguration`

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #949 from timothy65535/ky-942.

Closes #942

5d4723b3 [timothy65535] update config
b41461bb [timothy65535] [KYUUBI #942] Support for adding internal attributes to ConfigBuilder

Authored-by: timothy65535 <timothy65535@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-08-18 10:24:59 +08:00
timothy65535
f21e118a44 [KYUUBI #940] Fix process-logger-capture thread consumes high usage CPU
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

After several benchmarks in the offline environment, kyuubi consumes a lot of CPU.

```
[bigdata]$ jps
3188 Main
6036 Jps
4189 KyuubiServer

[bigdata]$ top
top - 09:34:21 up 23 min,  1 user,  load average: 10.67, 7.85, 4.98
Tasks: 320 total,   1 running, 319 sleeping,   0 stopped,   0 zombie
%Cpu(s): 48.2 us, 23.9 sy,  0.0 ni, 27.4 id,  0.0 wa,  0.5 hi,  0.0 si,  0.0 st
MiB Mem :  15806.0 total,   4738.4 free,   6085.0 used,   4982.5 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   9306.5 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   4189 bigdata   20   0   10.1g 311272  22484 S 633.3   1.9  48:12.21 java
```
```
Threads:  53 total,   6 running,  47 sleeping,   0 stopped,   0 zombie
%Cpu(s): 31.9 us, 22.8 sy,  0.0 ni, 44.8 id,  0.0 wa,  0.4 hi,  0.0 si,  0.0 st
MiB Mem :  15806.0 total,   4770.5 free,   6052.9 used,   4982.5 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   9338.7 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   5222 bigdata   20   0   10.1g 311272  22484 R  99.3   1.9  10:00.91 process-logger-
   5044 bigdata   20   0   10.1g 311272  22484 R  99.0   1.9  10:03.98 process-logger-
   4730 bigdata   20   0   10.1g 311272  22484 R  98.3   1.9  10:13.24 process-logger-
   4911 bigdata   20   0   10.1g 311272  22484 R  98.3   1.9  10:06.42 process-logger-
   4523 bigdata   20   0   10.1g 311272  22484 R  98.0   1.9  10:41.55 process-logger-
   5769 bigdata   20   0   10.1g 311272  22484 R  96.7   1.9   2:46.80 process-logger-
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #941 from timothy65535/ky-940.

Closes #940

331b9423 [timothy65535] [KYUUBI #940] Fix process-logger-capture thread consumes high usage CPU

Authored-by: timothy65535 <timothy65535@163.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-08-17 16:59:43 +08:00
Cheng Pan
98f38ba517
[BUILD] Bump 1.4.0-SNAPSHOT 2021-08-17 01:39:06 +08:00
timothy65535
9efa78b65e
[KYUUBI #930] Extract the zookeeper part from ServiceDiscovery
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
### Description

`ServiceDiscovery` contains `discovery` and `zookeeper` part, `ServiceDiscovery` should only has one responsibility.

![image](https://user-images.githubusercontent.com/86483005/129339811-9cfaeabf-10dc-4962-9d59-4ca7835ccce5.png)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #931 from timothy65535/ky-930.

Closes #930

29d48921 [timothy65535] Add ZooKeeperClientProvider
17fbe5a2 [timothy65535] [KYUUBI #930] Extract the zookeeper part from ServiceDiscovery

Authored-by: timothy65535 <timothy65535@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-08-14 23:11:18 +08:00
timothy65535
5526898f7b [KYUUBI #894] Enhance kyuubi-zookeeper module
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### 1. Describe the feature

Kyuubi use `TestingServer` as the EmbeddedZookeeper service in codebase, and require many args to init the service. These `electionPort`, `quorumPort`, `serverId` parameters only work in cluster mode, these ports are not bound to the host.

From hadoop and bookkeeper project, they use `ZooKeeperServer` as a built-in zk service, especially the bookeeper project.
- https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/server/services/MicroZookeeperService.java
- https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/shims/zk/ZooKeeperServerShimImpl.java

### 2. Describe the solution

Improve
- Remove `electionPort`, `quorumPort`, `serverId` which are nerver be used, hadoop and bookkeer do the same
- Replace `TestingServer` with `ZooKeeperServer`
- Remove `curator-test` dependency

### 3. `deleteDataDirectoryOnClose`

The name style for `deleteDataDirectoryOnClose` option, refer to hadoop `core-default.xml`
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/core-default.xml
```
hadoop.security.dns.log-slow-lookups.enabled
hadoop.security.dns.log-slow-lookups.threshold.ms
dfs.client.block.write.replace-datanode-on-failure.policy
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #907 from timothy65535/ky-894.

Closes #894

7d7fde9f [timothy65535] remove useless config
ce06a0fd [timothy65535] limit travis build only master branch
ff825c3f [timothy65535] revert deleteDataDirectoryOnClose config
a336544a [timothy65535] remove javassist
3da0bd55 [timothy65535] remove javassist
0abb44df [timothy65535] [KYUUBI #894] Enhance kyuubi-zookeeper module

Authored-by: timothy65535 <timothy65535@163.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-08-11 10:06:44 +08:00
Yue Boyang
0f8c70b8f1
[KYUUBI #905] [MINOR] clear handleToTSessionHandle in KyuubiOperationManager
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
To clear deprecated handleToTSessionHandle in KyuubiOperationManager.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #911 from BoyangYue/test-branch.

Closes #905

3da8add4 [Yue Boyang] [KYUUBI #905] [MINOR] clear handleToTSessionHandle in KyuubiOperationManager

Authored-by: Yue Boyang <boyang.yue@student.unimelb.edu.au>
Signed-off-by: Kent Yao <yao@apache.org>
2021-08-10 09:59:18 +08:00
ulysses-you
c86767f336
[KYUUBI #901] Skip download module generate config docs
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Imporve perf of generate config docs.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #901 from ulysses-you/enhance-comment.

Closes #901

a86a9aa6 [ulysses-you] skip-download

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-08-07 01:55:32 +08:00
Cheng Pan
4fecf13902 [KYUUBI #888] Add scala version suffix on scala modules
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
- Add scala version suffix on scala modules to match the name pattern.
- Enable `kyuubi-extension-spark-3-1` in daily publish snapshot

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #888 from pan3793/scala.

Closes #888

571eb9c3 [Cheng Pan] comments
16cb263a [Cheng Pan] remove hardcode scala version in pom
6090c798 [Cheng Pan] Fix shade
5178847f [Cheng Pan] Fixup
a1547753 [Cheng Pan] Fix engine jar name
2cf4793e [Cheng Pan] fix
cac96531 [Cheng Pan] Enable kyuubi-extension-spark-3-1 in daily publish snapshot
1ca31de5 [Cheng Pan] Add scala version suffix on scala modules

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-08-04 09:31:19 +08:00
Kent Yao
6c3b722174
[KYUUBI #869] Add Kyuubi Query Engine for Spark UI with basic stats
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

This PR adds the Kyuubi Query Engine tab on Spark UI, it gathers basic stats currently. We will put more details in the following PRs

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate
![image](https://user-images.githubusercontent.com/8326978/127176837-d0bb6433-c5fc-4402-a2f2-76131332fdf6.png)

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #869 from yaooqinn/ui.

Closes #869

aaba5f02 [Kent Yao] Merge branch 'master' of github.com:apache/incubator-kyuubi into ui
0a80a352 [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats
ff877fe6 [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats
f2e2cabb [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats
7f2b5931 [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats
cff28784 [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-07-30 12:10:36 +08:00
ulysses-you
f4fb822743
[KYUUBI #871] Change module name kyuubi-main to kyuubi-server
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
As we will release soon, it's better to format the module name and artifactId.

The name of kyuubi-main sounds like a bootstrap of Kyuubi but not a server.

### _How was this patch tested?_
Pass CI

Closes #872 from ulysses-you/kyuubi-server.

Closes #871

1437f9b5 [ulysses-you] address comment
117f2c7c [ulysses-you] server
86986baa [ulysses-you] master
eba660a2 [ulysses-you] Merge branch 'master' of https://github.com/NetEase/kyuubi into kyuubi-server
e649012b [ulysses-you] change module name

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-07-29 13:41:43 +08:00
Kent Yao
91f622f94f Remove Kyuubi 0.8 and before 2020-11-16 20:12:08 +08:00
Kent Yao
bd319f4216 remove most tests in kyuubi-server module 2020-11-05 14:14:11 +08:00
Kent Yao
cdb46e90db Add Configurations Guide 2020-11-04 22:40:02 +08:00
Kent Yao
67858e8c15 [DOCS] Add qucik start documentation 2020-11-02 21:07:11 +08:00
Kent Yao
cf255c79c6 add and rm some tests 2020-10-27 10:41:48 +08:00
Kent Yao
11fd6f9263 PlainSASLHelperSuite 2020-10-26 15:17:53 +08:00
Kent Yao
e59a751b04 PlainSASLServerSuite 2020-10-26 14:21:49 +08:00
Kent Yao
50f2d85954 LdapAuthenticationProviderImplSuite 2020-10-23 17:59:13 +08:00
Kent Yao
7eb84b1800 add more tests 2020-10-23 17:31:04 +08:00
Kent Yao
76a7d55685 add ServiceUtilsSuite 2020-10-23 10:34:44 +08:00
Kent Yao
1f52dbd656 FrontendServiceSuite nit 2020-10-22 18:48:31 +08:00
Kent Yao
64cfb8cd95 FrontendServiceSuite get catalogs 2020-10-22 10:34:30 +08:00
Kent Yao
69b2d55d24 Thread Suites 2020-10-19 11:03:25 +08:00
Kent Yao
0b07dfd6dd improve test cov 2020-10-14 11:33:56 +08:00
Kent Yao
164a984b97 rm useless test compile version info 2020-09-25 19:33:02 +08:00
Kent Yao
53205a43cc Revert "Revert "fix""
This reverts commit 9f4ccdd70c.
2020-09-23 05:08:26 +08:00
Kent Yao
9f4ccdd70c Revert "fix"
This reverts commit d97f71456b.
2020-09-22 18:33:11 +08:00
Kent Yao
d97f71456b fix 2020-09-22 14:15:58 +08:00
Kent Yao
eb2a7050d9 build info 2020-09-21 21:26:16 +08:00
Kent Yao
e22dda7dbd Add more build infomation 2020-09-11 09:55:11 +08:00
Kent Yao
83d1ab7450 bump up spark to 3.0.1 2020-09-10 10:58:52 +08:00
Kent Yao
0468fd4d4e Implementaton of KyuubiOperationManager 2020-08-21 10:51:55 +08:00
Kent Yao
86e1a7aac3 SparkProcessBuilder 2020-08-18 17:22:08 +08:00
Kent Yao
3617657bed Add support for operation_log 2020-08-13 10:41:46 +08:00
Kent Yao
7682225145
Add github action for pull request and commit (#240)
* Add github action for pull request and commit

* fix

* ut
2020-08-11 10:23:53 +08:00
Kent Yao
a3891b3c53 Implementaion of Spark SQL engine 2020-08-10 17:52:10 +08:00
Kent Yao
eea65272e2 Add Operation for spark engine 2020-07-31 11:09:11 +08:00
Kent Yao
4d2ec8a00a Add new FrontendService 2020-07-21 15:26:48 +08:00
Kent Yao
934a9d3158 Support RowSet 2020-07-08 11:38:48 +08:00
Kent Yao
e3e1a8f00b Add OperationState 2020-06-30 15:29:59 +08:00
Kent Yao
1a3e886627 drop module kyuubi-hive-thrift 2020-06-23 17:32:43 +08:00
Kent Yao
c2271ac05b Bumpup Spark to 3.0.0 from 3.0.0-preview2 2020-06-18 17:41:43 +08:00
Kent Yao
e1d55f8351 Introduce ServiceDiscovery 2020-06-17 16:14:20 +08:00