Commit Graph

268 Commits

Author SHA1 Message Date
ulysses-you
59d9f06a46 [KYUUBI #1032][FOLLOWUP] Refine session hande in AbstractSession
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Remove session hande in `AbstractSession`

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1046 from ulysses-you/kyuubi-1032-f.

Closes #1032

1d7543a9 [ulysses-you] order
36a0a1cc [ulysses-you] refine

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-09-07 12:08:09 +08:00
Cheng Pan
f584953f7f
[KYUUBI #703][FOLLOWUP][DEPS][TEST] Reduce Hudi dependencies
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

HBase deps is optional for creating hudi table and query, but required for writing data to hudi table, as currently we only test create table and query for data lake formats, it's reasonable to remove the hbase deps to reduce the deps pulled from hudi.

This PR also change all data lake test suites in spark engine from `hive` catalog to `in-memory` catalog.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1028 from pan3793/hudi.

Closes #703

56c41a30 [Cheng Pan] Use in-memory catalog for DataLake test
b582a865 [Cheng Pan] Revert "Default use in-memory in Spark engine test"
e62c403a [Cheng Pan] Default use in-memory in Spark engine test
e2b8e0fd [Cheng Pan] Remove unused hudi spark conf
069fbacc [Cheng Pan] Revert "HFileBootstrapIndex"
ca1b20e1 [Cheng Pan] HFileBootstrapIndex
da37b0fc [Cheng Pan] Remove log4j
41ce8526 [Cheng Pan] Align jackson version
f56f5e89 [Cheng Pan] Remove HBase dependencies
c2b76b54 [Cheng Pan] Exclude hudi-timeline-service
56641762 [Cheng Pan] nit
e588c12f [Cheng Pan] [DEPS][TEST] Reduce Hudi dependencies

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-09-07 01:37:18 +08:00
simon
098e660a3d
[KYUUBI #703] support hudi-0.9.0
### _Why are the changes needed?_
Hudi has add Spark Sql Support in version 0.9.0.
This PR add Hudi-0.9.0 dependency and Unit Testing  #703

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #994 from simon824/master.

Closes #703

a04b2e42 [simon] exclude  jdk.tools dep
50e1ab01 [simon] fix jdk11 dep
f33ebdf1 [simon] fix hudi dep conflicts
ff2585c2 [simon] fix yarn dep conflicts
32dd1ea8 [simon] fix scala version conflicts
b8a37401 [simon] add spark.sql.catalogImplementation
82f5422f [simon] fix spark3 dependency
2a6c497d [simon] fix spark3 dependency
4dbec8be [Simon] Merge branch 'apache:master' into master
3f180157 [simon] fix maven
73e48d0a [simon] add spark3 support maven dependency
3def658b [simon] fix missing spark-sql-engine hudi maven dependency
524132d3 [simon] bugfix
e98998a6 [simon] fix pom error
59fc6669 [Simon] Merge branch 'apache:master' into master
d9e17ebf [simon] fix conflict
2466ece3 [simon] add sparkSQL test on hudi-0.9.0

Lead-authored-by: simon <zhangshiming@cvte.com>
Co-authored-by: Simon <3656562@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-09-04 10:48:18 +08:00
yanghua
32111a3014
[KYUUBI #999] Build the basic framework for rest frontend service
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1003 from yanghua/KYUUBI-999.

Closes #999

78fdf542 [yanghua] Changed rest frontend service port from 10009 to 10099
73929d5d [yanghua] [KYUUBI #999] Build the basic framework for rest frontend service

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-09-03 19:19:17 +08:00
zhouyifan279
f8056f458b
[KYUUBI #1007] Implement delegation token renewal framework
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
This PR finishes sub-task #1007 under umbrella issue #913.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1015 from zhouyifan279/KYUUBI#1007.

Closes #1007

94263804 [zhouyifan279] [KYUUBI #1007] Implement delegation token renewal framework
6cea96d0 [zhouyifan279] [KYUUBI #1007] Implement delegation token renewal framework
dbffcf15 [zhouyifan279] [KYUUBI #1007] Implement delegation token renewal framework
57100271 [zhouyifan279] [KYUUBI #1007] Implement delegation token renewal framework

Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-09-03 18:40:30 +08:00
yanghua
01bebe36e8 [KYUUBI #1011] Fallback some frontend service config option about thrift
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1016 from yanghua/KYUUBI-1011.

Closes #1011

78107b76 [yanghua] Addressed review suggestion
651b5989 [yanghua] [KYUUBI #1011] Fallback some frontend service config option about thrift

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-09-03 09:30:29 +08:00
Kent Yao
d375eb0cfc
[KYUUBI #1010] Rename engine.share.level.sub.domain to engine.share.level.subdomain
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

`subdomain` is a widely used term in computer science

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1010 from yaooqinn/subdomain.

Closes #1010

72ed19a3 [Kent Yao] address comments
4dcae414 [Kent Yao] fix test
c3bb788d [Kent Yao] Rename engine.share.level.sub.domain to engine.share.level.subdomain
b7762357 [Kent Yao] Rename engine.share.level.sub.domain to engine.share.level.subdomain

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-09-02 11:46:07 +08:00
hongdongdong
94d60ea505 [KYUUBI #1000] Use underscore instead hyphen in path
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Use underscore instead hyphen in path, friendly to sql.

### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1000 from hddong/fix-underscore.

Closes #1000

8b3459a5 [hongdongdong] Use underscore instead hyphen in path

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-08-30 16:55:03 +08:00
ulysses-you
d49f8bf06d [KYUUBI #988] Add Kyuubi session event
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
closes https://github.com/apache/incubator-kyuubi/issues/988

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #989 from ulysses-you/kyuubi-session-event.

Closes #988

baebf2eb [ulysses-you] address comment
283d574c [ulysses-you] fix
c5d1d5f7 [ulysses-you] adddress comment
44cfd06f [ulysses-you] docs
43639993 [ulysses-you] simplify session
2618b764 [ulysses-you] config name
a2d4608b [ulysses-you] docs
5adbe5c8 [ulysses-you] config
ea5c70e1 [ulysses-you] nit
eea73ed5 [ulysses-you] config
cda6d38b [ulysses-you] test
a754c76f [ulysses-you] init

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-08-30 16:54:01 +08:00
yanghua
4ed5704590
[KYUUBI #983] Refactor the basic abstraction of frontend service
…

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #984 from yanghua/KYUUBI-983.

Closes #983

db953b8a [yanghua] Rename KyuubiFrontendService to be KyuubiFrontendServices
8c3d7f2a [yanghua] [KYUUBI #983] Refactor the basic abstraction of frontend service

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-08-30 09:34:12 +08:00
timothy65535
acb4f29906
[KYUUBI #962] Support engine pool feature
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

For detail, please go to _Originally posted by yaooqinn in https://github.com/apache/incubator-kyuubi/issues/962#issuecomment-905116261_

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

```
./bin/beeline -u 'jdbc:hive2://localhost:10009/;?kyuubi.engine.pool.enabled=true;kyuubi.engine.pool.size=3;'
```

Closes #986 from timothy65535/ky-962-v2.

Closes #962

b79f8778 [timothy65535] fix test name
9a435607 [timothy65535] fix style
abc746b8 [timothy65535] update tests
71b6933d [timothy65535] revert changes
018dee8b [timothy65535] revert changes
e3492934 [timothy65535] revert changes
67fc8fd9 [timothy65535] improve threshold
fa52acb2 [timothy65535] update pool threshold
d8b5977d [timothy65535] remove dup get
7f39b8d7 [timothy65535] fix test error
4fdf8d76 [timothy65535] update thread pool size to 1 -> threshold
2477348a [timothy65535] rich engine pool tests
8eee427f [timothy65535] fix update error msg
104be5ea [timothy65535] fix error tests
8c2d1441 [timothy65535] fix doc
ddd0eda9 [timothy65535] improve doc & tests
83d7c5cf [timothy65535] update doc
4b6c7ee3 [timothy65535] improve subdomain
169d9d06 [timothy65535] update todo
1424e565 [timothy65535] increase pool size range
3523cd6c [timothy65535] update todo
2cbeafb2 [timothy65535] remove enable option
424b5364 [timothy65535] revert ServiceDiscovery
6fc1c510 [timothy65535] remove random policy
9feef8f3 [timothy65535] [KYUUBI #962] Support engine pool feature

Authored-by: timothy65535 <timothy65535@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-08-27 11:18:00 +08:00
hongdongdong
d704e4d49d [KYUUBI #922] Add OperationEvent for server to track operation state
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Track operation state change and support output to custom system.

### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #968 from hddong/add-server-events.

Closes #922

8be8f268 [hongdongdong] Add prefix for StatementEvent
9fdcc504 [hongdongdong] fix comment
a0cdb35d [hongdongdong] fix test
4d340c67 [hongdongdong] add dependency
00920751 [hongdongdong] [KYUUBI#922]Add OperationEvent for server to track operation state

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-08-25 14:28:48 +08:00
yanghua
cb70e062c3
[KYUUBI #979] Introduce AbstractFrontendService and KyuubiFrontendService
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #980 from yanghua/KPIP-1.

Closes #979

e67e37a4 [yanghua] [KYUUBI 979] Introduce AbstractFrontendService and KyuubiFrontendService

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-08-24 21:36:30 +08:00
odone
f1ae664910
[KYUUBI #961] Fixed: opHanle's operation may not be release
Align with `sessionManager.operationManager.closeOperation(operationHandle)` and remove first

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #975 from iodone/fix-opHanle-operation-not-release.

Closes #961

bd798e0d [odone] [KYUUBI #961] fixed: opHanle's operation may not be release

Authored-by: odone <odone.zhang@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-08-23 14:05:54 +08:00
Cheng Pan
33be4516b6
[KYUUBI #973] [LICENSE] Fix license issue
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
- Remove the `scala.util.Using`
- Remove redundant license declaration

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #973 from pan3793/license.

Closes #973

b02206ea [Cheng Pan] Remove redundant license declaration
3a1401d9 [Cheng Pan] [LICENSE] Remove Scala Using

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-08-23 09:45:52 +08:00
Cheng Pan
ae4878349c
[KYUUBI #659][FOLLOWUP] Temporarily ignore udf engine_name test
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Workaround to recover Minikube test until #937 is solved.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #969 from pan3793/747-ignore.

Closes #659

e26153f6 [Cheng Pan] [KYUUBI #659][FOLLOWUP] Temporarily ignore udf engine_name test

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-08-22 01:00:49 +08:00
Min Zhao
8bab5b8d6a
[KYUUBI #659] Add UDF engine_name
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Add UDF engine_name to show the runtime information about execution engine

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #747 from zhaomin1423/udf.

Closes #659

8c3b3c16 [Min Zhao] Merge branch 'master' into udf
f184a661 [Min Zhao] Merge branch 'master' into udf
9ea0c190 [Min Zhao] get appName from driver side
09f07523 [Min Zhao] get conf by SparkEnv.get.conf in the engine_name udf
6c000784 [Min Zhao] update the description of the engine_name udf
63ef2762 [Min Zhao] update class to org.apache.kyuubi.engine.spark.udf.KyuubiDefinedFunctionSuite in functions.md
db02884c [Min Zhao] [KYUUBI #659] Add UDF engine_name

Lead-authored-by: Min Zhao <zhaomin1423@163.com>
Co-authored-by: Min Zhao <49054376+zhaomin1423@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-08-22 00:28:45 +08:00
Baoqi Wu
484bb3c6bd
[KYUUBI #956] Fail to get table list if there is _ in schema's name
Just write some basic tests out in other projects.

Since this toJavaRegex is not a static function, which is hard to write unit test if don't move it out as  as static function in object.  (But in that case, the change may be too large for this).

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #964 from Baoqi/bwu_fix_956.

Closes #956

985a73b5 [Baoqi Wu] [KYUUBI #956] add a unit test for this
cb111893 [Baoqi Wu] [KYUUBI #956] Fail to get table list if there is _ in schema's name

Authored-by: Baoqi Wu <wubaoqi@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-08-21 00:17:54 +08:00
yanghua
6b23c1c1f5
[KYUUBI #952] Rename FrontendService and FrontendServiceSuite with a Thrift prefix
…

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #953 from yanghua/KYUUBI-952.

Closes #952

a3029011 [yanghua] [KYUUBI 952] Rename FrontendService and FrontendServiceSuite with a Thrift prefix

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-08-18 22:19:51 +08:00
ulysses-you
960ac89776 [KYUUBI #936] Add date info for json event path
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Add date info helps collect metrics across some days.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #936 from ulysses-you/event-date-partition.

Closes #936

6d204084 [ulysses-you] time
f142e052 [ulysses-you] test
fd220112 [ulysses-you] statement
31f9f0fb [ulysses-you] empty
67acbd5e [ulysses-you] fix
4cb0ee91 [ulysses-you] fix
42159c08 [ulysses-you] partitions
3500d91c [ulysses-you] remove enent=
04c3dacb [ulysses-you] fix
5e7f19ec [ulysses-you] nit
83eaf529 [ulysses-you] date partition
a2d6a38d [ulysses-you] start time
693e8ee9 [ulysses-you] Add date info for json event path

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-08-18 18:20:52 +08:00
timothy65535
11d1de2a0e
[KYUUBI #942] Support for adding internal attributes to ConfigBuilder
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### Description

`ConfigBuilder` currently only supports public configuration by default. In practice, there are also some configurations, which are internal configurations and do not need to be reserved for users.

refer to https://github.com/apache/incubator-kyuubi/pull/935#issuecomment-900117832

### Modification

- Add `internal` to ConfigBuilder
- Update `AllKyuubiConfiguration`

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #949 from timothy65535/ky-942.

Closes #942

5d4723b3 [timothy65535] update config
b41461bb [timothy65535] [KYUUBI #942] Support for adding internal attributes to ConfigBuilder

Authored-by: timothy65535 <timothy65535@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-08-18 10:24:59 +08:00
Cheng Pan
c228ecc2ef
[KYUUBI #946] [TEST] Fix version test
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Fix #944

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #946 from pan3793/ver.

Closes #946

81f69c8e [Cheng Pan] Fix version test

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-08-18 09:23:01 +08:00
ulysses-you
5fb143e50b
[KYUUBI #938] Disable Kyuubi version udf test
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Kyuubi docker image has not updated to lastest version, the kubernetes integration tests failed in master branch.

### _How was this patch tested?_
Pass CI

Closes #938 from ulysses-you/disable-udf.

Closes #938

3260ed73 [ulysses-you] disable version test

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-08-17 10:58:22 +08:00
Cheng Pan
98f38ba517
[BUILD] Bump 1.4.0-SNAPSHOT 2021-08-17 01:39:06 +08:00
Kent Yao
ad7eca1281 [KYUUBI #920] Add COMPILED State for ExecuteStatement af engine side
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
RUNNING -> COMPILED can be used to calculate to the duration of a statement being parsed, analyzed, and optimized

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #920 from yaooqinn/compiled.

Closes #920

05c355d6 [Kent Yao] Add COMPILED State for ExecuteStatement af engine side
dfba0481 [Kent Yao] Add COMPILED State for ExecuteStatement af engine side

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-08-12 14:49:38 +08:00
timothy65535
5526898f7b [KYUUBI #894] Enhance kyuubi-zookeeper module
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### 1. Describe the feature

Kyuubi use `TestingServer` as the EmbeddedZookeeper service in codebase, and require many args to init the service. These `electionPort`, `quorumPort`, `serverId` parameters only work in cluster mode, these ports are not bound to the host.

From hadoop and bookkeeper project, they use `ZooKeeperServer` as a built-in zk service, especially the bookeeper project.
- https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/server/services/MicroZookeeperService.java
- https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/shims/zk/ZooKeeperServerShimImpl.java

### 2. Describe the solution

Improve
- Remove `electionPort`, `quorumPort`, `serverId` which are nerver be used, hadoop and bookkeer do the same
- Replace `TestingServer` with `ZooKeeperServer`
- Remove `curator-test` dependency

### 3. `deleteDataDirectoryOnClose`

The name style for `deleteDataDirectoryOnClose` option, refer to hadoop `core-default.xml`
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/core-default.xml
```
hadoop.security.dns.log-slow-lookups.enabled
hadoop.security.dns.log-slow-lookups.threshold.ms
dfs.client.block.write.replace-datanode-on-failure.policy
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #907 from timothy65535/ky-894.

Closes #894

7d7fde9f [timothy65535] remove useless config
ce06a0fd [timothy65535] limit travis build only master branch
ff825c3f [timothy65535] revert deleteDataDirectoryOnClose config
a336544a [timothy65535] remove javassist
3da0bd55 [timothy65535] remove javassist
0abb44df [timothy65535] [KYUUBI #894] Enhance kyuubi-zookeeper module

Authored-by: timothy65535 <timothy65535@163.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-08-11 10:06:44 +08:00
Kent Yao
6dd8dfb70c
[KYUUBI #914] Use isEmpty to trigger default init queries
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

1 use isEmpty instead show for the init SQLs
2 add job desc for SQL tab of Spark UI
3 make it simple.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #914 from yaooqinn/isempty.

Closes #914

70785da7 [Kent Yao] Use isEmpty to trigger default init queries
438009e4 [Kent Yao] Use isEmpty to trigger default init queries

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-08-10 17:27:47 +08:00
Kent Yao
f96256ced6
[KYUUBI #906] Add SessionEvent to track session status changes
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #909 from yaooqinn/906.

Closes #906

e202b70d [Kent Yao] nit
1271e841 [Kent Yao] fix test
d696021f [Kent Yao] [KYUUBI #906] Add SessionEvent to track session status changes
bdd2b875 [Kent Yao] [KYUUBI #906] Add SessionEvent to track session status changes

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-08-10 17:26:04 +08:00
ulysses-you
065c98a259
[KYUUBI #897] Support stop Spark engine through web ui
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Closes https://github.com/apache/incubator-kyuubi/issues/897

### _How was this patch tested?_
The following two images show the stop feature is enabled:
![image](https://user-images.githubusercontent.com/12025282/128304854-20f3619a-6cae-4975-837b-34858ba4ef37.png)
![image](https://user-images.githubusercontent.com/12025282/128304875-987fec0d-5242-4d09-acf5-c6af8aad0068.png)

The following image show the stop feature is disabled:
![image](https://user-images.githubusercontent.com/12025282/128304893-0547fb54-da58-4691-a0a6-38feabc3930d.png)

Closes #898 from ulysses-you/kyuubi-897.

Closes #897

61ad25c1 [ulysses-you] address comment
b63997ec [ulysses-you] maven
dbb36014 [ulysses-you] maven
3c91307c [ulysses-you] simplify
6e9e5bca [ulysses-you] pom
4e143657 [ulysses-you] config
e579e227 [ulysses-you] Support stop Spark engine through web ui

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-08-07 01:53:49 +08:00
Cheng Pan
4fecf13902 [KYUUBI #888] Add scala version suffix on scala modules
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
- Add scala version suffix on scala modules to match the name pattern.
- Enable `kyuubi-extension-spark-3-1` in daily publish snapshot

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #888 from pan3793/scala.

Closes #888

571eb9c3 [Cheng Pan] comments
16cb263a [Cheng Pan] remove hardcode scala version in pom
6090c798 [Cheng Pan] Fix shade
5178847f [Cheng Pan] Fixup
a1547753 [Cheng Pan] Fix engine jar name
2cf4793e [Cheng Pan] fix
cac96531 [Cheng Pan] Enable kyuubi-extension-spark-3-1 in daily publish snapshot
1ca31de5 [Cheng Pan] Add scala version suffix on scala modules

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-08-04 09:31:19 +08:00
Kent Yao
f6d5069bb6
[KYUUBI #885] Add Event for Engine
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #885 from yaooqinn/engineevent.

Closes #885

6a7ac341 [Kent Yao] refine
41baf0e7 [Kent Yao] refine
2c141a06 [Kent Yao] test history server
d1b0823a [Kent Yao] test history server
3aae8417 [Kent Yao] dump the final engine status
257e50f5 [Kent Yao] Add Event for Engine
f1f1f0e8 [Kent Yao] Add Event for Engine
85addc6e [Kent Yao] Add Event for Engine
cece5888 [Kent Yao] Add Event for Engine

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-08-03 15:51:04 +08:00
Cheng Pan
1a8b4ebaae
[KYUUBI #884] Rewrite kyuubi-hive-jdbc in Java
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
close #765

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #884 from pan3793/jdbc-java.

Closes #884

ca7bad48 [Cheng Pan] Rewrite kyuubi-hive-jdbc in Java

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-08-02 17:35:00 +08:00
timothy65535
4b4cfa6ff6
[KYUUBI #821] Improper exception for closing a non-existent session
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

**ROOT CAUSE**
When server receive request of close session, will call `FrontendService#CloseSession`. When an error is encountered when closing the session, `CURRENT_SERVER_CONTEXT` will never be called.

```
2021-08-02 09:41:50.945 INFO service.FrontendService: Received request of closing SessionHandle [f3545c54-2e94-4090-aaa6-4397e653bb72]
2021-08-02 09:41:50.946 INFO session.KyuubiSessionManager: SessionHandle [f3545c54-2e94-4090-aaa6-4397e653bb72] is closed, current opening sessions 0
2021-08-02 09:41:50.949 WARN service.FrontendService: Error closing session:
org.apache.kyuubi.KyuubiSQLException: Error while cleaning up the engine resources
	at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:68)
	at org.apache.kyuubi.session.KyuubiSessionImpl.close(KyuubiSessionImpl.scala:89)
	at org.apache.kyuubi.session.SessionManager.closeSession(SessionManager.scala:74)
	at org.apache.kyuubi.service.AbstractBackendService.closeSession(AbstractBackendService.scala:49)
	at org.apache.kyuubi.service.FrontendService.CloseSession(FrontendService.scala:218)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1397)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1382)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.thrift.transport.TTransportException
	at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
	at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
	at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376)
	at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453)
	at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435)
	at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
	at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
	at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
	at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
	at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
	at org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_CloseSession(TCLIService.java:191)
	at org.apache.hive.service.rpc.thrift.TCLIService$Client.CloseSession(TCLIService.java:178)
	at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$closeSession$1(KyuubiSyncThriftClient.scala:67)
	at org.apache.kyuubi.client.KyuubiSyncThriftClient.withLockAcquired(KyuubiSyncThriftClient.scala:47)
	at org.apache.kyuubi.client.KyuubiSyncThriftClient.closeSession(KyuubiSyncThriftClient.scala:67)
	at org.apache.kyuubi.session.KyuubiSessionImpl.close(KyuubiSessionImpl.scala:86)
	... 12 more
2021-08-02 09:41:50.953 INFO service.FrontendService: Finished closing SessionHandle [f3545c54-2e94-4090-aaa6-4397e653bb72]
2021-08-02 09:41:50.957 INFO service.FrontendService: Session [SessionHandle [f3545c54-2e94-4090-aaa6-4397e653bb72]] disconnected without closing properly, close it now
2021-08-02 09:41:50.963 INFO service.FrontendService: Session [SessionHandle [f3545c54-2e94-4090-aaa6-4397e653bb72]] disconnected without closing properly, close it now
2021-08-02 09:41:50.964 WARN service.FrontendService: Failed closing session
org.apache.kyuubi.KyuubiSQLException: Invalid SessionHandle [f3545c54-2e94-4090-aaa6-4397e653bb72]
	at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:68)
	at org.apache.kyuubi.session.SessionManager.closeSession(SessionManager.scala:71)
	at org.apache.kyuubi.service.AbstractBackendService.closeSession(AbstractBackendService.scala:49)
	at org.apache.kyuubi.service.FrontendService$FeTServerEventHandler.deleteContext(FrontendService.scala:529)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:300)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2021-08-02 09:42:30.908 INFO server.ZooKeeperServer: Expiring session 0x1000da7829f0002, timeout of 60000ms exceeded
2021-08-02 09:42:30.909 INFO server.PrepRequestProcessor: Processed session termination for sessionid: 0x1000da7829f0002
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #879 from timothy65535/ky-821.

Closes #821

f485f580 [timothy65535] [KYUUBI #821] Improper exception for closing a non-existent session

Authored-by: timothy65535 <timothy65535@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-08-02 11:34:57 +08:00
Kent Yao
6c3b722174
[KYUUBI #869] Add Kyuubi Query Engine for Spark UI with basic stats
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

This PR adds the Kyuubi Query Engine tab on Spark UI, it gathers basic stats currently. We will put more details in the following PRs

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate
![image](https://user-images.githubusercontent.com/8326978/127176837-d0bb6433-c5fc-4402-a2f2-76131332fdf6.png)

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #869 from yaooqinn/ui.

Closes #869

aaba5f02 [Kent Yao] Merge branch 'master' of github.com:apache/incubator-kyuubi into ui
0a80a352 [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats
ff877fe6 [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats
f2e2cabb [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats
7f2b5931 [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats
cff28784 [Kent Yao] Add Kyuubi Query Engine for Spark UI with basic stats

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-07-30 12:10:36 +08:00
Cheng Pan
b963b75d58
[KYUUBI #857] Support init SQL for each session
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #857 from pan3793/seesion-init-sql.

Closes #857

f6f59196 [Cheng Pan] remove unnecessary change
4960830a [Cheng Pan] run ENGINE_SESSION_INITIALIZE_SQL when engine startup
6e8100fa [Cheng Pan] address comments
08f4af62 [Cheng Pan] address comments
2bb1bb85 [Cheng Pan] fix import
880b38ec [Cheng Pan] session initialize sql will only execute once in single session mode
2a646890 [Cheng Pan] test session initialize sql will not execute in single session mode
f03f3eff [Cheng Pan] update doc
6e4a72f2 [Cheng Pan] address comments
34285e25 [Cheng Pan] format
6d978025 [Cheng Pan] revert log4j conf change
881a89e0 [Cheng Pan] improve test
b9ec18c9 [Cheng Pan] fix ut and add log
b4f586d8 [Cheng Pan] Support init SQL for each session

Lead-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-07-28 21:32:48 +08:00
Kent Yao
d5104ed277
[KYUUBI #868] Avoid calling a stopped SparkContext
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

We shall avoid calling cancelJobGroup if the sc is already stopped.

We shall stop the engine to stop serving before we stop the sc during the shutdown hook

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #868 from yaooqinn/shutdownhook.

Closes #868

14810875 [Kent Yao] comment
972e2741 [Kent Yao] Avoid calling a stopped SparkContext

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-07-27 19:58:25 +08:00
ulysses-you
536e5fde72
[KYUUBI #444][KYUUBI #854] Add minikube for k8s integration test
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
* Use github action to install `MiniKube`
* Start Kyuubi Server as Pod with `MiniKube`
* Add a new module `kubernetes-integration-tests` to run basic JDBC test

Since there exists many `MiniKube` app in github market, this PR use [Setup Minikube Kubernetes Cluster](https://github.com/marketplace/actions/setup-minikube-kubernetes-cluster) , we can change this if necessary in future.

Note that, this PR is for Kyuubi test with k8s not for Apache Spark.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #853 from ulysses-you/k8s-it.

Closes #444

Closes #854

4a73a8fa [ulysses-you] ignore
a501482d [ulysses-you] ignore
242f60c4 [ulysses-you] todo
994e52e4 [ulysses-you] pom
3f60fb2e [ulysses-you] Merge branch 'master' of https://github.com/NetEase/kyuubi into k8s-it
3febdcd2 [ulysses-you] fix
f564792b [ulysses-you] Add minikuube for k8s integration test

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-07-26 18:47:11 +08:00
Cheng Pan
ce5fb97036
[KYUUBI #849] ASF Nexus SNAPSHOT publish
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
https://infra.apache.org/publishing-maven-artifacts.html
https://issues.apache.org/jira/browse/INFRA-22114

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #849 from pan3793/asf.

Closes #849

c609f6ba [Cheng Pan] update pom
31f1f2e1 [Cheng Pan] rename root project name to kyuubi-parent
9c5616d8 [Cheng Pan] [INFRA] Update POM for Apache release

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-07-23 13:15:05 +08:00
hongdongdong
8cef5c809a [KYUUBI #839]Fix load user specific defaults from properties file
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
loadFromMap ignored user specific defaults that start with '___'.

### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #840 from hddong/kyuubi-839.

Closes #839

ba3683e3 [hongdongdong] fix comments
a7753849 [hongdongdong] fix test
07f82667 [hongdongdong] fix test
b2a4c3cb [hongdongdong] [KYUUBI#839]Fix load user specific defaults from properties file

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-07-21 12:13:46 +08:00
Min Zhao
cd308f38ad
[KYUUBI #758] [KYUUBI 661] Add UDF system_user
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

add udf system_user.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #758 from zhaomin1423/661_add-system_user.

Closes #758

70eba56a [Min Zhao] update system user name
f5edc621 [Min Zhao] [KYUUBI 661] Add UDF system_user

Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-07-19 18:16:38 +08:00
张宇翔
e7e77f9277
[KYUUBI #814] Event Tracking: For jonInfo
### _Why are the changes needed?_

Store all jobInfos for each statement in mem.
- First, according to jobInfo, you can get how long did it run, and get which stage took the longest time by stageIds.
- Second, if this job failed that you can get from jobResult field, you can look up which stage cause this situation by stageIds.

Some interfaces:
- KyuubiJobInfo: job's summary info, contains statementId, startTime, endTime, jobResult, stageIds.
- KyuubiStatementListener: singleton pattern, used for getting metrics about job, stage, executor and so on.
- KyuubiStatementMonitor: for storing data in mem and dumpping them to a file when reached threshold.

Test case:
externals/kyuubi-spark-sql-engine/src/test/scala/org/apache/kyuubi/engine/spark/KyuubiStatementMonitorSuite.scala

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #814 from zhang1002/branch-1.2_add-jobinfo.

Closes #814

7d8ece23 [张宇翔] 1. Change the log level from INFO to DEBUG 2. Code modify: avoid producing NullPointerException
0ddd83a7 [张宇翔] 1. Remove waitForOperationToComplete(client, opHandle)
bbc36bb3 [张宇翔] 1. Merge statementListener to sqlEngineListener
741fcd0f [张宇翔] Add interval
b61b9406 [张宇翔] Code changes
cb6f92c4 [张宇翔] 1. Remove object-lock 2. Add eventually code
c1a1d732 [张宇翔] 1. Remove volatile for jobInfo
9547d49e [张宇翔] 1. Remove volatile for jobInfo
970c0002 [张宇翔] 1. Add volatile for jobInfo
62ecb53d [张宇翔] 1. Remove kyuubiJobInfoQueue 2. Modify some unit test 3. Add some code annotations
db1de381 [张宇翔] 1. Add some log 2. Add some annotations
8deb61f2 [张宇翔] change some test
87d9c102 [张宇翔] 1. Event tracking: for jonInfo 2. Add unit test
86c49ebc [张宇翔] Merge master branch
a0a99b3a [张宇翔] Merge master branch
f248bef7 [张宇翔] Merge remote-tracking branch 'upstream/master'
5d3b9afb [张宇翔] Code optimization
db3a0b6f [张宇翔] Format changes
08d6d1fc [张宇翔] Modify some annotations
f62f00e9 [张宇翔] Event Tracking: For job
85025193 [张宇翔] Merge branch 'branch-1.2_spark-monitor' into branch-1.2_add-jobinfo
3c8d9af1 [张宇翔] 1. change directory structure 2. code optimizing
b4290bfa [张宇翔] Event tracking: For jobInfo
fe1f7cce [张宇翔] Change directory structure
5ffb54f3 [张宇翔] Add kyuubi-spark-monitor module for nightly.yml
71d33b9a [张宇翔] Add some comment
6b87dff0 [张宇翔] 1. Remove some unused code 2. Add some comment
f364d433 [张宇翔] Format change: Add newline in KyuubiStatementInfo
8bddb202 [张宇翔] 1. Remove the relationship between executionId and operationId 2. Get each state by the function: setState 3. Get this statement's physicalPlan in ExecuteStatement 4. Add sparkUser item 5. Remove java code
f43b3c8f [张宇翔] merge master branch
55522613 [张宇翔] Merge branch 'master' into branch-1.2_spark-monitor
6f0be547 [张宇翔] format change
bbfba274 [张宇翔] Even tracking
5a62f586 [张宇翔] remove some unused conf
b26345ac [张宇翔] Merge master branch and resolve conflict
fdc12bbe [张宇翔] Event tracking: for kStatement 1. Store the relationship between executionId and operationId 2. Store the relationship between operationId and statement 3. Store the relationship between executionId and physicalPlan 4. Store each state and its happen time for this statement: initialized, running, finished
d1676268 [张宇翔] event tracking: for SQLExecutionStart

Authored-by: 张宇翔 <zhang1002@126.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-07-19 18:15:15 +08:00
Min Zhao
3100e8c172
[KYUUBI #749] fix wrong hint in test for update doc
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
fix wrong hint in test for update doc

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #827 from zhaomin1423/kyuubi_749.

Closes #749

a2b10247 [Min Zhao] fix wrong hint in test for update doc

Authored-by: Min Zhao <zhaomin1423@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-07-18 21:44:04 +08:00
Kent Yao
d5be0a4005
[KYUUBI #283] Add a sync client to ensure all requests sent to engine within a session are in order
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Add a sync client to ensure all requests sent to engine within a session are in order

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #819 from yaooqinn/283.

Closes #283

16cb8632 [Kent Yao] [KYUUBI #283] Add a sync client to ensure all requests in order
8bb2c81f [Kent Yao] [KYUUBI #283] Add a sync client to ensure all request in order
5c73fd0b [Kent Yao] [KYUUBI #283] Add a sync client to ensure all request in order

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-07-18 20:59:40 +08:00
yanghua
129521c9a9
[KYUUBI #820] [KYUUBI 817] Fix KyuubiDriverSuite UT failure because of NPE
### _Why are the changes needed?_

See issue #817 , when we pull the project and do not invoke any build action, e.g. `mvn package` or `mvn generate-resources`, we would miss the `kyuubi-version-info.properties` file, it caused the NPE.

As discussed under issue #817 , this PR tries to check it and enhance the exception message to make it more friendly to newbie developers.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

- [ ] Verified in the local env and no need to add test cases.

Closes #820 from yanghua/KYUUBI-817.

Closes #820

9464d07b [yanghua] [KYUUBI 817] Fix KyuubiDriverSuite UT failure because of missing kyuubi-version-info.properties

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-07-17 11:56:12 +08:00
wuchunfu
0798a07612
[KYUUBI #816] [KYUUBI 813] Fix requestTimeoutUnit in FrontendService
Fix requestTimeoutUnit in FrontendService #813

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #816 from wuchunfu/813.

Closes #816

d15797ca [wuchunfu] [KYUUBI 813] Fix requestTimeoutUnit in FrontendService

Authored-by: wuchunfu <319355703@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
(cherry picked from commit 0a87a254f7)
Signed-off-by: Kent Yao <yao@apache.org>
2021-07-16 13:13:36 +08:00
Kent Yao
a13b770f85
Revert "[KYUUBI #816] [KYUUBI 813] Fix requestTimeoutUnit in FrontendService"
This reverts commit 0a87a254f7.
2021-07-16 12:47:20 +08:00
wuchunfu
0a87a254f7
[KYUUBI #816] [KYUUBI 813] Fix requestTimeoutUnit in FrontendService
Fix requestTimeoutUnit in FrontendService #813

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #816 from wuchunfu/813.

Closes #816

d15797ca [wuchunfu] [KYUUBI 813] Fix requestTimeoutUnit in FrontendService

Authored-by: wuchunfu <319355703@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-07-16 12:45:57 +08:00
Kent Yao
89e5650269
Find local InetAddress (#801)
* Find local InetAddress

* Find local InetAddress

* Find local InetAddress

* Find local InetAddress
2021-07-15 13:27:48 +08:00
wForget
9654a936d2
[KYUUBI #751] Support single session mode
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

#751

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #752 from wForget/dev.

Closes #751

ae9dc873 [wForget] [KYUUBI #751] Support single session mode

Authored-by: wForget <643348094@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-07-06 19:09:20 +08:00
Kent Yao
ccce2d2792
[KYUUBI #761] Reduce explicit hang for test
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

1. reduce one server bootstrap
2. remove explicitly hanging a test for 1 min long to wait for an unnecessary message

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #761 from yaooqinn/test2.

Closes #761

1d5c8cb4 [Kent Yao] style
e877348a [Kent Yao] simpler
7ade8ead [Kent Yao] simpler
46a11b8c [Kent Yao] Reduce explicit hang for test

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-07-06 13:23:45 +08:00