Commit Graph

1540 Commits

Author SHA1 Message Date
yanghua
35ea7505a7
[KYUUBI #1774] Export FLINK_HOME in load-kyuubi-env script
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1793 from yanghua/KYUUBI-1774.

Closes #1774

178a6289 [yanghua] minor fix
ba7cc202 [yanghua] reorder code
7f15ffbb [yanghua] refactor code
85aa1856 [yanghua] [KYUUBI #1774] Export FLINK_HOME in load-kyuubi-env script

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-01-18 17:55:52 +08:00
Wang Zhen
916f81abaf
[KYUUBI #1775] Add kyuubi.server.name configuration
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Add `kyuubi.server.name` configuration. #1775

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [X] Add screenshots for manual tests if appropriate

![image](https://user-images.githubusercontent.com/17894939/149863381-0f71c33a-f2b2-48ef-a53f-1f48ce35fc7f.png)

![image](https://user-images.githubusercontent.com/17894939/149863308-677e54cd-d766-47e5-a284-7b2a475da82d.png)

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1789 from wForget/KYUUBI-1775.

Closes #1775

8b895044 [Wang Zhen] [KYUUBI-1775] Add kyuubi.server.name configuration

Authored-by: Wang Zhen <wangzhen07@qiyi.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-01-18 12:06:41 +08:00
Kent Yao
3aba64d9bb
[KYUUBI #1785] Stop Discovery services properly
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Fix some issues:

1. EngineServiceDiscovery calls deregister twice
2. When LOST zk-server, the delay for gracefully stop shall be calculated via retry policy not retrywait * maxWaitMs
     - engine default using n time retry policy, we can avoid unnecessary cache for engines when the whole test file finished and server stopped
     - otherwise, we are very likey to hit the GA memory limits
```
     KyuubiOperationPerGroupSuite:
./build/mvn: line 99:  1725 Killed                  ${MVN_BIN} $MAVEN_CLI_OPTS "$"
Error: Process completed with exit code 137.
```
3. When LOST, we shall skip connection retries, closing znode & closingclient

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1785 from yaooqinn/stop.

Closes #1785

afdd4b47 [Kent Yao] Stop Discovery services properly
5efa493a [Kent Yao] Stop Discovery services properly
45f9312e [Kent Yao] Stop Discovery services properly
d899012e [Kent Yao] Stop Discovery services properly
925ff3e9 [Kent Yao] Stop Discovery services properly
1f766116 [Kent Yao] Stop Discovery services properly
0d25f6a1 [Kent Yao] Stop Discovery services properly
ff833751 [Kent Yao] Stop Discovery services properly
55d64c02 [Kent Yao] Stop Discovery services properly

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-01-18 10:41:57 +08:00
SteNicholas
012fc75cad
[KYUUBI #1643][FOLLOWUP] Fix Flink GetFunctions result schema
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Implement GetFunctions operation.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1773 from SteNicholas/KYUUBI-1643.

Closes #1643

498c7f99 [SteNicholas] [KYUUBI #1643] Implement GetFunctions operation

Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-01-18 09:54:06 +08:00
Kent Yao
7ca1665bfc
[KYUUBI #1788] Update greetings.yml
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1788 from yaooqinn/yaooqinn-patch-2.

Closes #1788

4af27b0b [Kent Yao] Update greetings.yml

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-17 21:27:05 +08:00
Fu Chen
250d2f17b6
[KYUUBI #1776] Improve EnginePage
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Improve the UI page of `Kyuubi Query Engine`.

Currently, the column named `Query Execution` in tab `SQL Statistics` is not readable

1. Replace column `Query Execution` with `Query Details` and point it to the Spark SQL detail page
2. Add a new column `Failure Reason` to display an exception message when the query fails.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

![截屏2022-01-17 下午2 53 43](https://user-images.githubusercontent.com/8537877/149723476-7dd1aaab-b141-4ca3-a824-cbd393378dd8.png)

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1776 from cfmcgrady/ui.

Closes #1776

efab75a7 [Fu Chen] trigger GitHub actions
5b7001b2 [Fu Chen] fix ut
d9ccff89 [Fu Chen] fix style
717b916d [Fu Chen] Improve EnginePage

Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-01-17 18:40:32 +08:00
SteNicholas
7dcddafa6c
[KYUUBI #1648] Implement Flink GetTypeInfo operation
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Implement GetTypeInfo operation.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1777 from SteNicholas/KYUUBI-1648.

Closes #1648

1c7c0417 [SteNicholas] [KYUUBI #1648] Implement GetTypeInfo operation
abdba44c [SteNicholas] [KYUUBI #1648] Implement GetTypeInfo operation

Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-01-17 18:14:34 +08:00
zhenjiaguo
61e6022162
[KYUUBI #1771] Change rate unit in JsonReporterService
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Change rate unit in `JsonReporterService`.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate
A part of the report JSON file  content  shows like this:
```
    "kyuubi.backend_service.close_operation" : {
      "count" : 2196,
      "max" : 211.072349,
      "mean" : 1.2610130266188537,
      "min" : 0.172128,
      "p50" : 0.492044,
      "p75" : 0.718252,
      "p95" : 1.6039109999999999,
      "p98" : 5.8341069999999995,
      "p99" : 9.43642,
      "p999" : 211.072349,
      "stddev" : 10.214272758857668,
      "m15_rate" : 2.405930386704415,
      "m1_rate" : 21.71755165233277,
      "m5_rate" : 6.395622468399835,
      "mean_rate" : 15.886573053353114,
      "duration_units" : "milliseconds",
      "rate_units" : "calls/second"
    },
    "kyuubi.backend_service.execute_statement" : {
      "count" : 2207,
      "max" : 12.308489999999999,
      "mean" : 1.0070540441064393,
      "min" : 0.255337,
      "p50" : 0.8148869999999999,
      "p75" : 1.133641,
      "p95" : 1.794405,
      "p98" : 2.698518,
      "p99" : 5.280635999999999,
      "p999" : 11.317784999999999,
      "stddev" : 0.8658011924327101,
      "m15_rate" : 2.4059209510427615,
      "m1_rate" : 21.719940889226297,
      "m5_rate" : 6.3955828312258785,
      "mean_rate" : 15.928117712442724,
      "duration_units" : "milliseconds",
      "rate_units" : "calls/second"
    },
```
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1772 from zhenjiaguo/change_rate_unit.

Closes #1771

87be9a7b [zhenjiaguo] change rate unit

Authored-by: zhenjiaguo <zhenjiaguo@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-17 10:14:13 +08:00
zhenjiaguo
fb0c3a9794
[KYUUBI #1696] Add fetch logs and results rows rate
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Add fetch results and logs rows rate metric.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1707 from zhenjiaguo/add_fetch_logs_and_result.

Closes #1696

3b7d1fca [zhenjiaguo] Merge branch 'add_fetch_logs_and_result' of github.com:zhenjiaguo/kyuubi into add_fetch_logs_and_result
653e3b00 [zhenjiaguo] comment
e93d595a [zhenjiaguo] rerun action
6db7a768 [zhenjiaguo] add rowset not null judgment
eff57dec [zhenjiaguo] change fetch name
5543b92a [zhenjiaguo] add fetch logs and results rate
b99f5a69 [zhenjiaguo] comment
3703e350 [zhenjiaguo] rerun action
082f41a6 [zhenjiaguo] add rowset not null judgment
1ca562bc [zhenjiaguo] change fetch name
c6fef427 [zhenjiaguo] add fetch logs and results rate

Authored-by: zhenjiaguo <zhenjiaguo@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-17 10:12:18 +08:00
Paul Lin
4f2696d300
[KYUUBI #1704] Complete Flink-Hive data type mapping
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Currently, only a subset of Flink data types is supported. Queries that return results with unsupported data types would fail with Thrift TTransportExecption. We should fix this by completing Flink data type to Hive data type mapping.

This is a sub-task of KPIP-2 #1322.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1770 from link3280/feature/KYUUBI-1704.

Closes #1704

2d53a3b1 [Paul Lin] [KYUUBI #1704] Complete Flink-Hive data type mapping

Authored-by: Paul Lin <paullin3280@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-17 10:10:50 +08:00
SteNicholas
cab0762c6c
[KYUUBI #1645] Implement Flink engine GetSchemas operation
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Implement GetSchemas operation.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1768 from SteNicholas/KYUUBI-1645.

Closes #1645

7ff66db9 [SteNicholas] [KYUUBI apache#1645] Implement GetSchemas operation

Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-17 10:07:22 +08:00
Fei Wang
36b95d333f [KYUUBI #1714] Add executeScala api for KyuubiStatement
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Add executeScala api for KyuubiStatement, so that user can invoke this method to execute scala code directly.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1714 from turboFei/execute_scala.

Closes #1714

a62616df [Fei Wang] Add executeScala api for KyuubiStatement

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-01-16 20:23:51 +08:00
Cheng Pan
84839e0279
[KYUUBI #1756] Clean up code of FlinkOperation
### _Why are the changes needed?_

Minor code cleanup.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1756 from pan3793/cleanup.

Closes #1756

319a9859 [Cheng Pan] nit
57804ea0 [Cheng Pan] nit
1f8db1e5 [Cheng Pan] Clean up code of FlinkOperation

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-01-16 18:15:30 +08:00
Kent Yao
783fc16d3a
[KYUUBI #1766] [TEST] Remove SessionManagerSuite to reduce duplicated tests
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

SessionManagerSuite re-runs all tests in the super class, it's unnecessary.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1766 from yaooqinn/test2.

Closes #1766

28250732 [Kent Yao] [TEST] Remove SessionManagerSuite to reduce duplicated tests

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-01-15 21:50:02 +08:00
Kent Yao
38814691b2
[KYUUBI #1763] [INFRA] Remove scala installation inbuild/mvn and using warn level for the ci
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

1. remove the unnecessary scala installation
2. maven compiler log to warn level to filter a lot of annoying message

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1763 from yaooqinn/ci2.

Closes #1763

c0f11145 [Kent Yao] fix
dfd75dd7 [Kent Yao] fix
01fcc3f1 [Kent Yao] fix
94c62e46 [Kent Yao] Merge branch 'master' of github.com:apache/incubator-kyuubi into ci2
aab62e62 [Kent Yao] fix
b38b52cc [Kent Yao] ci
51cc7db8 [Kent Yao] nit

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-14 17:30:58 +08:00
SteNicholas
ce42d34ebb
[KYUUBI #1643] Implement GetFunctions operation
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Implement GetFunctions operation.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1749 from SteNicholas/KYUUBI-1643.

Closes #1643

442f4f1a [SteNicholas] [KYUUBI #1643] Implement GetFunctions operation
efc380ef [SteNicholas] [KYUUBI #1643] Implement GetFunctions operation
af43756b [SteNicholas] [KYUUBI #1643] Implement GetFunctions operation

Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-01-14 17:08:19 +08:00
Kent Yao
b91854e33b
[KYUUBI #1761] [SubTask][THTTP] Make abstract TFrontendService for thrift http protocol support
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

This PR mainly focus on refactoring
1. Use Apache Thrift Naming style for thrift-based classes which is `TXyzAbc`, e.g. `TFrontendService`
2. Created `TFrontendService` and move the `TCLIService.Iface` APIs and other common fields in `TFrontendService`, later we will use it to implement the `KyuubiTHttpFrontendService` in the server module
3. introduce a new conf also for server FEs to be able to pick IP first

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1761 from yaooqinn/http1.

Closes #1761

98b4afdd [Kent Yao] address comment
3138af29 [Kent Yao] [WIP] Thrift Http protocol support 1
a06026af [Kent Yao] [WIP] Thrift Http protocol support 1
efa899fa [Kent Yao] [WIP] Thrift Http protocol support 1
d1403e30 [Kent Yao] [WIP] Thrift Http protocol support 1
d114efd4 [Kent Yao] [WIP] Thrift Http protocol support 1
86e30c47 [Kent Yao] [WIP] Thrift Http protocol support 1
8117d12d [Kent Yao] [WIP] Thrift Http protocol support 1
62194323 [Kent Yao] [WIP] Thrift Http protocol support 1

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-14 17:03:44 +08:00
yanghua
e032a6218f
[KYUUBI #1757] Extract workingDir from engine specific process builde…
…r into ProcBuilder

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1758 from yanghua/KYUUBI-1757.

Closes #1757

14d3673c [yanghua] Fix CI issue
d6696919 [yanghua] [KYUUBI #1757] Extract workingDir from engine specific process builder into ProcBuilder

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-14 12:46:15 +08:00
hongdongdong
11556034e3
[KYUUBI #1510] Add TRINO engine type and TrinoProcessBuilder
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Add TRINO engine type and TrinoProcessBuilder

### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1695 from hddong/add-trino-process-builder.

Closes #1510

394b853e [hongdongdong] fix
044ca8cc [hongdongdong] fix
568b1b8e [hongdongdong] use java options to pass args
fbe7e60d [hongdongdong] [KYUUBI #1510] Add TRINO engine type and TrinoProcessBuilder

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-14 12:43:16 +08:00
Cheng Pan
9709832706
[KYUUBI #1748] bin/kyuubi-zk-cli should has execute permission
### _Why are the changes needed?_

`bin/kyuubi-zk-cli` should has execute permission

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1748 from pan3793/execute.

Closes #1748

cba11d02 [Cheng Pan] bin/kyuubi-zk-cli should has execute permission

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-14 12:40:24 +08:00
simon
f5205c6bbb
[KYUUBI #1759] Remove duplicate close operation api
### _Why are the changes needed?_
Remove duplicate close operation api, closes #1759

 #1445 is duplicate with #1517 where action='close'.

```
/${version}/operations/${operation_identifier}
desc: apply an action for an operation based on a given session identifier and operation identifier
method: PUT
params:
action: "CANCEL" or "ADD" or "CLOSE"
returns: status code

/${version}/sessions/${session_identifier}/operations/${operation_identifier}
desc: remove operation based on a given session identifier and operation identifier
method: DELETE
params: none
returns: an instance of OperationHandle
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1760 from simon824/closeoperation.

Closes #1759

9af70a07 [simon] closeoperation

Authored-by: simon <zhangshiming@cvte.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-14 12:39:39 +08:00
zhouyifan279
c17dd9d490
[KYUUBI #1731][FEATURE][FOLLOWUP] Use validated HadoopFs URI in HadoopFsDelegationTokenProvider#obtainDelegationTokens
### _Why are the changes needed?_

If core-site.xml contains a FileSystem uri, whose FileSystem implementation class is not present, following error will appear repeatedly in Kyuubi log after any user connects to Kyuubi and executed any SQL:

```
22/01/12 19:45:08 WARN credentials.HadoopFsDelegationTokenProvider: Failed to get Hadoop FileSystem instance by URI: alluxio://localhost
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class alluxio.hadoop.FileSystem not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2667)
	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3431)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466)
	at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
	at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider$.$anonfun$hadoopFSsToAccess$2(HadoopFsDelegationTokenProvider.scala:100)
	at scala.util.Try$.apply(Try.scala:213)
	at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider$.$anonfun$hadoopFSsToAccess$1(HadoopFsDelegationTokenProvider.scala:100)
	at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
	at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
	at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider$.hadoopFSsToAccess(HadoopFsDelegationTokenProvider.scala:99)
	at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.$anonfun$obtainDelegationTokens$1(HadoopFsDelegationTokenProvider.scala:62)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider$$anon$1.run(HadoopFsDelegationTokenProvider.scala:125)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
	at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider$.doAsProxyUser(HadoopFsDelegationTokenProvider.scala:124)
	at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.obtainDelegationTokens(HadoopFsDelegationTokenProvider.scala:60)
	at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.$anonfun$run$1(HadoopCredentialsManager.scala:210)
	at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.$anonfun$run$1$adapted(HadoopCredentialsManager.scala:210)
	at scala.collection.Iterator.foreach(Iterator.scala:943)
	at scala.collection.Iterator.foreach$(Iterator.scala:943)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
	at scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:214)
	at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.run(HadoopCredentialsManager.scala:210)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: Class alluxio.hadoop.FileSystem not found
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2571)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2665)
	... 40 more

```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1739 from zhouyifan279/1731.

Closes #1731

387036d8 [zhouyifan279] [KYUUBI #1731][FEATURE] Kyuubi server should not fail to start when Hadoop FileSystem class not found
6d80b8e9 [zhouyifan279] [KYUUBI #1731][FEATURE] Kyuubi server should not fail to start when Hadoop FileSystem class not found

Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-01-13 20:23:02 +08:00
ulysses-you
a533b2fa37
[KYUUBI #1738] Unify kubernetes integration test
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
- Move kubernetes integration test into the new module `integration-test`
- move the kubernetes profile  into `integration-test` module since it is only used in test

### _How was this patch tested?_
Pass CI

Closes #1738 from ulysses-you/unify-it.

Closes #1738

2582823a [ulysses-you] Unify kubernetes integration test

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-01-13 17:29:30 +08:00
mask
dfd9f9438e
[KYUUBI #1660] Add restart option for bin/kyuubi
- Add restart option for bin/kyuubi

This closes #1660

Closes #1752 from Narcasserun/shell_restart.

Closes #1660

6e86f171 [mask] [KYUUBI #1660] Add restart option for bin/kyuubi
15203a06 [mask] [KYUUBI #1660] Add restart option for bin/kyuubi

Authored-by: mask <Narcasserun@hotmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-01-13 16:58:53 +08:00
yanghua
37e17f206f
[KYUUBI #1746] Move sparkEngineMajorMinorVersion into SparkQueryTests to make HiveJDBCTestHelper spark free
…

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1747 from yanghua/KYUUBI-1746.

Closes #1746

f10fc378 [yanghua] [KYUUBI #1746] Move sparkEngineMajorMinorVersion into SparkQueryTests to make HiveJDBCTestHelper spark free

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-13 16:11:05 +08:00
Cheng Pan
01f0ea8609
[KYUUBI #1743] Fix parallelism of DataGenerator and other enhancements
### _Why are the changes needed?_

The parallelism of DataGenerator always is `spark.sparkContext.defaultParallelism`, it does not make sense for generating large scale data.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1743 from pan3793/tpcds.

Closes #1743

62f7c866 [Cheng Pan] nit
fdcf8329 [Cheng Pan] nit
a52ff489 [Cheng Pan] Fix parallelism of DataGenerator and other enhancements

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-01-13 11:41:22 +08:00
hongdongdong
ab4184dbe3
[KYUUBI #1740] Use testcontainers scalatest in trino module
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Use testcontainers scalatest instead testcontainers

### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1741 from hddong/use-testcontainers-scala-trino.

Closes #1740

3a12747f [hongdongdong] [KYUUBI #1740] Use testcontainers scalatest in trino module

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-01-13 11:39:18 +08:00
hongdongdong
97f2e0e181
[KYUUBI #1745] [KYUBBI #1744] Fix trino client flaky test
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
```scala
TrinoStatementSuite:
- test query *** FAILED ***
  resultSet.toIterator.hasNext was false (TrinoStatementSuite.scala:32)
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1745 from hddong/fix-trino-flasky-test.

Closes #1745

5368a8ec [hongdongdong] [KYUBBI #1744] Fix trino client flaky test

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-13 11:26:27 +08:00
zhouyifan279
a550ac705b
[KYUUBI #1737] [WIP][KYUUBI #1731][FEATURE] Kyuubi server should not fail to start when Hadoop FileSystem class not found
### _Why are the changes needed?_
Kyuubi server fails to start when Hadoop FileSystem class not found.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1737 from zhouyifan279/1731.

Closes #1737

Closes #1731

477ca37d [zhouyifan279] [KYUUBI #1731][FEATURE] Kyuubi server should not fail to start when Hadoop FileSystem class not found

Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-01-12 16:37:18 +08:00
Kent Yao
c1a6cd587c
[KYUUBI #1716] Unify Hive deps for server and client
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

This PR make the hive deps be consistent between server and clients for a single release, we shall not play client-server compatibility tricky in one release

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1716 from yaooqinn/hive.

Closes #1716

2eb3bdf7 [Kent Yao] Merge branch 'master' into hive
c1bf6369 [Kent Yao] dep
c7f0f26f [Kent Yao] dep
645d76ab [Kent Yao] fix test
3ac4f55c [Kent Yao] Merge branch 'master' into hive
f1ca475a [Kent Yao] delta test
01d30291 [Kent Yao] refine
1e93f33a [Kent Yao] Merge branch 'master' of github.com:apache/incubator-kyuubi into hive
5bd96313 [Kent Yao] Unify Hive deps for server and client

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-01-12 16:36:40 +08:00
Kent Yao
90e974637a
[KYUUBI #1729] Extract java lint workflow to a single yml and disable it in other jobs
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

quick fail on lint issue, no need to wait unit test

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1729 from yaooqinn/javalint.

Closes #1729

f5f1acd4 [Kent Yao] Merge branch 'master' into javalint
5b774f0b [Kent Yao] Merge branch 'master' into javalint
6cbd5c67 [Kent Yao] Merge branch 'master' into javalint
cc515a6a [Kent Yao] Extract java lint workflow to a single yml and disable it in other jobs

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-12 14:55:26 +08:00
Kent Yao
77f860407b
[KYUUBI #1735] [CI] Add current ip address, long hostname and short hostname to /etc/hosts
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Fixes reverse DNS lookup for the current hostname. This makes GitHub Actions flacky currently.

```scala
DDLTPCDSSuite:
*** RUN ABORTED ***
  java.net.BindException: Cannot assign requested address
  at sun.nio.ch.Net.bind0(Native Method)
  at sun.nio.ch.Net.bind(Net.java:461)
  at sun.nio.ch.Net.bind(Net.java:453)
  at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222)
  at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85)
  at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:78)
  at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:90)
  at org.apache.kyuubi.zookeeper.EmbeddedZookeeper.initialize(EmbeddedZookeeper.scala:53)
  at org.apache.kyuubi.WithKyuubiServer.beforeAll(WithKyuubiServer.scala:48)
  at org.apache.kyuubi.WithKyuubiServer.beforeAll$(WithKyuubiServer.scala:38)

```
See apache/pulsar#10232 and apache/pulsar#10233 for more details.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1735 from yaooqinn/ci.

Closes #1735

03e9f51f [Kent Yao] [CI] Add current ip address, long hostname and short hostname to /etc/hosts
7503db72 [Kent Yao] Merge branch 'master' into ci
b3b6d8bc [Kent Yao] [CI] Add current ip address, long hostname and short hostname to /etc/hosts

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-12 13:24:01 +08:00
Fu Chen
88463772c0
[KYUUBI #1736] Bump up GA actions/checkout to latest (v2.3.5)
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Fix Github Action, see: https://github.com/actions/checkout/issues/672#issuecomment-1010571414

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1736 from cfmcgrady/fix-ga.

Closes #1736

9a6216cd [Fu Chen] bump up GA actions/checkout to latest (v2.3.5)

Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-12 10:59:29 +08:00
yanghua
60ea99225b
[KYUUBI #1732] Reduce spark driver memory configuration to protect fa…
…iling from OOM

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1734 from yanghua/KYUUBI-1732.

Closes #1732

49f88df9 [yanghua] [KYUUBI #1732] Reduce spark driver memory configuration to protect failing from OOM

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-12 00:12:14 +08:00
Kent Yao
4f9922b3d8
[KYUUBI #1730] Use getHostName instead of getCanonicalHostName
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1730 from yaooqinn/zk.

Closes #1730

37bfe10c [Kent Yao] do not merge
fbf05df7 [Kent Yao] do not merge

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-11 22:24:21 +08:00
Kent Yao
4c682df05f
[KYUUBI #1672] Add VersionInfo in REST API
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1728 from yaooqinn/1672.

Closes #1672

511d8ceb [Kent Yao] [KYUUBI #1672] Add VersionInfo in REST API

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-11 21:16:59 +08:00
Kent Yao
b0d7dacc77
[KYUUBI #1726] Extract dep workflow to a single yml and disable it in other jobs
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1726 from yaooqinn/dep2.

Closes #1726

c214c067 [Kent Yao] Extract dep workflow to a single yml and disable it in other jobs

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-11 18:50:58 +08:00
sundapeng
89689c0221 [KYUUBI #1719] javax.activation:activation:jar duplicates
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1725 from SunDapeng1/issues-1719.

Closes #1719

d31d4e49 [sundapeng] [KYUUBI #1719] javax.activation:activation:jar duplicates

Authored-by: sundapeng <sundapeng@cmss.chinamobile.com>
Signed-off-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
2022-01-11 17:14:01 +08:00
Kent Yao
64118881aa
[KYUUBI #1720] Extract scalastyle workflow to a single yml and disable it in other jobs
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

quick fail on style issue and reduce ga overhead

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

```
=========dev/kyuubi-tpcds/target/scalastyle-output.xml=========
=========dev/kyuubi-extension-spark-3-1/target/scalastyle-output.xml=========
=========dev/kyuubi-extension-spark-common/target/scalastyle-output.xml=========
=========dev/kyuubi-extension-spark-3-2/target/scalastyle-output.xml=========
=========externals/kyuubi-download/target/scalastyle-output.xml=========
=========externals/kyuubi-trino-engine/target/scalastyle-output.xml=========
=========externals/kyuubi-spark-sql-engine/target/scalastyle-output.xml=========
=========externals/kyuubi-flink-sql-engine/target/scalastyle-output.xml=========
=========kubernetes/integration-tests/target/scalastyle-output.xml=========
=========kyuubi-assembly/target/scalastyle-output.xml=========
=========kyuubi-common/target/scalastyle-output.xml=========
  <error line="22" source="org.scalastyle.file.FileLineLengthChecker" severity="error" message="File line length exceeds 100 characters"/>
=========kyuubi-ctl/target/scalastyle-output.xml=========
=========kyuubi-ha/target/scalastyle-output.xml=========
=========kyuubi-metrics/target/scalastyle-output.xml=========
=========kyuubi-server/target/scalastyle-output.xml=========
=========kyuubi-zookeeper/target/scalastyle-output.xml=========
=========target/scalastyle-output.xml=========
=========tools/spark-block-cleaner/target/scalastyle-output.xml=========

```

Closes #1720 from yaooqinn/style.

Closes #1720

36ca3983 [Kent Yao] Extract scalastyle workflow to a single yml and disable it in other jobs
d83b8077 [Kent Yao] Extract scalastyle workflow to a single yml and disable it in other jobs
7366ca85 [Kent Yao] Extract scalastyle workflow to a single yml and disable it in other jobs
a0b44e26 [Kent Yao] Extract scalastyle workflow to a single yml and disable it in other jobs

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-11 15:12:23 +08:00
xifeng yang
d2f10e9da8
[KYUUBI #1646] support flink get tables
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

![image](https://user-images.githubusercontent.com/12961802/148889184-e56fc78b-fb03-42f5-b866-1d06b8c335a0.png)

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1713 from xifeng/kyuubi-1646.

Closes #1646

01522433 [xifeng yang] [KYUUBI #1646] support flink get tables

Authored-by: xifeng yang <xifeng.yang@hotmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-11 15:11:32 +08:00
Kent Yao
4f17f1b1af
[KYUUBI #1723] Create greetings.yml
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1723 from yaooqinn/yaooqinn-patch-1.

Closes #1723

ee49ffd6 [Kent Yao] Create greetings.yml

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-11 14:44:52 +08:00
Cheng Pan
e9a9a2fd71
[KYUUBI #1722] [NOTICE] Update NOTICE for 2022
### _Why are the changes needed?_

Welcome to 2022

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1722 from pan3793/notice.

Closes #1722

3a8730b8 [Cheng Pan] [NOTICE] Update NOTICE for 2022

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-11 14:11:04 +08:00
ulysses-you
33a2ff64b6
[KYUUBI #1721] Recover maven deplop plugin in all module
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Seems it no harmful push them to maven redundantly

### _How was this patch tested?_
Can see the 1.5.0-snapshot in
https://repository.apache.org/content/groups/snapshots/org/apache/kyuubi/kyuubi-extension-spark-common_2.12/

Closes #1721 from ulysses-you/maven-deploy.

Closes #1721

bcad87a7 [ulysses-you] recover

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-11 13:50:26 +08:00
yanghua
d63eed2749
[KYUUBI #1709] Introduce a IT module for Kyuubi and FlinkSQL Engine t…
…o decouple with kyuubi server

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1711 from yanghua/KYUUBI-1709.

Closes #1709

95466130 [yanghua] refactor code
89f9c572 [yanghua] Reorder module list in root pom file
b664a4bb [yanghua] Rename IT module names
6b7fa88f [yanghua] Rename IT module names
b133e8e9 [yanghua] Reformatted pom indent
38444416 [yanghua] [KYUUBI #1709] Introduce a IT module for Kyuubi and FlinkSQL Engine to decouple with kyuubi server

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-01-11 12:45:41 +08:00
Kent Yao
ee4473da3f
[KYUUBI #1718] Extract rat workflow to a single yml and disable it in other jobs
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1718 from yaooqinn/rat2.

Closes #1718

c19ba09d [Kent Yao] Extract rat workflow to a single yml and disable it in other jobs
7436964c [Kent Yao] Extract rat workflow to a single yml and disable it in other jobs
8e5f127e [Kent Yao] Extract rat workflow to a single yml and disable it in other jobs
83f08173 [Kent Yao] Extract rat workflow to a single yml and disable it in other jobs

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-11 12:15:22 +08:00
Kent Yao
32990b707f
[KYUUBI #1717] Extract rat configurations from pom to file
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Extract rat configurations from pom to file

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1717 from yaooqinn/rat.

Closes #1717

3f6836fe [Kent Yao] Extract rat configuation from pom to file
cb267b37 [Kent Yao] Extract rat configuation from pom to file

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-11 11:52:18 +08:00
Fei Wang
fbdbcb0276 [KYUUBI #1715] For kyuubi-beeline, using kyuubi-hive-jdbc instead of kyuubi-hive-jdbc-shaded
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

For kyuubi-beeline module, shaded kyuubi-hive-jdbc is not needed, using kyuubi-hive-jdbc to decrease the binary size.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1715 from turboFei/lighten_beeline.

Closes #1715

ebcde1df [Fei Wang] use kyuubi-hive-jdbc instead of kyuubi-hive-jdbc-shaded

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-01-11 00:29:53 +08:00
Kent Yao
a9d8fb2e7c
[KYUUBI #1712] Bump up maven.version to 3.8.4
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1712 from yaooqinn/mvnv.

Closes #1712

93793468 [Kent Yao] Bump up maven.version to 3.8.4

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2022-01-10 19:21:10 +08:00
Fei Wang
e0eeab07eb
[KYUUBI #1710] Support to specify OPERATION_LANGUAGE with TExecuteStatementReq confOverlay
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Now kyuubi supports SQL and SCALA language.

Now the way to switch OPERATION_LANAUAGE:

```
// for SCALA
spark.sql("set kyuubi.operation.language=SCALA")

// for SQL
set kyuubi.operation.language=SCALA
```

It is more user friendly that user can specify the OPERATION_LANGUAGE for each ExecuteStatement.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1710 from turboFei/conf_overlay.

Closes #1710

385264dc [Fei Wang] fix ut
58fbdbdf [Fei Wang] fix npe
2718fb48 [Fei Wang] Support to get OPERATION_LANGUAGE from execute statement confOverlay

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-01-10 17:21:25 +08:00
yanghua
58dd4ca71f
[KYUUBI #1647] Change GetCatalog returned schema key name to support JDBC specification
…

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1688 from yanghua/KYUUBI-1687.

Closes #1647

d6fda463 [yanghua] [KYUUBI #1647] Change GetCatalog returned schema key name to support JDBC specification

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-01-10 16:18:22 +08:00