master
305 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Kent Yao
|
1b48b1874f
|
[KYUUBI #1601] Align SparkStatementEvent to KyuubiOperationEvent
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Align SparkStatementEvent to KyuubiOperationEvent ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1601 from yaooqinn/soe. Closes #1601 090dee8f [Kent Yao] Merge branch 'master' of github.com:apache/incubator-kyuubi into soe a61ce3b7 [Kent Yao] Align SparkStatementEvent to KyuubiOperationEven efbbdf79 [Kent Yao] Align SparkStatementEvent to KyuubiOperationEven Authored-by: Kent Yao <yao@apache.org> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Fu Chen
|
2b8304d154
|
[KYUUBI #1591] Watchdog support for Spark-3.2
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> <!-- 2. move spark-3.1 `MarkAggregateOrderRule` to spark-common and rename to `MarkAggregateOrderBase` --> 1. move spark-3.1 `ForcedMaxOutputRowsRule` to spark-common and rename to `ForcedMaxOutputRowsBase` 2. handle `WithCTE` logical plan in spark-3.2 3. move spark-3.1 `MaxPartitionStrategy` to spark-common 4. add netsted cte unit test for `ForcedMaxOutputRowsRule` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1591 from cfmcgrady/watchdog-spark32. Closes #1591 5399a3f0 [Fu Chen] fix style 0ce83ba5 [Fu Chen] remove MarkAggregateOrderRule 364fc26e [Fu Chen] add license header 44726dee [Fu Chen] fix style 4847dbf3 [Fu Chen] watchdog support for spark-3.2 Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Wang Zhen
|
df1d9f3bb2
|
[KYUUBI #1577] Add DropIgnoreNonexistent Rule.
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Like the `hive.exec.drop.ignorenonexistent` configuration in hive, do not report an error if DROP DATABASE/TABLE/VIEW/Function/Partition specifies a non-existent database/table/view/function/partition. For details: #1577. ### _How was this patch tested?_ - [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1583 from wForget/KYUUBI-1577. Closes #1577 63382660 [Wang Zhen] delete useless comment adadf424 [Wang Zhen] add DropIgnoreNonexistent rule using injectPostHocResolutionRule 860b15d3 [Wang Zhen] add DropIgnoreNonexistent rule for spark 3.2 aad43cdb [Wang Zhen] adjust code ca1b9f11 [Wang Zhen] make some adjustments and update rules.md 804bf40e [Wang Zhen] [KYUUBI-1577] Add DropIgnoreNonexistent Rule. Authored-by: Wang Zhen <wangzhen07@qiyi.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
77a278707f
|
[KYUUBI #1559] Bump Jersey 2.34
### _Why are the changes needed?_ #1561 and align Spark 3.2.0 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1559 from pan3793/jersey. Closes #1559 a86d4d9f [Cheng Pan] Bump Jersey 2.34 Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
f841da4b4e
|
[KYUUBI #1558] Bump Jackson 2.12.5
### _Why are the changes needed?_ Because Spark 3.2, Trino 363, Jersey 2.34 (we use Jersey 2.30 now, will update to 2.34 soon) use Jackson 2.12.x ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1558 from pan3793/jackson. Closes #1558 7bdd65b0 [Cheng Pan] Bump Jackson 2.12.5 Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
6fa1327f4e
|
[KYUUBI #1560] Align all Jackson libs version
### _Why are the changes needed?_ Fix missing version of `jackson-module-jaxb-annotations` in `dependencyManagement` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1560 from pan3793/jackson-1. Closes #1560 49529d47 [Cheng Pan] Update deps b3610007 [Cheng Pan] fix 67044840 [Cheng Pan] Algin all Jackson libs version Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
ulysses-you
|
37a4e5c0da
|
[KYUUBI #1496] Support tpcds benchmark
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Support tpcds benchmark in `dev/kyuubi-tpcds` module. Add a `README.md` in `dev/kyuubi-tpcds` module to show how to use. The mian code is from [databricks-spark-sql-perf](https://github.com/databricks/spark-sql-perf) ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1496 from ulysses-you/tpcds-benchmark. Closes #1496 d4afe2d1 [ulysses-you] comment 54a146ef [ulysses-you] pom 91e71692 [ulysses-you] docs 20eadc49 [ulysses-you] benchmark Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Fu Chen
|
78c9cf3756
|
[KYUUBI #1443] MaxPartitionStrategy support file source based table.
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Fix [comment](https://github.com/apache/incubator-kyuubi/pull/1086#discussion_r707103617) ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1443 from cfmcgrady/max-partition-strategy. Closes #1443 e071ebc2 [Fu Chen] typo f7d4ea16 [Fu Chen] refactor 97a36016 [Fu Chen] truncate RootPaths c3fe93aa [Fu Chen] remove dpp suite 9075e5ba [Fu Chen] refactor 5a71f459 [Fu Chen] trigger GitHub actions 17df25d2 [Fu Chen] fix style 4dfe6a7e [Fu Chen] fix style 0b2df063 [Fu Chen] update docs 305c6e88 [Fu Chen] fix style 33b9b3d5 [Fu Chen] fix style 885e8ed7 [Fu Chen] data source support Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
fwang12
|
119e7f2c2c |
[KYUUBI #1441] [BUILD] Bump 1.5.0-SNAPSHOT
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1441 from turboFei/bump_1.5.0. Closes #1441 0c03c75d [fwang12] [BUILD] Bump 1.5.0-SNAPSHOT Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
zhouyifan279
|
ffe41631ce
|
[KYUUBI #1383] Leverage Scalafmt to auto format scala code
### _Why are the changes needed?_ #1383 As IntelliJ IDEA's `Optimize Imports` result does not meet rules in `scalastyle-config.xml`, currently developers have to arrange package imports manually. In this PR, we introduce a `.scalafmt.conf` file to solve the problem. It works as follows: 1. Set scala code style formatter to `scalafmt`. 2. Execute `Reformat Code` 3. IDEA picks up configurations in `.scalafmt.conf` and format both code lines and imports. We also formatted all historical scala codes. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1391 from zhouyifan279/1383. Closes #1383 e0696a9e [zhouyifan279] [KYUUBI #1383] Leverage Scalafmt to auto format scala code 18b8e229 [zhouyifan279] [KYUUBI #1383] Leverage Scalafmt to auto format scala code Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
sunfangbin
|
fb8bf49dbc
|
[KYUUBI #1413] Improve usage of zookeeper cli
### _Why are the changes needed?_ This change enabled the JLine support so that the `kyuubi-zookeeper-shell` command can be easier to use with history, prompt and completion, etc. Note that the `jline` version `0.9.94` is dependent on `zookeeper` version `3.4.14` which is a lower version. BTW, users can consider upgrading the `zookeeper` version for the sake of performance improvements and new features. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1413 from murong00/branch-1413. Closes #1413 2d1ec084 [sunfangbin] Rename shell script to zk-cli d09f64dc [sunfangbin] Fixes failed ci ce88357a [sunfangbin] Improve usage of zookeeper cli Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
09f04d6de8
|
[KYUUBI #1410] Add reformat shell script
### _Why are the changes needed?_ Following up #1403 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1410 from pan3793/reformat. Closes #1410 288fc1b5 [Cheng Pan] Add reformat shell script Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Fu Chen
|
a5ef94accc
|
[KYUUBI #1354] Expose OpenAPI Specifications
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. add a new endpoint `/openapi.json` which describes the entire restful API of Kyuubi. 2. add a static web page `/api/v1/swagger-ui`. see [swagger-ui](https://github.com/swagger-api/swagger-ui) ``` curl http://localhost:10099/openapi.json | jq . ``` ```json { "openapi": "3.0.1", "info": { "title": "Apache Kyuubi RESTful Api Documentation", "description": "Apache Kyuubi RESTful Api Documentation", "contact": { "name": "Apache Kyuubi Community", "url": "https://kyuubi.apache.org/issue_tracking.html", "email": "devkyuubi.apache.org" }, "license": { "name": "Apache 2.0", "url": "https://www.apache.org/licenses/LICENSE-2.0.html" }, "version": "1.4.0" }, "tags": [ { "name": "Session" } ], "paths": { "/api/v1/ping": { "get": { "operationId": "ping", "responses": { "default": { "description": "default response", "content": { "text/plain": { "schema": { "type": "string" } } } } } } }, "/api/v1/sessions/{sessionHandle}": { "get": { "tags": [ "Session" ], "operationId": "sessionInfo", "parameters": [ { "name": "sessionHandle", "in": "path", "required": true, "schema": { "type": "string" } } ], "responses": { "200": { "description": "get a session via session handle identifier", "content": { "application/json": {} } } } }, "delete": { "tags": [ "Session" ], "operationId": "closeSession", "parameters": [ { "name": "sessionHandle", "in": "path", "required": true, "schema": { "type": "string" } } ], "responses": { "200": { "description": "close a Session", "content": { "application/json": {} } } } } }, "/api/v1/sessions/execPool/statistic": { "get": { "tags": [ "Session" ], "operationId": "execPoolStatistic", "responses": { "200": { "description": "get some statistic info of sessions", "content": { "application/json": {} } } } } }, "/api/v1/sessions/{sessionHandle}/info/{infoType}": { "get": { "tags": [ "Session" ], "operationId": "getInfo", "parameters": [ { "name": "sessionHandle", "in": "path", "required": true, "schema": { "type": "string" } }, { "name": "infoType", "in": "path", "required": true, "schema": { "type": "integer", "format": "int32" } } ], "responses": { "200": { "description": "get a information detail via session handle identifier and a specific information type", "content": { "application/json": {} } } } } }, "/api/v1/sessions": { "get": { "tags": [ "Session" ], "operationId": "sessionInfoList", "responses": { "200": { "description": "get all the session list hosted in SessionManager", "content": { "application/json": {} } } } }, "post": { "tags": [ "Session" ], "operationId": "openSession", "requestBody": { "content": { "application/json": { "schema": { "$ref": "#/components/schemas/SessionOpenRequest" } } } }, "responses": { "200": { "description": "Open(create) a Session", "content": { "application/json": {} } } } } }, "/api/v1/sessions/count": { "get": { "tags": [ "Session" ], "operationId": "sessionCount", "responses": { "200": { "description": "get open session count", "content": { "application/json": {} } } } } }, "/api/v1/exception": { "get": { "operationId": "test", "responses": { "default": { "description": "default response", "content": { "text/plain": {} } } } } }, "/application.wadl/{path}": { "get": { "operationId": "getExternalGrammar", "parameters": [ { "name": "path", "in": "path", "required": true, "schema": { "type": "string" } } ], "responses": { "default": { "description": "default response", "content": { "application/xml": {} } } } } }, "/application.wadl": { "get": { "operationId": "getWadl", "responses": { "default": { "description": "default response", "content": { "application/vnd.sun.wadl+xml": {}, "application/xml": {} } } } } } }, "components": { "schemas": { "ExecPoolStatistic": { "required": [ "execPoolSize" ], "type": "object", "properties": { "execPoolSize": { "type": "integer", "format": "int32" }, "execPoolActiveCount": { "type": "integer", "format": "int32" } } }, "InfoDetail": { "required": [ "infoType" ], "type": "object", "properties": { "infoType": { "type": "string" }, "infoValue": { "type": "string" } } }, "HandleIdentifier": { "required": [ "publicId" ], "type": "object", "properties": { "publicId": { "type": "string", "format": "uuid" }, "secretId": { "type": "string", "format": "uuid" } } }, "SessionHandle": { "required": [ "identifier", "protocol" ], "type": "object", "properties": { "identifier": { "$ref": "#/components/schemas/HandleIdentifier" }, "protocol": { "type": "string", "enum": [ "HIVE_CLI_SERVICE_PROTOCOL_V1", "HIVE_CLI_SERVICE_PROTOCOL_V2", "HIVE_CLI_SERVICE_PROTOCOL_V3", "HIVE_CLI_SERVICE_PROTOCOL_V4", "HIVE_CLI_SERVICE_PROTOCOL_V5", "HIVE_CLI_SERVICE_PROTOCOL_V6", "HIVE_CLI_SERVICE_PROTOCOL_V7", "HIVE_CLI_SERVICE_PROTOCOL_V8", "HIVE_CLI_SERVICE_PROTOCOL_V9", "HIVE_CLI_SERVICE_PROTOCOL_V10" ] } } }, "MapStringString": { "type": "object", "additionalProperties": { "type": "string" } }, "SessionOpenRequest": { "required": [ "configs", "protocolVersion", "user" ], "type": "object", "properties": { "protocolVersion": { "type": "integer", "format": "int32" }, "user": { "type": "string" }, "password": { "type": "string" }, "ipAddr": { "type": "string" }, "configs": { "type": "object", "additionalProperties": { "type": "string" } } } }, "SessionOpenCount": { "required": [ "openSessionCount" ], "type": "object", "properties": { "openSessionCount": { "type": "integer", "format": "int32" } } }, "SessionDetail": { "required": [ "configs", "createTime", "user" ], "type": "object", "properties": { "user": { "type": "string" }, "ipAddr": { "type": "string" }, "createTime": { "type": "integer", "format": "int64" }, "sessionHandle": { "$ref": "#/components/schemas/SessionHandle" }, "lastAccessTime": { "type": "integer", "format": "int64" }, "lastIdleTime": { "type": "integer", "format": "int64" }, "noOperationTime": { "type": "integer", "format": "int64" }, "configs": { "type": "object", "additionalProperties": { "type": "string" } } } }, "SeqSessionOverview": { "type": "array", "items": { "$ref": "#/components/schemas/SessionOverview" } }, "SessionList": { "required": [ "sessionList" ], "type": "object", "properties": { "sessionList": { "type": "array", "items": { "$ref": "#/components/schemas/SessionOverview" } } } }, "SessionOverview": { "required": [ "createTime", "user" ], "type": "object", "properties": { "user": { "type": "string" }, "ipAddr": { "type": "string" }, "createTime": { "type": "integer", "format": "int64" }, "sessionHandle": { "$ref": "#/components/schemas/SessionHandle" } } } } } } ``` http://localhost:10099/api/v1/swagger-ui > Note that this request will redirect to http://localhost:10099/swagger-ui-redirected/index.html?url=http://localhost:10099/openapi.json  ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1363 from cfmcgrady/kyuubi-1354. Closes #1354 8895d7fc [Fu Chen] typo 8f5dea9a [Fu Chen] addressed comment 4d325a45 [Fu Chen] constant MediaType.APPLICATION_JSON 0d96951e [Fu Chen] RESTful Api -> REST API be7d02dc [Fu Chen] add LIABILITY-edl.txt 4ff1053e [Fu Chen] update dev/dependencyList b126f7e6 [Fu Chen] swagger-ui 4.1.0 a35c2f29 [Fu Chen] addressed comment 7bde0534 [Fu Chen] update LICENSE-binary 38397dc2 [Fu Chen] align jackson version 923f3e7e [Fu Chen] update dev/dependencyList 7d5338c8 [Fu Chen] remove kyuubi-server static files 5cb66b33 [Fu Chen] update dev/dependencyList e8a8c421 [Fu Chen] fix style 5b13f69c [Fu Chen] add license and remove .map files c04d7a75 [Fu Chen] rebase master 2388dae8 [Fu Chen] fix style 8dcbd3db [Fu Chen] add static resource 4b0d6d0a [Fu Chen] expose OpenAPI Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
ulysses-you
|
1dc755e450
|
[KYUUBI #1366] Add a todo in zorder that we can do rebalance partitions before local sort of zorder
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Rebalance partitions can resolve data skew issue before writing, so it should be good to do before we do the local sort of zorder, and also we can support dynamic partition insertion with zorder. However Spark only support this after 3.3.0, see [PR](https://github.com/apache/spark/pull/34542). So add a todo first. ### _How was this patch tested?_ not needed Closes #1366 from ulysses-you/zorder. Closes #1366 972229eb [ulysses-you] todo Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
c122a23a0d
|
[KYUUBI #1348] [DEPS] Add netty and mysql jdbc driver to kyuubi-server module
### _Why are the changes needed?_ This PR is one part of #1334. Add netty to `kyuubi-server` module compile scope, and mysql jdbc driver to test scope, it's required for implementing MySQL frontend protocol ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1348 from pan3793/mysql-fe-deps. Closes #1348 288ad9c0 [Cheng Pan] [DEPS] Add netty to kyuubi-server module Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
ulysses-you
|
8813c4cf92
|
[KYUUBI #1333] Improve zorder interleave perf
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Make interleave function more faster. The basic idea is from http://graphics.stanford.edu/~seander/bithacks.html#InterleaveTableObvious ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1333 from ulysses-you/zorder-perf. Closes #1333 a9411877 [ulysses-you] address comment bce82fc7 [ulysses-you] simplify 05f0b6e1 [ulysses-you] benchmark result f470a043 [ulysses-you] while a63cfae2 [ulysses-you] improve Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
ulysses-you
|
1003f5296c
|
[KYUUBI #1321] Add config to control if zorder using global sort
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add a new config `spark.sql.optimizer.zorderGlobalSort.enabled` to control if we do a global sort using zorder. This is a trade-off with data skew if the zorder columns has low cardinality. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1321 from ulysses-you/zorder-config. Closes #1321 422599e4 [ulysses-you] delete table c11e2f4f [ulysses-you] doc b984b537 [ulysses-you] Add config to control if zorder using global sort Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Fu Chen
|
1dd341fda2
|
[KYUUBI #1307] Fix zorder boolean column default value
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> As `true > false` and optimize command using nulls last as null ordering strategy, we should set true as the default value of boolean column instead of false ```sql select true > false ``` ``` +--------------+ |(true > false)| +--------------+ | true| +--------------+ ``` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1307 from cfmcgrady/boolean-zorder. Closes #1307 b4ca5a66 [Fu Chen] fix zorder boolean column default value Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Brian Yue
|
7665339244
|
[KYUUBI #460] Upgrade Hive dependency to 2.3.9
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Close #460. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1179 from byyue/feature-460. Closes #460 92eb71b8 [Brian Yue] Upgrade Hive dependency to 2.3.9 Authored-by: Brian Yue <code.byyue@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
589b23f578
|
[KYUUBI #1257] [BUILD] Reduce unnecessary maven profile
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> The changes based on https://github.com/apache/incubator-kyuubi/pull/1226#issuecomment-942979767 In this PR, I'm going to remove profiles `kyuubi-extension-spark-3-1` and `kyuubi-extension-spark-3-2`, and keep `spark-3.0`, `spark-3.1`, `spark-3.2`. After changes, when `spark-3.1` is active, set `spark.version` to 3.1.2 and enable `kyuubi-extension-spark-common` module and `kyuubi-extension-spark-3-1` module, `spark-3.2` does the same thing ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1257 from pan3793/pom. Closes #1257 1d871b6c [Cheng Pan] nit 874f30c1 [Cheng Pan] Address comments 00a32b25 [Cheng Pan] nit 445666ee [Cheng Pan] typo bd96097e [Cheng Pan] nit e85116cd [Cheng Pan] nit 98347c8c [Cheng Pan] Update doc 6615d7cc [Cheng Pan] Exclude HudiTest in cross test 087d3fde [Cheng Pan] Add spark 3.2 binary test ccff63c1 [Cheng Pan] update GA 23b81dcd [Cheng Pan] Address comments 3055c2a8 [Cheng Pan] nit 49a41e86 [Cheng Pan] nit 8d400798 [Cheng Pan] Reduce unnecessary maven profile ced8d987 [Cheng Pan] Reduce unnecessary maven profile Lead-authored-by: Cheng Pan <chengpan@apache.org> Co-authored-by: Cheng Pan <chengpan@ChengdeMac-mini.local> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
ulysses-you
|
f0f065b983
|
[KYUUBI #1268] Simplify extension code with Spark3.2 and Spark3.1 by unify zorder code
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Reduce file and code size to help maintain the extension. - move all zorder code into common module - add KyuubiSparkSQLCommonExtension in common module ### _How was this patch tested?_ Psss CI Closes #1268 from ulysses-you/simplify-extension. Closes #1268 0eeb7161 [ulysses-you] add link c4d35c5c [ulysses-you] simplify Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
d0d5fb6d4f
|
[KYUUBI #1260] Avoid calling javax.xml API for better JDK compatible
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> `javax.xml` API has been separated from JDK since JDK9, avoid calling it for better JDK compatible ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1260 from pan3793/javax. Closes #1260 c4f7d277 [Cheng Pan] Avoid calling javax.xml API for better JDK compatible Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
f03a1443a3
|
[KYUUBI #757] [DEPS] Bump Hadoop 3.3.1
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Spark 3.2.0 is out, which bundles Hadoop 3.3.1 shaded client in default. https://github.com/apache/spark/pull/30135 The test failed when using Hadoop 3.3.1 client connects to Yarn Mini Cluster 3.2.2 ``` Cause: java.lang.RuntimeException: org.apache.kyuubi.KyuubiSQLException:java.lang.ClassCastException: org.apache.hadoop.yarn.proto.YarnServiceProtos$GetClusterMetricsRequestProto cannot be cast to org.apache.hadoop.shaded.com.google.protobuf.Message at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:123) at com.sun.proxy.$Proxy12.getClusterMetrics(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterMetrics(ApplicationClientProtocolPBClientImpl.java:271) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy13.getClusterMetrics(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getYarnClusterMetrics(YarnClientImpl.java:638) at org.apache.spark.deploy.yarn.Client.$anonfun$submitApplication$1(Client.scala:179) at org.apache.spark.internal.Logging.logInfo(Logging.scala:57) at org.apache.spark.internal.Logging.logInfo$(Logging.scala:56) at org.apache.spark.deploy.yarn.Client.logInfo(Client.scala:65) at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:179) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62) at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:220) at org.apache.spark.SparkContext.<init>(SparkContext.scala:581) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690) at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943) at org.apache.kyuubi.engine.spark.SparkSQLEngine$.createSpark(SparkSQLEngine.scala:103) at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:155) at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955) at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:165) at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:163) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #757 from pan3793/hadoop-3.3. Closes #757 7ec9313e [Cheng Pan] [DEPS] Bump Hadoop 3.3.1 Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
senmiaoliu
|
f4094d6524
|
[KYUUBI #1085] Fix-Enforce maxOutputRows for Resolve View, Multi Insert and Distribute By
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
1. Should not add limit when multi insert tables.
```sql
FROM tmp_table
insert into tmp_table1 select * limit 2
insert into tmp_table2 select *
```
We should not add any limit when insert table
2. Should add limit when distribute by
```sql
select * from tmp_table distribute by a
```
3. Should not add limit when resolve view
```sql
SELECT * FROM
(select * from tmp_view1 union select * from tmp_view2)
ORDER BY a DESC
```
Like this, when the limit is added in resolve view, the complete data cannot be obtained, so the final result is incorrect
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes #1247 from lsm1/fix/add_limit.
Closes #1085
a02a99c0 [senmiaoliu] Fix-Enforce maxOutputRows for Resolve View, Multi Insert and Distribute By
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
|
||
|
ulysses-you
|
7626973e15
|
[KYUUBI #1245] Improve paddingTo8Byte perf
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add a new benchmark for `paddingTo8Byte` method Before this PR: ``` Java HotSpot(TM) 64-Bit Server VM 1.8.0_271-b09 on Mac OS X 10.16 Intel(R) Core(TM) i7-4770HQ CPU 2.20GHz 10000000 iterations paddingTo8Byte benchmark: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ---------------------------------------------------------------------------------------------------------------------------- 2 length benchmark 2112 2180 78 4.7 211.2 1.0X 16 length benchmark 454 459 6 22.0 45.4 4.6X ``` After this PR: ``` Java HotSpot(TM) 64-Bit Server VM 1.8.0_271-b09 on Mac OS X 10.16 Intel(R) Core(TM) i7-4770HQ CPU 2.20GHz 10000000 iterations paddingTo8Byte benchmark: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ---------------------------------------------------------------------------------------------------------------------------- 2 length benchmark 167 170 3 59.9 16.7 1.0X 16 length benchmark 162 164 3 61.7 16.2 1.0X ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1245 from ulysses-you/improve-zorder-perf. Closes #1245 487c6900 [ulysses-you] improve paddingTo8Byte perf Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
ulysses-you
|
7961b32166
|
[KYUUBI #1241] Build Spark extension fat jar with common module
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> see https://github.com/apache/incubator-kyuubi/issues/1241 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1242 from ulysses-you/KYUUBI-1241. Closes #1241 a8e25c37 [ulysses-you] address comment d25a3664 [ulysses-you] version 3c982938 [ulysses-you] Build Spark extension fat jar with common module Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
eb5749d34b
|
[KYUUBI #1238] [BUILD] Add back Scala stuff to root POM
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This partial revert [KYUUBI #1203], recover codecov ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1238 from pan3793/mvn. Closes #1238 b3ba87b2 [Cheng Pan] remove unnecessary changes c124340a [Cheng Pan] nit 39c436be [Cheng Pan] pom 28546cbd [Cheng Pan] CI f22e9c91 [Cheng Pan] nit 72e9542d [Cheng Pan] Rebase master 7cd1cba3 [Cheng Pan] [BUILD] Add back Scala stuff to root POM Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
ulysses-you
|
8a5134e322
|
[KYUUBI #1225] Add kyuubi-extension-spark-3-2 module
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> see https://github.com/apache/incubator-kyuubi/issues/1225 Note that: - this PR only supports `Zorder` and some common rules with Spark 3.2.0. The others `watchdog` and `sqlclassification` should be supported in future. - this PR dones't supprpt shade jar and we also need to update build/dist script in a new PR. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1226 from ulysses-you/extension-3.2. Closes #1225 384a0711 [ulysses-you] address comment 08629f20 [ulysses-you] address comment 15f15f6d [ulysses-you] Add kyuubi-extension-spark-3-2 module Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
8a8f49d987
|
[KYUUBI #1233] Add checkout_pr.sh script
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> A convenient tool for checkout PR to local branch. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1233 from pan3793/pr. Closes #1233 97489300 [Cheng Pan] Add checkout_pr.sh script Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
ulysses-you
|
36aab5b95f
|
[KYUUBI #1218] Z-order int and long should respect negative number
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Made a mistake at https://github.com/apache/incubator-kyuubi/pull/1192 that we can't reverse the short highest bit for int and int highest bit for long. Otherwise the negative number will get the wrong order. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1218 from ulysses-you/fix-zorder. Closes #1218 3f7f22d3 [ulysses-you] fix Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
ulysses-you
|
b26ef2ba9e
|
[KYUUBI #1216] Skip zorder if only requires one column
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Improve the perf. If user only requires one column as zorde columns, it has no meaning to wrap a zorder expression. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1216 from ulysses-you/improve-zorder. Closes #1216 9975fccd [ulysses-you] skip zorder if only requires one column Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
ulysses-you
|
6160b9de30
|
[KYUUBI #1214] Refine sql extension and avoid setup data for unused test
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> - Refine rule order - Speed up test ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1214 from ulysses-you/tests. Closes #1214 261271f0 [ulysses-you] test Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
h
|
fdff2b6240
|
[KYUUBI #1085][FOLLOWUP] Fix-Enforce maxOutputRows for aggregate with having statement
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Support `Union` case as below ``` SELECT * FROM t1 UNION [ALL] SELECT * FROM t2 ``` Support `Distinct` case as below ``` SELECT DISTINCT * FROM t1 ``` Fix The bug of watchdog with maxOutputRows happens in this situation as below Having and Sort ``` SELECT c1, COUNT(c2) AS cnt FROM t1 GROUP BY c1 HAVING cnt > 0 [ORDER BY c1, [c2 ...]] ``` It throws Exception as ``` org.apache.spark.sql.catalyst.plans.logical.GlobalLimit cannot be cast to org.apache.spark.sql.catalyst.plans.logical.Aggregate java.lang.ClassCastException: org.apache.spark.sql.catalyst.plans.logical.GlobalLimit cannot be cast to org.apache.spark.sql.catalyst.plans.logical.Aggregate at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$.resolvedAggregateFilter$1(Analyzer.scala:2451) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$.resolveFilterCondInAggregate(Analyzer.scala:2460) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$.resolveHaving(Analyzer.scala:2496) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$$anonfun$apply$21.applyOrElse(Analyzer.scala:2353) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$$anonfun$apply$21.applyOrElse(Analyzer.scala:2345) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUp$3(AnalysisHelper.scala:90) at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:74) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUp$1(AnalysisHelper.scala:90) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:221) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUp(AnalysisHelper.scala:86) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUp$(AnalysisHelper.scala:84) at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsUp(LogicalPlan.scala:29) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$.apply(Analyzer.scala:2345) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$.apply(Analyzer.scala:2344) at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:216) at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126) at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122) at scala.collection.immutable.List.foldLeft(List.scala:91) at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:213) at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1$adapted(RuleExecutor.scala:205) at scala.collection.immutable.List.foreach(List.scala:431) at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:205) at org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:196) at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:190) at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:155) at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$executeAndTrack$1(RuleExecutor.scala:183) at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:88) at org.apache.spark.sql.catalyst.rules.RuleExecutor.executeAndTrack(RuleExecutor.scala:183) at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$executeAndCheck$1(Analyzer.scala:174) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:228) at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:173) at org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:73) at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111) at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:143) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:143) at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:73) at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:71) at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:63) at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:98) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:96) at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:618) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613) at org.apache.spark.sql.test.SQLTestUtilsBase.$anonfun$sql$1(SQLTestUtils.scala:231) at org.apache.spark.sql.KyuubiExtensionSuite.$anonfun$new$55(KyuubiExtensionSuite.scala:1331) at org.apache.spark.sql.catalyst.plans.SQLHelper.withSQLConf(SQLHelper.scala:54) at org.apache.spark.sql.catalyst.plans.SQLHelper.withSQLConf$(SQLHelper.scala:38) at ``` Reference related issue: https://issues.apache.org/jira/browse/SPARK-31519 Spark SQL Ananlyzer transform aggregate with having to ``` Filter +- Aggregate ``` Solution: 1. Skip the aggregate with havingCondition 2. Match Filter for Adding Limit ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate <img width="1440" alt="截屏2021-09-21 下午8 35 16" src="https://user-images.githubusercontent.com/635169/134171308-2842f0d4-acfa-4817-a03c-a7ef5e38df12.png"> - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1129 from i7xh/fixAggWithHavingInMaxOutput. Closes #1085 7577f4d3 [h] update 5955c89e [h] update 384b2333 [h] fix issue 5b0af156 [h] update 46327fc6 [h] Fix issue 6119a039 [h] fix issue a7b87dd7 [h] Fix issue 2570444e [h] BugFix: Aggregate with having statement Authored-by: h <h@zhihu.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
9e6eb086b1
|
[KYUUBI #1203] [SUB-TASK][BUILD] Remove Scala stuff from kyuubi-hive-jdbc pom
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> #1131 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1203 from pan3793/pom. Closes #1203 8e6b8f9b [Cheng Pan] fix d31fd198 [Cheng Pan] fix 9da82843 [Cheng Pan] fix 6d150de9 [Cheng Pan] fix 9faf77e5 [Cheng Pan] Remove Scala stuff from kyuubi-hive-jdbc pom Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
ulysses-you
|
afb46ec152
|
[KYUUBI #1192] Refine zorder utils toByte
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Fix some issue: 1. int/long type miss update highest byte of byte array 2. short/int/long same value may have different byte array 3. the mask value of long is error. (1L << 31 instead of 1 << 31) 4. the default value of string should be Long.Max since we pad it to 8 bytes so we can keep the ordering of null value (null last) 5. boolean to byte doesn't need 4 bytes, 1 byte is enough ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1192 from ulysses-you/improve-zorder. Closes #1192 7a0cba28 [ulysses-you] byte 2271b427 [ulysses-you] refine b13d2652 [ulysses-you] refine zorder Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
191d30e2ac
|
[KYUUBI #1167][FOLLOWUP] Refine ZorderCoreBenchmark doc
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Minor change, update `ZorderCoreBenchmark` doc. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1172 from pan3793/t. Closes #1167 8372a3a7 [Cheng Pan] nit 1b6ee9da [Cheng Pan] typo bd1817a7 [Cheng Pan] [KYUUBI #1167][FOLLOWUP] Refine ZorderCoreBenchmark doc Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
ulysses-you
|
627d271fc6
|
[KYUUBI #1167] Add zorder core benchmark
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add zorder core benchmark, so we can get the perf number of zorder. This PR adds two column type `int` and `long` benchmark: - 2 int cols - 3 int cols - 4 int cols - 2 long cols - 3 long cols - 4 long cols ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1167 from ulysses-you/benchmark. Closes #1167 27ecf51f [ulysses-you] Add zorder core benchmark Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
zhouyifan279
|
114ec9a046
|
[KYUUBI #1009][FOLLOWUP] Kyuubi server failed to start when connecting to secured HMS
### _Why are the changes needed?_ Fix #1161 [Bug] Kyuubi server failed to start when connecting to a secured Hive metastore ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate  - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1162 from zhouyifan279/KYUUBI#1161. Closes #1009 f5b54818 [zhouyifan279] [KYUUBI #1161][Bug] Kyuubi server failed to start when connecting to a secured Hive metastore Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Min Zhao
|
d829bae526
|
[KYUUBI #939][FOLLOWUP][TEST] Add string column test for Z-Order
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Added string column test for zorder. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1157 from zhaomin1423/zorder_test. Closes #939 59543776 [Min Zhao] add string column test for zorder Authored-by: Min Zhao <zhaomin1423@163.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
ulysses-you
|
7a3545e748
|
[KYUUBI #1002][FOLLOWUP] Refine sql classification rule
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Follow up [KYUUBI #1002] [KYUUBI #1035] [KYUUBI #1037] - correct the config key; we should use `spark.` as the prefix - change the config to false by default - correct the class loader; thread context is not safe, spark and hive often change it - update the log and exception following is the failed error msg: ``` Caused by: java.io.FileNotFoundException: file:/Users/cathy/Desktop/tmp/spark-3.1.2-bin-hadoop2.7/jars/kyuubi-extension-spark-3-1_2.12-1.4.0-SNAPSHOT.jar!/sql-classification-default.json (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:916) at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2745) at org.apache.kyuubi.sql.sqlclassification.KyuubiGetSqlClassification$.<init>(KyuubiGetSqlClassification.scala:51) at org.apache.kyuubi.sql.sqlclassification.KyuubiGetSqlClassification$.<clinit>(KyuubiGetSqlClassification.scala) ... 59 more ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1140 from ulysses-you/refine. Closes #1002 f39ef2cf [ulysses-you] address comment 5cb94459 [ulysses-you] nit de9fd932 [ulysses-you] test 78e8c147 [ulysses-you] default 6f23c63e [ulysses-you] refine Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Simon
|
7b7d228fb4
|
[KYUUBI #1123][TEST] Split KyuubiExtensionSuite according to different extensions
split KyuubiExtensionSuite according to different extensions #1123 drop： `KyuubiSparkExtensionTest` add： - `FinalStageConfigIsolationSuite` - `InsertShuffleNodeBeforeJoinSuite` - `WatchDogSuite` - `RepartitionBeforeWriteSuite` - `SqlClassificationSuite` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1124 from simon824/extension-ut. Closes #1123 4b1ae4db [Simon] Update FinalStageConfigIsolationSuite.scala 2068a74d [Simon] Update SparkProcessBuilder.scala cd6aa7d2 [Simon] Merge branch 'apache:master' into extension-ut 1401d69c [simon] rename test to suite 98095823 [Simon] Rename WatchDogSuit.scala to WatchDogSuite.scala 0392d5b5 [Simon] Rename SqlClassificationSuit.scala to SqlClassificationSuite.scala a8776df9 [Simon] Rename RepartitionBeforeWriteSuit.scala to RepartitionBeforeWriteSuite.scala 4d06966c [Simon] Rename InsertShuffleNodeBeforeJoinSuit.scala to InsertShuffleNodeBeforeJoinSuite.scala 60b2932e [Simon] Rename FinalStageConfigIsolationSuit.scala to FinalStageConfigIsolationSuite.scala 175fb47f [Simon] Rename WatchDogTest.scala to WatchDogSuit.scala 755ad22c [Simon] Rename SqlClassificationTest.scala to SqlClassificationSuit.scala b36171b8 [Simon] Rename RepartitionBeforeWriteTest.scala to RepartitionBeforeWriteSuit.scala 74c110da [Simon] Rename InsertShuffleNodeBeforeJoinTest.scala to InsertShuffleNodeBeforeJoinSuit.scala 405d8b72 [Simon] Rename FinalStageConfigIsolationTest.scala to FinalStageConfigIsolationSuit.scala 7a158057 [simon] fix codestyle 86148f3e [simon] fix ut 291c93aa [simon] split KyuubiExtensionSuite to different extension ut class 2098ce89 [simon] split KyuubiExtensionSuite to different extension ut class 2aaed1c8 [simon] Merge remote-tracking branch 'upstream/master' into extension-ut Lead-authored-by: Simon <3656562@qq.com> Co-authored-by: simon <zhangshiming@cvte.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Fu Chen
|
208c74d197
|
[KYUUBI #1105][FOLLOWUP] Minor improve ZorderBytesUtils.defaultValue
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> fix [comment](https://github.com/apache/incubator-kyuubi/pull/1109#discussion_r711007709) ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1126 from cfmcgrady/zorder. Closes #1105 48e4d73a [Fu Chen] review a58c5e52 [Fu Chen] minor improve ZorderBytesUtils.defaultValue Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
h
|
cc3ca56ab2
|
[KYUUBI #1085] Add forcedMaxOutputRows rule for limitation to avoid huge output unexpectly
Add MaxOutputRows rule for output rows limitation to avoid huge output unexpectedly <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add <img width="1440" alt="截屏2021-09-12 下午12 19 28" src="https://user-images.githubusercontent.com/635169/132972063-b12937bb-807a-47bd-8d21-835d83031191.png"> '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> We support the PR feature with limitation that avoid huge output rows in user ad-hoc query unexpected，Generally ad-hoc query seems needle in a Haystack, user pick few computed result data in huge data from warehouse, we mainly used in below cases: - CASE 1: ``` SELECT [c1, c2, ...] ``` - CASE 2: ``` WITH CTE AS (...) SELECT [c1, c2, ...] FROM Express(CTE) ... ``` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate <img width="1440" alt="截屏2021-09-12 下午12 19 28" src="https://user-images.githubusercontent.com/635169/132972078-c4821135-0520-420d-9ab8-24e124f6c6c9.png"> - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1085 from i7xh/watchdogMaxOutputRows. Closes #1085 77aa6ff3 [h] Resolve issue: unify maxOutputRows and using unittest setupData 939d2556 [h] Resolve issue: well-format code 5b0383dd [h] Fix issue: Final Limit according to spark.sql.watchdog.forcedMaxOutputRows 7dcbb0a4 [h] Resolve issue: remove origin match from rule f4dff4cf [h] Resolve issue: update forcedMaxOutputRows config doc ae21c1ac [h] Resovled issue: Support Aggregate force limitation and Remove InsertIntoDataSourceDirCommand process a9d3640b [h] Resolved code review issue 01c87fd2 [h] Add MaxOutputRows rule for output rows limitation to avoid huge output rows of query unexpectedly Authored-by: h <h@zhihu.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Brian Yue
|
cbe5bee6ec
|
[KYUUBI #1112] Upgrade scala to 2.12.15
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Close #1112. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1120 from byyue/feature-1112. Closes #1112 9bb10858 [Brian Yue] [FEATURE] [Dependency] Upgrade scala to 2.14.15 Authored-by: Brian Yue <code.byyue@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
timothy65535
|
6983139487
|
[KYUUBI #1111] Remove duplicate keys
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Remove duplicate keys. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1111 from timothy65535/remove. Closes #1111 75b001e3 [timothy65535] Remove duplicate keys Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Fu Chen
|
7da30bd992
|
[KYUUBI #1105][FOLLOWUP][TEST] Refine KyuubiSparkSQLExtension suites
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> fix [comment](https://github.com/apache/incubator-kyuubi/pull/1109#discussion_r709838217) ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1118 from cfmcgrady/refine-extension-test. Closes #1105 a3f31e5d [Fu Chen] minor 853bf8f2 [Fu Chen] fix style 89f2338a [Fu Chen] minor d4d57ebd [Fu Chen] refine KyuubiSparkSQLExtension suites Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Fu Chen
|
5602c29b36
|
[KYUUBI #1105] Zorder codegen support
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. zorder codegen support 2. move all of the zorder tests to a separate file. 3. add more zorder unit tests. generate.java ```java /* 001 */ public java.lang.Object generate(Object[] references) { /* 002 */ return new SpecificUnsafeProjection(references); /* 003 */ } /* 004 */ /* 005 */ class SpecificUnsafeProjection extends org.apache.spark.sql.catalyst.expressions.UnsafeProjection { /* 006 */ /* 007 */ private Object[] references; /* 008 */ private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[] mutableStateArray_0 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[1]; /* 009 */ /* 010 */ public SpecificUnsafeProjection(Object[] references) { /* 011 */ this.references = references; /* 012 */ mutableStateArray_0[0] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(1, 32); /* 013 */ /* 014 */ } /* 015 */ /* 016 */ public void initialize(int partitionIndex) { /* 017 */ /* 018 */ } /* 019 */ /* 020 */ // Scala.Function1 need this /* 021 */ public java.lang.Object apply(java.lang.Object row) { /* 022 */ return apply((InternalRow) row); /* 023 */ } /* 024 */ /* 025 */ public UnsafeRow apply(InternalRow i) { /* 026 */ mutableStateArray_0[0].reset(); /* 027 */ /* 028 */ /* 029 */ /* 030 */ /* 031 */ byte[] value_0 = null; /* 032 */ byte[][] binaryArray_0 = new byte[2][]; /* 033 */ /* 034 */ boolean isNull_1 = i.isNullAt(0); /* 035 */ long value_1 = isNull_1 ? /* 036 */ -1L : (i.getLong(0)); /* 037 */ if (isNull_1) { /* 038 */ binaryArray_0[0] = (byte[]) ((byte[][]) references[0] /* defaultValues */)[0]; /* 039 */ } else { /* 040 */ binaryArray_0[0] = org.apache.kyuubi.sql.zorder.ZorderBytesUtils.toByte(value_1); /* 041 */ } /* 042 */ /* 043 */ /* 044 */ boolean isNull_2 = i.isNullAt(1); /* 045 */ long value_2 = isNull_2 ? /* 046 */ -1L : (i.getLong(1)); /* 047 */ if (isNull_2) { /* 048 */ binaryArray_0[1] = (byte[]) ((byte[][]) references[0] /* defaultValues */)[1]; /* 049 */ } else { /* 050 */ binaryArray_0[1] = org.apache.kyuubi.sql.zorder.ZorderBytesUtils.toByte(value_2); /* 051 */ } /* 052 */ /* 053 */ value_0 = org.apache.kyuubi.sql.zorder.ZorderBytesUtils.interleaveMultiByteArray(binaryArray_0); /* 054 */ mutableStateArray_0[0].write(0, value_0); /* 055 */ return (mutableStateArray_0[0].getRow()); /* 056 */ } /* 057 */ /* 058 */ /* 059 */ } ``` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1109 from cfmcgrady/zorder-codegen. Closes #1105 f06da25a [Fu Chen] update dependency 8e04dc4e [Fu Chen] JMap -> Array da00f86c [Fu Chen] remove Logging trait 266b65b3 [Fu Chen] fix style and revert Expression e15165f3 [Fu Chen] ev.isNull = FalseLiteral 0d77d9de [Fu Chen] review 0d8467f4 [Fu Chen] fix style 5ede5df9 [Fu Chen] refactor test 56e12172 [Fu Chen] clean up 88573102 [Fu Chen] update 5f3d1c02 [Fu Chen] zorder codegen support Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
ulysses-you
|
43390834b1
|
[KYUUBI #1100] Refine zorder expression and add test
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> - refine the zorder expression - improve coverage ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1100 from ulysses-you/zorder. Closes #1100 d07ad8ae [ulysses-you] value 5b160384 [ulysses-you] refine zorder expression Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
h
|
9dfbd2b830
|
[KYUUBI #1086] Add maxHivePartitions Strategy to avoid scanning excessive hive partitions on partitioned table
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> The PR support this Strategy to avoid scan huge hive partitions or unused necessary partition filter on partitioned table，It will abort the SQL query with thrown exception via scan hive table partitions exceed `spark.sql.watchdog.maxHivePartitions`, meanwhile show the related hive table partition struct to assist user to optimize sql according the given suggestion ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate <img width="1440" alt="截屏2021-09-12 下午2 30 20" src="https://user-images.githubusercontent.com/635169/132974627-7879ee2b-4624-4bb4-bec9-3ed0940c3986.png"> - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1086 from i7xh/watchdogMaxHivePartition. Closes #1086 d535b57e [h] Fixed: scala style newline of end 774a486f [h] Resolved isssue: shrink unnesscary test data 0541dcc5 [h] Resovled issue: replace listPartitions with listPartitionNames 5a97c5d0 [h] Resolved issue: unify Exceed maxHivePartitions without partition filter 1581b297 [h] resovled the code review issues d0b0fc46 [h] Add maxHivePartitions Strategy to avoid scan excessive hive partitions on partitioned table Authored-by: h <h@zhihu.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
ulysses-you
|
f6d4bba101
|
[KYUUBI #1070] Support insert zorder by table properties
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Closes #1070 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1071 from ulysses-you/tblproperties. Closes #1070 37800666 [ulysses-you] add test 3dec3f81 [ulysses-you] support insert zorder by table properties Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
zhouyifan279
|
1b89272749
|
[KYUUBI #1009] Implement Hive metastore server Delegation Token Provider
### _Why are the changes needed?_ This PR finishes the work of issue #1009 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1053 from zhouyifan279/KYUUBI#1009. Closes #1009 342fca91 [zhouyifan279] [KYUUBI #1009] Implement Hive metastore server Delegation Token Provider a9e1b34d [zhouyifan279] [KYUUBI #1009] Implement Hive metastore server Delegation Token Provider 82da9007 [zhouyifan279] [KYUUBI #1009] Implement Hive metastore server Delegation Token Provider 2c00dd3a [zhouyifan279] [KYUUBI #1009] Implement Hive metastore server Delegation Token Provider Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
张宇翔
|
86aa4691c5
|
[KYUUBI #1037] Add classification of SQL_Type: for dql and auxiliary statement
This PR, we have two different of sql classification: - other: all auxiliary statement fail into this classification - dql We support the user use the self-defined matching rule: sql-classification.json. If there have no this named jsonFile, the service will upload the default matching rule: sql-classification-default.json. ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1037 from zhang1002/branch-1.3_get-sql-type-for-aux. Closes #1037 e6e9e906 [张宇翔] merge master f971c9ad [张宇翔] merge master af28520b [张宇翔] Merge remote-tracking branch 'upstream/master' e94ad829 [张宇翔] 1. Remove NullPointException 2. If the url is null, use the default jsonFile 9d346c61 [张宇翔] 1. Add analyzeColumnCommand and AnalyzePartitionCommand 2. change ExplainCommand to other a73941ed [张宇翔] order the matching rule in jsonFile 72a18ece [张宇翔] change format d7f75387 [张宇翔] change format e6f9b6b2 [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-aux d258e806 [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-aux c328c884 [张宇翔] Merge remote-tracking branch 'upstream/master' 47a6dc5a [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-aux abcaae1f [张宇翔] Merge remote-tracking branch 'upstream/master' 524a3801 [张宇翔] Support user use the self-defined matching rule, named sql-classification.json 8c132ff6 [张宇翔] Support user use the self-defined matching rule, named sql-classification.json aec0d368 [张宇翔] Merge branch 'branch-1.3_get-sql-type-for-dml' into branch-1.3_get-sql-type-for-aux eb6a96d8 [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-dml f0a7e566 [张宇翔] Merge remote-tracking branch 'upstream/master' 5e986922 [张宇翔] 1. Add sql classification of auxiliary statement 2. Add sql classification of dql 8e586df1 [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-dml ba0cd0a3 [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-dml 112aa6b6 [张宇翔] Merge remote-tracking branch 'upstream/master' 18f35f2b [张宇翔] Add the classification of sql: DML 025bc3f8 [张宇翔] Add the classification of sql: DML 55ef6af7 [张宇翔] some modification f1f8b355 [张宇翔] change other to undefined 1052ae45 [张宇翔] Change some code standards 5e21dc62 [张宇翔] Change some code standards f531744d [张宇翔] Add dml test 3017b96c [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-ddl c55652fd [张宇翔] Merge remote-tracking branch 'upstream/master' c2572b98 [张宇翔] 1. Use RuleBuilder to develop this function 2. Use analyzed logical plan to judge this sql's classification 3. Change the matching rule: use map, the key is simpleClassName, the value is classification of this sql 93b5624a [张宇翔] Exclude license check for json d8187ced [张宇翔] Exclude license check for json e46bc86e [张宇翔] Add exception 3b358bf0 [张宇翔] Add licence 1125b600 [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-ddl ef5e8c55 [张宇翔] Merge remote-tracking branch 'upstream/master' ba8f99eb [张宇翔] Use extension to get simpleName c0bdea7b [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-ddl 5a75384c [张宇翔] Merge remote-tracking branch 'upstream/master' 55b85849 [张宇翔] Update settings.md ecbd8000 [张宇翔] Repair the scalastyle violations. 76edd20d [张宇翔] Repair the scalastyle violations. d8e820ee [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-ddl 8da4f7ed [张宇翔] Merge remote-tracking branch 'upstream/master' 65a90958 [张宇翔] Classification for sqlType: DDL a7ba1bfc [张宇翔] Merge remote-tracking branch 'upstream/master' b662a989 [张宇翔] Merge remote-tracking branch 'upstream/master' 4c8f3b87 [张宇翔] Merge remote-tracking branch 'upstream/master' 8b686767 [张宇翔] Merge remote-tracking branch 'upstream/master' cf99e309 [张宇翔] Merge remote-tracking branch 'upstream/master' 0afaa578 [张宇翔] Merge remote-tracking branch 'upstream/master' b24fea07 [张宇翔] Merge remote-tracking branch 'upstream/master' e517cfc5 [张宇翔] Merge remote-tracking branch 'upstream/master' 18aebe76 [张宇翔] Merge remote-tracking branch 'upstream/master' f248bef7 [张宇翔] Merge remote-tracking branch 'upstream/master' 5ffb54f3 [张宇翔] Add kyuubi-spark-monitor module for nightly.yml Authored-by: 张宇翔 <zhang1002@126.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
ulysses-you
|
ba2880cd66
|
[KYUUBI #1055] Refine zorder plan
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> - Change origin `OptimizeZorderCommand` to `OptimizeZorderStatement` which holds the parsed plan - Add `OptimizeZorderCommand` which delegate an another command(e.g. `InsertIntoHive`) to execute; the reason we should add a new command (not reuse `InserIntoStatement`) is we may need to implement read and overwrite same table in future. And also it's tricky to add `InserIntoStatement` during analyze. - Change `ZorderBeforeWrite` to `ResolveZorder`, so we can resolve `OptimizeZorderStatement` to `OptimizeZorderCommand` So the code should look cleaner ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1055 from ulysses-you/zorder-insert. Closes #1055 d252d35e [ulysses-you] simplify 223f68ed [ulysses-you] refine zorder plan Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
ulysses-you
|
eecf074bf1 |
[KYUUBI #1045] Clean up zorder .g4
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Clean up .g4 file. Remove some grammar we donn't need ### _How was this patch tested?_ Pass CI Closes #1045 from ulysses-you/clean-up-zorder. Closes #1045 cf1a5cc3 [ulysses-you] eight indention d6d7eb6e [ulysses-you] pom 8da9df92 [ulysses-you] clean up Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
yanghua
|
10d8ab90dd |
[KYUUBI #1031] Implement api: /sessions/count
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1038 from yanghua/KYUUBI-1031. Closes #1031 36ad5b97 [yanghua] Addressed the review concerns 66d8b0fe [yanghua] [KYUUBI #1031] Implement api: /sessions/count Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Cheng Pan
|
19072f9642 |
[KYUUBI #1042] [DEPS] Bump jersey 2.30
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Align with `spark-core` https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.12/3.1.2 https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.12/3.0.3 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1042 from pan3793/jersey. Closes #1042 f5a05cd0 [Cheng Pan] Update dependencyList abb517e2 [Cheng Pan] [DEPS] Bump jersey 2.30 Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Cheng Pan
|
4621842283 |
[KYUUBI #939][FOLLOWUP] Refine pom.xml
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1041 from pan3793/deps. Closes #939 29d566c8 [Cheng Pan] Address comments e18f2c4e [Cheng Pan] Refine pom.xml Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
qphien
|
4cd83790ae |
[KYUUBI #939] Add Z-Order extensions to support optimize SQL
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ This PR adds Z-Order extension to support `OPTIMIZE` SQL in https://github.com/apache/incubator-kyuubi/issues/939. Currently this PR only supports optimize `HiveTableRelation`. We need to set `hive.exec.dynamic.partition` to `true` and `hive.exec.dynamic.partition.mode` to `nonstrict` when a partitioned table is optimized. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #990 from qphien/zorder. Closes #939 9190e315 [qphien] Add Z-Order extensions to support optimize SQL Authored-by: qphien <clxidian@126.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
张宇翔
|
121de7fbb0
|
[KYUUBI #1035] Add classification of SQL_Type: this PR is just for DML
Add classification of SQL_Type: this PR is just for DML ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1035 from zhang1002/branch-1.3_get-sql-type-for-dml. Closes #1035 eb6a96d8 [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-dml f0a7e566 [张宇翔] Merge remote-tracking branch 'upstream/master' 8e586df1 [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-dml ba0cd0a3 [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-dml 112aa6b6 [张宇翔] Merge remote-tracking branch 'upstream/master' 18f35f2b [张宇翔] Add the classification of sql: DML 025bc3f8 [张宇翔] Add the classification of sql: DML 55ef6af7 [张宇翔] some modification f1f8b355 [张宇翔] change other to undefined 1052ae45 [张宇翔] Change some code standards 5e21dc62 [张宇翔] Change some code standards f531744d [张宇翔] Add dml test 3017b96c [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-ddl c55652fd [张宇翔] Merge remote-tracking branch 'upstream/master' c2572b98 [张宇翔] 1. Use RuleBuilder to develop this function 2. Use analyzed logical plan to judge this sql's classification 3. Change the matching rule: use map, the key is simpleClassName, the value is classification of this sql 93b5624a [张宇翔] Exclude license check for json d8187ced [张宇翔] Exclude license check for json e46bc86e [张宇翔] Add exception 3b358bf0 [张宇翔] Add licence 1125b600 [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-ddl ef5e8c55 [张宇翔] Merge remote-tracking branch 'upstream/master' ba8f99eb [张宇翔] Use extension to get simpleName c0bdea7b [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-ddl 5a75384c [张宇翔] Merge remote-tracking branch 'upstream/master' 55b85849 [张宇翔] Update settings.md ecbd8000 [张宇翔] Repair the scalastyle violations. 76edd20d [张宇翔] Repair the scalastyle violations. d8e820ee [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-ddl 8da4f7ed [张宇翔] Merge remote-tracking branch 'upstream/master' 65a90958 [张宇翔] Classification for sqlType: DDL a7ba1bfc [张宇翔] Merge remote-tracking branch 'upstream/master' b662a989 [张宇翔] Merge remote-tracking branch 'upstream/master' 4c8f3b87 [张宇翔] Merge remote-tracking branch 'upstream/master' 8b686767 [张宇翔] Merge remote-tracking branch 'upstream/master' cf99e309 [张宇翔] Merge remote-tracking branch 'upstream/master' 0afaa578 [张宇翔] Merge remote-tracking branch 'upstream/master' b24fea07 [张宇翔] Merge remote-tracking branch 'upstream/master' e517cfc5 [张宇翔] Merge remote-tracking branch 'upstream/master' 18aebe76 [张宇翔] Merge remote-tracking branch 'upstream/master' f248bef7 [张宇翔] Merge remote-tracking branch 'upstream/master' 5ffb54f3 [张宇翔] Add kyuubi-spark-monitor module for nightly.yml Authored-by: 张宇翔 <zhang1002@126.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
yanghua
|
32111a3014
|
[KYUUBI #999] Build the basic framework for rest frontend service
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1003 from yanghua/KYUUBI-999. Closes #999 78fdf542 [yanghua] Changed rest frontend service port from 10009 to 10099 73929d5d [yanghua] [KYUUBI #999] Build the basic framework for rest frontend service Authored-by: yanghua <yanghua1127@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
张宇翔
|
7b289ae6ff
|
[KYUUBI #1002] Add classification of SQL_Type: this PR is just for DDL
Add classification of SQL_Type: this PR is just for DDL
Now we use the simple name to judge this operation belong to which classification of SQL_Type.
The matching rule store in #{sql-classification-default.json}.
The key is the simpleClassName of this analyzed logical plan, the value is the classification of this sql.
It will store in the #{sparksession.conf}, the configuration item is #{spark.sql.classification}.
Why we use the file to store the whole rules?
If we need to change the matching rule, we only need to revise this file.
When the kyuubi started, it will load this json file only once.
When user execute a sql, it will generate a logicalPlan and get its simple name.
By the rules, the function will return the classification that belong to this sql, then it will set this value into #{sparksession.conf}.
For this function, I use SparkSessionExtension to develop it.
### _Why are the changes needed?_
By these SQL_Type, we can do something below:
1. Reduce some log print:
For Auxiliary Statements(the sql type is others), we don't need to print operation log.
2. Optimizing some configuration item.
For example, in final stage, the conf is different between DML and DQL.
Through this configuration item, we can use different conf for them.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes #1002 from zhang1002/branch-1.3_get-sql-type-for-ddl.
Closes #1002
55ef6af7 [张宇翔] some modification
f1f8b355 [张宇翔] change other to undefined
1052ae45 [张宇翔] Change some code standards
5e21dc62 [张宇翔] Change some code standards
3017b96c [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-ddl
c55652fd [张宇翔] Merge remote-tracking branch 'upstream/master'
c2572b98 [张宇翔] 1. Use RuleBuilder to develop this function 2. Use analyzed logical plan to judge this sql's classification 3. Change the matching rule: use map, the key is simpleClassName, the value is classification of this sql
93b5624a [张宇翔] Exclude license check for json
d8187ced [张宇翔] Exclude license check for json
e46bc86e [张宇翔] Add exception
3b358bf0 [张宇翔] Add licence
1125b600 [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-ddl
ef5e8c55 [张宇翔] Merge remote-tracking branch 'upstream/master'
ba8f99eb [张宇翔] Use extension to get simpleName
c0bdea7b [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-ddl
5a75384c [张宇翔] Merge remote-tracking branch 'upstream/master'
55b85849 [张宇翔] Update settings.md
ecbd8000 [张宇翔] Repair the scalastyle violations.
76edd20d [张宇翔] Repair the scalastyle violations.
d8e820ee [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-ddl
8da4f7ed [张宇翔] Merge remote-tracking branch 'upstream/master'
65a90958 [张宇翔] Classification for sqlType: DDL
a7ba1bfc [张宇翔] Merge remote-tracking branch 'upstream/master'
b662a989 [张宇翔] Merge remote-tracking branch 'upstream/master'
4c8f3b87 [张宇翔] Merge remote-tracking branch 'upstream/master'
8b686767 [张宇翔] Merge remote-tracking branch 'upstream/master'
cf99e309 [张宇翔] Merge remote-tracking branch 'upstream/master'
0afaa578 [张宇翔] Merge remote-tracking branch 'upstream/master'
b24fea07 [张宇翔] Merge remote-tracking branch 'upstream/master'
e517cfc5 [张宇翔] Merge remote-tracking branch 'upstream/master'
18aebe76 [张宇翔] Merge remote-tracking branch 'upstream/master'
f248bef7 [张宇翔] Merge remote-tracking branch 'upstream/master'
5ffb54f3 [张宇翔] Add kyuubi-spark-monitor module for nightly.yml
Authored-by: 张宇翔 <zhang1002@126.com>
Signed-off-by: Kent Yao <yao@apache.org>
|
||
|
hongdongdong
|
d704e4d49d |
[KYUUBI #922] Add OperationEvent for server to track operation state
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Track operation state change and support output to custom system. ### _How was this patch tested?_ - [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #968 from hddong/add-server-events. Closes #922 8be8f268 [hongdongdong] Add prefix for StatementEvent 9fdcc504 [hongdongdong] fix comment a0cdb35d [hongdongdong] fix test 4d340c67 [hongdongdong] add dependency 00920751 [hongdongdong] [KYUUBI#922]Add OperationEvent for server to track operation state Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Cheng Pan
|
98f38ba517
|
[BUILD] Bump 1.4.0-SNAPSHOT | ||
|
ulysses-you
|
7e2d67ed61 |
[KYUUBI #926] Support add repartition for OptimizedCreateHiveTableAsSelectCommand
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> We missed a ctas node `OptimizedCreateHiveTableAsSelectCommand`. This PR aims to support add repartition for `OptimizedCreateHiveTableAsSelectCommand`. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #926 from ulysses-you/cats. Closes #926 5ff28eed [ulysses-you] nit f530fb0d [ulysses-you] style 4bf0a1cf [ulysses-you] ctas Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
timothy65535
|
5526898f7b |
[KYUUBI #894] Enhance kyuubi-zookeeper module
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### 1. Describe the feature Kyuubi use `TestingServer` as the EmbeddedZookeeper service in codebase, and require many args to init the service. These `electionPort`, `quorumPort`, `serverId` parameters only work in cluster mode, these ports are not bound to the host. From hadoop and bookkeeper project, they use `ZooKeeperServer` as a built-in zk service, especially the bookeeper project. - https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/server/services/MicroZookeeperService.java - https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/shims/zk/ZooKeeperServerShimImpl.java ### 2. Describe the solution Improve - Remove `electionPort`, `quorumPort`, `serverId` which are nerver be used, hadoop and bookkeer do the same - Replace `TestingServer` with `ZooKeeperServer` - Remove `curator-test` dependency ### 3. `deleteDataDirectoryOnClose` The name style for `deleteDataDirectoryOnClose` option, refer to hadoop `core-default.xml` https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/core-default.xml ``` hadoop.security.dns.log-slow-lookups.enabled hadoop.security.dns.log-slow-lookups.threshold.ms dfs.client.block.write.replace-datanode-on-failure.policy ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #907 from timothy65535/ky-894. Closes #894 7d7fde9f [timothy65535] remove useless config ce06a0fd [timothy65535] limit travis build only master branch ff825c3f [timothy65535] revert deleteDataDirectoryOnClose config a336544a [timothy65535] remove javassist 3da0bd55 [timothy65535] remove javassist 0abb44df [timothy65535] [KYUUBI #894] Enhance kyuubi-zookeeper module Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Cheng Pan
|
4fecf13902 |
[KYUUBI #888] Add scala version suffix on scala modules
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> - Add scala version suffix on scala modules to match the name pattern. - Enable `kyuubi-extension-spark-3-1` in daily publish snapshot ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #888 from pan3793/scala. Closes #888 571eb9c3 [Cheng Pan] comments 16cb263a [Cheng Pan] remove hardcode scala version in pom 6090c798 [Cheng Pan] Fix shade 5178847f [Cheng Pan] Fixup a1547753 [Cheng Pan] Fix engine jar name 2cf4793e [Cheng Pan] fix cac96531 [Cheng Pan] Enable kyuubi-extension-spark-3-1 in daily publish snapshot 1ca31de5 [Cheng Pan] Add scala version suffix on scala modules Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Cheng Pan
|
962d721842
|
[KYUUBI #887] [GA] Daily publish snapshot
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #887 from pan3793/publish-snapshot. Closes #887 5b2044a1 [Cheng Pan] Pin ubuntu-20.04 in GA 98945a12 [Cheng Pan] Remove Kyuubi local cache before deploy f29cb6e4 [Cheng Pan] [GA] Daily publish snapshot Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
c6bee7c1b8
|
[KYUUBI #882] Rename kyuubi-extension-spark_3.1 to kyuubi-extension-spark-3-1
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> `kyuubi-extension-spark_3.1` is conflict with the scala naming convention, e.g. `spark-core_2.12` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #882 from pan3793/rename-spark-3-1. Closes #882 d3297409 [Cheng Pan] Rename kyuubi-extension-spark_3.1 to kyuubi-extension-spark-3-1 Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
1a8b4ebaae
|
[KYUUBI #884] Rewrite kyuubi-hive-jdbc in Java
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #765 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #884 from pan3793/jdbc-java. Closes #884 ca7bad48 [Cheng Pan] Rewrite kyuubi-hive-jdbc in Java Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
ulysses-you
|
f4fb822743
|
[KYUUBI #871] Change module name kyuubi-main to kyuubi-server
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> As we will release soon, it's better to format the module name and artifactId. The name of kyuubi-main sounds like a bootstrap of Kyuubi but not a server. ### _How was this patch tested?_ Pass CI Closes #872 from ulysses-you/kyuubi-server. Closes #871 1437f9b5 [ulysses-you] address comment 117f2c7c [ulysses-you] server 86986baa [ulysses-you] master eba660a2 [ulysses-you] Merge branch 'master' of https://github.com/NetEase/kyuubi into kyuubi-server e649012b [ulysses-you] change module name Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
ce5fb97036
|
[KYUUBI #849] ASF Nexus SNAPSHOT publish
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> https://infra.apache.org/publishing-maven-artifacts.html https://issues.apache.org/jira/browse/INFRA-22114 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #849 from pan3793/asf. Closes #849 c609f6ba [Cheng Pan] update pom 31f1f2e1 [Cheng Pan] rename root project name to kyuubi-parent 9c5616d8 [Cheng Pan] [INFRA] Update POM for Apache release Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
98e44636fd
|
[KYUUBI #823] [BUILD] [DEPS] Leverage maven to resolve transitive dependencies except hive
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #823 from pan3793/deps. Closes #823 51cdcb5d [Cheng Pan] restore thrift deps from spark-hive 148038a3 [Cheng Pan] [BUILD] [DEPS] Leverage maven to resolve transitive dependencies except hive Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
b840e374c2 |
[KYUUBI #809] [INFRA] Support reopened PR in pr merge tool
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Make the `dev/merge_kyuubi_pr.py` work as expected on reopened PR. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate Test passed on merge the reopened PR #807. - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #809 from pan3793/merge_pr. Closes #809 fd7176f8 [Cheng Pan] [INFRA] Support reopen pr in pr merge tool cc02b5e2 [Cheng Pan] [INFRA] Fix typo in merge_kyuubi_pr script Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
ulysses-you
|
ca94d7b46c
|
[KYUUBI #807] Make kyuubi-extension-spark_3.1 test pass with Spark 3.2.0-snapshot
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Make this extesion more robust. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #807 from ulysses-you/sql-extension. Closes #807 27155c1e [ulysses-you] fix Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Kent Yao
|
cc1254d648
|
[BUILD] Update PR Merge tool (#790)
* [BUILD] Update PR Merge tool * [BUILD] Update PR Merge tool * [BUILD] Update PR Merge tool * [BUILD] Update PR Merge tool |
||
|
hongdongdong
|
16b93e4720
|
[KYUUBI #753] Use scopt to parse arguments
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Use [`scopt`](https://github.com/scopt/scopt) replace custom parser to parse arguments. ### _How was this patch tested?_ - [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #754 from hddong/kyuubi-753. Closes #753 3d2645d8 [hongdongdong] fix check style fd181452 [hongdongdong] fix output ef7c44e4 [hongdongdong] use scopt help 682b3a2f [hongdongdong] [KYUUBI#753] Use scopt to parse arguments Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
5c5d43b126
|
[KYUUBI #693] [BUILD] [GA] Support JDK 11 and enable it in GitHub Action
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Close #550, relate to #559 #688 #689. Add `javax.xml.bind:jaxb-api` and `javax.activation:activation` since those apis has been removed by JDK 11. In GA, we saw a significant performance downgrade in the unit tests, and even with some changes like enlarging timeout, using in-memory catalog instead of hive catalog, the JDK11(~43min) tests cost 2 times than JDK8(~22min), this performance regression is tracked in #745. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #693 from pan3793/jdk11. Closes #693 5706daf1 [Cheng Pan] shrink scalatest memory 6ebac68e [Cheng Pan] address comments 2b479391 [Cheng Pan] address comments 5fa74aa0 [Cheng Pan] disable test in kyuubi-extension-spark_3.1 9f070a40 [Cheng Pan] maven profile for jdk versions 801e4cd1 [Cheng Pan] jdk11 in GA e86893cc [Cheng Pan] enlarge YarnCluster timeout c880edd1 [Cheng Pan] minor 38d2775c [Cheng Pan] enlarge engine crash timeout a945ade3 [Cheng Pan] use in-memory catalog in EngineRefSuite 51604d73 [Cheng Pan] enlarge ENGINE_INIT_TIMEOUT 7baeb965 [Cheng Pan] minicluster a7e4bdf1 [Cheng Pan] GA matrix b29aa2f4 [Cheng Pan] license for jaxb 4283777b [Cheng Pan] Add dependency jaxb-api c5b0e580 [Cheng Pan] [GA] Enable JDK 11 in GA Lead-authored-by: Cheng Pan <chengpan@apache.org> Co-authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
wuchunfu
|
4d2fee5dcc
|
[KYUUBI #694] Upgrade Scala to 2.12.14
### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #721 from wuchunfu/694. Closes #694 10b6774a [wuchunfu] [KYUUBI #694] replace dev/dependencyList efafeab5 [wuchunfu] [KYUUBI #694] Upgrade Scala to 2.12.14 Authored-by: wuchunfu <319355703@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
ac481a6c89 |
[KYUUBI #714] [TPCDS] kyuubi-tpcds should respect dsgen charset
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Fix #713. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate run `/opt/spark3/bin/spark-submit --conf spark.sql.tpcds.scale.factor=10 --conf spark.sql.tpcds.database=tpcds_sf10 --class org.apache.kyuubi.tpcds.DataGenerator kyuubi-tpcds-*.jar` before:  after:  - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #714 from pan3793/tpcds. Closes #714 3a44f6da [Cheng Pan] address comments f8f73e6d [Cheng Pan] [TPCDS] kyuubi-tpcds should respect dsgen charset Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Cheng Pan <379377944@qq.com> |
||
|
ulysses-you
|
6e9d30c48a |
[KYUUBI #715] [SQL] Check if need add repartition through project
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Avoid unnecessary shuffle ### _How was this patch tested?_ Add test Closes #715 from ulysses-you/through-project. Closes #715 5942320a [ulysses-you] through-project Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Cheng Pan <379377944@qq.com> |
||
|
Cheng Pan
|
5276cfbecb
|
[KYUUBI #711] [BUILD] Bump jetty 9.4.41.v20210516
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #710 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #711 from pan3793/jetty. Closes #711 95bd9a32 [Cheng Pan] [BUILD] Bump jetty 9.4.41.v20210516 Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
ulysses-you
|
cd32e4aeb6 |
[KYUUBI #700] Add class description for DataGenerator
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add usage description so we can use this tpcds data generator easily. ### _How was this patch tested?_ Not need. Closes #700 from ulysses-you/class-description. Closes #700 2e3d5945 [ulysses-you] * b2aea77e [ulysses-you] desc Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Cheng Pan <379377944@qq.com> |
||
|
Cheng Pan
|
10d72e02e5
|
[KYUUBI #645] [BUILD] [FOLLOWUP #631] Fix CI profile kyuubi-extension-spark_3.1
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Fix `-Pkyuubi-sql-spark_3.1` introduced by KYUUBI #631 in GH action ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #645 from pan3793/build. Closes #645 d20c6d8 [Cheng Pan] suppress CI log for kyuubi-extension-spark_3.1 1499d5b [Cheng Pan] [BUILD] Fix CI profile kyuubi-extension-spark_3.1 Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
ulysses-you
|
ccb891d2ae |
[KYUUBI #638] Refine kyuubi extension config docs
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Refine docs. ### _How was this patch tested?_ Not needed. Closes #638 from ulysses-you/refine-config-docs. Closes #638 e9f3f37c [ulysses-you] refine Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
ulysses-you
|
e77ea13c6d |
[KYUUBI #635] Fix stage level config miss ReusedExchange
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Avoid the reused exchange cause the wrong stage config. ### _How was this patch tested?_ Add test. Closes #635 from ulysses-you/fix-stage-level-config-miss-reusedexchange. Closes #635 7d514a90 [ulysses-you] fix Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou18@gmail.com> |
||
|
Cheng Pan
|
b60d18aeb1
|
[KYUUBI #633] [FOLLOWUP #631] [BUILD] Fix kyuubi-extension-spark_3.1 module version
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #633 from pan3793/minor. Closes #633 108aca0 [Cheng Pan] [BUILD] Fix kyuubi-extension-spark_3.1 module version Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
ulysses-you
|
43f40dcb8a
|
[KYUUBI #631] Add kyuubi-spark-extensions module
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> The added sql module structure looks like: ``` kyuubi | - dev | - kyuubi-extension-spark_3.1 ``` This PR mainly add 3 feature: * merging small files automatically (include dynamic partition insertion case) * insert shuffle node before Join to make AQE `OptimizeSkewedJoin` work * stage level config isolation in AQE Note that, the sql rule depend on the Apache Spark interface so we need make the sql module verion independence. Currently, this PR only supports the Spark 3.1.1. Due to the version issue, we need to check and deploy this extension manually currently. ### _How was this patch tested?_ Add new test. Closes #631 from ulysses-you/add-sql-module. Closes #631 2cf12f1 [ulysses-you] version cfbf72c [ulysses-you] address comment 7740ca6 [ulysses-you] module name 0f723eb [ulysses-you] workflow 45c23d8 [ulysses-you] line 80378f5 [ulysses-you] assembly 95528aa [ulysses-you] move module 5fe5d87 [ulysses-you] license 6578440 [ulysses-you] init work Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
bb9e8a50e5
|
Prepare Kyuubi v1.3.0 | ||
|
Cheng Pan
|
0b4f4f242c
|
[KYUUBI #630] [METRICS] Support expose Prometheus metrics
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Support expose Prometheus metrics ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #630 from pan3793/prometheus. Closes #630 fff68b0 [Cheng Pan] nit 47c19eb [Cheng Pan] simplify 203e776 [Cheng Pan] address comments 10a877c [Cheng Pan] synchronized 26eb26e [Cheng Pan] simplify random port 827427c [Cheng Pan] fix import 476ee3a [Cheng Pan] address comments 9787c77 [Cheng Pan] update ut c8e1b0e [Cheng Pan] update license 2387a97 [Cheng Pan] [METRICS] Support expose Prometheus metrics Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
34374f3b3e
|
[KYUUBI #613] [TEST] [DELTA] Add test for deltalake
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #613 from pan3793/delta. Closes #613 ccadedb [Cheng Pan] [TEST] [DELTA] Add test for deltalake Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
feiwang
|
4e81e6b230
|
[KYUUBI #531] Refactor repo key words (#531)
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Refactor the key words after repo renamed. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request |
||
|
Kent Yao
|
c6585b4708
|
[KYUUBI #471] Separate Embedded Zookeeper server from ha
 [](https://github.com/yaooqinn/kyuubi/pull/471)     [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. Separated embedded zk from ha module, as it's only used for setup a zk from Kyuubi server for now if there is not an external one 2. Support more configuration for the embedded zookeeper, for example, HOSTNAME, then it's more close to prod usage. 3. TODO: Make the embedded zookeeper of our own, which should support as quorum peers ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #471 from yaooqinn/qp. Closes #471 1c24d02 [Kent Yao] address comments 9666a40 [Kent Yao] refine f069fc0 [Kent Yao] refine 8bb2f3a [Kent Yao] refine c20593a [Kent Yao] refine 8fa9a40 [Kent Yao] Merge branch 'master' into qp 1e2e1a3 [Kent Yao] tmp 7d66d05 [Kent Yao] tmp Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
fwang12
|
0aafc55bba
|
[KYUUBI #304] Add Kyuubi Ctl arguments parser
 [](https://github.com/yaooqinn/kyuubi/pull/465)     [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ ``` bin/kyuubi-service <create|get|delete|list> <server|engine> --zkAddress ... --namespace ... --user ... --host ... --port ... --version Operations: - create - expose a service to a namespace, this case is rare but sometimes we may want one server to be reached in 2 or more namespaces by different user groups - get - get the service node info - delete - delete the specified serviceNode - list - list all the service nodes for a particular domain Role: - server default - engine Args: --zkAddress - one of the zk ensemble address, using kyuubi-defaults/conf if absent --namespace - the namespace, using kyuubi-defaults/conf if absent --user - --host --port --version ``` ### _How was this patch tested?_ UT Closes #465 from turboFei/KYUUBI_304_CMD. Closes #304 4bab34b [fwang12] retest pleaes 8083a12 [fwang12] to increase code converage c7e51a2 [fwang12] complete 7249cd6 [fwang12] add ut c809c27 [fwang12] enable set verbose at first bb3cbb6 [fwang12] fix ut 604820d [fwang12] validate for each action a01ac1f [fwang12] fix scala style issue 76c9b4c [fwang12] save 7139dd5 [fwang12] increase test converage 318ebce [fwang12] add more ut 72978a6 [fwang12] save 2931f93 [fwang12] address comments 10b855d [fwang12] save 420912a [fwang12] treat help as an action b27d0a6 [fwang12] treat help as action 896e20d [fwang12] add ctl module into codecov ea43d69 [fwang12] rename kyuubi-service to kyuubi-ctl 65a0e30 [fwang12] save db718b0 [fwang12] refactor 41b503e [fwang12] Add kyuubi-ctl arguments parser cb3f6a8 [fwang12] with log appender Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
9616a8ac36
|
[KYUUBI #464] Closes corresponding github issue with merge tool
 [](https://github.com/yaooqinn/kyuubi/pull/464)     [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> auto-link and close target issue ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #464 from yaooqinn/merge. Closes #464 3b282f4 [Kent Yao] Closes corresponding github issue with merge tool Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
9e7649296d
|
[KYUUBI #449] Add Basic Metrics Service framework
 [](https://github.com/yaooqinn/kyuubi/pull/450)    [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This PR adds a future-proofing Metrics Service framework close #499 add some reporters, like json file, jmx, etc.. add some basic metrics, like statement.error ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible WIP - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #450 from yaooqinn/449. ea21a43 [Kent Yao] add tests 6513d07 [Kent Yao] update codecov 7cfe318 [Kent Yao] update docs a8454d0 [Kent Yao] update docs 91c33f4 [Kent Yao] update deps 05b13fb [Kent Yao] fix dep issues babc739 [Kent Yao] Add Basic Metrics Service Platform acc716a [Kent Yao] Add Basic Metrics Service Platform Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
wangsheng
|
9f5c752402
|
[KYUUBI #442] [MINOR] Sort out the code logic of kyuubi/tpcds
 [](https://github.com/yaooqinn/kyuubi/pull/442)     [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> There is a tpcds generate in kyuubi/dev/kyuubi-tpcds. Helpe to sort out the code logic and modify the pom dependency of kyuubi-tpcds. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #442 from zwangsheng/change_dev_tpcds. cdc5604 [wangsheng] Unify the style of adding parameters to the string e7c7879 [wangsheng] Sort out the code logic and modify the pom dependency Authored-by: wangsheng <2213335496@qq.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
354f55a0a5
|
[KYUUBI #416] Detected Dependency List Change for better dependency management
 [](https://github.com/yaooqinn/kyuubi/pull/416)     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Detect Dependency List Change, to let us and users know what actually release with Kyuubi ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #416 from yaooqinn/dep. 5938423 [Kent Yao] update doc 037c897 [Kent Yao] nit 2b06bf9 [Kent Yao] Detected Dependency List Change for better dependency management Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
5e2a1290c3
|
Prepare Kyuubi v1.2.0 | ||
|
peterylh
|
af58331f67
|
[KYUUBI #320] Fix Merge PR with duplicated title
 [](https://github.com/yaooqinn/kyuubi/pull/320)    [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html --> duplicated ### _Which issue are you going to fix?_ <!-- Replace ${ID} below with the actual issue id from https://github.com/yaooqinn/kyuubi/issues, so that the issue will be linked and automatically closed after merging --> Fixes #${ID} ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the user case of it. 2. If you fix a bug, you can clarify why it is a bug. --> duplicated PR ID in the merged title ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #320 from yaooqinn/mergepr2. 8cbcf1b [peterylh] Fix Merge PR with duplicated title Authored-by: peterylh <peterylh@163.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Kent Yao
|
aa4c8bb6e5
|
[KYUUBI #317] [KYUUBI #317] merge pr script
 [](https://github.com/yaooqinn/kyuubi/pull/317)     [❨?❩](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html --> ### _Which issue are you going to fix?_ <!-- Replace ${ID} below with the actual issue id from https://github.com/yaooqinn/kyuubi/issues, so that the issue will be linked and automatically closed after merging --> Fixes #317 ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the user case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #317 from yaooqinn/mergepr. 4588d32 [Kent Yao] [KYUUBI #317] merge pr script Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
0eeb99774f
|
Tune pom
fixes #271 Squashed commit of the following: commit fe0f469b2068865398d6e5be1957fdf6c6f5eb87 Author: Cheng Pan <379377944@qq.com> Date: Sat Jan 9 22:28:43 2021 +0800 followup commit 527e32419ecb2ae584d1251cb345e3fc870965a9 Author: Cheng Pan <379377944@qq.com> Date: Sat Jan 9 22:21:29 2021 +0800 revert some stuffs commit 82f188972b8491c5d381ba076a53f84a2fbc1898 Author: Cheng Pan <379377944@qq.com> Date: Sat Jan 9 16:49:14 2021 +0800 remove unused dependencies commit c7eb27228d2749ab9100102467e94d3a6a98cf44 Author: Cheng Pan <379377944@qq.com> Date: Sat Jan 9 15:41:59 2021 +0800 minimize dependencies in kyuubi-spark-sql-engine commit da902b1b1e02c7e93810ca832f6c9e7109cf95b9 Author: Cheng Pan <379377944@qq.com> Date: Sat Jan 9 13:26:34 2021 +0800 exclude hive dependencies commit 55ae7f025a5ffcec402919e28b7b3af5987dfe16 Author: Cheng Pan <379377944@qq.com> Date: Sat Jan 9 01:08:09 2021 +0800 pin versions and excludes in <dependencyManagement> commit 9ef84a5f62afd66fb4b817d593ba1b13ff8348ed Author: Cheng Pan <379377944@qq.com> Date: Sat Jan 9 14:12:04 2021 +0800 trigger kyuubi-spark-sql-engine package commit d492e3fdce64e6554fe0db1d810c6f4bea198baa Author: Cheng Pan <379377944@qq.com> Date: Sat Jan 9 13:44:13 2021 +0800 trigger kyuubi-download commit 050281b8fc400f3ae5fa1d43d56da49846664ea4 Author: Cheng Pan <379377944@qq.com> Date: Sat Jan 9 15:30:58 2021 +0800 pin modules version in dependencyManagement commit 1655fc453ded4626ae5e12e7b17af6f10c30a0dd Author: Cheng Pan <379377944@qq.com> Date: Fri Jan 8 19:45:27 2021 +0800 tune modules in alphabetical order commit 7f3f4d987336d71d73eab059e93f3d008cfcd1a6 Author: Cheng Pan <379377944@qq.com> Date: Sat Jan 9 00:09:49 2021 +0800 change dependencies scope to test |
||
|
Kent Yao
|
ca1b452e9a
|
Prepare Kyuubi v1.1.0 | ||
|
Kent Yao
|
a8b70d0841
|
Add a one-click tpcds generating tool (#262)
* Add a one click tpcds generating tool * nit |
||
|
Kent Yao
|
a49660e769 | fix | ||
|
Kent Yao
|
e8ab5ab4a0 | Some build staff |
