1b48b1874f
105 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Kent Yao
|
1b48b1874f
|
[KYUUBI #1601] Align SparkStatementEvent to KyuubiOperationEvent
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Align SparkStatementEvent to KyuubiOperationEvent ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1601 from yaooqinn/soe. Closes #1601 090dee8f [Kent Yao] Merge branch 'master' of github.com:apache/incubator-kyuubi into soe a61ce3b7 [Kent Yao] Align SparkStatementEvent to KyuubiOperationEven efbbdf79 [Kent Yao] Align SparkStatementEvent to KyuubiOperationEven Authored-by: Kent Yao <yao@apache.org> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Fu Chen
|
2b8304d154
|
[KYUUBI #1591] Watchdog support for Spark-3.2
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> <!-- 2. move spark-3.1 `MarkAggregateOrderRule` to spark-common and rename to `MarkAggregateOrderBase` --> 1. move spark-3.1 `ForcedMaxOutputRowsRule` to spark-common and rename to `ForcedMaxOutputRowsBase` 2. handle `WithCTE` logical plan in spark-3.2 3. move spark-3.1 `MaxPartitionStrategy` to spark-common 4. add netsted cte unit test for `ForcedMaxOutputRowsRule` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1591 from cfmcgrady/watchdog-spark32. Closes #1591 5399a3f0 [Fu Chen] fix style 0ce83ba5 [Fu Chen] remove MarkAggregateOrderRule 364fc26e [Fu Chen] add license header 44726dee [Fu Chen] fix style 4847dbf3 [Fu Chen] watchdog support for spark-3.2 Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Wang Zhen
|
df1d9f3bb2
|
[KYUUBI #1577] Add DropIgnoreNonexistent Rule.
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Like the `hive.exec.drop.ignorenonexistent` configuration in hive, do not report an error if DROP DATABASE/TABLE/VIEW/Function/Partition specifies a non-existent database/table/view/function/partition. For details: #1577. ### _How was this patch tested?_ - [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1583 from wForget/KYUUBI-1577. Closes #1577 63382660 [Wang Zhen] delete useless comment adadf424 [Wang Zhen] add DropIgnoreNonexistent rule using injectPostHocResolutionRule 860b15d3 [Wang Zhen] add DropIgnoreNonexistent rule for spark 3.2 aad43cdb [Wang Zhen] adjust code ca1b9f11 [Wang Zhen] make some adjustments and update rules.md 804bf40e [Wang Zhen] [KYUUBI-1577] Add DropIgnoreNonexistent Rule. Authored-by: Wang Zhen <wangzhen07@qiyi.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
77a278707f
|
[KYUUBI #1559] Bump Jersey 2.34
### _Why are the changes needed?_ #1561 and align Spark 3.2.0 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1559 from pan3793/jersey. Closes #1559 a86d4d9f [Cheng Pan] Bump Jersey 2.34 Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Cheng Pan
|
f841da4b4e
|
[KYUUBI #1558] Bump Jackson 2.12.5
### _Why are the changes needed?_ Because Spark 3.2, Trino 363, Jersey 2.34 (we use Jersey 2.30 now, will update to 2.34 soon) use Jackson 2.12.x ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1558 from pan3793/jackson. Closes #1558 7bdd65b0 [Cheng Pan] Bump Jackson 2.12.5 Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
6fa1327f4e
|
[KYUUBI #1560] Align all Jackson libs version
### _Why are the changes needed?_ Fix missing version of `jackson-module-jaxb-annotations` in `dependencyManagement` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1560 from pan3793/jackson-1. Closes #1560 49529d47 [Cheng Pan] Update deps b3610007 [Cheng Pan] fix 67044840 [Cheng Pan] Algin all Jackson libs version Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
ulysses-you
|
37a4e5c0da
|
[KYUUBI #1496] Support tpcds benchmark
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Support tpcds benchmark in `dev/kyuubi-tpcds` module. Add a `README.md` in `dev/kyuubi-tpcds` module to show how to use. The mian code is from [databricks-spark-sql-perf](https://github.com/databricks/spark-sql-perf) ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1496 from ulysses-you/tpcds-benchmark. Closes #1496 d4afe2d1 [ulysses-you] comment 54a146ef [ulysses-you] pom 91e71692 [ulysses-you] docs 20eadc49 [ulysses-you] benchmark Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Fu Chen
|
78c9cf3756
|
[KYUUBI #1443] MaxPartitionStrategy support file source based table.
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Fix [comment](https://github.com/apache/incubator-kyuubi/pull/1086#discussion_r707103617) ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1443 from cfmcgrady/max-partition-strategy. Closes #1443 e071ebc2 [Fu Chen] typo f7d4ea16 [Fu Chen] refactor 97a36016 [Fu Chen] truncate RootPaths c3fe93aa [Fu Chen] remove dpp suite 9075e5ba [Fu Chen] refactor 5a71f459 [Fu Chen] trigger GitHub actions 17df25d2 [Fu Chen] fix style 4dfe6a7e [Fu Chen] fix style 0b2df063 [Fu Chen] update docs 305c6e88 [Fu Chen] fix style 33b9b3d5 [Fu Chen] fix style 885e8ed7 [Fu Chen] data source support Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
fwang12
|
119e7f2c2c |
[KYUUBI #1441] [BUILD] Bump 1.5.0-SNAPSHOT
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1441 from turboFei/bump_1.5.0. Closes #1441 0c03c75d [fwang12] [BUILD] Bump 1.5.0-SNAPSHOT Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
zhouyifan279
|
ffe41631ce
|
[KYUUBI #1383] Leverage Scalafmt to auto format scala code
### _Why are the changes needed?_ #1383 As IntelliJ IDEA's `Optimize Imports` result does not meet rules in `scalastyle-config.xml`, currently developers have to arrange package imports manually. In this PR, we introduce a `.scalafmt.conf` file to solve the problem. It works as follows: 1. Set scala code style formatter to `scalafmt`. 2. Execute `Reformat Code` 3. IDEA picks up configurations in `.scalafmt.conf` and format both code lines and imports. We also formatted all historical scala codes. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1391 from zhouyifan279/1383. Closes #1383 e0696a9e [zhouyifan279] [KYUUBI #1383] Leverage Scalafmt to auto format scala code 18b8e229 [zhouyifan279] [KYUUBI #1383] Leverage Scalafmt to auto format scala code Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
sunfangbin
|
fb8bf49dbc
|
[KYUUBI #1413] Improve usage of zookeeper cli
### _Why are the changes needed?_ This change enabled the JLine support so that the `kyuubi-zookeeper-shell` command can be easier to use with history, prompt and completion, etc. Note that the `jline` version `0.9.94` is dependent on `zookeeper` version `3.4.14` which is a lower version. BTW, users can consider upgrading the `zookeeper` version for the sake of performance improvements and new features. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1413 from murong00/branch-1413. Closes #1413 2d1ec084 [sunfangbin] Rename shell script to zk-cli d09f64dc [sunfangbin] Fixes failed ci ce88357a [sunfangbin] Improve usage of zookeeper cli Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
09f04d6de8
|
[KYUUBI #1410] Add reformat shell script
### _Why are the changes needed?_ Following up #1403 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1410 from pan3793/reformat. Closes #1410 288fc1b5 [Cheng Pan] Add reformat shell script Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Fu Chen
|
a5ef94accc
|
[KYUUBI #1354] Expose OpenAPI Specifications
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. add a new endpoint `/openapi.json` which describes the entire restful API of Kyuubi. 2. add a static web page `/api/v1/swagger-ui`. see [swagger-ui](https://github.com/swagger-api/swagger-ui) ``` curl http://localhost:10099/openapi.json | jq . ``` ```json { "openapi": "3.0.1", "info": { "title": "Apache Kyuubi RESTful Api Documentation", "description": "Apache Kyuubi RESTful Api Documentation", "contact": { "name": "Apache Kyuubi Community", "url": "https://kyuubi.apache.org/issue_tracking.html", "email": "devkyuubi.apache.org" }, "license": { "name": "Apache 2.0", "url": "https://www.apache.org/licenses/LICENSE-2.0.html" }, "version": "1.4.0" }, "tags": [ { "name": "Session" } ], "paths": { "/api/v1/ping": { "get": { "operationId": "ping", "responses": { "default": { "description": "default response", "content": { "text/plain": { "schema": { "type": "string" } } } } } } }, "/api/v1/sessions/{sessionHandle}": { "get": { "tags": [ "Session" ], "operationId": "sessionInfo", "parameters": [ { "name": "sessionHandle", "in": "path", "required": true, "schema": { "type": "string" } } ], "responses": { "200": { "description": "get a session via session handle identifier", "content": { "application/json": {} } } } }, "delete": { "tags": [ "Session" ], "operationId": "closeSession", "parameters": [ { "name": "sessionHandle", "in": "path", "required": true, "schema": { "type": "string" } } ], "responses": { "200": { "description": "close a Session", "content": { "application/json": {} } } } } }, "/api/v1/sessions/execPool/statistic": { "get": { "tags": [ "Session" ], "operationId": "execPoolStatistic", "responses": { "200": { "description": "get some statistic info of sessions", "content": { "application/json": {} } } } } }, "/api/v1/sessions/{sessionHandle}/info/{infoType}": { "get": { "tags": [ "Session" ], "operationId": "getInfo", "parameters": [ { "name": "sessionHandle", "in": "path", "required": true, "schema": { "type": "string" } }, { "name": "infoType", "in": "path", "required": true, "schema": { "type": "integer", "format": "int32" } } ], "responses": { "200": { "description": "get a information detail via session handle identifier and a specific information type", "content": { "application/json": {} } } } } }, "/api/v1/sessions": { "get": { "tags": [ "Session" ], "operationId": "sessionInfoList", "responses": { "200": { "description": "get all the session list hosted in SessionManager", "content": { "application/json": {} } } } }, "post": { "tags": [ "Session" ], "operationId": "openSession", "requestBody": { "content": { "application/json": { "schema": { "$ref": "#/components/schemas/SessionOpenRequest" } } } }, "responses": { "200": { "description": "Open(create) a Session", "content": { "application/json": {} } } } } }, "/api/v1/sessions/count": { "get": { "tags": [ "Session" ], "operationId": "sessionCount", "responses": { "200": { "description": "get open session count", "content": { "application/json": {} } } } } }, "/api/v1/exception": { "get": { "operationId": "test", "responses": { "default": { "description": "default response", "content": { "text/plain": {} } } } } }, "/application.wadl/{path}": { "get": { "operationId": "getExternalGrammar", "parameters": [ { "name": "path", "in": "path", "required": true, "schema": { "type": "string" } } ], "responses": { "default": { "description": "default response", "content": { "application/xml": {} } } } } }, "/application.wadl": { "get": { "operationId": "getWadl", "responses": { "default": { "description": "default response", "content": { "application/vnd.sun.wadl+xml": {}, "application/xml": {} } } } } } }, "components": { "schemas": { "ExecPoolStatistic": { "required": [ "execPoolSize" ], "type": "object", "properties": { "execPoolSize": { "type": "integer", "format": "int32" }, "execPoolActiveCount": { "type": "integer", "format": "int32" } } }, "InfoDetail": { "required": [ "infoType" ], "type": "object", "properties": { "infoType": { "type": "string" }, "infoValue": { "type": "string" } } }, "HandleIdentifier": { "required": [ "publicId" ], "type": "object", "properties": { "publicId": { "type": "string", "format": "uuid" }, "secretId": { "type": "string", "format": "uuid" } } }, "SessionHandle": { "required": [ "identifier", "protocol" ], "type": "object", "properties": { "identifier": { "$ref": "#/components/schemas/HandleIdentifier" }, "protocol": { "type": "string", "enum": [ "HIVE_CLI_SERVICE_PROTOCOL_V1", "HIVE_CLI_SERVICE_PROTOCOL_V2", "HIVE_CLI_SERVICE_PROTOCOL_V3", "HIVE_CLI_SERVICE_PROTOCOL_V4", "HIVE_CLI_SERVICE_PROTOCOL_V5", "HIVE_CLI_SERVICE_PROTOCOL_V6", "HIVE_CLI_SERVICE_PROTOCOL_V7", "HIVE_CLI_SERVICE_PROTOCOL_V8", "HIVE_CLI_SERVICE_PROTOCOL_V9", "HIVE_CLI_SERVICE_PROTOCOL_V10" ] } } }, "MapStringString": { "type": "object", "additionalProperties": { "type": "string" } }, "SessionOpenRequest": { "required": [ "configs", "protocolVersion", "user" ], "type": "object", "properties": { "protocolVersion": { "type": "integer", "format": "int32" }, "user": { "type": "string" }, "password": { "type": "string" }, "ipAddr": { "type": "string" }, "configs": { "type": "object", "additionalProperties": { "type": "string" } } } }, "SessionOpenCount": { "required": [ "openSessionCount" ], "type": "object", "properties": { "openSessionCount": { "type": "integer", "format": "int32" } } }, "SessionDetail": { "required": [ "configs", "createTime", "user" ], "type": "object", "properties": { "user": { "type": "string" }, "ipAddr": { "type": "string" }, "createTime": { "type": "integer", "format": "int64" }, "sessionHandle": { "$ref": "#/components/schemas/SessionHandle" }, "lastAccessTime": { "type": "integer", "format": "int64" }, "lastIdleTime": { "type": "integer", "format": "int64" }, "noOperationTime": { "type": "integer", "format": "int64" }, "configs": { "type": "object", "additionalProperties": { "type": "string" } } } }, "SeqSessionOverview": { "type": "array", "items": { "$ref": "#/components/schemas/SessionOverview" } }, "SessionList": { "required": [ "sessionList" ], "type": "object", "properties": { "sessionList": { "type": "array", "items": { "$ref": "#/components/schemas/SessionOverview" } } } }, "SessionOverview": { "required": [ "createTime", "user" ], "type": "object", "properties": { "user": { "type": "string" }, "ipAddr": { "type": "string" }, "createTime": { "type": "integer", "format": "int64" }, "sessionHandle": { "$ref": "#/components/schemas/SessionHandle" } } } } } } ``` http://localhost:10099/api/v1/swagger-ui > Note that this request will redirect to http://localhost:10099/swagger-ui-redirected/index.html?url=http://localhost:10099/openapi.json  ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1363 from cfmcgrady/kyuubi-1354. Closes #1354 8895d7fc [Fu Chen] typo 8f5dea9a [Fu Chen] addressed comment 4d325a45 [Fu Chen] constant MediaType.APPLICATION_JSON 0d96951e [Fu Chen] RESTful Api -> REST API be7d02dc [Fu Chen] add LIABILITY-edl.txt 4ff1053e [Fu Chen] update dev/dependencyList b126f7e6 [Fu Chen] swagger-ui 4.1.0 a35c2f29 [Fu Chen] addressed comment 7bde0534 [Fu Chen] update LICENSE-binary 38397dc2 [Fu Chen] align jackson version 923f3e7e [Fu Chen] update dev/dependencyList 7d5338c8 [Fu Chen] remove kyuubi-server static files 5cb66b33 [Fu Chen] update dev/dependencyList e8a8c421 [Fu Chen] fix style 5b13f69c [Fu Chen] add license and remove .map files c04d7a75 [Fu Chen] rebase master 2388dae8 [Fu Chen] fix style 8dcbd3db [Fu Chen] add static resource 4b0d6d0a [Fu Chen] expose OpenAPI Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
ulysses-you
|
1dc755e450
|
[KYUUBI #1366] Add a todo in zorder that we can do rebalance partitions before local sort of zorder
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Rebalance partitions can resolve data skew issue before writing, so it should be good to do before we do the local sort of zorder, and also we can support dynamic partition insertion with zorder. However Spark only support this after 3.3.0, see [PR](https://github.com/apache/spark/pull/34542). So add a todo first. ### _How was this patch tested?_ not needed Closes #1366 from ulysses-you/zorder. Closes #1366 972229eb [ulysses-you] todo Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
c122a23a0d
|
[KYUUBI #1348] [DEPS] Add netty and mysql jdbc driver to kyuubi-server module
### _Why are the changes needed?_ This PR is one part of #1334. Add netty to `kyuubi-server` module compile scope, and mysql jdbc driver to test scope, it's required for implementing MySQL frontend protocol ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1348 from pan3793/mysql-fe-deps. Closes #1348 288ad9c0 [Cheng Pan] [DEPS] Add netty to kyuubi-server module Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
ulysses-you
|
8813c4cf92
|
[KYUUBI #1333] Improve zorder interleave perf
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Make interleave function more faster. The basic idea is from http://graphics.stanford.edu/~seander/bithacks.html#InterleaveTableObvious ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1333 from ulysses-you/zorder-perf. Closes #1333 a9411877 [ulysses-you] address comment bce82fc7 [ulysses-you] simplify 05f0b6e1 [ulysses-you] benchmark result f470a043 [ulysses-you] while a63cfae2 [ulysses-you] improve Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
ulysses-you
|
1003f5296c
|
[KYUUBI #1321] Add config to control if zorder using global sort
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add a new config `spark.sql.optimizer.zorderGlobalSort.enabled` to control if we do a global sort using zorder. This is a trade-off with data skew if the zorder columns has low cardinality. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1321 from ulysses-you/zorder-config. Closes #1321 422599e4 [ulysses-you] delete table c11e2f4f [ulysses-you] doc b984b537 [ulysses-you] Add config to control if zorder using global sort Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Fu Chen
|
1dd341fda2
|
[KYUUBI #1307] Fix zorder boolean column default value
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> As `true > false` and optimize command using nulls last as null ordering strategy, we should set true as the default value of boolean column instead of false ```sql select true > false ``` ``` +--------------+ |(true > false)| +--------------+ | true| +--------------+ ``` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1307 from cfmcgrady/boolean-zorder. Closes #1307 b4ca5a66 [Fu Chen] fix zorder boolean column default value Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Brian Yue
|
7665339244
|
[KYUUBI #460] Upgrade Hive dependency to 2.3.9
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Close #460. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1179 from byyue/feature-460. Closes #460 92eb71b8 [Brian Yue] Upgrade Hive dependency to 2.3.9 Authored-by: Brian Yue <code.byyue@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
589b23f578
|
[KYUUBI #1257] [BUILD] Reduce unnecessary maven profile
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> The changes based on https://github.com/apache/incubator-kyuubi/pull/1226#issuecomment-942979767 In this PR, I'm going to remove profiles `kyuubi-extension-spark-3-1` and `kyuubi-extension-spark-3-2`, and keep `spark-3.0`, `spark-3.1`, `spark-3.2`. After changes, when `spark-3.1` is active, set `spark.version` to 3.1.2 and enable `kyuubi-extension-spark-common` module and `kyuubi-extension-spark-3-1` module, `spark-3.2` does the same thing ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1257 from pan3793/pom. Closes #1257 1d871b6c [Cheng Pan] nit 874f30c1 [Cheng Pan] Address comments 00a32b25 [Cheng Pan] nit 445666ee [Cheng Pan] typo bd96097e [Cheng Pan] nit e85116cd [Cheng Pan] nit 98347c8c [Cheng Pan] Update doc 6615d7cc [Cheng Pan] Exclude HudiTest in cross test 087d3fde [Cheng Pan] Add spark 3.2 binary test ccff63c1 [Cheng Pan] update GA 23b81dcd [Cheng Pan] Address comments 3055c2a8 [Cheng Pan] nit 49a41e86 [Cheng Pan] nit 8d400798 [Cheng Pan] Reduce unnecessary maven profile ced8d987 [Cheng Pan] Reduce unnecessary maven profile Lead-authored-by: Cheng Pan <chengpan@apache.org> Co-authored-by: Cheng Pan <chengpan@ChengdeMac-mini.local> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
ulysses-you
|
f0f065b983
|
[KYUUBI #1268] Simplify extension code with Spark3.2 and Spark3.1 by unify zorder code
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Reduce file and code size to help maintain the extension. - move all zorder code into common module - add KyuubiSparkSQLCommonExtension in common module ### _How was this patch tested?_ Psss CI Closes #1268 from ulysses-you/simplify-extension. Closes #1268 0eeb7161 [ulysses-you] add link c4d35c5c [ulysses-you] simplify Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
d0d5fb6d4f
|
[KYUUBI #1260] Avoid calling javax.xml API for better JDK compatible
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> `javax.xml` API has been separated from JDK since JDK9, avoid calling it for better JDK compatible ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1260 from pan3793/javax. Closes #1260 c4f7d277 [Cheng Pan] Avoid calling javax.xml API for better JDK compatible Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
f03a1443a3
|
[KYUUBI #757] [DEPS] Bump Hadoop 3.3.1
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Spark 3.2.0 is out, which bundles Hadoop 3.3.1 shaded client in default. https://github.com/apache/spark/pull/30135 The test failed when using Hadoop 3.3.1 client connects to Yarn Mini Cluster 3.2.2 ``` Cause: java.lang.RuntimeException: org.apache.kyuubi.KyuubiSQLException:java.lang.ClassCastException: org.apache.hadoop.yarn.proto.YarnServiceProtos$GetClusterMetricsRequestProto cannot be cast to org.apache.hadoop.shaded.com.google.protobuf.Message at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:123) at com.sun.proxy.$Proxy12.getClusterMetrics(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterMetrics(ApplicationClientProtocolPBClientImpl.java:271) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy13.getClusterMetrics(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getYarnClusterMetrics(YarnClientImpl.java:638) at org.apache.spark.deploy.yarn.Client.$anonfun$submitApplication$1(Client.scala:179) at org.apache.spark.internal.Logging.logInfo(Logging.scala:57) at org.apache.spark.internal.Logging.logInfo$(Logging.scala:56) at org.apache.spark.deploy.yarn.Client.logInfo(Client.scala:65) at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:179) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62) at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:220) at org.apache.spark.SparkContext.<init>(SparkContext.scala:581) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690) at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943) at org.apache.kyuubi.engine.spark.SparkSQLEngine$.createSpark(SparkSQLEngine.scala:103) at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:155) at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955) at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:165) at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:163) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #757 from pan3793/hadoop-3.3. Closes #757 7ec9313e [Cheng Pan] [DEPS] Bump Hadoop 3.3.1 Authored-by: Cheng Pan <379377944@qq.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
senmiaoliu
|
f4094d6524
|
[KYUUBI #1085] Fix-Enforce maxOutputRows for Resolve View, Multi Insert and Distribute By
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
1. Should not add limit when multi insert tables.
```sql
FROM tmp_table
insert into tmp_table1 select * limit 2
insert into tmp_table2 select *
```
We should not add any limit when insert table
2. Should add limit when distribute by
```sql
select * from tmp_table distribute by a
```
3. Should not add limit when resolve view
```sql
SELECT * FROM
(select * from tmp_view1 union select * from tmp_view2)
ORDER BY a DESC
```
Like this, when the limit is added in resolve view, the complete data cannot be obtained, so the final result is incorrect
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes #1247 from lsm1/fix/add_limit.
Closes #1085
a02a99c0 [senmiaoliu] Fix-Enforce maxOutputRows for Resolve View, Multi Insert and Distribute By
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
|
||
|
ulysses-you
|
7626973e15
|
[KYUUBI #1245] Improve paddingTo8Byte perf
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add a new benchmark for `paddingTo8Byte` method Before this PR: ``` Java HotSpot(TM) 64-Bit Server VM 1.8.0_271-b09 on Mac OS X 10.16 Intel(R) Core(TM) i7-4770HQ CPU 2.20GHz 10000000 iterations paddingTo8Byte benchmark: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ---------------------------------------------------------------------------------------------------------------------------- 2 length benchmark 2112 2180 78 4.7 211.2 1.0X 16 length benchmark 454 459 6 22.0 45.4 4.6X ``` After this PR: ``` Java HotSpot(TM) 64-Bit Server VM 1.8.0_271-b09 on Mac OS X 10.16 Intel(R) Core(TM) i7-4770HQ CPU 2.20GHz 10000000 iterations paddingTo8Byte benchmark: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ---------------------------------------------------------------------------------------------------------------------------- 2 length benchmark 167 170 3 59.9 16.7 1.0X 16 length benchmark 162 164 3 61.7 16.2 1.0X ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1245 from ulysses-you/improve-zorder-perf. Closes #1245 487c6900 [ulysses-you] improve paddingTo8Byte perf Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
ulysses-you
|
7961b32166
|
[KYUUBI #1241] Build Spark extension fat jar with common module
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> see https://github.com/apache/incubator-kyuubi/issues/1241 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1242 from ulysses-you/KYUUBI-1241. Closes #1241 a8e25c37 [ulysses-you] address comment d25a3664 [ulysses-you] version 3c982938 [ulysses-you] Build Spark extension fat jar with common module Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
eb5749d34b
|
[KYUUBI #1238] [BUILD] Add back Scala stuff to root POM
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> This partial revert [KYUUBI #1203], recover codecov ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1238 from pan3793/mvn. Closes #1238 b3ba87b2 [Cheng Pan] remove unnecessary changes c124340a [Cheng Pan] nit 39c436be [Cheng Pan] pom 28546cbd [Cheng Pan] CI f22e9c91 [Cheng Pan] nit 72e9542d [Cheng Pan] Rebase master 7cd1cba3 [Cheng Pan] [BUILD] Add back Scala stuff to root POM Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
ulysses-you
|
8a5134e322
|
[KYUUBI #1225] Add kyuubi-extension-spark-3-2 module
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> see https://github.com/apache/incubator-kyuubi/issues/1225 Note that: - this PR only supports `Zorder` and some common rules with Spark 3.2.0. The others `watchdog` and `sqlclassification` should be supported in future. - this PR dones't supprpt shade jar and we also need to update build/dist script in a new PR. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1226 from ulysses-you/extension-3.2. Closes #1225 384a0711 [ulysses-you] address comment 08629f20 [ulysses-you] address comment 15f15f6d [ulysses-you] Add kyuubi-extension-spark-3-2 module Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
8a8f49d987
|
[KYUUBI #1233] Add checkout_pr.sh script
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> A convenient tool for checkout PR to local branch. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1233 from pan3793/pr. Closes #1233 97489300 [Cheng Pan] Add checkout_pr.sh script Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
ulysses-you
|
36aab5b95f
|
[KYUUBI #1218] Z-order int and long should respect negative number
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Made a mistake at https://github.com/apache/incubator-kyuubi/pull/1192 that we can't reverse the short highest bit for int and int highest bit for long. Otherwise the negative number will get the wrong order. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1218 from ulysses-you/fix-zorder. Closes #1218 3f7f22d3 [ulysses-you] fix Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
ulysses-you
|
b26ef2ba9e
|
[KYUUBI #1216] Skip zorder if only requires one column
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Improve the perf. If user only requires one column as zorde columns, it has no meaning to wrap a zorder expression. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1216 from ulysses-you/improve-zorder. Closes #1216 9975fccd [ulysses-you] skip zorder if only requires one column Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
ulysses-you
|
6160b9de30
|
[KYUUBI #1214] Refine sql extension and avoid setup data for unused test
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> - Refine rule order - Speed up test ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1214 from ulysses-you/tests. Closes #1214 261271f0 [ulysses-you] test Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
h
|
fdff2b6240
|
[KYUUBI #1085][FOLLOWUP] Fix-Enforce maxOutputRows for aggregate with having statement
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Support `Union` case as below ``` SELECT * FROM t1 UNION [ALL] SELECT * FROM t2 ``` Support `Distinct` case as below ``` SELECT DISTINCT * FROM t1 ``` Fix The bug of watchdog with maxOutputRows happens in this situation as below Having and Sort ``` SELECT c1, COUNT(c2) AS cnt FROM t1 GROUP BY c1 HAVING cnt > 0 [ORDER BY c1, [c2 ...]] ``` It throws Exception as ``` org.apache.spark.sql.catalyst.plans.logical.GlobalLimit cannot be cast to org.apache.spark.sql.catalyst.plans.logical.Aggregate java.lang.ClassCastException: org.apache.spark.sql.catalyst.plans.logical.GlobalLimit cannot be cast to org.apache.spark.sql.catalyst.plans.logical.Aggregate at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$.resolvedAggregateFilter$1(Analyzer.scala:2451) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$.resolveFilterCondInAggregate(Analyzer.scala:2460) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$.resolveHaving(Analyzer.scala:2496) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$$anonfun$apply$21.applyOrElse(Analyzer.scala:2353) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$$anonfun$apply$21.applyOrElse(Analyzer.scala:2345) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUp$3(AnalysisHelper.scala:90) at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:74) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUp$1(AnalysisHelper.scala:90) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:221) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUp(AnalysisHelper.scala:86) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUp$(AnalysisHelper.scala:84) at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsUp(LogicalPlan.scala:29) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$.apply(Analyzer.scala:2345) at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions$.apply(Analyzer.scala:2344) at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:216) at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126) at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122) at scala.collection.immutable.List.foldLeft(List.scala:91) at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:213) at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1$adapted(RuleExecutor.scala:205) at scala.collection.immutable.List.foreach(List.scala:431) at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:205) at org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:196) at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:190) at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:155) at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$executeAndTrack$1(RuleExecutor.scala:183) at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:88) at org.apache.spark.sql.catalyst.rules.RuleExecutor.executeAndTrack(RuleExecutor.scala:183) at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$executeAndCheck$1(Analyzer.scala:174) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:228) at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:173) at org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:73) at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111) at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:143) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:143) at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:73) at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:71) at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:63) at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:98) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:96) at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:618) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613) at org.apache.spark.sql.test.SQLTestUtilsBase.$anonfun$sql$1(SQLTestUtils.scala:231) at org.apache.spark.sql.KyuubiExtensionSuite.$anonfun$new$55(KyuubiExtensionSuite.scala:1331) at org.apache.spark.sql.catalyst.plans.SQLHelper.withSQLConf(SQLHelper.scala:54) at org.apache.spark.sql.catalyst.plans.SQLHelper.withSQLConf$(SQLHelper.scala:38) at ``` Reference related issue: https://issues.apache.org/jira/browse/SPARK-31519 Spark SQL Ananlyzer transform aggregate with having to ``` Filter +- Aggregate ``` Solution: 1. Skip the aggregate with havingCondition 2. Match Filter for Adding Limit ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate <img width="1440" alt="截屏2021-09-21 下午8 35 16" src="https://user-images.githubusercontent.com/635169/134171308-2842f0d4-acfa-4817-a03c-a7ef5e38df12.png"> - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1129 from i7xh/fixAggWithHavingInMaxOutput. Closes #1085 7577f4d3 [h] update 5955c89e [h] update 384b2333 [h] fix issue 5b0af156 [h] update 46327fc6 [h] Fix issue 6119a039 [h] fix issue a7b87dd7 [h] Fix issue 2570444e [h] BugFix: Aggregate with having statement Authored-by: h <h@zhihu.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
9e6eb086b1
|
[KYUUBI #1203] [SUB-TASK][BUILD] Remove Scala stuff from kyuubi-hive-jdbc pom
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> #1131 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1203 from pan3793/pom. Closes #1203 8e6b8f9b [Cheng Pan] fix d31fd198 [Cheng Pan] fix 9da82843 [Cheng Pan] fix 6d150de9 [Cheng Pan] fix 9faf77e5 [Cheng Pan] Remove Scala stuff from kyuubi-hive-jdbc pom Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
ulysses-you
|
afb46ec152
|
[KYUUBI #1192] Refine zorder utils toByte
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Fix some issue: 1. int/long type miss update highest byte of byte array 2. short/int/long same value may have different byte array 3. the mask value of long is error. (1L << 31 instead of 1 << 31) 4. the default value of string should be Long.Max since we pad it to 8 bytes so we can keep the ordering of null value (null last) 5. boolean to byte doesn't need 4 bytes, 1 byte is enough ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1192 from ulysses-you/improve-zorder. Closes #1192 7a0cba28 [ulysses-you] byte 2271b427 [ulysses-you] refine b13d2652 [ulysses-you] refine zorder Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Cheng Pan
|
191d30e2ac
|
[KYUUBI #1167][FOLLOWUP] Refine ZorderCoreBenchmark doc
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Minor change, update `ZorderCoreBenchmark` doc. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1172 from pan3793/t. Closes #1167 8372a3a7 [Cheng Pan] nit 1b6ee9da [Cheng Pan] typo bd1817a7 [Cheng Pan] [KYUUBI #1167][FOLLOWUP] Refine ZorderCoreBenchmark doc Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
ulysses-you
|
627d271fc6
|
[KYUUBI #1167] Add zorder core benchmark
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add zorder core benchmark, so we can get the perf number of zorder. This PR adds two column type `int` and `long` benchmark: - 2 int cols - 3 int cols - 4 int cols - 2 long cols - 3 long cols - 4 long cols ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1167 from ulysses-you/benchmark. Closes #1167 27ecf51f [ulysses-you] Add zorder core benchmark Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
zhouyifan279
|
114ec9a046
|
[KYUUBI #1009][FOLLOWUP] Kyuubi server failed to start when connecting to secured HMS
### _Why are the changes needed?_ Fix #1161 [Bug] Kyuubi server failed to start when connecting to a secured Hive metastore ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate  - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1162 from zhouyifan279/KYUUBI#1161. Closes #1009 f5b54818 [zhouyifan279] [KYUUBI #1161][Bug] Kyuubi server failed to start when connecting to a secured Hive metastore Authored-by: zhouyifan279 <zhouyifan279@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Min Zhao
|
d829bae526
|
[KYUUBI #939][FOLLOWUP][TEST] Add string column test for Z-Order
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Added string column test for zorder. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1157 from zhaomin1423/zorder_test. Closes #939 59543776 [Min Zhao] add string column test for zorder Authored-by: Min Zhao <zhaomin1423@163.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
ulysses-you
|
7a3545e748
|
[KYUUBI #1002][FOLLOWUP] Refine sql classification rule
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Follow up [KYUUBI #1002] [KYUUBI #1035] [KYUUBI #1037] - correct the config key; we should use `spark.` as the prefix - change the config to false by default - correct the class loader; thread context is not safe, spark and hive often change it - update the log and exception following is the failed error msg: ``` Caused by: java.io.FileNotFoundException: file:/Users/cathy/Desktop/tmp/spark-3.1.2-bin-hadoop2.7/jars/kyuubi-extension-spark-3-1_2.12-1.4.0-SNAPSHOT.jar!/sql-classification-default.json (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:916) at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2745) at org.apache.kyuubi.sql.sqlclassification.KyuubiGetSqlClassification$.<init>(KyuubiGetSqlClassification.scala:51) at org.apache.kyuubi.sql.sqlclassification.KyuubiGetSqlClassification$.<clinit>(KyuubiGetSqlClassification.scala) ... 59 more ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1140 from ulysses-you/refine. Closes #1002 f39ef2cf [ulysses-you] address comment 5cb94459 [ulysses-you] nit de9fd932 [ulysses-you] test 78e8c147 [ulysses-you] default 6f23c63e [ulysses-you] refine Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Simon
|
7b7d228fb4
|
[KYUUBI #1123][TEST] Split KyuubiExtensionSuite according to different extensions
split KyuubiExtensionSuite according to different extensions #1123 drop： `KyuubiSparkExtensionTest` add： - `FinalStageConfigIsolationSuite` - `InsertShuffleNodeBeforeJoinSuite` - `WatchDogSuite` - `RepartitionBeforeWriteSuite` - `SqlClassificationSuite` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1124 from simon824/extension-ut. Closes #1123 4b1ae4db [Simon] Update FinalStageConfigIsolationSuite.scala 2068a74d [Simon] Update SparkProcessBuilder.scala cd6aa7d2 [Simon] Merge branch 'apache:master' into extension-ut 1401d69c [simon] rename test to suite 98095823 [Simon] Rename WatchDogSuit.scala to WatchDogSuite.scala 0392d5b5 [Simon] Rename SqlClassificationSuit.scala to SqlClassificationSuite.scala a8776df9 [Simon] Rename RepartitionBeforeWriteSuit.scala to RepartitionBeforeWriteSuite.scala 4d06966c [Simon] Rename InsertShuffleNodeBeforeJoinSuit.scala to InsertShuffleNodeBeforeJoinSuite.scala 60b2932e [Simon] Rename FinalStageConfigIsolationSuit.scala to FinalStageConfigIsolationSuite.scala 175fb47f [Simon] Rename WatchDogTest.scala to WatchDogSuit.scala 755ad22c [Simon] Rename SqlClassificationTest.scala to SqlClassificationSuit.scala b36171b8 [Simon] Rename RepartitionBeforeWriteTest.scala to RepartitionBeforeWriteSuit.scala 74c110da [Simon] Rename InsertShuffleNodeBeforeJoinTest.scala to InsertShuffleNodeBeforeJoinSuit.scala 405d8b72 [Simon] Rename FinalStageConfigIsolationTest.scala to FinalStageConfigIsolationSuit.scala 7a158057 [simon] fix codestyle 86148f3e [simon] fix ut 291c93aa [simon] split KyuubiExtensionSuite to different extension ut class 2098ce89 [simon] split KyuubiExtensionSuite to different extension ut class 2aaed1c8 [simon] Merge remote-tracking branch 'upstream/master' into extension-ut Lead-authored-by: Simon <3656562@qq.com> Co-authored-by: simon <zhangshiming@cvte.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Fu Chen
|
208c74d197
|
[KYUUBI #1105][FOLLOWUP] Minor improve ZorderBytesUtils.defaultValue
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> fix [comment](https://github.com/apache/incubator-kyuubi/pull/1109#discussion_r711007709) ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1126 from cfmcgrady/zorder. Closes #1105 48e4d73a [Fu Chen] review a58c5e52 [Fu Chen] minor improve ZorderBytesUtils.defaultValue Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
h
|
cc3ca56ab2
|
[KYUUBI #1085] Add forcedMaxOutputRows rule for limitation to avoid huge output unexpectly
Add MaxOutputRows rule for output rows limitation to avoid huge output unexpectedly <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add <img width="1440" alt="截屏2021-09-12 下午12 19 28" src="https://user-images.githubusercontent.com/635169/132972063-b12937bb-807a-47bd-8d21-835d83031191.png"> '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> We support the PR feature with limitation that avoid huge output rows in user ad-hoc query unexpected，Generally ad-hoc query seems needle in a Haystack, user pick few computed result data in huge data from warehouse, we mainly used in below cases: - CASE 1: ``` SELECT [c1, c2, ...] ``` - CASE 2: ``` WITH CTE AS (...) SELECT [c1, c2, ...] FROM Express(CTE) ... ``` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate <img width="1440" alt="截屏2021-09-12 下午12 19 28" src="https://user-images.githubusercontent.com/635169/132972078-c4821135-0520-420d-9ab8-24e124f6c6c9.png"> - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1085 from i7xh/watchdogMaxOutputRows. Closes #1085 77aa6ff3 [h] Resolve issue: unify maxOutputRows and using unittest setupData 939d2556 [h] Resolve issue: well-format code 5b0383dd [h] Fix issue: Final Limit according to spark.sql.watchdog.forcedMaxOutputRows 7dcbb0a4 [h] Resolve issue: remove origin match from rule f4dff4cf [h] Resolve issue: update forcedMaxOutputRows config doc ae21c1ac [h] Resovled issue: Support Aggregate force limitation and Remove InsertIntoDataSourceDirCommand process a9d3640b [h] Resolved code review issue 01c87fd2 [h] Add MaxOutputRows rule for output rows limitation to avoid huge output rows of query unexpectedly Authored-by: h <h@zhihu.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Brian Yue
|
cbe5bee6ec
|
[KYUUBI #1112] Upgrade scala to 2.12.15
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Close #1112. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1120 from byyue/feature-1112. Closes #1112 9bb10858 [Brian Yue] [FEATURE] [Dependency] Upgrade scala to 2.14.15 Authored-by: Brian Yue <code.byyue@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
timothy65535
|
6983139487
|
[KYUUBI #1111] Remove duplicate keys
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Remove duplicate keys. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1111 from timothy65535/remove. Closes #1111 75b001e3 [timothy65535] Remove duplicate keys Authored-by: timothy65535 <timothy65535@163.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Fu Chen
|
7da30bd992
|
[KYUUBI #1105][FOLLOWUP][TEST] Refine KyuubiSparkSQLExtension suites
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> fix [comment](https://github.com/apache/incubator-kyuubi/pull/1109#discussion_r709838217) ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1118 from cfmcgrady/refine-extension-test. Closes #1105 a3f31e5d [Fu Chen] minor 853bf8f2 [Fu Chen] fix style 89f2338a [Fu Chen] minor d4d57ebd [Fu Chen] refine KyuubiSparkSQLExtension suites Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
Fu Chen
|
5602c29b36
|
[KYUUBI #1105] Zorder codegen support
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> 1. zorder codegen support 2. move all of the zorder tests to a separate file. 3. add more zorder unit tests. generate.java ```java /* 001 */ public java.lang.Object generate(Object[] references) { /* 002 */ return new SpecificUnsafeProjection(references); /* 003 */ } /* 004 */ /* 005 */ class SpecificUnsafeProjection extends org.apache.spark.sql.catalyst.expressions.UnsafeProjection { /* 006 */ /* 007 */ private Object[] references; /* 008 */ private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[] mutableStateArray_0 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[1]; /* 009 */ /* 010 */ public SpecificUnsafeProjection(Object[] references) { /* 011 */ this.references = references; /* 012 */ mutableStateArray_0[0] = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(1, 32); /* 013 */ /* 014 */ } /* 015 */ /* 016 */ public void initialize(int partitionIndex) { /* 017 */ /* 018 */ } /* 019 */ /* 020 */ // Scala.Function1 need this /* 021 */ public java.lang.Object apply(java.lang.Object row) { /* 022 */ return apply((InternalRow) row); /* 023 */ } /* 024 */ /* 025 */ public UnsafeRow apply(InternalRow i) { /* 026 */ mutableStateArray_0[0].reset(); /* 027 */ /* 028 */ /* 029 */ /* 030 */ /* 031 */ byte[] value_0 = null; /* 032 */ byte[][] binaryArray_0 = new byte[2][]; /* 033 */ /* 034 */ boolean isNull_1 = i.isNullAt(0); /* 035 */ long value_1 = isNull_1 ? /* 036 */ -1L : (i.getLong(0)); /* 037 */ if (isNull_1) { /* 038 */ binaryArray_0[0] = (byte[]) ((byte[][]) references[0] /* defaultValues */)[0]; /* 039 */ } else { /* 040 */ binaryArray_0[0] = org.apache.kyuubi.sql.zorder.ZorderBytesUtils.toByte(value_1); /* 041 */ } /* 042 */ /* 043 */ /* 044 */ boolean isNull_2 = i.isNullAt(1); /* 045 */ long value_2 = isNull_2 ? /* 046 */ -1L : (i.getLong(1)); /* 047 */ if (isNull_2) { /* 048 */ binaryArray_0[1] = (byte[]) ((byte[][]) references[0] /* defaultValues */)[1]; /* 049 */ } else { /* 050 */ binaryArray_0[1] = org.apache.kyuubi.sql.zorder.ZorderBytesUtils.toByte(value_2); /* 051 */ } /* 052 */ /* 053 */ value_0 = org.apache.kyuubi.sql.zorder.ZorderBytesUtils.interleaveMultiByteArray(binaryArray_0); /* 054 */ mutableStateArray_0[0].write(0, value_0); /* 055 */ return (mutableStateArray_0[0].getRow()); /* 056 */ } /* 057 */ /* 058 */ /* 059 */ } ``` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1109 from cfmcgrady/zorder-codegen. Closes #1105 f06da25a [Fu Chen] update dependency 8e04dc4e [Fu Chen] JMap -> Array da00f86c [Fu Chen] remove Logging trait 266b65b3 [Fu Chen] fix style and revert Expression e15165f3 [Fu Chen] ev.isNull = FalseLiteral 0d77d9de [Fu Chen] review 0d8467f4 [Fu Chen] fix style 5ede5df9 [Fu Chen] refactor test 56e12172 [Fu Chen] clean up 88573102 [Fu Chen] update 5f3d1c02 [Fu Chen] zorder codegen support Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
ulysses-you
|
43390834b1
|
[KYUUBI #1100] Refine zorder expression and add test
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> - refine the zorder expression - improve coverage ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1100 from ulysses-you/zorder. Closes #1100 d07ad8ae [ulysses-you] value 5b160384 [ulysses-you] refine zorder expression Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
h
|
9dfbd2b830
|
[KYUUBI #1086] Add maxHivePartitions Strategy to avoid scanning excessive hive partitions on partitioned table
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> The PR support this Strategy to avoid scan huge hive partitions or unused necessary partition filter on partitioned table，It will abort the SQL query with thrown exception via scan hive table partitions exceed `spark.sql.watchdog.maxHivePartitions`, meanwhile show the related hive table partition struct to assist user to optimize sql according the given suggestion ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate <img width="1440" alt="截屏2021-09-12 下午2 30 20" src="https://user-images.githubusercontent.com/635169/132974627-7879ee2b-4624-4bb4-bec9-3ed0940c3986.png"> - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1086 from i7xh/watchdogMaxHivePartition. Closes #1086 d535b57e [h] Fixed: scala style newline of end 774a486f [h] Resolved isssue: shrink unnesscary test data 0541dcc5 [h] Resovled issue: replace listPartitions with listPartitionNames 5a97c5d0 [h] Resolved issue: unify Exceed maxHivePartitions without partition filter 1581b297 [h] resovled the code review issues d0b0fc46 [h] Add maxHivePartitions Strategy to avoid scan excessive hive partitions on partitioned table Authored-by: h <h@zhihu.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
||
|
ulysses-you
|
f6d4bba101
|
[KYUUBI #1070] Support insert zorder by table properties
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Closes #1070 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1071 from ulysses-you/tblproperties. Closes #1070 37800666 [ulysses-you] add test 3dec3f81 [ulysses-you] support insert zorder by table properties Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: ulysses-you <ulyssesyou@apache.org> |
