apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity
1,350 Commits 23 Branches 0 Tags 65 MiB
10582c8b96
Commit Graph

1350 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
wForget
10582c8b96
[KYUUBI #1417] Parse instance information in a compatible way 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Fix bug #1417.

After fix, run the `bin/kyuubi-ctl list server` command successfully.

![image](https://user-images.githubusercontent.com/17894939/142574175-161e40d8-ada2-4a8f-bd80-1835118ac281.png)

### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [X] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1418 from wForget/KYUUBI-1417.

Closes #1417

4c662617 [wForget] [KYUUBI-1417] fix checkstyle
44759fea [wForget] [KYUUBI-1417] Parse instance information in a compatible way

Authored-by: wForget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-19 15:27:49 +08:00
fwang12
e5e49b5b1d
[KYUUBI #1414] Add kyuubi-hive-beeline module to support KyuubiConnection and KyuubiStatement 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Add kyuubi-hive-beeline module to support KyuubiConnection and KyuubiStatement.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate
<img width="1410" alt="Screen Shot 2021-11-19 at 1 56 21 PM" src="https://user-images.githubusercontent.com/6757692/142572735-409afe25-b96a-49da-8191-e9a8ced3947b.png">

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1414 from turboFei/rebase-kyuubi_beeline.

Closes #1414

4694bbca [fwang12] return engine launch handle anyway
20ca1400 [fwang12] address comments
2999c30a [fwang12] fix ut
4d3abf4f [fwang12] jars.target.dir to target
4659f1f6 [fwang12] add readme
7e74c5c8 [fwang12] integrate beeline and server
969af045 [fwang12] fix beeline shell
90395ade [fwang12] BeeLine version By Apache Kyuubi
4e63695e [fwang12] fix ut
026e41ef [fwang12] refactor properties
a49677af [fwang12] refactor build and beeline script
ca68eba1 [fwang12] use beeline mode property instead of static method
4a1932e7 [fwang12] cli -> beeline
624281e3 [fwang12] nit
d7d27a09 [fwang12] refactor kyuubi cli assembly
9d9b25a9 [fwang12] simplify the waitLaunchEngineToComplete
cc025ecf [fwang12] Rebase PR_1407

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-19 15:05:10 +08:00
sunfangbin
e54fabab68
[KYUUBI #1416] Refine jline dependency 
### _Why are the changes needed?_
Follows on #1413.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1416 from murong00/branch-1415.

Closes #1416

05335c45 [sunfangbin] Refine jline dependency

Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-19 10:40:40 +08:00
sunfangbin
fb8bf49dbc
[KYUUBI #1413] Improve usage of zookeeper cli 
### _Why are the changes needed?_
This change enabled the JLine support so that the `kyuubi-zookeeper-shell` command can be easier to use with history, prompt and completion, etc. Note that the `jline` version `0.9.94` is dependent on `zookeeper` version `3.4.14` which is a lower version. BTW, users can consider upgrading the `zookeeper` version for the sake of performance improvements and new features.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1413 from murong00/branch-1413.

Closes #1413

2d1ec084 [sunfangbin] Rename shell script to zk-cli
d09f64dc [sunfangbin] Fixes failed ci
ce88357a [sunfangbin] Improve usage of zookeeper cli

Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-18 19:17:11 +08:00
Cheng Pan
09f04d6de8
[KYUUBI #1410] Add reformat shell script 
### _Why are the changes needed?_

Following up #1403

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1410 from pan3793/reformat.

Closes #1410

288fc1b5 [Cheng Pan] Add reformat shell script

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-18 18:39:22 +08:00
Cheng Pan
1a3228f132 [KYUUBI #1415] Immediately wrap the thrift client with a thread-safe proxy 
### _Why are the changes needed?_

Sync before async engine log request has been sent
```
- open session with KyuubiConnection *** FAILED ***
  java.sql.SQLException: org.apache.kyuubi.shade.org.apache.thrift.transport.TTransportException: Read a negative frame size (-2147418110)!
  at org.apache.kyuubi.jdbc.hive.KyuubiStatement.runAsyncOnServer(KyuubiStatement.java:271)
  at org.apache.kyuubi.jdbc.hive.KyuubiStatement.execute(KyuubiStatement.java:193)
  at org.apache.kyuubi.operation.KyuubiOperationPerConnectionSuite.$anonfun$new$20(KyuubiOperationPerConnectionSuite.scala:145)
  at org.apache.kyuubi.operation.HiveJDBCTestHelper.withSessionConf(HiveJDBCTestHelper.scala:60)
  at org.apache.kyuubi.operation.HiveJDBCTestHelper.withSessionConf$(HiveJDBCTestHelper.scala:53)
  at org.apache.kyuubi.operation.KyuubiOperationPerConnectionSuite.withSessionConf(KyuubiOperationPerConnectionSuite.scala:33)
  at org.apache.kyuubi.operation.KyuubiOperationPerConnectionSuite.$anonfun$new$19(KyuubiOperationPerConnectionSuite.scala:140)
  at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
  at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
  at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
  ...
  Cause: org.apache.kyuubi.shade.org.apache.thrift.transport.TTransportException: Read a negative frame size (-2147418110)!
  at org.apache.kyuubi.shade.org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:456)
  at org.apache.kyuubi.shade.org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435)
  at org.apache.kyuubi.shade.org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
  at org.apache.kyuubi.shade.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
  at org.apache.kyuubi.shade.org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
  at org.apache.kyuubi.shade.org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
  at org.apache.kyuubi.shade.org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
  at org.apache.kyuubi.shade.org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
  at org.apache.kyuubi.shade.org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_ExecuteStatement(TCLIService.java:237)
  at org.apache.kyuubi.shade.org.apache.hive.service.rpc.thrift.TCLIService$Client.ExecuteStatement(TCLIService.java:224)
  ...
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1415 from pan3793/sync.

Closes #1415

5b7ca1bd [Cheng Pan] Immediately wrap the thrift client with a thread-safe proxy

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-11-18 18:25:01 +08:00
yanyu34946
93e9745a39
[KYUUBI #1293] Fix potential failure of engine discovery when mixed use subdomain 
### _Why are the changes needed?_

Fix #1293

This PR is a rework of #1303, also related to #962

We have two options to fix this issue. This PR implements option 2.

```
[option 1]
kyuubi.engine.share.level.subdomain: String -- default: "default"
kyuubi.engine.pool.size: Int                -- default: -1

val subdomain =
    if (kyuubi.engine.share.level.subdomain == "default" && kyuubi.engine.pool.size > 0) {
        s"engine-pool-[num]"
    } else {
        kyuubi.engine.share.level.subdomain
    }

[option 2]
kyuubi.engine.share.level.subdomain: Option[String] -- default: None
kyuubi.engine.pool.size: Int                        -- default: -1

val subdomain =
    kyuubi.engine.share.level.subdomain match {
        case Some(value) => value
        case None if kyuubi.engine.pool.size > 0 => s"engine-pool-[num]"
        case None => "default"
    }
```
```
[diff]
[[case1]]
kyuubi.engine.share.level.subdomain=default
kyuubi.engine.pool.size=2

option 1 result: subdomain="engine-pool-[num]"
option 2 result: subdomain="default"

[[case2]]
kyuubi.engine.share.level.subdomain=hello
kyuubi.engine.pool.size=2

option 1 result: subdomain="hello"
option 2 result: subdomain="hello"

[[case3]]
kyuubi.engine.share.level.subdomain=
kyuubi.engine.pool.size=2

option 1 result: subdomain="engine-pool-[num]"
option 2 result: subdomain="engine-pool-[num]"

[[case4]]
kyuubi.engine.share.level.subdomain=
kyuubi.engine.pool.size=

option 1 result: subdomain="default"
option 2 result: subdomain="default"

[[case5]]
kyuubi.engine.share.level.subdomain="hello"
kyuubi.engine.pool.size=

option 1 result: subdomain="hello"
option 2 result: subdomain="hello"
```

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1402 from pan3793/1293.

Closes #1293

f6c4a7db [Cheng Pan] nit
fd427bfb [yanyu34946] [KYUUBI #1293] Fix bootstrap potential failure when mixed use subdomain

Lead-authored-by: yanyu34946 <875082356@qq.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-18 18:00:30 +08:00
simon
dd343fb476
[KYUUBI #1354][FOLLOWUP] Update url to camel case style 
### _Why are the changes needed?_
Update url to camel case style
[#1354](https://github.com/apache/incubator-kyuubi/pull/1408#discussion_r751948135)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1412 from simon824/r3.

Closes #1354

aa933713 [Simon] Merge branch 'apache:master' into r3
5260926a [simon] camel case style
6349ac5a [simon] ApiResponse
b1f906e5 [simon] init

Lead-authored-by: simon <zhangshiming@cvte.com>
Co-authored-by: Simon <3656562@qq.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-18 16:40:01 +08:00
sunfangbin
995c74b8ee
[KYUUBI #1409] [MINOR] Fix the missing suffix of tarball 
### _Why are the changes needed?_
Fixes the missing suffix of tarball, which should be `$TARDIR_NAME.tgz`.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1409 from murong00/branch-1409.

Closes #1409

2ba2718a [sunfangbin] Fix the missing suffix of tarball

Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-18 14:49:45 +08:00
simon
5bf08495b9
[KYUUBI #1354][FOLLOWUP] Add ApiResponse for operation apis 
### _Why are the changes needed?_
Add ApiResponse for operation apis
#1354

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1408 from simon824/r3.

Closes #1354

6349ac5a [simon] ApiResponse
b1f906e5 [simon] init

Authored-by: simon <zhangshiming@cvte.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-18 14:48:06 +08:00
yanghua
2baabf46ca
[KYUUBI #1384] Introduce spotless and google java format for java code 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1403 from yanghua/KYUUBI-1384.

Closes #1384

e4e12088 [yanghua] address review suggestion
17a4cb86 [yanghua] reformat all the java files based on the offical google java code style
10a2ca04 [yanghua] [KYUUBI #1384] Introduce Java code style and formatter

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-18 10:45:51 +08:00
Cheng Pan
56a88f1e48
[KYUUBI #1395] Implement MySQL protocol codec 
### _Why are the changes needed?_

Sub task of #1219

The required `MySQLPacket`s are split into 4 groups, as Kyuubi is designed to be a "MySQL Server", only part of packets need to be encodable and others just need to be decodable.

- Generic
  - `MySQLOKPacket` with `SupportsEncode`
  - `MySQLErrPacket` with `SupportsEncode`
  - `MySQLEofPacket` with `SupportsEncode`
- Authentication
  - `MySQLHandshakePacket` with `SupportsEncode`
  - `MySQLHandshakeResponse41Packet` with `SupportsDecode`
  - `MySQLAuthSwitchRequestPacket` with `SupportsEncode`
  - `MySQLAuthSwitchResponsePacket` with `SupportsDecode`
- Command
  - `MySQLComPingPacket` with `SupportsDecode`
  - `MySQLComInitDbPacket` with `SupportsDecode`
  - `MySQLComFieldListPacket` with `SupportsDecode`
  - `MySQLComQueryPacket` with `SupportsDecode`
  - `MySQLComQuitPacket` with `SupportsDecode`
- Data
  - `MySQLFieldCountPacket`  with`SupportsEncode`
  - `MySQLColumnDefinition41Packet` with `SupportsEncode`
  - `MySQLTextResultSetRowPacket` with `SupportsEncode`

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1395 from pan3793/mysql-protocol.

Closes #1395

25b7b3d8 [Cheng Pan] Nit
90637ea1 [Cheng Pan] Fix compile
7fae786e [Cheng Pan] Use while
47a1da82 [Cheng Pan] Add MySQL links
b6d91f7b [Cheng Pan] Implement MySQL protocol codec

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-17 18:42:01 +08:00
Fu Chen
a5ef94accc
[KYUUBI #1354] Expose OpenAPI Specifications 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

1. add a new endpoint `/openapi.json` which describes the entire restful API of Kyuubi.
2. add a static web page `/api/v1/swagger-ui`. see [swagger-ui](https://github.com/swagger-api/swagger-ui)

```
curl http://localhost:10099/openapi.json | jq .
```

```json
{
  "openapi": "3.0.1",
  "info": {
    "title": "Apache Kyuubi RESTful Api Documentation",
    "description": "Apache Kyuubi RESTful Api Documentation",
    "contact": {
      "name": "Apache Kyuubi Community",
      "url": "https://kyuubi.apache.org/issue_tracking.html",
      "email": "devkyuubi.apache.org"
    },
    "license": {
      "name": "Apache 2.0",
      "url": "https://www.apache.org/licenses/LICENSE-2.0.html"
    },
    "version": "1.4.0"
  },
  "tags": [
    {
      "name": "Session"
    }
  ],
  "paths": {
    "/api/v1/ping": {
      "get": {
        "operationId": "ping",
        "responses": {
          "default": {
            "description": "default response",
            "content": {
              "text/plain": {
                "schema": {
                  "type": "string"
                }
              }
            }
          }
        }
      }
    },
    "/api/v1/sessions/{sessionHandle}": {
      "get": {
        "tags": [
          "Session"
        ],
        "operationId": "sessionInfo",
        "parameters": [
          {
            "name": "sessionHandle",
            "in": "path",
            "required": true,
            "schema": {
              "type": "string"
            }
          }
        ],
        "responses": {
          "200": {
            "description": "get a session via session handle identifier",
            "content": {
              "application/json": {}
            }
          }
        }
      },
      "delete": {
        "tags": [
          "Session"
        ],
        "operationId": "closeSession",
        "parameters": [
          {
            "name": "sessionHandle",
            "in": "path",
            "required": true,
            "schema": {
              "type": "string"
            }
          }
        ],
        "responses": {
          "200": {
            "description": "close a Session",
            "content": {
              "application/json": {}
            }
          }
        }
      }
    },
    "/api/v1/sessions/execPool/statistic": {
      "get": {
        "tags": [
          "Session"
        ],
        "operationId": "execPoolStatistic",
        "responses": {
          "200": {
            "description": "get some statistic info of sessions",
            "content": {
              "application/json": {}
            }
          }
        }
      }
    },
    "/api/v1/sessions/{sessionHandle}/info/{infoType}": {
      "get": {
        "tags": [
          "Session"
        ],
        "operationId": "getInfo",
        "parameters": [
          {
            "name": "sessionHandle",
            "in": "path",
            "required": true,
            "schema": {
              "type": "string"
            }
          },
          {
            "name": "infoType",
            "in": "path",
            "required": true,
            "schema": {
              "type": "integer",
              "format": "int32"
            }
          }
        ],
        "responses": {
          "200": {
            "description": "get a information detail via session handle identifier and a specific information type",
            "content": {
              "application/json": {}
            }
          }
        }
      }
    },
    "/api/v1/sessions": {
      "get": {
        "tags": [
          "Session"
        ],
        "operationId": "sessionInfoList",
        "responses": {
          "200": {
            "description": "get all the session list hosted in SessionManager",
            "content": {
              "application/json": {}
            }
          }
        }
      },
      "post": {
        "tags": [
          "Session"
        ],
        "operationId": "openSession",
        "requestBody": {
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/SessionOpenRequest"
              }
            }
          }
        },
        "responses": {
          "200": {
            "description": "Open(create) a Session",
            "content": {
              "application/json": {}
            }
          }
        }
      }
    },
    "/api/v1/sessions/count": {
      "get": {
        "tags": [
          "Session"
        ],
        "operationId": "sessionCount",
        "responses": {
          "200": {
            "description": "get open session count",
            "content": {
              "application/json": {}
            }
          }
        }
      }
    },
    "/api/v1/exception": {
      "get": {
        "operationId": "test",
        "responses": {
          "default": {
            "description": "default response",
            "content": {
              "text/plain": {}
            }
          }
        }
      }
    },
    "/application.wadl/{path}": {
      "get": {
        "operationId": "getExternalGrammar",
        "parameters": [
          {
            "name": "path",
            "in": "path",
            "required": true,
            "schema": {
              "type": "string"
            }
          }
        ],
        "responses": {
          "default": {
            "description": "default response",
            "content": {
              "application/xml": {}
            }
          }
        }
      }
    },
    "/application.wadl": {
      "get": {
        "operationId": "getWadl",
        "responses": {
          "default": {
            "description": "default response",
            "content": {
              "application/vnd.sun.wadl+xml": {},
              "application/xml": {}
            }
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "ExecPoolStatistic": {
        "required": [
          "execPoolSize"
        ],
        "type": "object",
        "properties": {
          "execPoolSize": {
            "type": "integer",
            "format": "int32"
          },
          "execPoolActiveCount": {
            "type": "integer",
            "format": "int32"
          }
        }
      },
      "InfoDetail": {
        "required": [
          "infoType"
        ],
        "type": "object",
        "properties": {
          "infoType": {
            "type": "string"
          },
          "infoValue": {
            "type": "string"
          }
        }
      },
      "HandleIdentifier": {
        "required": [
          "publicId"
        ],
        "type": "object",
        "properties": {
          "publicId": {
            "type": "string",
            "format": "uuid"
          },
          "secretId": {
            "type": "string",
            "format": "uuid"
          }
        }
      },
      "SessionHandle": {
        "required": [
          "identifier",
          "protocol"
        ],
        "type": "object",
        "properties": {
          "identifier": {
            "$ref": "#/components/schemas/HandleIdentifier"
          },
          "protocol": {
            "type": "string",
            "enum": [
              "HIVE_CLI_SERVICE_PROTOCOL_V1",
              "HIVE_CLI_SERVICE_PROTOCOL_V2",
              "HIVE_CLI_SERVICE_PROTOCOL_V3",
              "HIVE_CLI_SERVICE_PROTOCOL_V4",
              "HIVE_CLI_SERVICE_PROTOCOL_V5",
              "HIVE_CLI_SERVICE_PROTOCOL_V6",
              "HIVE_CLI_SERVICE_PROTOCOL_V7",
              "HIVE_CLI_SERVICE_PROTOCOL_V8",
              "HIVE_CLI_SERVICE_PROTOCOL_V9",
              "HIVE_CLI_SERVICE_PROTOCOL_V10"
            ]
          }
        }
      },
      "MapStringString": {
        "type": "object",
        "additionalProperties": {
          "type": "string"
        }
      },
      "SessionOpenRequest": {
        "required": [
          "configs",
          "protocolVersion",
          "user"
        ],
        "type": "object",
        "properties": {
          "protocolVersion": {
            "type": "integer",
            "format": "int32"
          },
          "user": {
            "type": "string"
          },
          "password": {
            "type": "string"
          },
          "ipAddr": {
            "type": "string"
          },
          "configs": {
            "type": "object",
            "additionalProperties": {
              "type": "string"
            }
          }
        }
      },
      "SessionOpenCount": {
        "required": [
          "openSessionCount"
        ],
        "type": "object",
        "properties": {
          "openSessionCount": {
            "type": "integer",
            "format": "int32"
          }
        }
      },
      "SessionDetail": {
        "required": [
          "configs",
          "createTime",
          "user"
        ],
        "type": "object",
        "properties": {
          "user": {
            "type": "string"
          },
          "ipAddr": {
            "type": "string"
          },
          "createTime": {
            "type": "integer",
            "format": "int64"
          },
          "sessionHandle": {
            "$ref": "#/components/schemas/SessionHandle"
          },
          "lastAccessTime": {
            "type": "integer",
            "format": "int64"
          },
          "lastIdleTime": {
            "type": "integer",
            "format": "int64"
          },
          "noOperationTime": {
            "type": "integer",
            "format": "int64"
          },
          "configs": {
            "type": "object",
            "additionalProperties": {
              "type": "string"
            }
          }
        }
      },
      "SeqSessionOverview": {
        "type": "array",
        "items": {
          "$ref": "#/components/schemas/SessionOverview"
        }
      },
      "SessionList": {
        "required": [
          "sessionList"
        ],
        "type": "object",
        "properties": {
          "sessionList": {
            "type": "array",
            "items": {
              "$ref": "#/components/schemas/SessionOverview"
            }
          }
        }
      },
      "SessionOverview": {
        "required": [
          "createTime",
          "user"
        ],
        "type": "object",
        "properties": {
          "user": {
            "type": "string"
          },
          "ipAddr": {
            "type": "string"
          },
          "createTime": {
            "type": "integer",
            "format": "int64"
          },
          "sessionHandle": {
            "$ref": "#/components/schemas/SessionHandle"
          }
        }
      }
    }
  }
}
```
http://localhost:10099/api/v1/swagger-ui

> Note that this request will redirect to http://localhost:10099/swagger-ui-redirected/index.html?url=http://localhost:10099/openapi.json

![截屏2021-11-09 上午10 43 49](https://user-images.githubusercontent.com/8537877/141281596-e47f5552-75b9-47d1-8a36-67337cb6eb3a.png)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1363 from cfmcgrady/kyuubi-1354.

Closes #1354

8895d7fc [Fu Chen] typo
8f5dea9a [Fu Chen] addressed comment
4d325a45 [Fu Chen] constant MediaType.APPLICATION_JSON
0d96951e [Fu Chen] RESTful Api -> REST API
be7d02dc [Fu Chen] add LIABILITY-edl.txt
4ff1053e [Fu Chen] update dev/dependencyList
b126f7e6 [Fu Chen] swagger-ui 4.1.0
a35c2f29 [Fu Chen] addressed comment
7bde0534 [Fu Chen] update LICENSE-binary
38397dc2 [Fu Chen] align jackson version
923f3e7e [Fu Chen] update dev/dependencyList
7d5338c8 [Fu Chen] remove kyuubi-server static files
5cb66b33 [Fu Chen] update dev/dependencyList
e8a8c421 [Fu Chen] fix style
5b13f69c [Fu Chen] add license and remove .map files
c04d7a75 [Fu Chen] rebase master
2388dae8 [Fu Chen] fix style
8dcbd3db [Fu Chen] add static resource
4b0d6d0a [Fu Chen] expose OpenAPI

Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-17 12:17:58 +08:00
fwang12
09c8a4093f [KYUUBI #1400] Expose hasMoreEngineLogs and getEngineLog for BeeLine mode 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Expose hasMoreEngineLogs and getEngineLog for BeeLine mode.

Refer the API of HiveStatement/KyuubiStatement:
- hasMoreLogs
- getQueryLog

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1400 from turboFei/refactor_kyuubi_connection.

Closes #1400

9716e8f4 [fwang12] address comments
1da636d3 [fwang12] public
acaf0c7a [fwang12] refactor
54cc1125 [fwang12] save
b66f2585 [fwang12] Make preparation for beeline mode

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-11-17 10:01:23 +08:00
Kent Yao
bf9736e31b
[KYUUBI #1399] [DOCS] Add doc for engine share level 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

doc improvement

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

![image](https://user-images.githubusercontent.com/8326978/141964500-f0121a54-4ce9-4e9d-84a5-c4a4ab1f0ac9.png)

---

![image](https://user-images.githubusercontent.com/8326978/141965482-f367e486-c68b-44d8-85d9-9c7f0e7e63aa.png)
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1399 from yaooqinn/sldoc.

Closes #1399

d8bcce8a [Kent Yao] [DOCS] Add doc for engine share level
21e2548f [Kent Yao] [DOCS] Add doc for engine share level
bba8dc9d [Kent Yao] [DOCS] Add doc for engine share level

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-16 19:58:25 +08:00
simon
178c39e221
[KYUUBI #1351] Implement api: /${version}/sessions/${identifier}/operations 
### _Why are the changes needed?_
This is a subtask of umbrella issue #KPIP-1

- /${version}/sessions/${identifier}/operations
  - mapping:
    - ICLIService#executeStatement
    - ICLIService#executeStatementAsync
    - ICLIService#getTypeInfo
    - ICLIService#getCatelogs
    - ICLIService#getSchemas
    - ICLIService#getTables
    - ICLIService#getTableTypes
    - ICLIService#getColumns
    - ICLIService#getFunctions
  - desc: create a new operation under a session
  - method: POST
  - params:
  - operation: e.g. GetTableTypes
  - addition: required params to create an Operation
  - returns: an instance of OperationHandle

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1353 from simon824/r1.

Closes #1351

98ffcda4 [simon] rename
7c0c3b40 [simon] ut fix
586a0b8a [simon] rename url
1ea6f322 [simon] codestyle
48b5d79d [simon] fix
543caf3c [simon] separate
0ff30af2 [simon] codestyle
203cb164 [simon] Merge remote-tracking branch 'upstream/master' into r1
e578cb42 [simon] fix codestype
006ce4d4 [simon] bugfix
aeb0de63 [simon] operationtype
3b6dceec [simon] init

Authored-by: simon <zhangshiming@cvte.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-16 19:55:36 +08:00
hongdongdong
3fa3004aef
[KYUUBI #1397] Move bind host and port to configmap 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Setting `frontend.bind.host` and `frontend.bind.port` in `kyuubi-configmap.yaml`, instead of put them in ENV.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1398 from hddong/move-bind.

Closes #1397

0741f08a [hongdongdong] [KYUUBI#1397] Move bind host and port to configmap

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-11-16 18:05:12 +08:00
fwang12
d187391a1e [KYUUBI #1377] Support to get launch engine log asynchronously with KyuubiConnection 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Support to get launch engine log in client side.

When open session with async mode, return the launch engine operation handle by open session response configuration.

For the KyuubiConnection, it will launch a thread to fetch engine log.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate
![image](https://user-images.githubusercontent.com/6757692/141752286-95fbcc91-d7ec-4951-ba33-6b4974f21726.png)

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1377 from turboFei/kyuubi_session_async_logs.

Closes #1377

4c6a66e1 [fwang12] remove thread local directly
cbb653ea [fwang12] revert operation idle time
2afb282f [fwang12] nit
901f0ec3 [fwang12] add comments
3c9bc70c [fwang12] delaying close launch engine when close session
4c6775c2 [fwang12] address comments
ef12757b [fwang12] rewrite opensession in KyuubiThriftBinaryFrontendService
880b5bd7 [fwang12] address comments
bb18c90d [fwang12] fix
b59cc04a [fwang12] address nit
3f4917ed [fwang12] support to add extra log path for operation log
b39e3a85 [fwang12] use unknown type for kyuubi defined type
cd0648d6 [fwang12] refactor code
0321a426 [fwang12] only return engine op handle when run async
f0251ba0 [fwang12] complete ut
1a98d2c5 [fwang12] use Base64 encode & decode
eb86c8fb [fwang12] fix ut
329818c1 [fwang12] refactor import order
b419e547 [fwang12] support Kyuubi connection to get engine log

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-11-16 16:00:35 +08:00
sunfangbin
f5e8152a9e
[KYUUBI #1389] Refine helm service template to support LoadBalancer type 
### _Why are the changes needed?_
The helm service template is currently hard-coded with `NodePort`, we should make it configurable to support other service type such as `LoadBalancer`.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1389 from murong00/branch-1389.

Closes #1389

5855c546 [sunfangbin] Refine toYaml usage
d659ad4c [sunfangbin] Refine helm service template to support LoadBalancer type

Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-16 12:04:55 +08:00
yanghua
20ee845c0b
[KYUUBI #1386] Refactor SparkThriftBinaryFrontendService to extract connectionUrl into super class 
…

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1387 from yanghua/KYUUBI-1386.

Closes #1386

29415904 [yanghua] [KYUUBI #1386] Refactor SparkThriftBinaryFrontendService to extract connectionUrl into super class

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-16 09:41:55 +08:00
sunfangbin
481903fb31
[KYUUBI #1388] Added support a zookeeper shell 
### _Why are the changes needed?_
It is useful to support a zookeeper shell so that we can view some metadata if necessary. For example:
```
$ bin/kyuubi-zookeeper-shell -server localhost:2181
Connecting to localhost:2181
Welcome to ZooKeeper!
...
ls /
[zookeeper, kyuubi]
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1388 from murong00/branch-1387.

Closes #1388

7698e79c [sunfangbin] Added support a zookeeper shell

Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-15 18:25:43 +08:00
dependabot[bot]
4073716ac4
[KYUUBI #1385] Bump junit from 4.11 to 4.13.1 in /kyuubi-hive-jdbc 
Bumps [junit](https://github.com/junit-team/junit4) from 4.11 to 4.13.1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/junit-team/junit4/releases">junit's releases</a>.</em></p>
<blockquote>
<h2>JUnit 4.13.1</h2>
<p>Please refer to the <a href="https://github.com/junit-team/junit/blob/HEAD/doc/ReleaseNotes4.13.1.md">release notes</a> for details.</p>
<h2>JUnit 4.13</h2>
<p>Please refer to the <a href="https://github.com/junit-team/junit/blob/HEAD/doc/ReleaseNotes4.13.md">release notes</a> for details.</p>
<h2>JUnit 4.13 RC 2</h2>
<p>Please refer to the <a href="https://github.com/junit-team/junit4/wiki/4.13-Release-Notes">release notes</a> for details.</p>
<h2>JUnit 4.13 RC 1</h2>
<p>Please refer to the <a href="https://github.com/junit-team/junit4/wiki/4.13-Release-Notes">release notes</a> for details.</p>
<h2>JUnit 4.13 Beta 3</h2>
<p>Please refer to the <a href="https://github.com/junit-team/junit4/wiki/4.13-Release-Notes">release notes</a> for details.</p>
<h2>JUnit 4.13 Beta 2</h2>
<p>Please refer to the <a href="https://github.com/junit-team/junit4/wiki/4.13-Release-Notes">release notes</a> for details.</p>
<h2>JUnit 4.13 Beta 1</h2>
<p>Please refer to the <a href="https://github.com/junit-team/junit4/wiki/4.13-Release-Notes">release notes</a> for details.</p>
<h2>JUnit 4.12</h2>
<p>Please refer to the <a href="https://github.com/junit-team/junit/blob/HEAD/doc/ReleaseNotes4.12.md">release notes</a> for details.</p>
<h2>JUnit 4.12 Beta 3</h2>
<p>Please refer to the <a href="https://github.com/junit-team/junit/blob/HEAD/doc/ReleaseNotes4.12.md">release notes</a> for details.</p>
<h2>JUnit 4.12 Beta 2</h2>
<p>No release notes provided.</p>
<h2>JUnit 4.12 Beta 1</h2>
<p>No release notes provided.</p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="1b683f4ec0"><code>1b683f4</code></a> [maven-release-plugin] prepare release r4.13.1</li>
<li><a href="ce6ce3aadc"><code>ce6ce3a</code></a> Draft 4.13.1 release notes</li>
<li><a href="c29dd8239d"><code>c29dd82</code></a> Change version to 4.13.1-SNAPSHOT</li>
<li><a href="1d174861f0"><code>1d17486</code></a> Add a link to assertThrows in exception testing</li>
<li><a href="543905df72"><code>543905d</code></a> Use separate line for annotation in Javadoc</li>
<li><a href="510e906b39"><code>510e906</code></a> Add sub headlines to class Javadoc</li>
<li><a href="610155b8c2"><code>610155b</code></a> Merge pull request from GHSA-269g-pwp5-87pp</li>
<li><a href="b6cfd1e3d7"><code>b6cfd1e</code></a> Explicitly wrap float parameter for consistency (<a href="https://github-redirect.dependabot.com/junit-team/junit4/issues/1671">#1671</a>)</li>
<li><a href="a5d205c795"><code>a5d205c</code></a> Fix GitHub link in FAQ (<a href="https://github-redirect.dependabot.com/junit-team/junit4/issues/1672">#1672</a>)</li>
<li><a href="3a5c6b4d08"><code>3a5c6b4</code></a> Deprecated since jdk9 replacing constructor instance of Double and Float (<a href="https://github-redirect.dependabot.com/junit-team/junit4/issues/1660">#1660</a>)</li>
<li>Additional commits viewable in <a href="https://github.com/junit-team/junit4/compare/r4.11...r4.13.1">compare view</a></li>
</ul>
</details>
<br />

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=junit:junit&package-manager=maven&previous-version=4.11&new-version=4.13.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `dependabot rebase` will rebase this PR
- `dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `dependabot merge` will merge this PR after your CI passes on it
- `dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `dependabot cancel merge` will cancel a previously requested merge and block automerging
- `dependabot reopen` will reopen this PR if it is closed
- `dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/incubator-kyuubi/network/alerts).

</details>

Closes #1385 from dependabot[bot]/dependabot/maven/kyuubi-hive-jdbc/junit-junit-4.13.1.

Closes #1385

adc26825 [dependabot[bot]] Bump junit from 4.11 to 4.13.1 in /kyuubi-hive-jdbc

Authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-15 17:30:24 +08:00
Cheng Pan
cb9e55a9b4
[KYUUBI #1380] Fix kyuubi download duplicated plugins 
### _Why are the changes needed?_
Fix maven pom.xml issue

```
(apache-kyuubi) ➜  apache-kyuubi git:(mysql-fe-impl) ✗ build/mvn clean install
Using `mvn` from path: /Users/chengpan/.sdkman/candidates/maven/current/bin/mvn
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.apache.kyuubi:kyuubi-download:pom:1.4.0-SNAPSHOT
[WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but found duplicate declaration of plugin com.googlecode.maven-download-plugin:download-maven-plugin  line 62, column 21
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
[INFO] ------------------------------------------------------------------------
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

```
(apache-kyuubi) ➜  apache-kyuubi git:(download) ✗ build/mvn compile -pl :kyuubi-download -Pspark-provided,flink-provided | grep -A 100 'scala-maven-plugin:4.3.0:compile'
Using `mvn` from path: /Users/chengpan/.sdkman/candidates/maven/current/bin/mvn
[INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first)  kyuubi-download ---
[INFO] compile in 0.0 s
[INFO] No sources to compile
[INFO]
[INFO] --- download-maven-plugin:1.6.6:wget (download-spark-release)  kyuubi-download ---
[INFO] maven-download-plugin:wget skipped
[INFO]
[INFO] --- download-maven-plugin:1.6.6:wget (download-flink-release)  kyuubi-download ---
[INFO] maven-download-plugin:wget skipped
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.932 s
[INFO] Finished at: 2021-11-15T10:41:05+08:00
[INFO] ------------------------------------------------------------------------
```

```
(apache-kyuubi) ➜  apache-kyuubi git:(download) ✗ build/mvn compile -pl :kyuubi-download -Pflink-provided | grep -A 100 'scala-maven-plugin:4.3.0:compile'
Using `mvn` from path: /Users/chengpan/.sdkman/candidates/maven/current/bin/mvn
[INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first)  kyuubi-download ---
[INFO] compile in 0.0 s
[INFO] No sources to compile
[INFO]
[INFO] --- download-maven-plugin:1.6.6:wget (download-spark-release)  kyuubi-download ---
[INFO] Expanding: /Users/chengpan/Projects/apache-kyuubi/externals/kyuubi-download/target/spark-3.1.2-bin-hadoop3.2.tgz into /Users/chengpan/Projects/apache-kyuubi/externals/kyuubi-download/target
[INFO]
[INFO] --- download-maven-plugin:1.6.6:wget (download-flink-release)  kyuubi-download ---
[INFO] maven-download-plugin:wget skipped
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  4.486 s
[INFO] Finished at: 2021-11-15T10:42:22+08:00
[INFO] ------------------------------------------------------------------------
```

```
(apache-kyuubi) ➜  apache-kyuubi git:(download) ✗ build/mvn compile -pl :kyuubi-download -Pspark-provided | grep -A 100 'scala-maven-plugin:4.3.0:compile'
Using `mvn` from path: /Users/chengpan/.sdkman/candidates/maven/current/bin/mvn
[INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first)  kyuubi-download ---
[INFO] compile in 0.0 s
[INFO] No sources to compile
[INFO]
[INFO] --- download-maven-plugin:1.6.6:wget (download-spark-release)  kyuubi-download ---
[INFO] maven-download-plugin:wget skipped
[INFO]
[INFO] --- download-maven-plugin:1.6.6:wget (download-flink-release)  kyuubi-download ---
[INFO] Expanding: /Users/chengpan/Projects/apache-kyuubi/externals/kyuubi-download/target/flink-1.12.5-bin-scala_2.12.tgz into /Users/chengpan/Projects/apache-kyuubi/externals/kyuubi-download/target
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  5.114 s
[INFO] Finished at: 2021-11-15T10:41:53+08:00
[INFO] ------------------------------------------------------------------------
```

Closes #1380 from pan3793/download.

Closes #1380

21c7cc22 [Cheng Pan] Fix kyuubi download duplicated plugins

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-15 15:40:50 +08:00
fwang12
ed43391f8c [KYUUBI #1381] Rewrite kyuubi hive jdbc module based on hive branch-2.3 code base 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Hive code base:
8e7f23f34b

commit 8e7f23f34b2ce7328c9d571a13c336f0c8cdecb6 (HEAD, origin/branch-2.3)
Author: Neelesh Srinivas Salian <nssalianusers.noreply.github.com>
Date:   Tue Oct 19 12:50:31 2021 -0700

    HIVE-25616: Hive-24741 backport to 2.3 (Neelesh Srinivas Salian reviewed by Vihang Karajgaonkar)

    Closes (#2730)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1381 from turboFei/kyuubi_hive_jdbc.

Closes #1381

f39f1a1d [fwang12] adreess comments
5612f490 [fwang12] remove kyuubi-hive-jdbc from code cov
f5a6376e [fwang12] do not fail if no tests
4e99ffcb [fwang12] check kyuubi-hive-jdbc code coverage
1c5f2e16 [fwang12] refactor the test classes
d7f9c7d7 [fwang12] copy test classes from hive branch-2.3
a28c077a [fwang12] rename classname in comments
42f7c5ea [fwang12] try to remove hive-jdbc dependency
fcf5953a [fwang12] use kyuubi hive jdbc utils
17402f4d [fwang12] replace licenses
5cbca43d [fwang12] add missing class ignored by .gitignore
b2c19334 [fwang12] revert docs change
bdc860b4 [fwang12] backport kyuubiHiveDriver
04731e72 [fwang12] make some rename
9d57076b [fwang12] remove from org.apache.hive.jdbc to org.apache.kyuubi.jdbc.hive
86813e6b [fwang12] Backport KyuubiDriver
42f748d6 [fwang12] Backport KyuubiDatabaseMetaData.getTables and for KyuubiConnection.getMetaData, it is not needed
f0d2aedb [fwang12] rename hive classes to kyuubi version
3ea8ba73 [fwang12] copy the source code from hive branch-2.3
7db4e5ac [fwang12] remove at first

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-11-15 15:38:38 +08:00
Brian Yue
e4b8e961bb
[KYUUBI #1369] Remove _config.yml 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Close #1369.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1375 from byyue/feature-1369.

Closes #1369

03007076 [Brian Yue] Remove _config.yml

Authored-by: Brian Yue <code.byyue@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-14 12:48:30 +08:00
fwang12
1554032df2 [KYUUBI #1346] Support launch query engine asynchronously during opening session 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Support launch query engine asynchronously during opening session, make it more user-friendly.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1346 from turboFei/kyuubi_hive_jdbc_ext.

Closes #1346

513691d6 [fwang12] address comments
63b8f901 [fwang12] remove hanle to client
4590ad98 [fwang12] remove synchronized
f3b136b4 [fwang12] add ut for open session with failure
2cae9afa [fwang12] address comments
35113d0f [fwang12] add engine session id to session event
34c27f9c [fwang12] refactor log
b5f7e639 [fwang12] block
b8b0cf9d [fwang12] to support aync client
66eb07f3 [fwang12] address comments
623c0511 [fwang12] volatile
282c9f07 [fwang12] refactor conf
5047d91d [fwang12] server session handle is not same with that of engine
dc1cb8bc [fwang12] update docs
b773fd9d [fwang12] refactor
5f657e44 [fwang12] add ut
9e56cbab [fwang12] exclude engine init engine
6abdb52c [fwang12] refactor
d3e37a8c [fwang12] block until engine finished
085b324d [fwang12] save

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
 2021-11-13 17:25:57 +08:00
zhouyifan279
884de795c2
[KYUUBI #1361] Update available version of backported configurations to 1.3.2 
### _Why are the changes needed?_
Sub task of #1361

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1373 from zhouyifan279/1361-4.

Closes #1361

86b92e86 [zhouyifan279] [KYUUBI #1361] Update available version of backported configurations to 1.3.2

Authored-by: zhouyifan279 <zhouyifan279@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-12 22:40:10 +08:00
sunfangbin
9a55bf5e40
[KYUUBI #1368] Turn off debugging in dist scripts 
### _Why are the changes needed?_
`set -x` is currently used in dist scripts for debugging and the console looks a bit messy, it seems better to turn it off and users can enable it themselves if necessary.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1368 from murong00/branch-1366.

Closes #1368

6373fe92 [sunfangbin] Turn off debugging in dist scripts

Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-12 19:45:56 +08:00
sunfangbin
ff1f78fd57
[KYUUBI #1365] Refine print environment variables to console 
### _Why are the changes needed?_
Currently, the enviroment variables are also printed to console when using `bin/kyuubi status` or `bin/kyuubi stop` like below:
```
$ bin/kyuubi status
Using kyuubi environment file /data01/apache-kyuubi-1.4.0-SNAPSHOT-bin-incubating/conf/kyuubi-env.sh to initialize...
JAVA_HOME: /usr/jdk1.8.0_172
KYUUBI_HOME: /data01/apache-kyuubi-1.4.0-SNAPSHOT-bin-incubating
KYUUBI_CONF_DIR: /data01/apache-kyuubi-1.4.0-SNAPSHOT-bin-incubating/conf
KYUUBI_LOG_DIR: /data01/apache-kyuubi-1.4.0-SNAPSHOT-bin-incubating/logs
KYUUBI_PID_DIR: /data01/apache-kyuubi-1.4.0-SNAPSHOT-bin-incubating/pid
KYUUBI_WORK_DIR_ROOT: /data01/apache-kyuubi-1.4.0-SNAPSHOT-bin-incubating/work
SPARK_HOME: /data01/apache-kyuubi-1.4.0-SNAPSHOT-bin-incubating/externals/spark-3.1.2-bin-hadoop3.2
SPARK_CONF_DIR: /data01/apache-kyuubi-1.4.0-SNAPSHOT-bin-incubating/externals/spark-3.1.2-bin-hadoop3.2/conf
HADOOP_CONF_DIR:
Kyuubi is not running
```
In general, these variables are only needed to display when starting a cluster to keep clean style, something like below:
```
$ bin/kyuubi status
Kyuubi is running (pid: 95472)
$ bin/kyuubi stop
Stopping org.apache.kyuubi.server.KyuubiServer
...
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1365 from murong00/branch-1364.

Closes #1365

3856494f [sunfangbin] add checking run command
92686fe1 [sunfangbin] Refine print environment variables to console

Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-12 17:39:41 +08:00
ulysses-you
1dc755e450
[KYUUBI #1366] Add a todo in zorder that we can do rebalance partitions before local sort of zorder 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Rebalance partitions can resolve data skew issue before writing, so it should be good to do before we do the local sort of zorder, and also we can support dynamic partition insertion with zorder. However Spark only support this after 3.3.0, see [PR](https://github.com/apache/spark/pull/34542).

So add a todo first.

### _How was this patch tested?_
not needed

Closes #1366 from ulysses-you/zorder.

Closes #1366

972229eb [ulysses-you] todo

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-12 17:36:36 +08:00
sunfangbin
9e1f9c2e98
[KYUUBI #1359] Support setting zk chroot path when initialize cluster 
### _Why are the changes needed?_
Users may encounter the following exception when trying to set `kyuubi.ha.zookeeper.quorum` to `localhost:2181/lakehouse` with chroot path `/lakehouse` nonexisted:
```
2021-11-10 10:56:34.510 ERROR server.KyuubiThriftBinaryFrontendService: Error starting service KyuubiServiceDiscovery
org.apache.kyuubi.KyuubiException: Failed to create namespace '/kyuubi'
        at org.apache.kyuubi.ha.client.ServiceDiscovery$.createServiceNode(ServiceDiscovery.scala:225)
        at org.apache.kyuubi.ha.client.ServiceDiscovery.start(ServiceDiscovery.scala:101)
......
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /kyuubi
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:114)
......
```
It is wonderful to support this since zookeeper connection with chroot path is generally recommended in production environments. With this feture the znodes in zookeeper likes below:
```
[zk: localhost:2181(CONNECTED) 28] ls /lakehouse
[kyuubi, kyuubi_USER, kyuubi_USER_SPARK_SQL]
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1359 from murong00/branch-1358.

Closes #1359

42459ba0 [sunfangbin] Add a test to cover the changes
0191ab66 [sunfangbin] Fail the invalid cases
d04c532b [sunfangbin] address all the comments
8e7b5a64 [sunfangbin] Support setting zk chroot path when initialize cluster

Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-12 12:04:35 +08:00
Cheng Pan
1061d176c8
[KYUUBI #1358] Add KyuubiMySQLFrontendService stub 
### _Why are the changes needed?_

Add KyuubiMySQLFrontendService stub, without Netty pipeline, part of #1334

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1358 from pan3793/mysql-fe-stub.

Closes #1358

c091d955 [Cheng Pan] nit
61abb0f6 [Cheng Pan] Address comments
9ac70456 [Cheng Pan] Update conf
3d75acdf [Cheng Pan] nit
4a4d8a24 [Cheng Pan] Address comments
0136dd52 [Cheng Pan] nit
477474ff [Cheng Pan] KyuubiMySQLFrontendService Stub

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-11 16:50:52 +08:00
yanghua
1d9b98e119
[KYUUBI #1360] [MINOR] Change log level from warn to error when executing statement failed 
…

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1360 from yanghua/MINOR-LOG-LEVEL.

Closes #1360

8e846217 [yanghua] [MINOR] Change log level from warn to error when executing statement failed

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-11-11 14:41:32 +08:00
Kent Yao
9db2e5c054
[KYUUBI #1356] Refine KyuubiStatementEvent 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
1. When a statement error happened, the event will be logged twice, in `setState` and `setException`, we fix it
2. We break down KyuubiStatementEvent into three parts - statement basis, status, and sessionId/user, see the improved comments
3. log the remote operation handle too

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1356 from yaooqinn/se.

Closes #1356

4b5d0de2 [Kent Yao] Refine KyuubiStatementEvent
e7228ddd [Kent Yao] Refine KyuubiStatementEvent

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-10 12:09:04 +08:00
Cheng Pan
32c3f5cb2e
[KYUUBI #1349] Add new conf kyuubi.frontend.protocols 
### _Why are the changes needed?_

Adapt to multiple frontends protocol.

This PR also marks REST API as experimental and exposes it to users.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1349 from pan3793/frontend-conf.

Closes #1349

d113c4a6 [Cheng Pan] Simplify code by SAM
129c2c9a [Cheng Pan] Fix compile
fa8f7f67 [Cheng Pan] Add new conf kyuubi.frontend.protocols

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-09 22:09:50 +08:00
Cheng Pan
c122a23a0d
[KYUUBI #1348] [DEPS] Add netty and mysql jdbc driver to kyuubi-server module 
### _Why are the changes needed?_

This PR is one part of #1334.

Add netty to `kyuubi-server` module compile scope, and mysql jdbc driver to test scope, it's required for implementing MySQL frontend protocol

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1348 from pan3793/mysql-fe-deps.

Closes #1348

288ad9c0 [Cheng Pan] [DEPS] Add netty to kyuubi-server module

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-09 20:31:09 +08:00
Cheng Pan
4ac8b09b2f
[KYUUBI #1331] [TEST] Refactor test classes to match multi frontend protocol 
### _Why are the changes needed?_

As Kyuubi supports REST frontend protocol now, and MySQL frontend protocol is coming soon, Thrift is not the only one, this PR refacotor the test classes to match the changes.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1331 from pan3793/test.

Closes #1331

67c65810 [Cheng Pan] nit
f9168358 [Cheng Pan] [TEST] Refactor test classes to match multi frontend protocol

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-09 13:28:25 +08:00
Cheng Pan
0b43a8bb78
[KYUUBI #1338] Bump Iceberg 0.12.1 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Test Iceberg 0.12.1 RC0

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1338 from pan3793/iceberg-0.12.1.

Closes #1338

d2dd4c1f [Cheng Pan] Remove staging repo
a16103e8 [Cheng Pan] Test Iceberg 0.12.1 RC0

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-09 12:28:36 +08:00
zwangsheng
82cca0e481
[KYUUBI #1350] [BUILD] Put spark-block-cleaner jar into sub dir kubernetes 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Before PR, when you add profile `spark-block-cleaner`, you will get catalog list like
![POPO20211108-152830](https://user-images.githubusercontent.com/52876270/140700856-71f482ae-81b8-4c2e-a963-11c284a61e68.jpg)
You can observe that Kubernetes is an empty directory, and docker is created on its same layer.

After PR,
![popo_2021-11-08  15-30-53](https://user-images.githubusercontent.com/52876270/140701162-302f5334-9332-49f4-bce4-addc2902369d.jpg)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1350 from zwangsheng/fix/mv_docker_into_kubernetes.

Closes #1350

533fa13f [zwangsheng] mvn docker into kubernetes

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-08 17:10:07 +08:00
zhenjiaguo
0f1d4ec0e0
[KYUUBI #1347] [DOC] Fix miscellaneous doc typos 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Some spelling & grammar format fix.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1347 from zhenjiaguo/typo-fix.

Closes #1347

5c20ae28 [zhenjiaguo] change serial to several
db3fa969 [zhenjiaguo] typo fix

Authored-by: zhenjiaguo <zhenjia_guo@163.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-08 09:27:37 +08:00
yanghua
f9b933a661
[KYUUBI #1340] Refactor ProcBuilder creation based on EngineType 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1341 from yanghua/KYUUBI-1340.

Closes #1340

aa0df59a [yanghua] Fixed test failure
ee2cd078 [yanghua] [KYUUBI #1340] Refactor ProcBuilder creation based on EngineType

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-11-06 11:33:20 +08:00
AnybodyHome
04d4179bbe
[KYUUBI #1336] Remove confusing output when kyuubi stop 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
When kyuubi stops, it will print a confuse message `Starting Kyuubi Server ...`.
When kyuubi starts, KYUUBI_HOME will be included in kyuubi environment output print.
So I think this 'echo' is redundant.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1343 from zhenjiaguo/kyuubi-script-fix.

Closes #1336

3b4cc75c [AnybodyHome] Remove confusing output when kyuubi stop

Authored-by: AnybodyHome <zhenjia_guo@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-11-05 23:33:32 +08:00
AnybodyHome
25becb02b9
[KYUUBI #1342] [DOC] Fix beeline incorrect use of double quotes 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
fix beeline incorrect use of double quotes. Detials see [discussions 1137](https://github.com/apache/incubator-kyuubi/discussions/1337)
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1342 from zhenjiaguo/architecture-doc-fix.

Closes #1342

03c89edf [AnybodyHome] fix beeline incorrect use of double quotes

Authored-by: AnybodyHome <zhenjia_guo@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-05 20:16:34 +08:00
ulysses-you
8813c4cf92
[KYUUBI #1333] Improve zorder interleave perf 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Make interleave function more faster.

The basic idea is from http://graphics.stanford.edu/~seander/bithacks.html#InterleaveTableObvious

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1333 from ulysses-you/zorder-perf.

Closes #1333

a9411877 [ulysses-you] address comment
bce82fc7 [ulysses-you] simplify
05f0b6e1 [ulysses-you] benchmark result
f470a043 [ulysses-you] while
a63cfae2 [ulysses-you] improve

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-05 18:04:47 +08:00
yanghua
aad575ce3e
[KYUUBI #1329] Support download Flink binary package in kyuubi-download module 
…

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1339 from yanghua/KYUUBI-1329.

Closes #1329

895815fb [yanghua] address review suggestion
eddf0605 [yanghua] [KYUUBI #1329] Support download Flink binary package in kyuubi-download module

Authored-by: yanghua <yanghua1127@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-05 18:03:57 +08:00
zwangsheng
60392349a7
[KYUUBI #1330] fix tool cleaner process bug 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
When using the tool to help clean up spark on K8s residual cache files, I encountered unreported sleep conditions.
After analysis, it was found that the SSD mounted when `needToDeepClean` was run encountered an incorrect catch exception.
Now, `Try Catch` is added to ensure the normal operation of the function, and deep cleaning is performed by default if a problem occurs. In this case, the disk space still overruns after the cleaning opportunity is missed.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1330 from zwangsheng/tools/fix-process.

Closes #1330

6daf483c [zwangsheng] catch nonfatal
89caf897 [zwangsheng] fix tool cleaner process bug
b487cf8c [zwangsheng] fix tool cleaner process bug

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-05 11:26:13 +08:00
AnybodyHome
34c58b9133
[KYUUBI #1335] Spell issue branch 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Spell check and punctuation check.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1335 from zhenjiaguo/spell-issue-branch.

Closes #1335

b4d48192 [AnybodyHome] recover beeline change
85603b6f [AnybodyHome] spell check and punctuation check

Authored-by: AnybodyHome <zhenjia_guo@163.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
 2021-11-05 09:33:32 +08:00
Cheng Pan
96f843c3c0
[KYUUBI #1332] Fix regenerate document hint 
### _Why are the changes needed?_

The previous command does not work after #1131.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1332 from pan3793/hint.

Closes #1332

1c21a54a [Cheng Pan] Fix regenerate document hint

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-04 21:57:31 +08:00
simon
8f3cca9f97
[KYUUBI #1106] Implement /sessions/${identifier}/info/${infotype} 
### _Why are the changes needed?_
This is a subtask of umbrella issue #KPIP-1

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1317 from simon824/getinfo.

Closes #1106

d00b05f9 [simon] fix codestyle
312726a5 [simon] fix
0e7a2ce7 [simon] negativetest
6bb9b6f0 [simon] codestyle
24ca381a [simon] rename InfoDetail
32a836ee [Simon] Merge branch 'apache:master' into getinfo
474d858b [simon] getinfo
4f26c59a [simon] reduce sleep time
ebb6afb5 [simon] fixut
6c366c0b [simon] init

Lead-authored-by: simon <zhangshiming@cvte.com>
Co-authored-by: Simon <3656562@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
 2021-11-04 18:56:44 +08:00
Cheng Pan
5421b56440
[KYUUBI #1327] Fix socket timeout error when client sync execute statement cost time longer than engine.login.timeout 
### _Why are the changes needed?_

Fix #1327

#### Analysis

The typical error logs are

```log
2021-11-01 11:27:54.018 INFO client.KyuubiSyncThriftClient: TCloseOperationReq(operationHandle:TOperationHandle(operationId:THandleIdentifier(guid:47 69 37 B3 13 38 48 DA 87 7A 8A B6 BD 22 FA 57, secret:C1 01 AE 0B 6F 5F 48 F1 9A F0 FD 84 E3 0F 2B 1E), operationType:EXECUTE_STATEMENT, hasResultSet:true)) succeed on engine side
2021-11-01 11:27:54.019 INFO operation.ExecuteStatement: Processing sy.yao's query[c581db45-67f2-4f15-ac0d-aaecdb713fe9]: INITIALIZED_STATE -> PENDING_STATE, statement: [...SQL]
2021-11-01 11:27:54.019 INFO operation.ExecuteStatement: Processing sy.yao's query[c581db45-67f2-4f15-ac0d-aaecdb713fe9]: PENDING_STATE -> RUNNING_STATE, statement: [...SQL]
2021-11-01 11:28:09.034 INFO operation.ExecuteStatement: Processing sy.yao's query[c581db45-67f2-4f15-ac0d-aaecdb713fe9]: RUNNING_STATE -> ERROR_STATE, statement: [...SQL], time taken: 15.015 seconds
2021-11-01 11:28:09.035 INFO operation.ExecuteStatement: Processing sy.yao's query[c581db45-67f2-4f15-ac0d-aaecdb713fe9]: ERROR_STATE -> CLOSED_STATE, statement: [...SQL]
2021-11-01 11:28:09.035 WARN server.KyuubiThriftBinaryFrontendService: Error executing statement:
org.apache.kyuubi.KyuubiSQLException: Error operating EXECUTE_STATEMENT: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
[...omit detail stacktrace here]
```
The key points here are:

1. client execute query in sync mode, Hive JDBC client use async mode since 2.1.0 [HIVE-6535](https://issues.apache.org/jira/browse/HIVE-6535)
2. query execute cost time great than `kyuubi.session.engine.login.timeout`, which default is `15s`

Kyuubi server create Thrift clients use `kyuubi.session.engine.login.timeout` as both `socket_timeout` and `connect_timeout`, and there is no heartbeat/keepalive mechanism in Thrift Protocol layer, thus if engine does not send response to Kyuubi server in `socket_timeout`, the Thrift client(Kyuubi server) will fail with `SocketTimeoutException: Read timed out`.

In sync mode, when user send a execute statement request to Kyuubi server, Kyuubi server forword the request to the engine, engine return nothing until the execution finished or any other error happend.

In async mode, the query request passes in same way, but engine will return an `op_handle` immediately instead of waiting for query to finish, the client should use the `op_handle` to check query status periodically, in detail

1. Kyuubi server ask engine for query status, and keep the result in memory
2. User ask Kyuubi server for query status, Kyuubi server get result from memory

#### Solution

Option 1: change `socket_timeout` of Thrift clients created by Kyuubi server to infinite. This approach may cause another issue. if engine crash when executing the query, Kyuubi server will wait until the socket keepalive(controlled by OS) timeout, so the query status will always be `running` in Kyuubi server memory.

Option 2: always run a query in async mode, simulate sync mode in Server side.

This PR implement the option 2, also introduce a new conf `kyuubi.session.engine.request.timeout`

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1328 from pan3793/timeout.

Closes #1327

cc15e6d5 [Cheng Pan] Always execute statement in async mode
dc91d0d5 [Cheng Pan] Add new conf `kyuubi.session.engine.request.timeout`

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
 2021-11-04 18:23:10 +08:00