kyuubi/docs/deployment
Cheng Pan 5421b56440
[KYUUBI #1327] Fix socket timeout error when client sync execute statement cost time longer than engine.login.timeout
### _Why are the changes needed?_

Fix #1327

#### Analysis

The typical error logs are

```log
2021-11-01 11:27:54.018 INFO client.KyuubiSyncThriftClient: TCloseOperationReq(operationHandle:TOperationHandle(operationId:THandleIdentifier(guid:47 69 37 B3 13 38 48 DA 87 7A 8A B6 BD 22 FA 57, secret:C1 01 AE 0B 6F 5F 48 F1 9A F0 FD 84 E3 0F 2B 1E), operationType:EXECUTE_STATEMENT, hasResultSet:true)) succeed on engine side
2021-11-01 11:27:54.019 INFO operation.ExecuteStatement: Processing sy.yao's query[c581db45-67f2-4f15-ac0d-aaecdb713fe9]: INITIALIZED_STATE -> PENDING_STATE, statement: [...SQL]
2021-11-01 11:27:54.019 INFO operation.ExecuteStatement: Processing sy.yao's query[c581db45-67f2-4f15-ac0d-aaecdb713fe9]: PENDING_STATE -> RUNNING_STATE, statement: [...SQL]
2021-11-01 11:28:09.034 INFO operation.ExecuteStatement: Processing sy.yao's query[c581db45-67f2-4f15-ac0d-aaecdb713fe9]: RUNNING_STATE -> ERROR_STATE, statement: [...SQL], time taken: 15.015 seconds
2021-11-01 11:28:09.035 INFO operation.ExecuteStatement: Processing sy.yao's query[c581db45-67f2-4f15-ac0d-aaecdb713fe9]: ERROR_STATE -> CLOSED_STATE, statement: [...SQL]
2021-11-01 11:28:09.035 WARN server.KyuubiThriftBinaryFrontendService: Error executing statement:
org.apache.kyuubi.KyuubiSQLException: Error operating EXECUTE_STATEMENT: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
[...omit detail stacktrace here]
```
The key points here are:

1. client execute query in sync mode, Hive JDBC client use async mode since 2.1.0 [HIVE-6535](https://issues.apache.org/jira/browse/HIVE-6535)
2. query execute cost time great than `kyuubi.session.engine.login.timeout`, which default is `15s`

Kyuubi server create Thrift clients use `kyuubi.session.engine.login.timeout` as both `socket_timeout` and `connect_timeout`, and there is no heartbeat/keepalive mechanism in Thrift Protocol layer, thus if engine does not send response to Kyuubi server in `socket_timeout`, the Thrift client(Kyuubi server) will fail with `SocketTimeoutException: Read timed out`.

In sync mode, when user send a execute statement request to Kyuubi server, Kyuubi server forword the request to the engine, engine return nothing until the execution finished or any other error happend.

In async mode, the query request passes in same way, but engine will return an `op_handle` immediately instead of waiting for query to finish, the client should use the `op_handle` to check query status periodically, in detail

1. Kyuubi server ask engine for query status, and keep the result in memory
2. User ask Kyuubi server for query status, Kyuubi server get result from memory

#### Solution

Option 1: change `socket_timeout` of Thrift clients created by Kyuubi server to infinite. This approach may cause another issue. if engine crash when executing the query, Kyuubi server will wait until the socket keepalive(controlled by OS) timeout, so the query status will always be `running` in Kyuubi server memory.

Option 2: always run a query in async mode, simulate sync mode in Server side.

This PR implement the option 2, also introduce a new conf `kyuubi.session.engine.request.timeout`

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1328 from pan3793/timeout.

Closes #1327

cc15e6d5 [Cheng Pan] Always execute statement in async mode
dc91d0d5 [Cheng Pan] Add new conf `kyuubi.session.engine.request.timeout`

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-11-04 18:23:10 +08:00
..
spark [KYUUBI #951] [LICENSE] Add license header on all docs 2021-08-19 09:53:52 +08:00
high_availability_guide.md [KYUUBI #1320] Fix HA documents 2021-11-02 17:22:46 +08:00
hive_metastore.md [KYUUBI #951] [LICENSE] Add license header on all docs 2021-08-19 09:53:52 +08:00
index.rst [KYUUBI #951] [LICENSE] Add license header on all docs 2021-08-19 09:53:52 +08:00
on_kubernetes.md [KYUUBI #951] [LICENSE] Add license header on all docs 2021-08-19 09:53:52 +08:00
on_yarn.md [KYUUBI #951] [LICENSE] Add license header on all docs 2021-08-19 09:53:52 +08:00
settings.md [KYUUBI #1327] Fix socket timeout error when client sync execute statement cost time longer than engine.login.timeout 2021-11-04 18:23:10 +08:00