apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity
73c6b1b14a
kyuubi/dev/dependencyList
Fei Wang 73c6b1b14a [KYUUBI #2717] [SUB-TASK][KPIP-4] Introduce jdbc session state store for batch session multiple HA 
### _Why are the changes needed?_

Close #2715
Close #2374
Introduce jdbc based session state store to share the batch state information across kyuubi instances.

This pr is based on docs: https://docs.google.com/document/d/1_BMekMT88P2vCHCl4ZacWEeWslpIv4pxsJH32PAnWSI and  refactor it during implementation.

In this PR, I introduce `StateStore` interface and implement the default `JDBCStateStore`.

The data structures:
1. BatchState - the batch state info for multiple-HA, including batch operation state and application information
2. BatchMeta - the meta data used for batch recovery, including sessionConf, batchRequest information

For JDBCStateStore:
- support to specify data base type:
  - 1. DERBY - it will use derby jdbc driver by default
  - 2. MYSQL - it will use mysql jdbc by default
  - 3. CUSTOM - the user need specify the jdbc driver
- support to init schema

By default, the jdbc url is a derby in-memory database url for single kyuubi instance.

To support multiple HA in production env, we should config the production jdbc url to share batch info across kyuubi instances.

The datasource is managed by HiKariCP.

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2717 from turboFei/kyuubi_2715_hikaricp_v2.

Closes #2717

367b944a3 [Fei Wang] comments
9a7963dbd [Fei Wang] post session type in event
b1c130853 [Fei Wang] move session type to session package
66def0cb6 [Fei Wang] add JDBCStateStoreConf
839b72bf6 [Fei Wang] refactor
644cbcd28 [Fei Wang] filter by session type
58f1c7854 [Fei Wang] refactor
1c0035208 [Fei Wang] add session type
34a087ee5 [Fei Wang] refactor
fba38e473 [Fei Wang] refactor
fcbde25a0 [Fei Wang] refactor
332a67537 [Fei Wang] comments
135ba9b6d [Fei Wang] Revert "revert class utils change"
0e045a483 [Fei Wang] Revert "Revert "move""
05048416e [Fei Wang] Revert "test again"
01a50c35b [Fei Wang] test again
b89159d13 [Fei Wang] Revert "move"
3764a9051 [Fei Wang] move
eef17822c [Fei Wang] revert
4166db261 [Fei Wang] adjust engine idle timeout
5fc95a824 [Fei Wang] revert class utils change
23f88d63d [Fei Wang] refactor
a66cb811d [Fei Wang] refactor
89fa0edb4 [Fei Wang] fix
e5228fefd [Fei Wang] refactor
581908d86 [Fei Wang] exclude sql
3446aaa06 [Fei Wang] UPPER
ab980c677 [Fei Wang] [KYUUBI #2715] Introduce session state store for batch session multiple HA

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-05-31 15:06:39 +08:00

123 lines
6.5 KiB
Plaintext
Raw Blame History

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
HikariCP/4.0.3//HikariCP-4.0.3.jar
aopalliance-repackaged/2.6.1//aopalliance-repackaged-2.6.1.jar
classgraph/4.8.95//classgraph-4.8.95.jar
commons-codec/1.15//commons-codec-1.15.jar
commons-collections/3.2.2//commons-collections-3.2.2.jar
commons-lang/2.6//commons-lang-2.6.jar
commons-lang3/3.10//commons-lang3-3.10.jar
commons-logging/1.1.3//commons-logging-1.1.3.jar
curator-client/2.12.0//curator-client-2.12.0.jar
curator-framework/2.12.0//curator-framework-2.12.0.jar
curator-recipes/2.12.0//curator-recipes-2.12.0.jar
derby/10.14.2.0//derby-10.14.2.0.jar
guava/30.1-jre//guava-30.1-jre.jar
hadoop-client-api/3.3.2//hadoop-client-api-3.3.2.jar
hadoop-client-runtime/3.3.2//hadoop-client-runtime-3.3.2.jar
hive-common/3.1.3//hive-common-3.1.3.jar
hive-metastore/3.1.3//hive-metastore-3.1.3.jar
hive-serde/3.1.3//hive-serde-3.1.3.jar
hive-service-rpc/3.1.3//hive-service-rpc-3.1.3.jar
hive-shims-0.23/3.1.3//hive-shims-0.23-3.1.3.jar
hive-shims-common/3.1.3//hive-shims-common-3.1.3.jar
hive-standalone-metastore/3.1.3//hive-standalone-metastore-3.1.3.jar
hk2-api/2.6.1//hk2-api-2.6.1.jar
hk2-locator/2.6.1//hk2-locator-2.6.1.jar
hk2-utils/2.6.1//hk2-utils-2.6.1.jar
httpclient/4.5.13//httpclient-4.5.13.jar
httpcore/4.4.13//httpcore-4.4.13.jar
jackson-annotations/2.13.3//jackson-annotations-2.13.3.jar
jackson-core/2.13.3//jackson-core-2.13.3.jar
jackson-databind/2.13.3//jackson-databind-2.13.3.jar
jackson-dataformat-yaml/2.13.3//jackson-dataformat-yaml-2.13.3.jar
jackson-datatype-jsr310/2.13.3//jackson-datatype-jsr310-2.13.3.jar
jackson-jaxrs-base/2.13.3//jackson-jaxrs-base-2.13.3.jar
jackson-jaxrs-json-provider/2.13.3//jackson-jaxrs-json-provider-2.13.3.jar
jackson-module-jaxb-annotations/2.13.3//jackson-module-jaxb-annotations-2.13.3.jar
jackson-module-scala_2.12/2.13.3//jackson-module-scala_2.12-2.13.3.jar
jakarta.activation-api/1.2.1//jakarta.activation-api-1.2.1.jar
jakarta.annotation-api/1.3.5//jakarta.annotation-api-1.3.5.jar
jakarta.inject/2.6.1//jakarta.inject-2.6.1.jar
jakarta.servlet-api/4.0.4//jakarta.servlet-api-4.0.4.jar
jakarta.validation-api/2.0.2//jakarta.validation-api-2.0.2.jar
jakarta.ws.rs-api/2.1.6//jakarta.ws.rs-api-2.1.6.jar
jakarta.xml.bind-api/2.3.2//jakarta.xml.bind-api-2.3.2.jar
javassist/3.25.0-GA//javassist-3.25.0-GA.jar
jcl-over-slf4j/1.7.35//jcl-over-slf4j-1.7.35.jar
jersey-client/2.35//jersey-client-2.35.jar
jersey-common/2.35//jersey-common-2.35.jar
jersey-container-servlet-core/2.35//jersey-container-servlet-core-2.35.jar
jersey-entity-filtering/2.35//jersey-entity-filtering-2.35.jar
jersey-hk2/2.35//jersey-hk2-2.35.jar
jersey-media-json-jackson/2.35//jersey-media-json-jackson-2.35.jar
jersey-server/2.35//jersey-server-2.35.jar
jetty-http/9.4.41.v20210516//jetty-http-9.4.41.v20210516.jar
jetty-io/9.4.41.v20210516//jetty-io-9.4.41.v20210516.jar
jetty-security/9.4.41.v20210516//jetty-security-9.4.41.v20210516.jar
jetty-server/9.4.41.v20210516//jetty-server-9.4.41.v20210516.jar
jetty-servlet/9.4.41.v20210516//jetty-servlet-9.4.41.v20210516.jar
jetty-util-ajax/9.4.41.v20210516//jetty-util-ajax-9.4.41.v20210516.jar
jetty-util/9.4.41.v20210516//jetty-util-9.4.41.v20210516.jar
jline/0.9.94//jline-0.9.94.jar
libfb303/0.9.3//libfb303-0.9.3.jar
libthrift/0.9.3//libthrift-0.9.3.jar
log4j-1.2-api/2.17.2//log4j-1.2-api-2.17.2.jar
log4j-api/2.17.2//log4j-api-2.17.2.jar
log4j-core/2.17.2//log4j-core-2.17.2.jar
log4j-slf4j-impl/2.17.2//log4j-slf4j-impl-2.17.2.jar
metrics-core/4.2.8//metrics-core-4.2.8.jar
metrics-jmx/4.2.8//metrics-jmx-4.2.8.jar
metrics-json/4.2.8//metrics-json-4.2.8.jar
metrics-jvm/4.2.8//metrics-jvm-4.2.8.jar
netty-all/4.1.73.Final//netty-all-4.1.73.Final.jar
netty-buffer/4.1.73.Final//netty-buffer-4.1.73.Final.jar
netty-codec/4.1.73.Final//netty-codec-4.1.73.Final.jar
netty-common/4.1.73.Final//netty-common-4.1.73.Final.jar
netty-handler/4.1.73.Final//netty-handler-4.1.73.Final.jar
netty-resolver/4.1.73.Final//netty-resolver-4.1.73.Final.jar
netty-tcnative-classes/2.0.46.Final//netty-tcnative-classes-2.0.46.Final.jar
netty-transport-classes-epoll/4.1.73.Final//netty-transport-classes-epoll-4.1.73.Final.jar
netty-transport-classes-kqueue/4.1.73.Final//netty-transport-classes-kqueue-4.1.73.Final.jar
netty-transport-native-epoll/4.1.73.Final/linux-aarch_64/netty-transport-native-epoll-4.1.73.Final-linux-aarch_64.jar
netty-transport-native-epoll/4.1.73.Final/linux-x86_64/netty-transport-native-epoll-4.1.73.Final-linux-x86_64.jar
netty-transport-native-kqueue/4.1.73.Final/osx-aarch_64/netty-transport-native-kqueue-4.1.73.Final-osx-aarch_64.jar
netty-transport-native-kqueue/4.1.73.Final/osx-x86_64/netty-transport-native-kqueue-4.1.73.Final-osx-x86_64.jar
netty-transport-native-unix-common/4.1.73.Final//netty-transport-native-unix-common-4.1.73.Final.jar
netty-transport/4.1.73.Final//netty-transport-4.1.73.Final.jar
osgi-resource-locator/1.0.3//osgi-resource-locator-1.0.3.jar
paranamer/2.8//paranamer-2.8.jar
scala-library/2.12.15//scala-library-2.12.15.jar
scopt_2.12/4.0.1//scopt_2.12-4.0.1.jar
simpleclient/0.14.1//simpleclient-0.14.1.jar
simpleclient_common/0.14.1//simpleclient_common-0.14.1.jar
simpleclient_dropwizard/0.14.1//simpleclient_dropwizard-0.14.1.jar
simpleclient_servlet/0.14.1//simpleclient_servlet-0.14.1.jar
simpleclient_servlet_common/0.14.1//simpleclient_servlet_common-0.14.1.jar
simpleclient_tracer_common/0.14.1//simpleclient_tracer_common-0.14.1.jar
simpleclient_tracer_otel/0.14.1//simpleclient_tracer_otel-0.14.1.jar
simpleclient_tracer_otel_agent/0.14.1//simpleclient_tracer_otel_agent-0.14.1.jar
slf4j-api/1.7.35//slf4j-api-1.7.35.jar
snakeyaml/1.30//snakeyaml-1.30.jar
swagger-annotations/2.1.11//swagger-annotations-2.1.11.jar
swagger-core/2.1.11//swagger-core-2.1.11.jar
swagger-integration/2.1.11//swagger-integration-2.1.11.jar
swagger-jaxrs2/2.1.11//swagger-jaxrs2-2.1.11.jar
swagger-models/2.1.11//swagger-models-2.1.11.jar
zookeeper/3.4.14//zookeeper-3.4.14.jar
Reference in New Issue View Git Blame Copy Permalink