# 🔍 Description ## Issue References 🔗 This pull request fixes #6034 ## Describe Your Solution 🔧 Currently, use beeline to connect kyuubiServer with HA mode, the strategy only support random, this will lead to a high load on the machine. So i make this pr to support choose strategy. [description] First, we need know, beeline connect kyuubiServer dependency on kyuubi-hive-jdbc, it is isolated from the kyuubi cluster, so the code only support random choose serverHost from zk node /${namespace}. Because kyuubi-hive-jdbc is a stateless module, only run once, cannot store var about get serverHost from zk node. [Solution] This pr, we could implement a interface named ChooseServerStrategy to choose serverHost. I implement two strategy 1. poll: it will create a zk node named ${namespace}-counter, when a beeline client want connect kyuubiServer, the node will increment 1, use this value to take the remainder from serverHosts, like counter % serverHost.size, so we could get a order serverHost 2. random: random get serverHost from serverHosts 3. User Definied Class: implemented the ChooseServerStrategy, then put the jar to beeline-jars, it can use your strategy to choose serverHost ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Test the Strategy in my test Cluster #### Behavior Without This Pull Request ⚰️    #### Behavior With This Pull Request 🎉 [Use Case] 1. poll: `bin/beeline -u 'jdbc:hive2://xxx:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi;zooKeeperStrategy=poll?spark.yarn.queue=root.kylin;spark.app.name=testspark;spark.shuffle.useOldFetchProtocol=true' -n mfw_hadoop --verbose=true --showNestedErrs=true` 2. random: `bin/beeline -u 'jdbc:hive2://xxx:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi;zooKeeperStrategy=random?spark.yarn.queue=root.kylin;spark.app.name=testspark;spark.shuffle.useOldFetchProtocol=true' -n mfw_hadoop --verbose=true --showNestedErrs=true` or `bin/beeline -u 'jdbc:hive2://xxx:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi?spark.yarn.queue=root.kylin;spark.app.name=testspark;spark.shuffle.useOldFetchProtocol=true' -n mfw_hadoop --verbose=true --showNestedErrs=true` 3. YourStrategy: `bin/beeline -u 'jdbc:hive2://xxx:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi;zooKeeperStrategy=xxx.xxx.xxx.XxxChooseServerStrategy?spark.yarn.queue=root.kylin;spark.app.name=testspark;spark.shuffle.useOldFetchProtocol=true' -n mfw_hadoop --verbose=true --showNestedErrs=true` [Result: The Cluster have two Server (221,233)] 1. poll: 1.1. zkNode: counterValue  1.2. result:     2. random:    3. YourStrategy(the test case only get the first serverHost):    #### Related Unit Tests There is no Unit Tests. --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6213 from davidyuan1223/ha_zk_support_more_strategy. Closes #6034 961d3e989 [Bowen Liang] rename ServerStrategyFactory to ServerSelectStrategyFactory 353f94059 [Bowen Liang] repeat 8822ad471 [Bowen Liang] repeat 619339402 [Bowen Liang] nit e94f9e909 [Bowen Liang] nit 40f427ae5 [Bowen Liang] rename StrategyFactory to StrategyFactoryServerStrategyFactory 7668f99cc [Bowen Liang] test name e194ea62f [Bowen Liang] remove ZooKeeperHiveClientException from method signature of chooseServer 265965e5d [Bowen Liang] polling b39c56700 [Bowen Liang] style 1ab79b494 [Bowen Liang] strategyName 8f8ca28f2 [Bowen Liang] nit 228bf1091 [Bowen Liang] rename parameter zooKeeperStrategy to serverSelectStrategy 125c82358 [Bowen Liang] rename ChooseServerStrategy to ServerSelectStrategy b4aeb3dbd [Bowen Liang] repeat testing on pollingChooseStrategy 465548005 [davidyuan] update 09a84f1f9 [david yuan] remove the distirbuted lock 93f4a2699 [davidyuan] remove reset 7b0c1b811 [davidyuan] fix var not valid and counter getAndIncrement c95382a23 [davidyuan] fix var not valid and counter getAndIncrement 9ed2cac85 [david yuan] remove test comment 8eddd7682 [davidyuan] Add Strategy Unit Test Case and fix the polling strategy counter begin with 0 73952f878 [davidyuan] Kyuubi Server HA&ZK get server from serverHosts support more strategy 97b959776 [davidyuan] Kyuubi Server HA&ZK get server from serverHosts support more strategy ee5a9ad68 [davidyuan] Kyuubi Server HA&ZK get server from serverHosts support more strategy 6a0445357 [davidyuan] Kyuubi Server HA&ZK get server from serverHosts support more strategy 1892f148d [davidyuan] add common method to get session level config 7c0c6058d [yuanfuyuan] fix_4186 Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn> Co-authored-by: davidyuan <yuanfuyuan@mafengwo.com> Co-authored-by: davidyuan <davidyuan1223@gmail.com> Co-authored-by: david yuan <davidyuan1223@gmail.com> Co-authored-by: yuanfuyuan <1406957364@qq.com> Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
165 lines
5.5 KiB
XML
165 lines
5.5 KiB
XML
<?xml version="1.0" encoding="UTF-8"?>
|
|
<!--
|
|
~ Licensed to the Apache Software Foundation (ASF) under one or more
|
|
~ contributor license agreements. See the NOTICE file distributed with
|
|
~ this work for additional information regarding copyright ownership.
|
|
~ The ASF licenses this file to You under the Apache License, Version 2.0
|
|
~ (the "License"); you may not use this file except in compliance with
|
|
~ the License. You may obtain a copy of the License at
|
|
~
|
|
~ http://www.apache.org/licenses/LICENSE-2.0
|
|
~
|
|
~ Unless required by applicable law or agreed to in writing, software
|
|
~ distributed under the License is distributed on an "AS IS" BASIS,
|
|
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
~ See the License for the specific language governing permissions and
|
|
~ limitations under the License.
|
|
-->
|
|
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
|
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
|
<modelVersion>4.0.0</modelVersion>
|
|
<parent>
|
|
<groupId>org.apache.kyuubi</groupId>
|
|
<artifactId>kyuubi-parent</artifactId>
|
|
<version>1.10.0-SNAPSHOT</version>
|
|
<relativePath>../pom.xml</relativePath>
|
|
</parent>
|
|
|
|
<artifactId>kyuubi-ha_${scala.binary.version}</artifactId>
|
|
<packaging>jar</packaging>
|
|
<name>Kyuubi Project High Availability</name>
|
|
<url>https://kyuubi.apache.org/</url>
|
|
|
|
<dependencies>
|
|
<dependency>
|
|
<groupId>org.apache.kyuubi</groupId>
|
|
<artifactId>kyuubi-common_${scala.binary.version}</artifactId>
|
|
<version>${project.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.apache.kyuubi</groupId>
|
|
<artifactId>${kyuubi-relocated-zookeeper.artifacts}</artifactId>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>com.google.guava</groupId>
|
|
<artifactId>guava</artifactId>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>io.etcd</groupId>
|
|
<artifactId>jetcd-core</artifactId>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>io.grpc</groupId>
|
|
<artifactId>grpc-core</artifactId>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>io.grpc</groupId>
|
|
<artifactId>grpc-grpclb</artifactId>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>io.grpc</groupId>
|
|
<artifactId>grpc-netty</artifactId>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>io.grpc</groupId>
|
|
<artifactId>grpc-util</artifactId>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>io.netty</groupId>
|
|
<artifactId>netty-transport-native-epoll</artifactId>
|
|
<classifier>linux-aarch_64</classifier>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>io.netty</groupId>
|
|
<artifactId>netty-transport-native-epoll</artifactId>
|
|
<classifier>linux-x86_64</classifier>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>io.grpc</groupId>
|
|
<artifactId>grpc-protobuf</artifactId>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>io.grpc</groupId>
|
|
<artifactId>grpc-stub</artifactId>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>com.dimafeng</groupId>
|
|
<artifactId>testcontainers-scala-scalatest_${scala.binary.version}</artifactId>
|
|
<scope>test</scope>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>io.etcd</groupId>
|
|
<artifactId>jetcd-launcher</artifactId>
|
|
<scope>test</scope>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.apache.kyuubi</groupId>
|
|
<artifactId>kyuubi-common_${scala.binary.version}</artifactId>
|
|
<version>${project.version}</version>
|
|
<type>test-jar</type>
|
|
<scope>test</scope>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.apache.hadoop</groupId>
|
|
<artifactId>hadoop-minikdc</artifactId>
|
|
<scope>test</scope>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>com.unboundid</groupId>
|
|
<artifactId>unboundid-ldapsdk</artifactId>
|
|
<scope>test</scope>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.apache.kyuubi</groupId>
|
|
<artifactId>kyuubi-zookeeper_${scala.binary.version}</artifactId>
|
|
<version>${project.version}</version>
|
|
<scope>test</scope>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.apache.kyuubi</groupId>
|
|
<artifactId>kyuubi-hive-jdbc</artifactId>
|
|
<version>${project.version}</version>
|
|
<scope>test</scope>
|
|
</dependency>
|
|
</dependencies>
|
|
|
|
<build>
|
|
|
|
<plugins>
|
|
<plugin>
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
<artifactId>maven-jar-plugin</artifactId>
|
|
<executions>
|
|
<execution>
|
|
<id>prepare-test-jar</id>
|
|
<goals>
|
|
<goal>test-jar</goal>
|
|
</goals>
|
|
<phase>test-compile</phase>
|
|
</execution>
|
|
</executions>
|
|
</plugin>
|
|
</plugins>
|
|
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
|
|
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
|
|
</build>
|
|
</project>
|