4e81e6b230
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Refactor the key words after repo renamed. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request
7.4 KiB
Kyuubi High Availability Guide
As an enterprise-class ad-hoc SQL query service built on top of Apache Spark, Kyuubi takes high availability(HA) as a major characteristic, aiming to ensure an agreed level of service availability, such as a higher than normal period of uptime.
Running Kyuubi in HA mode is to use groups of computers or containers that support SQL query service on Kyuubi that can be reliably utilized with a minimum amount of down-time. Kyuubi operates by using Apache ZooKeeper to harness redundant service instances in groups that provide continuous service when one or more components fail.
Without HA, if a server crashes, Kyuubi will be unavailable until the crashed server is fixed. With HA, this situation will be remedied by hardware/software faults auto detecting, and immediately another Kyuubi service instance will be ready to serve without requiring human intervention.
High Availability Mode Types
Kyuubi supports two different types of HA mode. One is load balance mode, and the other active/standby failover.
Load balance mode means that all Kyuubi server instances are active at the first place and their service uri can be reached by clients through the Zookeeper. This mode can greatly reduce the load on the instance itself, on the contrary the load on the cluster manager(YARN) may go higher for there may be more than one Spark application running for a single user who is connected. Another thing you need to know is that applying resources from YARN to launch an application is time consuming.
Active/Standby failover is another option for you to make Kyuubi system highly available. Only one node is the primary node and visible for clients via ZooKeeper and all the others are secondary ones during the service period and invisble. In this mode, the standby will become serviceable only when it has gained the leadership then publish its service uri to ZooKeeper, meanwhile the previous active one's uri will be retired. There will be only one Spark application running on YARN for a single user connected. YARN's load will not be as heavy as in load balance mode and the overhead for launching applications will be certainly reduced. On the other side, the active server will withstand more traffic. Anyway, it is a good choice for those small Hadoop clusters to apply this mode to gain high availability.
The number of user concurrency and the size of your cluster may be two major indicators which you need to weigh against the online environment.
Load Balance Mode
Load balancing aims to optimize all Kyuubi service units usage, maximize throughput, minimize response time, and avoid overload of a single unit. Using multiple Kyuubi service units with load balancing instead of a single unit may increase reliability and availability through redundancy.
With Hive JDBC Driver, a client can specify service discovery mode in JDBC connection string, i.e. serviceDiscoveryMode=zooKeeper; and set zooKeeperNameSpace=kyuubiserver;, then it can randomly pick one of the Kyuubi service uris from the specified ZooKeeper address in the /kyuubiserver path.
When we set kyuubi.ha.enabled to true, load balance mode is activated by default. Please make sure that you specify the correct ZooKeeper address via kyuubi.ha.zookeeper.quorum and kyuubi.ha.zookeeper.client.port.
Active/Standby Failover
Active/Standby failover enables you to use a standby Kyuubi server to take over the functionality of a failed unit. When the active unit fails, it changes to the standby state after fixed while the standby unit changes to the active state.
A client need not to change any of its behaviours to support load balance or failover mode. But because only the active Kyuubi server will expose its service uri to ZooKeeper in /kyuubiserver, clients always randomly pick a server from one and the only choice.
When we set kyuubi.ha.enabled to true and kyuubi.ha.mode to failover, failover mode is activated then. Please make sure that you specify the correct ZooKeeper address via kyuubi.ha.zookeeper.quorum and kyuubi.ha.zookeeper.client.port.
Configuring High Availability
This section describes how to configure high availability. These configurations in the following table can be treat like normal Spark properties by setting them in spark-defaults.conf file or via --conf parameter in server starting scripts.
| Key | Default | Meaning | Since |
|---|---|---|---|
| kyuubi.ha.zookeeper .acl.enabled |
false |
Set to true if the zookeeper ensemble is kerberized |
1.0.0 |
| kyuubi.ha.zookeeper .connection.base.retry .wait |
1000 |
Initial amount of time to wait between retries to the zookeeper ensemble |
1.0.0 |
| kyuubi.ha.zookeeper .connection.max .retries |
3 |
Max retry times for connecting to the zookeeper ensemble |
1.0.0 |
| kyuubi.ha.zookeeper .connection.max.retry .wait |
30000 |
Max amount of time to wait between retries for BOUNDED_EXPONENTIAL_BACKOFF policy can reach, or max time until elapsed for UNTIL_ELAPSED policy to connect the zookeeper ensemble |
1.0.0 |
| kyuubi.ha.zookeeper .connection.retry .policy |
EXPONENTIAL_BACKOFF |
The retry policy for connecting to the zookeeper ensemble, all candidates are:
|
1.0.0 |
| kyuubi.ha.zookeeper .connection.timeout |
15000 |
The timeout(ms) of creating the connection to the zookeeper ensemble |
1.0.0 |
| kyuubi.ha.zookeeper .namespace |
kyuubi |
The root directory for the service to deploy its instance uri. Additionally, it will creates a -[username] suffixed root directory for each application |
1.0.0 |
| kyuubi.ha.zookeeper .quorum |
The connection string for the zookeeper ensemble |
1.0.0 |
|
| kyuubi.ha.zookeeper .session.timeout |
60000 |
The timeout(ms) of a connected session to be idled |
1.0.0 |