![](../imgs/kyuubi_logo_simple.png)

# Configurations Guide Kyuubi provides several ways to configure the system. ## Environments You can configure the environment variables in `$KYUUBI_HOME/conf/kyuubi-env.sh`, e.g, `JAVA_HOME`, then this java runtime will be used both for Kyuubi server instance and the applications it launches. You can also change the variable in the subprocess's env configuration file, e.g.`$SPARK_HOME/conf/spark-env.sh` to use more specific ENV for SQL engine applications. ```bash #!/usr/bin/env bash # # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # # - JAVA_HOME Java runtime to use. By default use "java" from PATH. # # # - KYUUBI_CONF_DIR Directory containing the Kyuubi configurations to use. # (Default: $KYUUBI_HOME/conf) # - KYUUBI_LOG_DIR Directory for Kyuubi server-side logs. # (Default: $KYUUBI_HOME/logs) # - KYUUBI_PID_DIR Directory stores the Kyuubi instance pid file. # (Default: $KYUUBI_HOME/pid) # - KYUUBI_MAX_LOG_FILES Maximum number of Kyuubi server logs can rotate to. # (Default: 5) # - KYUUBI_JAVA_OPTS JVM options for the Kyuubi server itself in the form "-Dx=y". # (Default: none). # - KYUUBI_NICENESS The scheduling priority for Kyuubi server. # (Default: 0) # - KYUUBI_WORK_DIR_ROOT Root directory for launching sql engine applications. # (Default: $KYUUBI_HOME/work) # - HADOOP_CONF_DIR Directory containing the Hadoop / YARN configuration to use. # # - SPARK_HOME Spark distribution which you would like to use in Kyuubi. # - SPARK_CONF_DIR Optional directory where the Spark configuration lives. # (Default: $SPARK_HOME/conf) # ``` ## Kyuubi Configurations You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.conf`. For example: ```bash # # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ## Kyuubi Configurations # # kyuubi.authentication NONE # kyuubi.frontend.bind.port 10009 # ## Spark Configurations # # spark.master local # spark.ui.enabled false ## Hadoop Configurations # # kyuubi.hadoop.authentication KERBEROS # ``` ### Authentication Key | Default | Meaning | Since --- | --- | --- | --- kyuubi\.authentication|

NONE

Client authentication types.

NONE: no authentication check.
KERBEROS: Kerberos/GSSAPI authentication.
LDAP: Lightweight Directory Access Protocol authentication.

1.0.0

kyuubi\.authentication
\.ldap\.base\.dn|

LDAP base DN.

1.0.0

kyuubi\.authentication
\.ldap\.domain|

LDAP base DN.

1.0.0

kyuubi\.authentication
\.ldap\.url|

SPACE character separated LDAP connection URL(s).

1.0.0

kyuubi\.authentication
\.sasl\.qop|

auth

Sasl QOP enable higher levels of protection for Kyuubi communication with clients.

auth - authentication only (default)
auth-int - authentication plus integrity protection
auth-conf - authentication plus integrity and confidentiality protection. This is applicable only if Kyuubi is configured to use Kerberos authentication.

1.0.0

### Delegation Key | Default | Meaning | Since --- | --- | --- | --- kyuubi\.delegation\.key
\.update\.interval|

PT24H

unused yet

1.0.0

kyuubi\.delegation
\.token\.gc\.interval|

PT1H

unused yet

1.0.0

kyuubi\.delegation
\.token\.max\.lifetime|

PT168H

unused yet

1.0.0

kyuubi\.delegation
\.token\.renew\.interval|

PT168H

unused yet

1.0.0

### Frontend Key | Default | Meaning | Since --- | --- | --- | --- kyuubi\.frontend
\.backoff\.slot\.length|

PT0.1S

Time to back off during login to the frontend service.

1.0.0

kyuubi\.frontend\.bind
\.host|

Hostname or IP of the machine on which to run the frontend service.

1.0.0

kyuubi\.frontend\.bind
\.port|

10009

Port of the machine on which to run the frontend service.

1.0.0

kyuubi\.frontend\.login
\.timeout|

PT20S

Timeout for Thrift clients during login to the frontend service.

1.0.0

kyuubi\.frontend\.max
\.message\.size|

104857600

Maximum message size in bytes a Kyuubi server will accept.

1.0.0

kyuubi\.frontend\.max
\.worker\.threads|

Maximum number of threads in the of frontend worker thread pool for the frontend service

1.0.0

kyuubi\.frontend\.min
\.worker\.threads|

Minimum number of threads in the of frontend worker thread pool for the frontend service

1.0.0

kyuubi\.frontend
\.worker\.keepalive\.time|

PT1M

Keep-alive time (in milliseconds) for an idle worker thread

1.0.0

### Ha Key | Default | Meaning | Since --- | --- | --- | --- kyuubi\.ha\.zookeeper
\.acl\.enabled|

false

Set to true if the zookeeper ensemble is kerberized

1.0.0

kyuubi\.ha\.zookeeper
\.connection\.base\.retry
\.wait|

1000

Initial amount of time to wait between retries to the zookeeper ensemble

1.0.0

kyuubi\.ha\.zookeeper
\.connection\.max
\.retries|

Max retry times for connecting to the zookeeper ensemble

1.0.0

kyuubi\.ha\.zookeeper
\.connection\.max\.retry
\.wait|

30000

Max amount of time to wait between retries for BONDED_EXPONENTIAL_BACKOFF policy can reach, or max time until elapsed for UNTIL_ELAPSED policy to connect the zookeeper ensemble

1.0.0

kyuubi\.ha\.zookeeper
\.connection\.retry
\.policy|

EXPONENTIAL_BACKOFF

The retry policy for connecting to the zookeeper ensemble, all candidates are:

ONE_TIME
N_TIME
EXPONENTIAL_BACKOFF
BONDED_EXPONENTIAL_BACKOFF
UNTIL_ELAPSED

1.0.0

kyuubi\.ha\.zookeeper
\.connection\.timeout|

15000

The timeout(ms) of creating the connection to the zookeeper ensemble

1.0.0

kyuubi\.ha\.zookeeper
\.namespace|

kyuubi

The root directory for the service to deploy its instance uri. Additionally, it will creates a -[username] suffixed root directory for each application

1.0.0

kyuubi\.ha\.zookeeper
\.quorum|

The connection string for the zookeeper ensemble

1.0.0

kyuubi\.ha\.zookeeper
\.session\.timeout|

60000

The timeout(ms) of a connected session to be idled

1.0.0

### Kinit Key | Default | Meaning | Since --- | --- | --- | --- kyuubi\.kinit\.interval|

PT1H

How often will Kyuubi server run `kinit -kt [keytab] [principal]` to renew the local Kerberos credentials cache

1.0.0

kyuubi\.kinit\.keytab|

Location of Kyuubi server's keytab.

1.0.0

kyuubi\.kinit\.max
\.attempts|

How many times will `kinit` process retry

1.0.0

kyuubi\.kinit
\.principal|

Name of the Kerberos principal.

1.0.0

### Operation Key | Default | Meaning | Since --- | --- | --- | --- kyuubi\.operation\.idle
\.timeout|

PT3H

Operation will be closed when it's not accessed for this duration of time

1.0.0

### Session Key | Default | Meaning | Since --- | --- | --- | --- kyuubi\.session\.check
\.interval|

PT5M

The check interval for session timeout.

kyuubi\.session\.engine
\.initialize\.timeout|

PT1M

Timeout for starting the background engine, e.g. SparkSQLEngine.

1.0.0

kyuubi\.session\.engine
\.login\.timeout|

PT15S

The timeout(ms) of creating the connection to remote sql query engine

1.0.0

kyuubi\.session\.engine
\.spark\.main\.resource|

The package used to create Spark SQL engine remote application. If it is undefined, Kyuubi will use the default

1.0.0

kyuubi\.session
\.timeout|

PT6H

session timeout, it will be closed when it's not accessed for this duration

### Zookeeper Key | Default | Meaning | Since --- | --- | --- | --- kyuubi\.zookeeper
\.embedded\.directory|

embedded_zookeeper

The temporary directory for the embedded zookeeper server

1.0.0

kyuubi\.zookeeper
\.embedded\.port|

2181

The port of the embedded zookeeper server

1.0.0

## Spark Configurations ### Via spark-defaults.conf Setting them in `$SPARK_HOME/conf/spark-defaults.conf` supplies with default values for SQL engine application. Available properties can be found at Spark official online documentation for [Spark Configurations](http://spark.apache.org/docs/latest/configuration.html) ### Via kyuubi-defaults.conf Setting them in `$KYUUBI_HOME/conf/kyuubi-defaults.conf` supplies with default values for SQL engine application too. These properties will override all settings in `$SPARK_HOME/conf/spark-defaults.conf` ### Via JDBC Connection URL Setting them in the JDBC Connection URL supplies session-specific for each SQL engine. For example: ```jdbc:hive2://localhost:10009/default;#spark.sql.shuffle.partitions=2;spark.executor.memory=5g``` - **Runtime SQL Configuration** - For [Runtime SQL Configurations](http://spark.apache.org/docs/latest/configuration.html#runtime-sql-configuration), they will take affect every time - **Static SQL and Spark Core Configuration** - For [Static SQL Configurations](http://spark.apache.org/docs/latest/configuration.html#static-sql-configuration) and other spark core configs, e.g. `spark.executor.memory`, they will take affect if there is no existing SQL engine application. Otherwise, they will just be ignored ### Via SET Syntax Please refer to the Spark official online documentation for [SET Command](http://spark.apache.org/docs/latest/sql-ref-syntax-aux-conf-mgmt-set.html) ## Logging Kyuubi uses [log4j](https://logging.apache.org/log4j/2.x/) for logging. You can configure it using `$KYUUBI_HOME/conf/log4j.properties`. ```bash # # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Set everything to be logged to the console log4j.rootCategory=INFO, console log4j.appender.console=org.apache.log4j.ConsoleAppender log4j.appender.console.target=System.err log4j.appender.console.layout=org.apache.log4j.PatternLayout log4j.appender.console.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss.SSS} %p %c{2}: %m%n ``` ## Other Configurations ### Hadoop Configurations Specifying `HADOOP_CONF_DIR` to the directory contains hadoop configuration files or treating them as Spark properties with a `spark.hadoop.` prefix. Please refer to the Spark official online documentation for [Inheriting Hadoop Cluster Configuration](http://spark.apache.org/docs/latest/configuration.html#inheriting-hadoop-cluster-configuration). Also, please refer to the [Apache Hadoop](http://hadoop.apache.org)'s online documentation for an overview on how to configure Hadoop. ### Hive Configurations These configurations are used for SQL engine application to talk to Hive MetaStore and could be configured in a `hive-site.xml`. Placed it in `$SPARK_HOME/conf` directory, or treating them as Spark properties with a `spark.hadoop.` prefix.