diff --git a/docs/deployment/settings.md b/docs/deployment/settings.md index 1aa0c8a0d..96553827b 100644 --- a/docs/deployment/settings.md +++ b/docs/deployment/settings.md @@ -1,5 +1,4 @@ - - +
@@ -7,15 +6,15 @@
- # Introduction to the Kyuubi Configurations System Kyuubi provides several ways to configure the system and corresponding engines. + ## Environments -You can configure the environment variables in `$KYUUBI_HOME/conf/kyuubi-env.sh`, e.g, `JAVA_HOME`, then this java runtime will be used both for Kyuubi server instance and the applications it launches. You can also change the variable in the subprocess's env configuration file, e.g.`$SPARK_HOME/conf/spark-env.sh` to use more specific ENV for SQL engine applications. +You can configure the environment variables in `$KYUUBI_HOME/conf/kyuubi-env.sh`, e.g, `JAVA_HOME`, then this java runtime will be used both for Kyuubi server instance and the applications it launches. You can also change the variable in the subprocess's env configuration file, e.g.`$SPARK_HOME/conf/spark-env.sh` to use more specific ENV for SQL engine applications. ```bash #!/usr/bin/env bash # @@ -66,10 +65,10 @@ You can configure the environment variables in `$KYUUBI_HOME/conf/kyuubi-env.sh` # export HADOOP_CONF_DIR=/usr/ndp/current/mapreduce_client/conf # export KYUUBI_JAVA_OPTS="-Xmx10g -XX:+UnlockDiagnosticVMOptions -XX:ParGCCardsPerStrideChunk=4096 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:+UseCondCardMark -XX:MaxDirectMemorySize=1024m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./logs -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -Xloggc:./logs/kyuubi-server-gc-%t.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=5M -XX:NewRatio=3 -XX:MetaspaceSize=512m" ``` + ## Kyuubi Configurations You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.conf`. For example: - ```bash # # Licensed to the Apache Software Foundation (ASF) under one or more @@ -98,6 +97,7 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co # Details in https://kyuubi.readthedocs.io/en/latest/deployment/settings.html ``` + ### Authentication Key | Default | Meaning | Since @@ -108,6 +108,7 @@ kyuubi\.authentication
\.ldap\.domain|
<undefined>
|
SPACE character separated LDAP connection URL(s).
|
1.0.0
kyuubi\.authentication
\.sasl\.qop|
auth
|
Sasl QOP enable higher levels of protection for Kyuubi communication with clients.
|
1.0.0
+ ### Backend Key | Default | Meaning | Since @@ -121,6 +122,7 @@ kyuubi\.backend\.server
\.exec\.pool\.shutdown
\.timeout|
100
|
Number of threads in the operation execution thread pool of Kyuubi server
|
1.0.0
kyuubi\.backend\.server
\.exec\.pool\.wait\.queue
\.size|
100
|
Size of the wait queue for the operation execution thread pool of Kyuubi server
|
1.0.0
+ ### Delegation Key | Default | Meaning | Since @@ -130,6 +132,7 @@ kyuubi\.delegation
\.token\.gc\.interval|
PT168H
|
unused yet
|
1.0.0
kyuubi\.delegation
\.token\.renew\.interval|
PT168H
|
unused yet
|
1.0.0
+ ### Frontend Key | Default | Meaning | Since @@ -143,6 +146,7 @@ kyuubi\.frontend\.max
\.worker\.threads|
9
|
Minimum number of threads in the of frontend worker thread pool for the frontend service
|
1.0.0
kyuubi\.frontend
\.worker\.keepalive\.time|
PT1M
|
Keep-alive time (in milliseconds) for an idle worker thread
|
1.0.0
+ ### Ha Key | Default | Meaning | Since @@ -157,6 +161,7 @@ kyuubi\.ha\.zookeeper
\.namespace|
|
The connection string for the zookeeper ensemble
|
1.0.0
kyuubi\.ha\.zookeeper
\.session\.timeout|
60000
|
The timeout(ms) of a connected session to be idled
|
1.0.0
+ ### Kinit Key | Default | Meaning | Since @@ -166,6 +171,7 @@ kyuubi\.kinit\.keytab|
10
|
How many times will `kinit` process retry
|
1.0.0
kyuubi\.kinit
\.principal|
<undefined>
|
Name of the Kerberos principal.
|
1.0.0
+ ### Metrics Key | Default | Meaning | Since @@ -175,6 +181,7 @@ kyuubi\.metrics\.json
\.report\.location|
PT5S
|
How often should report metrics to json/console. no effect on JMX
|
1.2.0
kyuubi\.metrics
\.reporters|
JSON
|
A comma separated list for all metrics reporters
|
1.2.0
+ ### Operation Key | Default | Meaning | Since @@ -184,6 +191,7 @@ kyuubi\.operation
\.interrupt\.on\.cancel|
PT0S
|
Set a query duration timeout in seconds in Kyuubi. If the timeout is set to a positive value, a running query will be cancelled automatically if timeout. Otherwise the query continues to run till completion. If timeout values are set for each statement via `java.sql.Statement.setQueryTimeout` and they are smaller than this configuration value, they take precedence. If you set this timeout and prefer to cancel the queries right away without waiting task to finish, consider enabling kyuubi.operation.interrupt.on.cancel together.
|
1.2.0
kyuubi\.operation
\.status\.polling
\.timeout|
PT5S
|
Timeout(ms) for long polling asynchronous running sql query's status
|
1.0.0
+ ### Session Key | Default | Meaning | Since @@ -199,6 +207,7 @@ kyuubi\.session\.engine
\.spark\.main\.resource|
8192
|
During engine bootstrapping, if error occurs, using this config to limit the length error message(characters).
|
1.1.0
kyuubi\.session
\.timeout|
PT6H
|
session timeout, it will be closed when it's not accessed for this duration
|
1.0.0
+ ### Zookeeper Key | Default | Meaning | Since @@ -221,16 +230,20 @@ Setting them in `$KYUUBI_HOME/conf/kyuubi-defaults.conf` supplies with default v Setting them in the JDBC Connection URL supplies session-specific for each SQL engine. For example: ```jdbc:hive2://localhost:10009/default;#spark.sql.shuffle.partitions=2;spark.executor.memory=5g``` - **Runtime SQL Configuration** + - For [Runtime SQL Configurations](http://spark.apache.org/docs/latest/configuration.html#runtime-sql-configuration), they will take affect every time + - **Static SQL and Spark Core Configuration** + - For [Static SQL Configurations](http://spark.apache.org/docs/latest/configuration.html#static-sql-configuration) and other spark core configs, e.g. `spark.executor.memory`, they will take affect if there is no existing SQL engine application. Otherwise, they will just be ignored + ### Via SET Syntax Please refer to the Spark official online documentation for [SET Command](http://spark.apache.org/docs/latest/sql-ref-syntax-aux-conf-mgmt-set.html) + ## Logging Kyuubi uses [log4j](https://logging.apache.org/log4j/2.x/) for logging. You can configure it using `$KYUUBI_HOME/conf/log4j.properties`. - ```bash # # Licensed to the Apache Software Foundation (ASF) under one or more @@ -256,6 +269,7 @@ log4j.appender.console.target=System.err log4j.appender.console.layout=org.apache.log4j.PatternLayout log4j.appender.console.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss.SSS} %p %c{2}: %m%n ``` + ## Other Configurations ### Hadoop Configurations @@ -269,9 +283,7 @@ These configurations are used for SQL engine application to talk to Hive MetaSto ## User Defaults In Kyuubi, we can configure user default settings to meet separate needs. These user defaults override system defaults, but will be overridden by those from [JDBC Connection URL](#via-jdbc-connection-url) or [Set Command](#via-set-syntax) if could be. They will take effect when creating the SQL engine application ONLY. - User default settings are in the form of `___{username}___.{config key}`. There are three continuous underscores(`_`) at both sides of the `username` and a dot(`.`) that separates the config key and the prefix. For example: - ```bash # For system defaults spark.master=local @@ -285,4 +297,3 @@ ___bob___.spark.executor.memory=8g ``` In the above case, if there are related configurations from [JDBC Connection URL](#via-jdbc-connection-url), `kent` will run his SQL engine application on YARN and prefer the Spark AQE to be off, while `bob` will activate his SQL engine application on a Spark standalone cluster with 8g heap memory for each executor and obey the Spark AQE behavior of Kyuubi system default. On the other hand, for those users who do not have custom configurations will use system defaults. - diff --git a/kyuubi-main/src/test/scala/org/apache/kyuubi/config/AllKyuubiConfiguration.scala b/kyuubi-main/src/test/scala/org/apache/kyuubi/config/AllKyuubiConfiguration.scala index de8520abe..772c37851 100644 --- a/kyuubi-main/src/test/scala/org/apache/kyuubi/config/AllKyuubiConfiguration.scala +++ b/kyuubi-main/src/test/scala/org/apache/kyuubi/config/AllKyuubiConfiguration.scala @@ -21,207 +21,262 @@ import java.nio.charset.StandardCharsets import java.nio.file.{Files, Path, Paths, StandardOpenOption} import scala.collection.JavaConverters._ +import scala.collection.mutable.ArrayBuffer import org.apache.kyuubi.KyuubiFunSuite import org.apache.kyuubi.ha.HighAvailabilityConf import org.apache.kyuubi.metrics.MetricsConf +// scalastyle:off line.size.limit +/** + * End-to-end test cases for configuration doc file + * The golden result file is "docs/deployment/settings.md". + * + * To run the entire test suite: + * {{{ + * build/mvn test -DwildcardSuites=org.apache.kyuubi.config.AllKyuubiConfiguration + * }}} + * + * To re-generate golden files for entire suite, run: + * {{{ + * KYUUBI_UPDATE=1 build/mvn test -DwildcardSuites=org.apache.kyuubi.config.AllKyuubiConfiguration + * }}} + */ +// scalastyle:on line.size.limit class AllKyuubiConfiguration extends KyuubiFunSuite { - - private val markdown = Paths.get("..", "docs", "deployment", "settings.md") + private val update: Boolean = System.getenv("KYUUBI_UPDATE") == "1" + private val kyuubiHome: String = + getClass.getProtectionDomain.getCodeSource.getLocation.getPath.split("kyuubi-main")(0) + private val markdown = Paths.get(kyuubiHome, "docs", "deployment", "settings.md") .toAbsolutePath - private val writer = Files.newBufferedWriter( - markdown, StandardCharsets.UTF_8, - StandardOpenOption.TRUNCATE_EXISTING, - StandardOpenOption.CREATE) - - def writeWithNewLine(content: String): Unit = { - writer.write(content) - writer.newLine() - writer.flush() - } - - def writeWith2Line(content: String): Unit = { - writeWithNewLine(content) - writer.newLine() - } - - def rewriteToConf(path: Path): Unit = { + def rewriteToConf(path: Path, buffer: ArrayBuffer[String]): Unit = { val env = Files.newBufferedReader(path, StandardCharsets.UTF_8) try { - writeWithNewLine("```bash") + buffer += "```bash" var line = env.readLine() while(line != null) { - writeWithNewLine(line) + buffer += line line = env.readLine() } - writeWithNewLine("```") + buffer += "```" } finally { env.close() } } - override def afterAll(): Unit = { - writer.close() - super.afterAll() - } - test("Check all kyuubi configs") { - KyuubiConf HighAvailabilityConf MetricsConf - writeWith2Line("") + val newOutput = new ArrayBuffer[String]() + newOutput += "" + newOutput += "" - writeWith2Line( - """ - |
- | - |![](../imgs/kyuubi_logo.png) - | - |
- |""".stripMargin) - - writeWith2Line("# Introduction to the Kyuubi Configurations System") - - writeWith2Line( - "Kyuubi provides several ways to configure the system and corresponding engines.") - - writeWith2Line("## Environments") - writeWith2Line("You can configure the environment variables in" + + newOutput += "
" + newOutput += "" + newOutput += "![](../imgs/kyuubi_logo.png)" + newOutput += "" + newOutput += "
" + newOutput += "" + newOutput += "# Introduction to the Kyuubi Configurations System" + newOutput += "" + newOutput += "Kyuubi provides several ways to configure the system and corresponding engines." + newOutput += "" + newOutput += "" + newOutput += "## Environments" + newOutput += "" + newOutput += "" + newOutput += "You can configure the environment variables in" + " `$KYUUBI_HOME/conf/kyuubi-env.sh`, e.g, `JAVA_HOME`, then this java runtime will be used" + " both for Kyuubi server instance and the applications it launches. You can also change" + " the variable in the subprocess's env configuration file, e.g." + - "`$SPARK_HOME/conf/spark-env.sh` to use more specific ENV for SQL engine applications.") + "`$SPARK_HOME/conf/spark-env.sh` to use more specific ENV for SQL engine applications." - rewriteToConf(Paths.get("..", "conf", "kyuubi-env.sh.template")) + rewriteToConf(Paths.get(kyuubiHome, "conf", "kyuubi-env.sh.template"), newOutput) - writeWith2Line("## Kyuubi Configurations") - writeWith2Line("You can configure the Kyuubi properties in" + - " `$KYUUBI_HOME/conf/kyuubi-defaults.conf`. For example:") + newOutput += "" + newOutput += "## Kyuubi Configurations" + newOutput += "" - rewriteToConf(Paths.get("..", "conf", "kyuubi-defaults.conf.template")) + newOutput += "You can configure the Kyuubi properties in" + + " `$KYUUBI_HOME/conf/kyuubi-defaults.conf`. For example:" + + rewriteToConf(Paths.get(kyuubiHome, "conf", "kyuubi-defaults.conf.template"), newOutput) KyuubiConf.kyuubiConfEntries.values().asScala .toSeq .groupBy(_.key.split("\\.")(1)) .toSeq.sortBy(_._1).foreach { case (category, entries) => - writeWith2Line(s"### ${category.capitalize}") + newOutput += "" + newOutput += s"### ${category.capitalize}" + newOutput += "" - writeWithNewLine("Key | Default | Meaning | Since") - writeWithNewLine("--- | --- | --- | ---") + newOutput += "Key | Default | Meaning | Since" + newOutput += "--- | --- | --- | ---" - entries.sortBy(_.key).foreach { c => - val key = { - val sb = new StringBuilder() - var curLen = 0 - c.key.split("\\.").foreach { str => - if (curLen + str.length > 21) { - sb.append("
\\." + str) - curLen = str.length + 1 - } else { - sb.append("\\." + str) - curLen += (str.length + 1) - } + entries.sortBy(_.key).foreach { c => + val key = { + val sb = new StringBuilder() + var curLen = 0 + c.key.split("\\.").foreach { str => + if (curLen + str.length > 21) { + sb.append("
\\." + str) + curLen = str.length + 1 + } else { + sb.append("\\." + str) + curLen += (str.length + 1) } - sb.toString().stripPrefix("\\.") } - val dft = c.defaultValStr.replace("<", "<").replace(">", ">") - val seq = Seq( - key, - s"
$dft
", - s"
${c.doc}
", - s"
${c.version}
") - writeWithNewLine(seq.mkString("|")) + sb.toString().stripPrefix("\\.") } - writer.newLine() + val dft = c.defaultValStr.replace("<", "<").replace(">", ">") + val seq = Seq( + key, + s"
$dft
", + s"
${c.doc}
", + s"
${c.version}
") + newOutput += seq.mkString("|") + } + newOutput += "" } - writeWith2Line("## Spark Configurations") - writeWith2Line("### Via spark-defaults.conf") - writeWith2Line("Setting them in `$SPARK_HOME/conf/spark-defaults.conf`" + + newOutput += ("## Spark Configurations") + newOutput += "" + + newOutput += ("### Via spark-defaults.conf") + newOutput += "" + + newOutput += ("Setting them in `$SPARK_HOME/conf/spark-defaults.conf`" + " supplies with default values for SQL engine application. Available properties can be" + " found at Spark official online documentation for" + " [Spark Configurations](http://spark.apache.org/docs/latest/configuration.html)") - writeWith2Line("### Via kyuubi-defaults.conf") - writeWith2Line("Setting them in `$KYUUBI_HOME/conf/kyuubi-defaults.conf`" + + newOutput += "" + newOutput += ("### Via kyuubi-defaults.conf") + newOutput += "" + newOutput += ("Setting them in `$KYUUBI_HOME/conf/kyuubi-defaults.conf`" + " supplies with default values for SQL engine application too. These properties will" + " override all settings in `$SPARK_HOME/conf/spark-defaults.conf`") - writeWith2Line("### Via JDBC Connection URL") - writeWith2Line("Setting them in the JDBC Connection URL" + + newOutput += "" + newOutput += ("### Via JDBC Connection URL") + newOutput += "" + newOutput += ("Setting them in the JDBC Connection URL" + " supplies session-specific for each SQL engine. For example: " + "```" + "jdbc:hive2://localhost:10009/default;#" + "spark.sql.shuffle.partitions=2;spark.executor.memory=5g" + "```") - writeWithNewLine("- **Runtime SQL Configuration**") - writeWithNewLine(" - For [Runtime SQL Configurations](" + + newOutput += "" + newOutput += ("- **Runtime SQL Configuration**") + newOutput += "" + newOutput += (" - For [Runtime SQL Configurations](" + "http://spark.apache.org/docs/latest/configuration.html#runtime-sql-configuration), they" + " will take affect every time") - writeWithNewLine("- **Static SQL and Spark Core Configuration**") - writeWithNewLine(" - For [Static SQL Configurations](" + + newOutput += "" + newOutput += ("- **Static SQL and Spark Core Configuration**") + newOutput += "" + newOutput += (" - For [Static SQL Configurations](" + "http://spark.apache.org/docs/latest/configuration.html#static-sql-configuration) and" + " other spark core configs, e.g. `spark.executor.memory`, they will take affect if there" + " is no existing SQL engine application. Otherwise, they will just be ignored") - writeWith2Line("### Via SET Syntax") - writeWithNewLine("Please refer to the Spark official online documentation for" + + newOutput += "" + newOutput += ("### Via SET Syntax") + newOutput += "" + newOutput += ("Please refer to the Spark official online documentation for" + " [SET Command](http://spark.apache.org/docs/latest/sql-ref-syntax-aux-conf-mgmt-set.html)") - - writeWith2Line("## Logging") - writeWith2Line("Kyuubi uses [log4j](https://logging.apache.org/log4j/2.x/) for logging." + + newOutput += "" + newOutput += ("## Logging") + newOutput += "" + newOutput += ("Kyuubi uses [log4j](https://logging.apache.org/log4j/2.x/) for logging." + " You can configure it using `$KYUUBI_HOME/conf/log4j.properties`.") - rewriteToConf(Paths.get("..", "conf", "log4j.properties.template")) + rewriteToConf(Paths.get(kyuubiHome, "conf", "log4j.properties.template"), newOutput) - writeWith2Line("## Other Configurations") - - writeWith2Line("### Hadoop Configurations") - writeWith2Line("Specifying `HADOOP_CONF_DIR` to the directory contains hadoop configuration" + + newOutput += "" + newOutput += ("## Other Configurations") + newOutput += "" + newOutput += ("### Hadoop Configurations") + newOutput += "" + newOutput += ("Specifying `HADOOP_CONF_DIR` to the directory contains hadoop configuration" + " files or treating them as Spark properties with a `spark.hadoop.` prefix." + " Please refer to the Spark official online documentation for" + " [Inheriting Hadoop Cluster Configuration](http://spark.apache.org/docs/latest/" + "configuration.html#inheriting-hadoop-cluster-configuration)." + " Also, please refer to the [Apache Hadoop](http://hadoop.apache.org)'s" + " online documentation for an overview on how to configure Hadoop.") - writeWith2Line("### Hive Configurations") - writeWith2Line("These configurations are used for SQL engine application to talk to" + + newOutput += "" + newOutput += ("### Hive Configurations") + newOutput += "" + newOutput += ("These configurations are used for SQL engine application to talk to" + " Hive MetaStore and could be configured in a `hive-site.xml`." + " Placed it in `$SPARK_HOME/conf` directory, or treating them as Spark properties with" + " a `spark.hadoop.` prefix.") - writeWith2Line("## User Defaults") - writeWith2Line("In Kyuubi, we can configure user default settings to meet separate needs." + + newOutput += "" + newOutput += ("## User Defaults") + newOutput += "" + newOutput += ("In Kyuubi, we can configure user default settings to meet separate needs." + " These user defaults override system defaults, but will be overridden by those from" + " [JDBC Connection URL](#via-jdbc-connection-url) or [Set Command](#via-set-syntax)" + " if could be. They will take effect when creating the SQL engine application ONLY.") - writeWith2Line("User default settings are in the form of `___{username}___.{config key}`." + + newOutput += ("User default settings are in the form of `___{username}___.{config key}`." + " There are three continuous underscores(`_`) at both sides of the `username` and" + " a dot(`.`) that separates the config key and the prefix. For example:") - writeWithNewLine("```bash") - writeWithNewLine("# For system defaults") - writeWithNewLine("spark.master=local") - writeWithNewLine("spark.sql.adaptive.enabled=true") - writeWithNewLine("# For a user named kent") - writeWithNewLine("___kent___.spark.master=yarn") - writeWithNewLine("___kent___.spark.sql.adaptive.enabled=false") - writeWithNewLine("# For a user named bob") - writeWithNewLine("___bob___.spark.master=spark://master:7077") - writeWithNewLine("___bob___.spark.executor.memory=8g") - writeWith2Line( "```") - writeWith2Line("In the above case, if there are related configurations from" + + newOutput += ("```bash") + newOutput += ("# For system defaults") + newOutput += ("spark.master=local") + newOutput += ("spark.sql.adaptive.enabled=true") + newOutput += ("# For a user named kent") + newOutput += ("___kent___.spark.master=yarn") + newOutput += ("___kent___.spark.sql.adaptive.enabled=false") + newOutput += ("# For a user named bob") + newOutput += ("___bob___.spark.master=spark://master:7077") + newOutput += ("___bob___.spark.executor.memory=8g") + newOutput += ( "```") + newOutput += "" + newOutput += "In the above case, if there are related configurations from" + " [JDBC Connection URL](#via-jdbc-connection-url), `kent` will run his SQL engine" + " application on YARN and prefer the Spark AQE to be off, while `bob` will activate" + " his SQL engine application on a Spark standalone cluster with 8g heap memory for each" + " executor and obey the Spark AQE behavior of Kyuubi system default. On the other hand," + - " for those users who do not have custom configurations will use system defaults.") + " for those users who do not have custom configurations will use system defaults." + + if (update) { + val writer = Files.newBufferedWriter( + markdown, StandardCharsets.UTF_8, + StandardOpenOption.TRUNCATE_EXISTING, + StandardOpenOption.CREATE) + try { + newOutput.foreach { line => + writer.write(line) + writer.newLine() + } + } finally { + writer.close() + } + } else { + val expected = new ArrayBuffer[String]() + + val reader = Files.newBufferedReader(markdown, StandardCharsets.UTF_8) + var line = reader.readLine() + while (line != null) { + expected += line + line = reader.readLine() + } + reader.close() + val hint = s"$markdown out of date, please update doc with build/mvn test" + + s" -DwildcardSuites=${getClass.getCanonicalName}" + assert(newOutput.size === expected.size, hint) + + newOutput.zip(expected).foreach { case (out, in) => assert(out === in, hint) } + } } }