[KYUUBI #459] Enable Configuration Checker
 [](https://github.com/yaooqinn/kyuubi/pull/459)      [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT --> <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Whenever ,there are new configs, the corresponding doc should also be updated ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #459 from yaooqinn/settinglock. 6e1e4d3 [Kent Yao] Enable Configuration Checker 30e7100 [Kent Yao] Enable Configuration Checker Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>
This commit is contained in:
Kent Yao
parent
392a10ce71
commit
768d583767
@ -1,5 +1,4 @@
|
||||
<!-- DO NOT MODIFY THIS FILE DIRECTORY, IT IS AUTO GENERATED BY [org.apache.kyuubi.config.AllKyuubiConfiguration] -->
|
||||
|
||||
<!-- DO NOT MODIFY THIS FILE DIRECTLY, IT IS AUTO GENERATED BY [org.apache.kyuubi.config.AllKyuubiConfiguration] -->
|
||||
|
||||
<div align=center>
|
||||
|
||||
@ -7,15 +6,15 @@
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
# Introduction to the Kyuubi Configurations System
|
||||
|
||||
Kyuubi provides several ways to configure the system and corresponding engines.
|
||||
|
||||
|
||||
## Environments
|
||||
|
||||
You can configure the environment variables in `$KYUUBI_HOME/conf/kyuubi-env.sh`, e.g, `JAVA_HOME`, then this java runtime will be used both for Kyuubi server instance and the applications it launches. You can also change the variable in the subprocess's env configuration file, e.g.`$SPARK_HOME/conf/spark-env.sh` to use more specific ENV for SQL engine applications.
|
||||
|
||||
You can configure the environment variables in `$KYUUBI_HOME/conf/kyuubi-env.sh`, e.g, `JAVA_HOME`, then this java runtime will be used both for Kyuubi server instance and the applications it launches. You can also change the variable in the subprocess's env configuration file, e.g.`$SPARK_HOME/conf/spark-env.sh` to use more specific ENV for SQL engine applications.
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
@ -66,10 +65,10 @@ You can configure the environment variables in `$KYUUBI_HOME/conf/kyuubi-env.sh`
|
||||
# export HADOOP_CONF_DIR=/usr/ndp/current/mapreduce_client/conf
|
||||
# export KYUUBI_JAVA_OPTS="-Xmx10g -XX:+UnlockDiagnosticVMOptions -XX:ParGCCardsPerStrideChunk=4096 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:+UseCondCardMark -XX:MaxDirectMemorySize=1024m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./logs -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -Xloggc:./logs/kyuubi-server-gc-%t.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=5M -XX:NewRatio=3 -XX:MetaspaceSize=512m"
|
||||
```
|
||||
|
||||
## Kyuubi Configurations
|
||||
|
||||
You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.conf`. For example:
|
||||
|
||||
```bash
|
||||
#
|
||||
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
@ -98,6 +97,7 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
|
||||
|
||||
# Details in https://kyuubi.readthedocs.io/en/latest/deployment/settings.html
|
||||
```
|
||||
|
||||
### Authentication
|
||||
|
||||
Key | Default | Meaning | Since
|
||||
@ -108,6 +108,7 @@ kyuubi\.authentication<br>\.ldap\.domain|<div style='width: 80pt;word-wrap: brea
|
||||
kyuubi\.authentication<br>\.ldap\.url|<div style='width: 80pt;word-wrap: break-word;white-space: normal'><undefined></div>|<div style='width: 200pt;word-wrap: break-word;white-space: normal'>SPACE character separated LDAP connection URL(s).</div>|<div style='width: 20pt'>1.0.0</div>
|
||||
kyuubi\.authentication<br>\.sasl\.qop|<div style='width: 80pt;word-wrap: break-word;white-space: normal'>auth</div>|<div style='width: 200pt;word-wrap: break-word;white-space: normal'>Sasl QOP enable higher levels of protection for Kyuubi communication with clients.<ul> <li>auth - authentication only (default)</li> <li>auth-int - authentication plus integrity protection</li> <li>auth-conf - authentication plus integrity and confidentiality protection. This is applicable only if Kyuubi is configured to use Kerberos authentication.</li> </ul></div>|<div style='width: 20pt'>1.0.0</div>
|
||||
|
||||
|
||||
### Backend
|
||||
|
||||
Key | Default | Meaning | Since
|
||||
@ -121,6 +122,7 @@ kyuubi\.backend\.server<br>\.exec\.pool\.shutdown<br>\.timeout|<div style='width
|
||||
kyuubi\.backend\.server<br>\.exec\.pool\.size|<div style='width: 80pt;word-wrap: break-word;white-space: normal'>100</div>|<div style='width: 200pt;word-wrap: break-word;white-space: normal'>Number of threads in the operation execution thread pool of Kyuubi server</div>|<div style='width: 20pt'>1.0.0</div>
|
||||
kyuubi\.backend\.server<br>\.exec\.pool\.wait\.queue<br>\.size|<div style='width: 80pt;word-wrap: break-word;white-space: normal'>100</div>|<div style='width: 200pt;word-wrap: break-word;white-space: normal'>Size of the wait queue for the operation execution thread pool of Kyuubi server</div>|<div style='width: 20pt'>1.0.0</div>
|
||||
|
||||
|
||||
### Delegation
|
||||
|
||||
Key | Default | Meaning | Since
|
||||
@ -130,6 +132,7 @@ kyuubi\.delegation<br>\.token\.gc\.interval|<div style='width: 80pt;word-wrap: b
|
||||
kyuubi\.delegation<br>\.token\.max\.lifetime|<div style='width: 80pt;word-wrap: break-word;white-space: normal'>PT168H</div>|<div style='width: 200pt;word-wrap: break-word;white-space: normal'>unused yet</div>|<div style='width: 20pt'>1.0.0</div>
|
||||
kyuubi\.delegation<br>\.token\.renew\.interval|<div style='width: 80pt;word-wrap: break-word;white-space: normal'>PT168H</div>|<div style='width: 200pt;word-wrap: break-word;white-space: normal'>unused yet</div>|<div style='width: 20pt'>1.0.0</div>
|
||||
|
||||
|
||||
### Frontend
|
||||
|
||||
Key | Default | Meaning | Since
|
||||
@ -143,6 +146,7 @@ kyuubi\.frontend\.max<br>\.worker\.threads|<div style='width: 80pt;word-wrap: br
|
||||
kyuubi\.frontend\.min<br>\.worker\.threads|<div style='width: 80pt;word-wrap: break-word;white-space: normal'>9</div>|<div style='width: 200pt;word-wrap: break-word;white-space: normal'>Minimum number of threads in the of frontend worker thread pool for the frontend service</div>|<div style='width: 20pt'>1.0.0</div>
|
||||
kyuubi\.frontend<br>\.worker\.keepalive\.time|<div style='width: 80pt;word-wrap: break-word;white-space: normal'>PT1M</div>|<div style='width: 200pt;word-wrap: break-word;white-space: normal'>Keep-alive time (in milliseconds) for an idle worker thread</div>|<div style='width: 20pt'>1.0.0</div>
|
||||
|
||||
|
||||
### Ha
|
||||
|
||||
Key | Default | Meaning | Since
|
||||
@ -157,6 +161,7 @@ kyuubi\.ha\.zookeeper<br>\.namespace|<div style='width: 80pt;word-wrap: break-wo
|
||||
kyuubi\.ha\.zookeeper<br>\.quorum|<div style='width: 80pt;word-wrap: break-word;white-space: normal'></div>|<div style='width: 200pt;word-wrap: break-word;white-space: normal'>The connection string for the zookeeper ensemble</div>|<div style='width: 20pt'>1.0.0</div>
|
||||
kyuubi\.ha\.zookeeper<br>\.session\.timeout|<div style='width: 80pt;word-wrap: break-word;white-space: normal'>60000</div>|<div style='width: 200pt;word-wrap: break-word;white-space: normal'>The timeout(ms) of a connected session to be idled</div>|<div style='width: 20pt'>1.0.0</div>
|
||||
|
||||
|
||||
### Kinit
|
||||
|
||||
Key | Default | Meaning | Since
|
||||
@ -166,6 +171,7 @@ kyuubi\.kinit\.keytab|<div style='width: 80pt;word-wrap: break-word;white-space:
|
||||
kyuubi\.kinit\.max<br>\.attempts|<div style='width: 80pt;word-wrap: break-word;white-space: normal'>10</div>|<div style='width: 200pt;word-wrap: break-word;white-space: normal'>How many times will `kinit` process retry</div>|<div style='width: 20pt'>1.0.0</div>
|
||||
kyuubi\.kinit<br>\.principal|<div style='width: 80pt;word-wrap: break-word;white-space: normal'><undefined></div>|<div style='width: 200pt;word-wrap: break-word;white-space: normal'>Name of the Kerberos principal.</div>|<div style='width: 20pt'>1.0.0</div>
|
||||
|
||||
|
||||
### Metrics
|
||||
|
||||
Key | Default | Meaning | Since
|
||||
@ -175,6 +181,7 @@ kyuubi\.metrics\.json<br>\.report\.location|<div style='width: 80pt;word-wrap: b
|
||||
kyuubi\.metrics\.report<br>\.interval|<div style='width: 80pt;word-wrap: break-word;white-space: normal'>PT5S</div>|<div style='width: 200pt;word-wrap: break-word;white-space: normal'>How often should report metrics to json/console. no effect on JMX</div>|<div style='width: 20pt'>1.2.0</div>
|
||||
kyuubi\.metrics<br>\.reporters|<div style='width: 80pt;word-wrap: break-word;white-space: normal'>JSON</div>|<div style='width: 200pt;word-wrap: break-word;white-space: normal'>A comma separated list for all metrics reporters<ul> <li>JSON - default reporter which outputs measurements to json file periodically</li> <li>CONSOLE - ConsoleReporter which outputs measurements to CONSOLE.</li> <li>SLF4J - Slf4jReporter which outputs measurements to system log.</li> <li>JMX - JmxReporter which listens for new metrics and exposes them as namespaced MBeans.</li> </ul></div>|<div style='width: 20pt'>1.2.0</div>
|
||||
|
||||
|
||||
### Operation
|
||||
|
||||
Key | Default | Meaning | Since
|
||||
@ -184,6 +191,7 @@ kyuubi\.operation<br>\.interrupt\.on\.cancel|<div style='width: 80pt;word-wrap:
|
||||
kyuubi\.operation<br>\.query\.timeout|<div style='width: 80pt;word-wrap: break-word;white-space: normal'>PT0S</div>|<div style='width: 200pt;word-wrap: break-word;white-space: normal'>Set a query duration timeout in seconds in Kyuubi. If the timeout is set to a positive value, a running query will be cancelled automatically if timeout. Otherwise the query continues to run till completion. If timeout values are set for each statement via `java.sql.Statement.setQueryTimeout` and they are smaller than this configuration value, they take precedence. If you set this timeout and prefer to cancel the queries right away without waiting task to finish, consider enabling kyuubi.operation.interrupt.on.cancel together.</div>|<div style='width: 20pt'>1.2.0</div>
|
||||
kyuubi\.operation<br>\.status\.polling<br>\.timeout|<div style='width: 80pt;word-wrap: break-word;white-space: normal'>PT5S</div>|<div style='width: 200pt;word-wrap: break-word;white-space: normal'>Timeout(ms) for long polling asynchronous running sql query's status</div>|<div style='width: 20pt'>1.0.0</div>
|
||||
|
||||
|
||||
### Session
|
||||
|
||||
Key | Default | Meaning | Since
|
||||
@ -199,6 +207,7 @@ kyuubi\.session\.engine<br>\.spark\.main\.resource|<div style='width: 80pt;word-
|
||||
kyuubi\.session\.engine<br>\.startup\.error\.max<br>\.size|<div style='width: 80pt;word-wrap: break-word;white-space: normal'>8192</div>|<div style='width: 200pt;word-wrap: break-word;white-space: normal'>During engine bootstrapping, if error occurs, using this config to limit the length error message(characters).</div>|<div style='width: 20pt'>1.1.0</div>
|
||||
kyuubi\.session<br>\.timeout|<div style='width: 80pt;word-wrap: break-word;white-space: normal'>PT6H</div>|<div style='width: 200pt;word-wrap: break-word;white-space: normal'>session timeout, it will be closed when it's not accessed for this duration</div>|<div style='width: 20pt'>1.0.0</div>
|
||||
|
||||
|
||||
### Zookeeper
|
||||
|
||||
Key | Default | Meaning | Since
|
||||
@ -221,16 +230,20 @@ Setting them in `$KYUUBI_HOME/conf/kyuubi-defaults.conf` supplies with default v
|
||||
Setting them in the JDBC Connection URL supplies session-specific for each SQL engine. For example: ```jdbc:hive2://localhost:10009/default;#spark.sql.shuffle.partitions=2;spark.executor.memory=5g```
|
||||
|
||||
- **Runtime SQL Configuration**
|
||||
|
||||
- For [Runtime SQL Configurations](http://spark.apache.org/docs/latest/configuration.html#runtime-sql-configuration), they will take affect every time
|
||||
|
||||
- **Static SQL and Spark Core Configuration**
|
||||
|
||||
- For [Static SQL Configurations](http://spark.apache.org/docs/latest/configuration.html#static-sql-configuration) and other spark core configs, e.g. `spark.executor.memory`, they will take affect if there is no existing SQL engine application. Otherwise, they will just be ignored
|
||||
|
||||
### Via SET Syntax
|
||||
|
||||
Please refer to the Spark official online documentation for [SET Command](http://spark.apache.org/docs/latest/sql-ref-syntax-aux-conf-mgmt-set.html)
|
||||
|
||||
## Logging
|
||||
|
||||
Kyuubi uses [log4j](https://logging.apache.org/log4j/2.x/) for logging. You can configure it using `$KYUUBI_HOME/conf/log4j.properties`.
|
||||
|
||||
```bash
|
||||
#
|
||||
# Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
@ -256,6 +269,7 @@ log4j.appender.console.target=System.err
|
||||
log4j.appender.console.layout=org.apache.log4j.PatternLayout
|
||||
log4j.appender.console.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss.SSS} %p %c{2}: %m%n
|
||||
```
|
||||
|
||||
## Other Configurations
|
||||
|
||||
### Hadoop Configurations
|
||||
@ -269,9 +283,7 @@ These configurations are used for SQL engine application to talk to Hive MetaSto
|
||||
## User Defaults
|
||||
|
||||
In Kyuubi, we can configure user default settings to meet separate needs. These user defaults override system defaults, but will be overridden by those from [JDBC Connection URL](#via-jdbc-connection-url) or [Set Command](#via-set-syntax) if could be. They will take effect when creating the SQL engine application ONLY.
|
||||
|
||||
User default settings are in the form of `___{username}___.{config key}`. There are three continuous underscores(`_`) at both sides of the `username` and a dot(`.`) that separates the config key and the prefix. For example:
|
||||
|
||||
```bash
|
||||
# For system defaults
|
||||
spark.master=local
|
||||
@ -285,4 +297,3 @@ ___bob___.spark.executor.memory=8g
|
||||
```
|
||||
|
||||
In the above case, if there are related configurations from [JDBC Connection URL](#via-jdbc-connection-url), `kent` will run his SQL engine application on YARN and prefer the Spark AQE to be off, while `bob` will activate his SQL engine application on a Spark standalone cluster with 8g heap memory for each executor and obey the Spark AQE behavior of Kyuubi system default. On the other hand, for those users who do not have custom configurations will use system defaults.
|
||||
|
||||
|
||||
@ -21,207 +21,262 @@ import java.nio.charset.StandardCharsets
|
||||
import java.nio.file.{Files, Path, Paths, StandardOpenOption}
|
||||
|
||||
import scala.collection.JavaConverters._
|
||||
import scala.collection.mutable.ArrayBuffer
|
||||
|
||||
import org.apache.kyuubi.KyuubiFunSuite
|
||||
import org.apache.kyuubi.ha.HighAvailabilityConf
|
||||
import org.apache.kyuubi.metrics.MetricsConf
|
||||
|
||||
// scalastyle:off line.size.limit
|
||||
/**
|
||||
* End-to-end test cases for configuration doc file
|
||||
* The golden result file is "docs/deployment/settings.md".
|
||||
*
|
||||
* To run the entire test suite:
|
||||
* {{{
|
||||
* build/mvn test -DwildcardSuites=org.apache.kyuubi.config.AllKyuubiConfiguration
|
||||
* }}}
|
||||
*
|
||||
* To re-generate golden files for entire suite, run:
|
||||
* {{{
|
||||
* KYUUBI_UPDATE=1 build/mvn test -DwildcardSuites=org.apache.kyuubi.config.AllKyuubiConfiguration
|
||||
* }}}
|
||||
*/
|
||||
// scalastyle:on line.size.limit
|
||||
class AllKyuubiConfiguration extends KyuubiFunSuite {
|
||||
|
||||
private val markdown = Paths.get("..", "docs", "deployment", "settings.md")
|
||||
private val update: Boolean = System.getenv("KYUUBI_UPDATE") == "1"
|
||||
private val kyuubiHome: String =
|
||||
getClass.getProtectionDomain.getCodeSource.getLocation.getPath.split("kyuubi-main")(0)
|
||||
private val markdown = Paths.get(kyuubiHome, "docs", "deployment", "settings.md")
|
||||
.toAbsolutePath
|
||||
|
||||
private val writer = Files.newBufferedWriter(
|
||||
markdown, StandardCharsets.UTF_8,
|
||||
StandardOpenOption.TRUNCATE_EXISTING,
|
||||
StandardOpenOption.CREATE)
|
||||
|
||||
def writeWithNewLine(content: String): Unit = {
|
||||
writer.write(content)
|
||||
writer.newLine()
|
||||
writer.flush()
|
||||
}
|
||||
|
||||
def writeWith2Line(content: String): Unit = {
|
||||
writeWithNewLine(content)
|
||||
writer.newLine()
|
||||
}
|
||||
|
||||
def rewriteToConf(path: Path): Unit = {
|
||||
def rewriteToConf(path: Path, buffer: ArrayBuffer[String]): Unit = {
|
||||
val env =
|
||||
Files.newBufferedReader(path, StandardCharsets.UTF_8)
|
||||
|
||||
try {
|
||||
writeWithNewLine("```bash")
|
||||
buffer += "```bash"
|
||||
var line = env.readLine()
|
||||
while(line != null) {
|
||||
writeWithNewLine(line)
|
||||
buffer += line
|
||||
line = env.readLine()
|
||||
}
|
||||
writeWithNewLine("```")
|
||||
buffer += "```"
|
||||
} finally {
|
||||
env.close()
|
||||
}
|
||||
}
|
||||
|
||||
override def afterAll(): Unit = {
|
||||
writer.close()
|
||||
super.afterAll()
|
||||
}
|
||||
|
||||
test("Check all kyuubi configs") {
|
||||
|
||||
KyuubiConf
|
||||
HighAvailabilityConf
|
||||
MetricsConf
|
||||
|
||||
writeWith2Line("<!-- DO NOT MODIFY THIS FILE DIRECTORY, IT IS AUTO GENERATED BY" +
|
||||
" [org.apache.kyuubi.config.AllKyuubiConfiguration] -->")
|
||||
val newOutput = new ArrayBuffer[String]()
|
||||
newOutput += "<!-- DO NOT MODIFY THIS FILE DIRECTLY, IT IS AUTO GENERATED BY" +
|
||||
" [org.apache.kyuubi.config.AllKyuubiConfiguration] -->"
|
||||
newOutput += ""
|
||||
|
||||
writeWith2Line(
|
||||
"""
|
||||
|<div align=center>
|
||||
|
|
||||
|
|
||||
|
|
||||
|</div>
|
||||
|""".stripMargin)
|
||||
|
||||
writeWith2Line("# Introduction to the Kyuubi Configurations System")
|
||||
|
||||
writeWith2Line(
|
||||
"Kyuubi provides several ways to configure the system and corresponding engines.")
|
||||
|
||||
writeWith2Line("## Environments")
|
||||
writeWith2Line("You can configure the environment variables in" +
|
||||
newOutput += "<div align=center>"
|
||||
newOutput += ""
|
||||
newOutput += ""
|
||||
newOutput += ""
|
||||
newOutput += "</div>"
|
||||
newOutput += ""
|
||||
newOutput += "# Introduction to the Kyuubi Configurations System"
|
||||
newOutput += ""
|
||||
newOutput += "Kyuubi provides several ways to configure the system and corresponding engines."
|
||||
newOutput += ""
|
||||
newOutput += ""
|
||||
newOutput += "## Environments"
|
||||
newOutput += ""
|
||||
newOutput += ""
|
||||
newOutput += "You can configure the environment variables in" +
|
||||
" `$KYUUBI_HOME/conf/kyuubi-env.sh`, e.g, `JAVA_HOME`, then this java runtime will be used" +
|
||||
" both for Kyuubi server instance and the applications it launches. You can also change" +
|
||||
" the variable in the subprocess's env configuration file, e.g." +
|
||||
"`$SPARK_HOME/conf/spark-env.sh` to use more specific ENV for SQL engine applications.")
|
||||
"`$SPARK_HOME/conf/spark-env.sh` to use more specific ENV for SQL engine applications."
|
||||
|
||||
rewriteToConf(Paths.get("..", "conf", "kyuubi-env.sh.template"))
|
||||
rewriteToConf(Paths.get(kyuubiHome, "conf", "kyuubi-env.sh.template"), newOutput)
|
||||
|
||||
writeWith2Line("## Kyuubi Configurations")
|
||||
writeWith2Line("You can configure the Kyuubi properties in" +
|
||||
" `$KYUUBI_HOME/conf/kyuubi-defaults.conf`. For example:")
|
||||
newOutput += ""
|
||||
newOutput += "## Kyuubi Configurations"
|
||||
newOutput += ""
|
||||
|
||||
rewriteToConf(Paths.get("..", "conf", "kyuubi-defaults.conf.template"))
|
||||
newOutput += "You can configure the Kyuubi properties in" +
|
||||
" `$KYUUBI_HOME/conf/kyuubi-defaults.conf`. For example:"
|
||||
|
||||
rewriteToConf(Paths.get(kyuubiHome, "conf", "kyuubi-defaults.conf.template"), newOutput)
|
||||
|
||||
KyuubiConf.kyuubiConfEntries.values().asScala
|
||||
.toSeq
|
||||
.groupBy(_.key.split("\\.")(1))
|
||||
.toSeq.sortBy(_._1).foreach { case (category, entries) =>
|
||||
|
||||
writeWith2Line(s"### ${category.capitalize}")
|
||||
newOutput += ""
|
||||
newOutput += s"### ${category.capitalize}"
|
||||
newOutput += ""
|
||||
|
||||
writeWithNewLine("Key | Default | Meaning | Since")
|
||||
writeWithNewLine("--- | --- | --- | ---")
|
||||
newOutput += "Key | Default | Meaning | Since"
|
||||
newOutput += "--- | --- | --- | ---"
|
||||
|
||||
entries.sortBy(_.key).foreach { c =>
|
||||
val key = {
|
||||
val sb = new StringBuilder()
|
||||
var curLen = 0
|
||||
c.key.split("\\.").foreach { str =>
|
||||
if (curLen + str.length > 21) {
|
||||
sb.append("<br>\\." + str)
|
||||
curLen = str.length + 1
|
||||
} else {
|
||||
sb.append("\\." + str)
|
||||
curLen += (str.length + 1)
|
||||
}
|
||||
entries.sortBy(_.key).foreach { c =>
|
||||
val key = {
|
||||
val sb = new StringBuilder()
|
||||
var curLen = 0
|
||||
c.key.split("\\.").foreach { str =>
|
||||
if (curLen + str.length > 21) {
|
||||
sb.append("<br>\\." + str)
|
||||
curLen = str.length + 1
|
||||
} else {
|
||||
sb.append("\\." + str)
|
||||
curLen += (str.length + 1)
|
||||
}
|
||||
sb.toString().stripPrefix("\\.")
|
||||
}
|
||||
val dft = c.defaultValStr.replace("<", "<").replace(">", ">")
|
||||
val seq = Seq(
|
||||
key,
|
||||
s"<div style='width: 80pt;word-wrap: break-word;white-space: normal'>$dft</div>",
|
||||
s"<div style='width: 200pt;word-wrap: break-word;white-space: normal'>${c.doc}</div>",
|
||||
s"<div style='width: 20pt'>${c.version}</div>")
|
||||
writeWithNewLine(seq.mkString("|"))
|
||||
sb.toString().stripPrefix("\\.")
|
||||
}
|
||||
writer.newLine()
|
||||
val dft = c.defaultValStr.replace("<", "<").replace(">", ">")
|
||||
val seq = Seq(
|
||||
key,
|
||||
s"<div style='width: 80pt;word-wrap: break-word;white-space: normal'>$dft</div>",
|
||||
s"<div style='width: 200pt;word-wrap: break-word;white-space: normal'>${c.doc}</div>",
|
||||
s"<div style='width: 20pt'>${c.version}</div>")
|
||||
newOutput += seq.mkString("|")
|
||||
}
|
||||
newOutput += ""
|
||||
}
|
||||
|
||||
writeWith2Line("## Spark Configurations")
|
||||
writeWith2Line("### Via spark-defaults.conf")
|
||||
writeWith2Line("Setting them in `$SPARK_HOME/conf/spark-defaults.conf`" +
|
||||
newOutput += ("## Spark Configurations")
|
||||
newOutput += ""
|
||||
|
||||
newOutput += ("### Via spark-defaults.conf")
|
||||
newOutput += ""
|
||||
|
||||
newOutput += ("Setting them in `$SPARK_HOME/conf/spark-defaults.conf`" +
|
||||
" supplies with default values for SQL engine application. Available properties can be" +
|
||||
" found at Spark official online documentation for" +
|
||||
" [Spark Configurations](http://spark.apache.org/docs/latest/configuration.html)")
|
||||
|
||||
writeWith2Line("### Via kyuubi-defaults.conf")
|
||||
writeWith2Line("Setting them in `$KYUUBI_HOME/conf/kyuubi-defaults.conf`" +
|
||||
newOutput += ""
|
||||
newOutput += ("### Via kyuubi-defaults.conf")
|
||||
newOutput += ""
|
||||
newOutput += ("Setting them in `$KYUUBI_HOME/conf/kyuubi-defaults.conf`" +
|
||||
" supplies with default values for SQL engine application too. These properties will" +
|
||||
" override all settings in `$SPARK_HOME/conf/spark-defaults.conf`")
|
||||
|
||||
writeWith2Line("### Via JDBC Connection URL")
|
||||
writeWith2Line("Setting them in the JDBC Connection URL" +
|
||||
newOutput += ""
|
||||
newOutput += ("### Via JDBC Connection URL")
|
||||
newOutput += ""
|
||||
newOutput += ("Setting them in the JDBC Connection URL" +
|
||||
" supplies session-specific for each SQL engine. For example: " +
|
||||
"```" +
|
||||
"jdbc:hive2://localhost:10009/default;#" +
|
||||
"spark.sql.shuffle.partitions=2;spark.executor.memory=5g" +
|
||||
"```")
|
||||
writeWithNewLine("- **Runtime SQL Configuration**")
|
||||
writeWithNewLine(" - For [Runtime SQL Configurations](" +
|
||||
newOutput += ""
|
||||
newOutput += ("- **Runtime SQL Configuration**")
|
||||
newOutput += ""
|
||||
newOutput += (" - For [Runtime SQL Configurations](" +
|
||||
"http://spark.apache.org/docs/latest/configuration.html#runtime-sql-configuration), they" +
|
||||
" will take affect every time")
|
||||
writeWithNewLine("- **Static SQL and Spark Core Configuration**")
|
||||
writeWithNewLine(" - For [Static SQL Configurations](" +
|
||||
newOutput += ""
|
||||
newOutput += ("- **Static SQL and Spark Core Configuration**")
|
||||
newOutput += ""
|
||||
newOutput += (" - For [Static SQL Configurations](" +
|
||||
"http://spark.apache.org/docs/latest/configuration.html#static-sql-configuration) and" +
|
||||
" other spark core configs, e.g. `spark.executor.memory`, they will take affect if there" +
|
||||
" is no existing SQL engine application. Otherwise, they will just be ignored")
|
||||
writeWith2Line("### Via SET Syntax")
|
||||
writeWithNewLine("Please refer to the Spark official online documentation for" +
|
||||
newOutput += ""
|
||||
newOutput += ("### Via SET Syntax")
|
||||
newOutput += ""
|
||||
newOutput += ("Please refer to the Spark official online documentation for" +
|
||||
" [SET Command](http://spark.apache.org/docs/latest/sql-ref-syntax-aux-conf-mgmt-set.html)")
|
||||
|
||||
|
||||
writeWith2Line("## Logging")
|
||||
writeWith2Line("Kyuubi uses [log4j](https://logging.apache.org/log4j/2.x/) for logging." +
|
||||
newOutput += ""
|
||||
newOutput += ("## Logging")
|
||||
newOutput += ""
|
||||
newOutput += ("Kyuubi uses [log4j](https://logging.apache.org/log4j/2.x/) for logging." +
|
||||
" You can configure it using `$KYUUBI_HOME/conf/log4j.properties`.")
|
||||
|
||||
rewriteToConf(Paths.get("..", "conf", "log4j.properties.template"))
|
||||
rewriteToConf(Paths.get(kyuubiHome, "conf", "log4j.properties.template"), newOutput)
|
||||
|
||||
writeWith2Line("## Other Configurations")
|
||||
|
||||
writeWith2Line("### Hadoop Configurations")
|
||||
writeWith2Line("Specifying `HADOOP_CONF_DIR` to the directory contains hadoop configuration" +
|
||||
newOutput += ""
|
||||
newOutput += ("## Other Configurations")
|
||||
newOutput += ""
|
||||
newOutput += ("### Hadoop Configurations")
|
||||
newOutput += ""
|
||||
newOutput += ("Specifying `HADOOP_CONF_DIR` to the directory contains hadoop configuration" +
|
||||
" files or treating them as Spark properties with a `spark.hadoop.` prefix." +
|
||||
" Please refer to the Spark official online documentation for" +
|
||||
" [Inheriting Hadoop Cluster Configuration](http://spark.apache.org/docs/latest/" +
|
||||
"configuration.html#inheriting-hadoop-cluster-configuration)." +
|
||||
" Also, please refer to the [Apache Hadoop](http://hadoop.apache.org)'s" +
|
||||
" online documentation for an overview on how to configure Hadoop.")
|
||||
writeWith2Line("### Hive Configurations")
|
||||
writeWith2Line("These configurations are used for SQL engine application to talk to" +
|
||||
newOutput += ""
|
||||
newOutput += ("### Hive Configurations")
|
||||
newOutput += ""
|
||||
newOutput += ("These configurations are used for SQL engine application to talk to" +
|
||||
" Hive MetaStore and could be configured in a `hive-site.xml`." +
|
||||
" Placed it in `$SPARK_HOME/conf` directory, or treating them as Spark properties with" +
|
||||
" a `spark.hadoop.` prefix.")
|
||||
|
||||
writeWith2Line("## User Defaults")
|
||||
writeWith2Line("In Kyuubi, we can configure user default settings to meet separate needs." +
|
||||
newOutput += ""
|
||||
newOutput += ("## User Defaults")
|
||||
newOutput += ""
|
||||
newOutput += ("In Kyuubi, we can configure user default settings to meet separate needs." +
|
||||
" These user defaults override system defaults, but will be overridden by those from" +
|
||||
" [JDBC Connection URL](#via-jdbc-connection-url) or [Set Command](#via-set-syntax)" +
|
||||
" if could be. They will take effect when creating the SQL engine application ONLY.")
|
||||
writeWith2Line("User default settings are in the form of `___{username}___.{config key}`." +
|
||||
newOutput += ("User default settings are in the form of `___{username}___.{config key}`." +
|
||||
" There are three continuous underscores(`_`) at both sides of the `username` and" +
|
||||
" a dot(`.`) that separates the config key and the prefix. For example:")
|
||||
writeWithNewLine("```bash")
|
||||
writeWithNewLine("# For system defaults")
|
||||
writeWithNewLine("spark.master=local")
|
||||
writeWithNewLine("spark.sql.adaptive.enabled=true")
|
||||
writeWithNewLine("# For a user named kent")
|
||||
writeWithNewLine("___kent___.spark.master=yarn")
|
||||
writeWithNewLine("___kent___.spark.sql.adaptive.enabled=false")
|
||||
writeWithNewLine("# For a user named bob")
|
||||
writeWithNewLine("___bob___.spark.master=spark://master:7077")
|
||||
writeWithNewLine("___bob___.spark.executor.memory=8g")
|
||||
writeWith2Line( "```")
|
||||
writeWith2Line("In the above case, if there are related configurations from" +
|
||||
newOutput += ("```bash")
|
||||
newOutput += ("# For system defaults")
|
||||
newOutput += ("spark.master=local")
|
||||
newOutput += ("spark.sql.adaptive.enabled=true")
|
||||
newOutput += ("# For a user named kent")
|
||||
newOutput += ("___kent___.spark.master=yarn")
|
||||
newOutput += ("___kent___.spark.sql.adaptive.enabled=false")
|
||||
newOutput += ("# For a user named bob")
|
||||
newOutput += ("___bob___.spark.master=spark://master:7077")
|
||||
newOutput += ("___bob___.spark.executor.memory=8g")
|
||||
newOutput += ( "```")
|
||||
newOutput += ""
|
||||
newOutput += "In the above case, if there are related configurations from" +
|
||||
" [JDBC Connection URL](#via-jdbc-connection-url), `kent` will run his SQL engine" +
|
||||
" application on YARN and prefer the Spark AQE to be off, while `bob` will activate" +
|
||||
" his SQL engine application on a Spark standalone cluster with 8g heap memory for each" +
|
||||
" executor and obey the Spark AQE behavior of Kyuubi system default. On the other hand," +
|
||||
" for those users who do not have custom configurations will use system defaults.")
|
||||
" for those users who do not have custom configurations will use system defaults."
|
||||
|
||||
if (update) {
|
||||
val writer = Files.newBufferedWriter(
|
||||
markdown, StandardCharsets.UTF_8,
|
||||
StandardOpenOption.TRUNCATE_EXISTING,
|
||||
StandardOpenOption.CREATE)
|
||||
try {
|
||||
newOutput.foreach { line =>
|
||||
writer.write(line)
|
||||
writer.newLine()
|
||||
}
|
||||
} finally {
|
||||
writer.close()
|
||||
}
|
||||
} else {
|
||||
val expected = new ArrayBuffer[String]()
|
||||
|
||||
val reader = Files.newBufferedReader(markdown, StandardCharsets.UTF_8)
|
||||
var line = reader.readLine()
|
||||
while (line != null) {
|
||||
expected += line
|
||||
line = reader.readLine()
|
||||
}
|
||||
reader.close()
|
||||
val hint = s"$markdown out of date, please update doc with build/mvn test" +
|
||||
s" -DwildcardSuites=${getClass.getCanonicalName}"
|
||||
assert(newOutput.size === expected.size, hint)
|
||||
|
||||
newOutput.zip(expected).foreach { case (out, in) => assert(out === in, hint) }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Loading…
Reference in New Issue
Block a user