[KYUUBI #4244] Improvement in auto-generated Markdown docs with MarkdownBuilder
### _Why are the changes needed?_ Note: - No output changes to existed generated docs as in `settings.md` and `functions.md`. Improvement: - readability in Scala code for doc auto-generation, and easy to maintain docs in group of sections - reline on markdown linting from `flexmark`, and reducing un-meaningful blank lines or alignment - declaration over operation by replacing repeated usages of `newOutput` itself by elegantly wrapped `.line`,`.lines`and etc. - less fragile and more handy for changing in a long single line, as now using auto margin stripping in `.line()` method - reusable extracted licence and auto-generation hints - more elegant and safer way to read and appending file content - possible less memory footprint by apply operators to Stream instead of to ArrayBuffer ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4244 from bowenliang123/config-regen. Closes #4244 3d90ad304 [liangbowen] make buffer private in MarkdownBuilder 8250a55f3 [liangbowen] remove licence.md a4c7baf78 [liangbowen] add MarkdownBuilder.apply 248a046a4 [liangbowen] Improvement in auto-generated Markdown docs with MarkdownBuilder Authored-by: liangbowen <liangbowen@gf.com.cn> Signed-off-by: Cheng Pan <chengpan@apache.org>
This commit is contained in:
liangbowen
Cheng Pan
parent
8760eeef0e
commit
5b70b41a01
@ -19,9 +19,7 @@ package org.apache.kyuubi.engine.spark.udf
|
||||
|
||||
import java.nio.file.Paths
|
||||
|
||||
import scala.collection.mutable.ArrayBuffer
|
||||
|
||||
import org.apache.kyuubi.{KyuubiFunSuite, TestUtils, Utils}
|
||||
import org.apache.kyuubi.{KyuubiFunSuite, MarkdownBuilder, MarkdownUtils, Utils}
|
||||
|
||||
// scalastyle:off line.size.limit
|
||||
/**
|
||||
@ -48,44 +46,25 @@ class KyuubiDefinedFunctionSuite extends KyuubiFunSuite {
|
||||
.toAbsolutePath
|
||||
|
||||
test("verify or update kyuubi spark sql functions") {
|
||||
val newOutput = new ArrayBuffer[String]()
|
||||
newOutput += "<!--"
|
||||
newOutput += "- Licensed to the Apache Software Foundation (ASF) under one or more"
|
||||
newOutput += "- contributor license agreements. See the NOTICE file distributed with"
|
||||
newOutput += "- this work for additional information regarding copyright ownership."
|
||||
newOutput += "- The ASF licenses this file to You under the Apache License, Version 2.0"
|
||||
newOutput += "- (the \"License\"); you may not use this file except in compliance with"
|
||||
newOutput += "- the License. You may obtain a copy of the License at"
|
||||
newOutput += "-"
|
||||
newOutput += "- http://www.apache.org/licenses/LICENSE-2.0"
|
||||
newOutput += "-"
|
||||
newOutput += "- Unless required by applicable law or agreed to in writing, software"
|
||||
newOutput += "- distributed under the License is distributed on an \"AS IS\" BASIS,"
|
||||
newOutput += "- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied."
|
||||
newOutput += "- See the License for the specific language governing permissions and"
|
||||
newOutput += "- limitations under the License."
|
||||
newOutput += "-->"
|
||||
newOutput += ""
|
||||
newOutput += ""
|
||||
newOutput += "<!-- DO NOT MODIFY THIS FILE DIRECTLY, IT IS AUTO-GENERATED BY" +
|
||||
" [org.apache.kyuubi.engine.spark.udf.KyuubiDefinedFunctionSuite] -->"
|
||||
newOutput += ""
|
||||
newOutput += "# Auxiliary SQL Functions"
|
||||
newOutput += ""
|
||||
newOutput += "Kyuubi provides several auxiliary SQL functions as supplement to Spark's " +
|
||||
"[Built-in Functions](https://spark.apache.org/docs/latest/api/sql/index.html#" +
|
||||
"built-in-functions)"
|
||||
newOutput += ""
|
||||
newOutput += "Name | Description | Return Type | Since"
|
||||
newOutput += "--- | --- | --- | ---"
|
||||
KDFRegistry
|
||||
val builder = MarkdownBuilder(licenced = true, getClass.getName)
|
||||
|
||||
builder
|
||||
.line("# Auxiliary SQL Functions")
|
||||
.line("""Kyuubi provides several auxiliary SQL functions as supplement to Spark's
|
||||
| [Built-in Functions](https://spark.apache.org/docs/latest/api/sql/index.html#
|
||||
|built-in-functions)""")
|
||||
.lines("""
|
||||
| Name | Description | Return Type | Since
|
||||
| --- | --- | --- | ---
|
||||
|
|
||||
|""")
|
||||
KDFRegistry.registeredFunctions.foreach { func =>
|
||||
newOutput += s"${func.name} | ${func.description} | ${func.returnType} | ${func.since}"
|
||||
builder.line(s"${func.name} | ${func.description} | ${func.returnType} | ${func.since}")
|
||||
}
|
||||
newOutput += ""
|
||||
TestUtils.verifyOutput(
|
||||
|
||||
MarkdownUtils.verifyOutput(
|
||||
markdown,
|
||||
newOutput,
|
||||
builder,
|
||||
getClass.getCanonicalName,
|
||||
"externals/kyuubi-spark-sql-engine")
|
||||
}
|
||||
|
||||
@ -0,0 +1,213 @@
|
||||
/*
|
||||
* Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
* contributor license agreements. See the NOTICE file distributed with
|
||||
* this work for additional information regarding copyright ownership.
|
||||
* The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
* (the "License"); you may not use this file except in compliance with
|
||||
* the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.kyuubi
|
||||
|
||||
import java.nio.charset.StandardCharsets
|
||||
import java.nio.file.{Files, Path, StandardOpenOption}
|
||||
|
||||
import scala.collection.JavaConverters._
|
||||
import scala.collection.mutable.ArrayBuffer
|
||||
import scala.compat.Platform.EOL
|
||||
|
||||
import com.vladsch.flexmark.formatter.Formatter
|
||||
import com.vladsch.flexmark.parser.{Parser, ParserEmulationProfile, PegdownExtensions}
|
||||
import com.vladsch.flexmark.profile.pegdown.PegdownOptionsAdapter
|
||||
import com.vladsch.flexmark.util.data.{MutableDataHolder, MutableDataSet}
|
||||
import org.scalatest.Assertions.{assertResult, withClue}
|
||||
|
||||
object MarkdownUtils {
|
||||
|
||||
def verifyOutput(
|
||||
markdown: Path,
|
||||
newOutput: MarkdownBuilder,
|
||||
agent: String,
|
||||
module: String): Unit = {
|
||||
val formatted = newOutput.formatMarkdown()
|
||||
if (System.getenv("KYUUBI_UPDATE") == "1") {
|
||||
Files.write(
|
||||
markdown,
|
||||
formatted.asJava,
|
||||
StandardOpenOption.CREATE,
|
||||
StandardOpenOption.TRUNCATE_EXISTING)
|
||||
} else {
|
||||
val linesInFile = Files.readAllLines(markdown, StandardCharsets.UTF_8)
|
||||
linesInFile.asScala.zipWithIndex.zip(formatted).foreach { case ((str1, index), str2) =>
|
||||
withClue(s"$markdown out of date, as line ${index + 1} is not expected." +
|
||||
" Please update doc with KYUUBI_UPDATE=1 build/mvn clean test" +
|
||||
s" -pl $module -am -Pflink-provided,spark-provided,hive-provided" +
|
||||
s" -DwildcardSuites=$agent") {
|
||||
assertResult(str2)(str1)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
def line(str: String): String = {
|
||||
str.stripMargin.replaceAll(EOL, "")
|
||||
}
|
||||
|
||||
def appendBlankLine(buffer: ArrayBuffer[String]): Unit = buffer += ""
|
||||
|
||||
def appendFileContent(buffer: ArrayBuffer[String], path: Path): Unit = {
|
||||
buffer += "```bash"
|
||||
buffer ++= Files.readAllLines(path).asScala
|
||||
buffer += "```"
|
||||
}
|
||||
}
|
||||
|
||||
class MarkdownBuilder {
|
||||
private val buffer = new ArrayBuffer[String]()
|
||||
|
||||
/**
|
||||
* append a single line
|
||||
* with replacing EOL to empty string
|
||||
* @param str single line
|
||||
* @return
|
||||
*/
|
||||
def line(str: String = ""): MarkdownBuilder = {
|
||||
buffer += str.stripMargin.replaceAll(EOL, "")
|
||||
this
|
||||
}
|
||||
|
||||
/**
|
||||
* append the multiline
|
||||
* with splitting EOL into single lines
|
||||
* @param multiline multiline with default line margin "|"
|
||||
* @return
|
||||
*/
|
||||
def lines(multiline: String): MarkdownBuilder = {
|
||||
buffer ++= multiline.stripMargin.split(EOL)
|
||||
this
|
||||
}
|
||||
|
||||
/**
|
||||
* append the licence
|
||||
* @return
|
||||
*/
|
||||
def licence(): MarkdownBuilder = {
|
||||
lines("""
|
||||
|<!--
|
||||
|- Licensed to the Apache Software Foundation (ASF) under one or more
|
||||
|- contributor license agreements. See the NOTICE file distributed with
|
||||
|- this work for additional information regarding copyright ownership.
|
||||
|- The ASF licenses this file to You under the Apache License, Version 2.0
|
||||
|- (the "License"); you may not use this file except in compliance with
|
||||
|- the License. You may obtain a copy of the License at
|
||||
|-
|
||||
|- http://www.apache.org/licenses/LICENSE-2.0
|
||||
|-
|
||||
|- Unless required by applicable law or agreed to in writing, software
|
||||
|- distributed under the License is distributed on an "AS IS" BASIS,
|
||||
|- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
|- See the License for the specific language governing permissions and
|
||||
|- limitations under the License.
|
||||
|-->
|
||||
|""")
|
||||
}
|
||||
|
||||
/**
|
||||
* append the auto-generation hint
|
||||
* @param className the full class name of agent suite
|
||||
* @return
|
||||
*/
|
||||
def generationHint(className: String): MarkdownBuilder = {
|
||||
lines(s"""
|
||||
|<!-- DO NOT MODIFY THIS FILE DIRECTLY, IT IS AUTO-GENERATED BY [$className] -->
|
||||
|
|
||||
|""")
|
||||
}
|
||||
|
||||
/**
|
||||
* append file content
|
||||
* @param path file path
|
||||
* @return
|
||||
*/
|
||||
def file(path: Path): MarkdownBuilder = {
|
||||
buffer ++= Files.readAllLines(path).asScala
|
||||
this
|
||||
}
|
||||
|
||||
/**
|
||||
* append file content with code block quote
|
||||
* @param path path to file
|
||||
* @param language language of codeblock
|
||||
* @return
|
||||
*/
|
||||
def fileWithBlock(path: Path, language: String = "bash"): MarkdownBuilder = {
|
||||
buffer += s"```$language"
|
||||
file(path)
|
||||
buffer += "```"
|
||||
this
|
||||
}
|
||||
|
||||
def formatMarkdown(): Stream[String] = {
|
||||
def createParserOptions(emulationProfile: ParserEmulationProfile): MutableDataHolder = {
|
||||
PegdownOptionsAdapter.flexmarkOptions(PegdownExtensions.ALL).toMutable
|
||||
.set(Parser.PARSER_EMULATION_PROFILE, emulationProfile)
|
||||
}
|
||||
|
||||
def createFormatterOptions(
|
||||
parserOptions: MutableDataHolder,
|
||||
emulationProfile: ParserEmulationProfile): MutableDataSet = {
|
||||
new MutableDataSet()
|
||||
.set(Parser.EXTENSIONS, Parser.EXTENSIONS.get(parserOptions))
|
||||
.set(Formatter.FORMATTER_EMULATION_PROFILE, emulationProfile)
|
||||
}
|
||||
|
||||
val emulationProfile = ParserEmulationProfile.COMMONMARK
|
||||
val parserOptions = createParserOptions(emulationProfile)
|
||||
val formatterOptions = createFormatterOptions(parserOptions, emulationProfile)
|
||||
val parser = Parser.builder(parserOptions).build
|
||||
val renderer = Formatter.builder(formatterOptions).build
|
||||
val document = parser.parse(buffer.mkString(EOL))
|
||||
val formattedLines = new ArrayBuffer[String](buffer.length)
|
||||
val formattedLinesAppendable = new Appendable {
|
||||
override def append(csq: CharSequence): Appendable = {
|
||||
if (csq.length() > 0) {
|
||||
formattedLines.append(csq.toString)
|
||||
}
|
||||
this
|
||||
}
|
||||
|
||||
override def append(csq: CharSequence, start: Int, end: Int): Appendable = {
|
||||
append(csq.toString.substring(start, end))
|
||||
}
|
||||
|
||||
override def append(c: Char): Appendable = {
|
||||
append(c.toString)
|
||||
}
|
||||
}
|
||||
renderer.render(document, formattedLinesAppendable)
|
||||
// trim the ending EOL appended by renderer for each line
|
||||
formattedLines.toStream.map(str =>
|
||||
if (str.endsWith(EOL)) {
|
||||
str.substring(0, str.length - 1)
|
||||
} else {
|
||||
str
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
object MarkdownBuilder {
|
||||
def apply(licenced: Boolean = true, className: String = null): MarkdownBuilder = {
|
||||
val builder = new MarkdownBuilder
|
||||
if (licenced) { builder.licence() }
|
||||
if (className != null) { builder.generationHint(className) }
|
||||
builder
|
||||
}
|
||||
}
|
||||
@ -14,99 +14,15 @@
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package org.apache.kyuubi
|
||||
|
||||
import java.nio.charset.StandardCharsets
|
||||
import java.nio.file.{Files, Path, StandardOpenOption}
|
||||
import java.sql.ResultSet
|
||||
|
||||
import scala.collection.JavaConverters._
|
||||
import scala.collection.mutable.ArrayBuffer
|
||||
|
||||
import com.jakewharton.fliptables.FlipTable
|
||||
import com.vladsch.flexmark.formatter.Formatter
|
||||
import com.vladsch.flexmark.parser.{Parser, ParserEmulationProfile, PegdownExtensions}
|
||||
import com.vladsch.flexmark.profile.pegdown.PegdownOptionsAdapter
|
||||
import com.vladsch.flexmark.util.data.{MutableDataHolder, MutableDataSet}
|
||||
import com.vladsch.flexmark.util.sequence.SequenceUtils
|
||||
import org.scalatest.Assertions.{assertResult, withClue}
|
||||
|
||||
object TestUtils {
|
||||
|
||||
private def formatMarkdown(lines: ArrayBuffer[String]): ArrayBuffer[String] = {
|
||||
def createParserOptions(emulationProfile: ParserEmulationProfile): MutableDataHolder = {
|
||||
PegdownOptionsAdapter.flexmarkOptions(PegdownExtensions.ALL).toMutable
|
||||
.set(Parser.PARSER_EMULATION_PROFILE, emulationProfile)
|
||||
}
|
||||
|
||||
def createFormatterOptions(
|
||||
parserOptions: MutableDataHolder,
|
||||
emulationProfile: ParserEmulationProfile): MutableDataSet = {
|
||||
new MutableDataSet()
|
||||
.set(Parser.EXTENSIONS, Parser.EXTENSIONS.get(parserOptions))
|
||||
.set(Formatter.FORMATTER_EMULATION_PROFILE, emulationProfile)
|
||||
}
|
||||
|
||||
val emulationProfile = ParserEmulationProfile.valueOf("COMMONMARK")
|
||||
val parserOptions = createParserOptions(emulationProfile)
|
||||
val formatterOptions = createFormatterOptions(parserOptions, emulationProfile)
|
||||
val parser = Parser.builder(parserOptions).build
|
||||
val renderer = Formatter.builder(formatterOptions).build
|
||||
val document = parser.parse(lines.mkString(SequenceUtils.EOL))
|
||||
val formattedLines = new ArrayBuffer[String]
|
||||
val formattedLinesAppendable = new Appendable {
|
||||
override def append(csq: CharSequence): Appendable = {
|
||||
if (csq.length() > 0) {
|
||||
formattedLines.append(csq.toString)
|
||||
}
|
||||
this
|
||||
}
|
||||
|
||||
override def append(csq: CharSequence, start: Int, end: Int): Appendable = {
|
||||
append(csq.toString.substring(start, end))
|
||||
}
|
||||
|
||||
override def append(c: Char): Appendable = {
|
||||
append(c.toString)
|
||||
}
|
||||
}
|
||||
renderer.render(document, formattedLinesAppendable)
|
||||
// trim the ending EOL appended by renderer for each line
|
||||
formattedLines.map(str =>
|
||||
if (str.nonEmpty && str.endsWith(SequenceUtils.EOL)) {
|
||||
str.substring(0, str.length - 1)
|
||||
} else {
|
||||
str
|
||||
})
|
||||
}
|
||||
|
||||
def verifyOutput(
|
||||
markdown: Path,
|
||||
newOutput: ArrayBuffer[String],
|
||||
agent: String,
|
||||
module: String): Unit = {
|
||||
if (System.getenv("KYUUBI_UPDATE") == "1") {
|
||||
val formatted = formatMarkdown(newOutput)
|
||||
Files.write(
|
||||
markdown,
|
||||
formatted.asJava,
|
||||
StandardOpenOption.CREATE,
|
||||
StandardOpenOption.TRUNCATE_EXISTING)
|
||||
} else {
|
||||
val linesInFile = Files.readAllLines(markdown, StandardCharsets.UTF_8)
|
||||
val formatted = formatMarkdown(newOutput)
|
||||
linesInFile.asScala.zipWithIndex.zip(formatted).foreach { case ((str1, index), str2) =>
|
||||
withClue(s"$markdown out of date, as line ${index + 1} is not expected." +
|
||||
" Please update doc with KYUUBI_UPDATE=1 build/mvn clean test" +
|
||||
s" -pl $module -am -Pflink-provided,spark-provided,hive-provided" +
|
||||
s" -DwildcardSuites=$agent") {
|
||||
assertResult(str2)(str1)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
def displayResultSet(resultSet: ResultSet): Unit = {
|
||||
if (resultSet == null) throw new NullPointerException("resultSet == null")
|
||||
val resultSetMetaData = resultSet.getMetaData
|
||||
|
||||
@ -17,13 +17,11 @@
|
||||
|
||||
package org.apache.kyuubi.config
|
||||
|
||||
import java.nio.charset.StandardCharsets
|
||||
import java.nio.file.{Files, Path, Paths}
|
||||
import java.nio.file.Paths
|
||||
|
||||
import scala.collection.JavaConverters._
|
||||
import scala.collection.mutable.ArrayBuffer
|
||||
|
||||
import org.apache.kyuubi.{KyuubiFunSuite, TestUtils, Utils}
|
||||
import org.apache.kyuubi.{KyuubiFunSuite, MarkdownBuilder, MarkdownUtils, Utils}
|
||||
import org.apache.kyuubi.ctl.CtlConf
|
||||
import org.apache.kyuubi.ha.HighAvailabilityConf
|
||||
import org.apache.kyuubi.metrics.MetricsConf
|
||||
@ -51,255 +49,196 @@ class AllKyuubiConfiguration extends KyuubiFunSuite {
|
||||
private val markdown = Paths.get(kyuubiHome, "docs", "deployment", "settings.md")
|
||||
.toAbsolutePath
|
||||
|
||||
def rewriteToConf(path: Path, buffer: ArrayBuffer[String]): Unit = {
|
||||
val env =
|
||||
Files.newBufferedReader(path, StandardCharsets.UTF_8)
|
||||
|
||||
try {
|
||||
buffer += "```bash"
|
||||
var line = env.readLine()
|
||||
while (line != null) {
|
||||
buffer += line
|
||||
line = env.readLine()
|
||||
}
|
||||
buffer += "```"
|
||||
} finally {
|
||||
env.close()
|
||||
}
|
||||
}
|
||||
private def loadConfigs = Array(
|
||||
KyuubiConf,
|
||||
CtlConf,
|
||||
HighAvailabilityConf,
|
||||
JDBCMetadataStoreConf,
|
||||
MetricsConf,
|
||||
ZookeeperConf)
|
||||
|
||||
test("Check all kyuubi configs") {
|
||||
KyuubiConf
|
||||
CtlConf
|
||||
HighAvailabilityConf
|
||||
JDBCMetadataStoreConf
|
||||
MetricsConf
|
||||
ZookeeperConf
|
||||
loadConfigs
|
||||
|
||||
val newOutput = new ArrayBuffer[String]()
|
||||
newOutput += "<!--"
|
||||
newOutput += " - Licensed to the Apache Software Foundation (ASF) under one or more"
|
||||
newOutput += " - contributor license agreements. See the NOTICE file distributed with"
|
||||
newOutput += " - this work for additional information regarding copyright ownership."
|
||||
newOutput += " - The ASF licenses this file to You under the Apache License, Version 2.0"
|
||||
newOutput += " - (the \"License\"); you may not use this file except in compliance with"
|
||||
newOutput += " - the License. You may obtain a copy of the License at"
|
||||
newOutput += " -"
|
||||
newOutput += " - http://www.apache.org/licenses/LICENSE-2.0"
|
||||
newOutput += " -"
|
||||
newOutput += " - Unless required by applicable law or agreed to in writing, software"
|
||||
newOutput += " - distributed under the License is distributed on an \"AS IS\" BASIS,"
|
||||
newOutput += " - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied."
|
||||
newOutput += " - See the License for the specific language governing permissions and"
|
||||
newOutput += " - limitations under the License."
|
||||
newOutput += " -->"
|
||||
newOutput += ""
|
||||
newOutput += "<!-- DO NOT MODIFY THIS FILE DIRECTLY, IT IS AUTO-GENERATED BY" +
|
||||
" [org.apache.kyuubi.config.AllKyuubiConfiguration] -->"
|
||||
newOutput += ""
|
||||
newOutput += ""
|
||||
newOutput += "# Introduction to the Kyuubi Configurations System"
|
||||
newOutput += ""
|
||||
newOutput += "Kyuubi provides several ways to configure the system and corresponding engines."
|
||||
newOutput += ""
|
||||
newOutput += ""
|
||||
newOutput += "## Environments"
|
||||
newOutput += ""
|
||||
newOutput += ""
|
||||
newOutput += "You can configure the environment variables in" +
|
||||
" `$KYUUBI_HOME/conf/kyuubi-env.sh`, e.g, `JAVA_HOME`, then this java runtime will be used" +
|
||||
" both for Kyuubi server instance and the applications it launches. You can also change" +
|
||||
" the variable in the subprocess's env configuration file, e.g." +
|
||||
"`$SPARK_HOME/conf/spark-env.sh` to use more specific ENV for SQL engine applications."
|
||||
val builder = MarkdownBuilder(licenced = true, getClass.getName)
|
||||
|
||||
rewriteToConf(Paths.get(kyuubiHome, "conf", "kyuubi-env.sh.template"), newOutput)
|
||||
|
||||
newOutput += ""
|
||||
newOutput += "For the environment variables that only needed to be transferred into engine" +
|
||||
" side, you can set it with a Kyuubi configuration item formatted" +
|
||||
" `kyuubi.engineEnv.VAR_NAME`. For example, with `kyuubi.engineEnv.SPARK_DRIVER_MEMORY=4g`," +
|
||||
" the environment variable `SPARK_DRIVER_MEMORY` with value `4g` would be transferred into" +
|
||||
" engine side. With `kyuubi.engineEnv.SPARK_CONF_DIR=/apache/confs/spark/conf`, the" +
|
||||
" value of `SPARK_CONF_DIR` on the engine side is set to `/apache/confs/spark/conf`."
|
||||
|
||||
newOutput += ""
|
||||
newOutput += "## Kyuubi Configurations"
|
||||
newOutput += ""
|
||||
|
||||
newOutput += "You can configure the Kyuubi properties in" +
|
||||
" `$KYUUBI_HOME/conf/kyuubi-defaults.conf`. For example:"
|
||||
|
||||
rewriteToConf(Paths.get(kyuubiHome, "conf", "kyuubi-defaults.conf.template"), newOutput)
|
||||
builder
|
||||
.lines(s"""
|
||||
|# Introduction to the Kyuubi Configurations System
|
||||
|
|
||||
|Kyuubi provides several ways to configure the system and corresponding engines.
|
||||
|
|
||||
|## Environments
|
||||
|
|
||||
|""")
|
||||
.line("""You can configure the environment variables in `$KYUUBI_HOME/conf/kyuubi-env.sh`,
|
||||
| e.g, `JAVA_HOME`, then this java runtime will be used both for Kyuubi server instance and
|
||||
| the applications it launches. You can also change the variable in the subprocess's env
|
||||
| configuration file, e.g.`$SPARK_HOME/conf/spark-env.sh` to use more specific ENV for
|
||||
| SQL engine applications.
|
||||
| """)
|
||||
.fileWithBlock(Paths.get(kyuubiHome, "conf", "kyuubi-env.sh.template"))
|
||||
.line(
|
||||
"""
|
||||
| For the environment variables that only needed to be transferred into engine
|
||||
| side, you can set it with a Kyuubi configuration item formatted
|
||||
| `kyuubi.engineEnv.VAR_NAME`. For example, with `kyuubi.engineEnv.SPARK_DRIVER_MEMORY=4g`,
|
||||
| the environment variable `SPARK_DRIVER_MEMORY` with value `4g` would be transferred into
|
||||
| engine side. With `kyuubi.engineEnv.SPARK_CONF_DIR=/apache/confs/spark/conf`, the
|
||||
| value of `SPARK_CONF_DIR` on the engine side is set to `/apache/confs/spark/conf`.
|
||||
| """)
|
||||
.line("## Kyuubi Configurations")
|
||||
.line(""" You can configure the Kyuubi properties in
|
||||
| `$KYUUBI_HOME/conf/kyuubi-defaults.conf`. For example: """)
|
||||
.fileWithBlock(Paths.get(kyuubiHome, "conf", "kyuubi-defaults.conf.template"))
|
||||
|
||||
KyuubiConf.getConfigEntries().asScala
|
||||
.toSeq
|
||||
.toStream
|
||||
.filterNot(_.internal)
|
||||
.groupBy(_.key.split("\\.")(1))
|
||||
.toSeq.sortBy(_._1).foreach { case (category, entries) =>
|
||||
newOutput += ""
|
||||
newOutput += s"### ${category.capitalize}"
|
||||
newOutput += ""
|
||||
|
||||
newOutput += "Key | Default | Meaning | Type | Since"
|
||||
newOutput += "--- | --- | --- | --- | ---"
|
||||
builder.lines(
|
||||
s"""### ${category.capitalize}
|
||||
| Key | Default | Meaning | Type | Since
|
||||
| --- | --- | --- | --- | ---
|
||||
|""")
|
||||
|
||||
entries.sortBy(_.key).foreach { c =>
|
||||
val dft = c.defaultValStr.replace("<", "<").replace(">", ">")
|
||||
val seq = Seq(
|
||||
builder.line(Seq(
|
||||
s"${c.key}",
|
||||
s"$dft",
|
||||
s"${c.doc}",
|
||||
s"${c.typ}",
|
||||
s"${c.version}")
|
||||
newOutput += seq.mkString("|")
|
||||
s"${c.version}").mkString("|"))
|
||||
}
|
||||
newOutput += ""
|
||||
}
|
||||
|
||||
newOutput += ("## Spark Configurations")
|
||||
newOutput += ""
|
||||
builder
|
||||
.lines("""
|
||||
|## Spark Configurations
|
||||
|### Via spark-defaults.conf
|
||||
|""")
|
||||
.line("""
|
||||
| Setting them in `$SPARK_HOME/conf/spark-defaults.conf`
|
||||
| supplies with default values for SQL engine application. Available properties can be
|
||||
| found at Spark official online documentation for
|
||||
| [Spark Configurations](https://spark.apache.org/docs/latest/configuration.html)
|
||||
| """)
|
||||
.line("### Via kyuubi-defaults.conf")
|
||||
.line("""
|
||||
| Setting them in `$KYUUBI_HOME/conf/kyuubi-defaults.conf`
|
||||
| supplies with default values for SQL engine application too. These properties will
|
||||
| override all settings in `$SPARK_HOME/conf/spark-defaults.conf`""")
|
||||
.line("### Via JDBC Connection URL")
|
||||
.line("""
|
||||
| Setting them in the JDBC Connection URL
|
||||
| supplies session-specific for each SQL engine. For example:
|
||||
| ```
|
||||
|jdbc:hive2://localhost:10009/default;#
|
||||
|spark.sql.shuffle.partitions=2;spark.executor.memory=5g
|
||||
|```""")
|
||||
.line()
|
||||
.line("- **Runtime SQL Configuration**")
|
||||
.line(""" - For [Runtime SQL Configurations](
|
||||
|https://spark.apache.org/docs/latest/configuration.html#runtime-sql-configuration), they
|
||||
| will take affect every time""")
|
||||
.line("- **Static SQL and Spark Core Configuration**")
|
||||
.line(""" - For [Static SQL Configurations](
|
||||
|https://spark.apache.org/docs/latest/configuration.html#static-sql-configuration) and
|
||||
| other spark core configs, e.g. `spark.executor.memory`, they will take effect if there
|
||||
| is no existing SQL engine application. Otherwise, they will just be ignored""")
|
||||
.line("### Via SET Syntax")
|
||||
.line("""Please refer to the Spark official online documentation for
|
||||
| [SET Command](https://spark.apache.org/docs/latest/sql-ref-syntax-aux-conf-mgmt-set.html)
|
||||
|""")
|
||||
|
||||
newOutput += ("### Via spark-defaults.conf")
|
||||
newOutput += ""
|
||||
builder
|
||||
.lines("""
|
||||
|## Flink Configurations
|
||||
|### Via flink-conf.yaml""")
|
||||
.line("""Setting them in `$FLINK_HOME/conf/flink-conf.yaml`
|
||||
| supplies with default values for SQL engine application.
|
||||
| Available properties can be found at Flink official online documentation for
|
||||
| [Flink Configurations]
|
||||
|(https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/config/)""")
|
||||
.line("### Via kyuubi-defaults.conf")
|
||||
.line("""Setting them in `$KYUUBI_HOME/conf/kyuubi-defaults.conf`
|
||||
| supplies with default values for SQL engine application too.
|
||||
| You can use properties with the additional prefix `flink.` to override settings in
|
||||
| `$FLINK_HOME/conf/flink-conf.yaml`.""")
|
||||
.lines("""
|
||||
|
|
||||
|For example:
|
||||
|```
|
||||
|flink.parallelism.default 2
|
||||
|flink.taskmanager.memory.process.size 5g
|
||||
|```""")
|
||||
.line("""The below options in `kyuubi-defaults.conf` will set `parallelism.default: 2`
|
||||
| and `taskmanager.memory.process.size: 5g` into flink configurations.""")
|
||||
.line("### Via JDBC Connection URL")
|
||||
.line("""Setting them in the JDBC Connection URL supplies session-specific
|
||||
| for each SQL engine. For example: ```jdbc:hive2://localhost:10009/default;
|
||||
|#parallelism.default=2;taskmanager.memory.process.size=5g```
|
||||
|""")
|
||||
.line("### Via SET Statements")
|
||||
.line("""Please refer to the Flink official online documentation for [SET Statements]
|
||||
|(https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/table/sql/set/)""")
|
||||
|
||||
newOutput += ("Setting them in `$SPARK_HOME/conf/spark-defaults.conf`" +
|
||||
" supplies with default values for SQL engine application. Available properties can be" +
|
||||
" found at Spark official online documentation for" +
|
||||
" [Spark Configurations](https://spark.apache.org/docs/latest/configuration.html)")
|
||||
builder
|
||||
.line("## Logging")
|
||||
.line("""Kyuubi uses [log4j](https://logging.apache.org/log4j/2.x/) for logging.
|
||||
| You can configure it using `$KYUUBI_HOME/conf/log4j2.xml`.""")
|
||||
.fileWithBlock(Paths.get(kyuubiHome, "conf", "log4j2.xml.template"))
|
||||
|
||||
newOutput += ""
|
||||
newOutput += ("### Via kyuubi-defaults.conf")
|
||||
newOutput += ""
|
||||
newOutput += ("Setting them in `$KYUUBI_HOME/conf/kyuubi-defaults.conf`" +
|
||||
" supplies with default values for SQL engine application too. These properties will" +
|
||||
" override all settings in `$SPARK_HOME/conf/spark-defaults.conf`")
|
||||
builder
|
||||
.lines("""
|
||||
|## Other Configurations
|
||||
|### Hadoop Configurations
|
||||
|""")
|
||||
.line("""Specifying `HADOOP_CONF_DIR` to the directory containing Hadoop configuration
|
||||
| files or treating them as Spark properties with a `spark.hadoop.` prefix.
|
||||
| Please refer to the Spark official online documentation for
|
||||
| [Inheriting Hadoop Cluster Configuration](https://spark.apache.org/docs/latest/
|
||||
|configuration.html#inheriting-hadoop-cluster-configuration).
|
||||
| Also, please refer to the [Apache Hadoop](https://hadoop.apache.org)'s
|
||||
| online documentation for an overview on how to configure Hadoop.""")
|
||||
.line("### Hive Configurations")
|
||||
.line("""These configurations are used for SQL engine application to talk to
|
||||
| Hive MetaStore and could be configured in a `hive-site.xml`.
|
||||
| Placed it in `$SPARK_HOME/conf` directory, or treat them as Spark properties with
|
||||
| a `spark.hadoop.` prefix.""")
|
||||
|
||||
newOutput += ""
|
||||
newOutput += ("### Via JDBC Connection URL")
|
||||
newOutput += ""
|
||||
newOutput += ("Setting them in the JDBC Connection URL" +
|
||||
" supplies session-specific for each SQL engine. For example: " +
|
||||
"```" +
|
||||
"jdbc:hive2://localhost:10009/default;#" +
|
||||
"spark.sql.shuffle.partitions=2;spark.executor.memory=5g" +
|
||||
"```")
|
||||
newOutput += ""
|
||||
newOutput += ("- **Runtime SQL Configuration**")
|
||||
newOutput += ""
|
||||
newOutput += (" - For [Runtime SQL Configurations](" +
|
||||
"https://spark.apache.org/docs/latest/configuration.html#runtime-sql-configuration), they" +
|
||||
" will take affect every time")
|
||||
newOutput += ("- **Static SQL and Spark Core Configuration**")
|
||||
newOutput += ""
|
||||
newOutput += (" - For [Static SQL Configurations](" +
|
||||
"https://spark.apache.org/docs/latest/configuration.html#static-sql-configuration) and" +
|
||||
" other spark core configs, e.g. `spark.executor.memory`, they will take effect if there" +
|
||||
" is no existing SQL engine application. Otherwise, they will just be ignored")
|
||||
newOutput += ""
|
||||
newOutput += ("### Via SET Syntax")
|
||||
newOutput += ""
|
||||
newOutput += ("Please refer to the Spark official online documentation for" +
|
||||
" [SET Command](https://spark.apache.org/docs/latest/sql-ref-syntax-aux-conf-mgmt-set.html)")
|
||||
newOutput += ""
|
||||
builder
|
||||
.line("## User Defaults")
|
||||
.line("""In Kyuubi, we can configure user default settings to meet separate needs.
|
||||
| These user defaults override system defaults, but will be overridden by those from
|
||||
| [JDBC Connection URL](#via-jdbc-connection-url) or [Set Command](#via-set-syntax)
|
||||
| if could be. They will take effect when creating the SQL engine application ONLY.""")
|
||||
.line("""User default settings are in the form of `___{username}___.{config key}`.
|
||||
| There are three continuous underscores(`_`) at both sides of the `username` and
|
||||
| a dot(`.`) that separates the config key and the prefix. For example:""")
|
||||
.lines("""
|
||||
|```bash
|
||||
|# For system defaults
|
||||
|spark.master=local
|
||||
|spark.sql.adaptive.enabled=true
|
||||
|# For a user named kent
|
||||
|___kent___.spark.master=yarn
|
||||
|___kent___.spark.sql.adaptive.enabled=false
|
||||
|# For a user named bob
|
||||
|___bob___.spark.master=spark://master:7077
|
||||
|___bob___.spark.executor.memory=8g
|
||||
|```
|
||||
|
|
||||
|""")
|
||||
.line("""In the above case, if there are related configurations from
|
||||
| [JDBC Connection URL](#via-jdbc-connection-url), `kent` will run his SQL engine
|
||||
| application on YARN and prefer the Spark AQE to be off, while `bob` will activate
|
||||
| his SQL engine application on a Spark standalone cluster with 8g heap memory for each
|
||||
| executor and obey the Spark AQE behavior of Kyuubi system default. On the other hand,
|
||||
| for those users who do not have custom configurations will use system defaults.""")
|
||||
|
||||
newOutput += ("## Flink Configurations")
|
||||
newOutput += ""
|
||||
|
||||
newOutput += ("### Via flink-conf.yaml")
|
||||
newOutput += ""
|
||||
newOutput += ("Setting them in `$FLINK_HOME/conf/flink-conf.yaml`" +
|
||||
" supplies with default values for SQL engine application." +
|
||||
" Available properties can be found at Flink official online documentation for" +
|
||||
" [Flink Configurations]" +
|
||||
"(https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/config/)")
|
||||
newOutput += ""
|
||||
|
||||
newOutput += ("### Via kyuubi-defaults.conf")
|
||||
newOutput += ""
|
||||
newOutput += ("Setting them in `$KYUUBI_HOME/conf/kyuubi-defaults.conf`" +
|
||||
" supplies with default values for SQL engine application too." +
|
||||
" You can use properties with the additional prefix `flink.` to override settings in" +
|
||||
" `$FLINK_HOME/conf/flink-conf.yaml`.")
|
||||
newOutput += ""
|
||||
newOutput += ("For example:")
|
||||
newOutput += ("```")
|
||||
newOutput += ("flink.parallelism.default 2")
|
||||
newOutput += ("flink.taskmanager.memory.process.size 5g")
|
||||
newOutput += ("```")
|
||||
newOutput += ""
|
||||
newOutput += ("The below options in `kyuubi-defaults.conf` will set `parallelism.default: 2`" +
|
||||
" and `taskmanager.memory.process.size: 5g` into flink configurations.")
|
||||
newOutput += ""
|
||||
|
||||
newOutput += ("### Via JDBC Connection URL")
|
||||
newOutput += ""
|
||||
newOutput += "Setting them in the JDBC Connection URL supplies session-specific" +
|
||||
" for each SQL engine. For example: ```jdbc:hive2://localhost:10009/default;" +
|
||||
"#parallelism.default=2;taskmanager.memory.process.size=5g```"
|
||||
newOutput += ""
|
||||
|
||||
newOutput += ("### Via SET Statements")
|
||||
newOutput += ""
|
||||
newOutput += ("Please refer to the Flink official online documentation for [SET Statements]" +
|
||||
"(https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/table/sql/set/)")
|
||||
newOutput += ""
|
||||
|
||||
newOutput += ("## Logging")
|
||||
newOutput += ""
|
||||
newOutput += ("Kyuubi uses [log4j](https://logging.apache.org/log4j/2.x/) for logging." +
|
||||
" You can configure it using `$KYUUBI_HOME/conf/log4j2.xml`.")
|
||||
|
||||
rewriteToConf(Paths.get(kyuubiHome, "conf", "log4j2.xml.template"), newOutput)
|
||||
|
||||
newOutput += ""
|
||||
newOutput += ("## Other Configurations")
|
||||
newOutput += ""
|
||||
newOutput += ("### Hadoop Configurations")
|
||||
newOutput += ""
|
||||
newOutput += ("Specifying `HADOOP_CONF_DIR` to the directory containing Hadoop configuration" +
|
||||
" files or treating them as Spark properties with a `spark.hadoop.` prefix." +
|
||||
" Please refer to the Spark official online documentation for" +
|
||||
" [Inheriting Hadoop Cluster Configuration](https://spark.apache.org/docs/latest/" +
|
||||
"configuration.html#inheriting-hadoop-cluster-configuration)." +
|
||||
" Also, please refer to the [Apache Hadoop](https://hadoop.apache.org)'s" +
|
||||
" online documentation for an overview on how to configure Hadoop.")
|
||||
newOutput += ""
|
||||
newOutput += ("### Hive Configurations")
|
||||
newOutput += ""
|
||||
newOutput += ("These configurations are used for SQL engine application to talk to" +
|
||||
" Hive MetaStore and could be configured in a `hive-site.xml`." +
|
||||
" Placed it in `$SPARK_HOME/conf` directory, or treat them as Spark properties with" +
|
||||
" a `spark.hadoop.` prefix.")
|
||||
|
||||
newOutput += ""
|
||||
newOutput += ("## User Defaults")
|
||||
newOutput += ""
|
||||
newOutput += ("In Kyuubi, we can configure user default settings to meet separate needs." +
|
||||
" These user defaults override system defaults, but will be overridden by those from" +
|
||||
" [JDBC Connection URL](#via-jdbc-connection-url) or [Set Command](#via-set-syntax)" +
|
||||
" if could be. They will take effect when creating the SQL engine application ONLY.")
|
||||
newOutput += ("User default settings are in the form of `___{username}___.{config key}`." +
|
||||
" There are three continuous underscores(`_`) at both sides of the `username` and" +
|
||||
" a dot(`.`) that separates the config key and the prefix. For example:")
|
||||
newOutput += ("```bash")
|
||||
newOutput += ("# For system defaults")
|
||||
newOutput += ("spark.master=local")
|
||||
newOutput += ("spark.sql.adaptive.enabled=true")
|
||||
newOutput += ("# For a user named kent")
|
||||
newOutput += ("___kent___.spark.master=yarn")
|
||||
newOutput += ("___kent___.spark.sql.adaptive.enabled=false")
|
||||
newOutput += ("# For a user named bob")
|
||||
newOutput += ("___bob___.spark.master=spark://master:7077")
|
||||
newOutput += ("___bob___.spark.executor.memory=8g")
|
||||
newOutput += ("```")
|
||||
newOutput += ""
|
||||
newOutput += "In the above case, if there are related configurations from" +
|
||||
" [JDBC Connection URL](#via-jdbc-connection-url), `kent` will run his SQL engine" +
|
||||
" application on YARN and prefer the Spark AQE to be off, while `bob` will activate" +
|
||||
" his SQL engine application on a Spark standalone cluster with 8g heap memory for each" +
|
||||
" executor and obey the Spark AQE behavior of Kyuubi system default. On the other hand," +
|
||||
" for those users who do not have custom configurations will use system defaults."
|
||||
|
||||
TestUtils.verifyOutput(markdown, newOutput, getClass.getCanonicalName, "kyuubi-server")
|
||||
MarkdownUtils.verifyOutput(markdown, builder, getClass.getCanonicalName, "kyuubi-server")
|
||||
}
|
||||
}
|
||||
|
||||
Loading…
Reference in New Issue
Block a user