apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

[KYUUBI #1209] kyuubi.engine.share.level.subdomain can be more tolerant

Browse Source 
feature: kyuubi.engine.share.level.subdomain can be more tolerant
now end-user can configure kyuubi.engine.share.level.subdomain by zookeeper valid path
I refer to the zookeeper doc http://zookeeper.apache.org/doc/r3.7.0/zookeeperProgrammers.html#:~:text=ZooKeeper%20Basic%20Operations.-,The%20ZooKeeper%20Data%20Model,-ZooKeeper%20has%20a
and source code:
https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/common/PathUtils.java#:~:text=public%20static%20void-,validatePath,-(String%20path)%20throws

Closes #1232 from TyrealBM/feature/valid-subdomain-path.

Closes #1209

8d00c260 [guoqing.yang] [KYUUBI #1209] kyuubi.engine.share.level.subdomain can be more tolerant

Authored-by: guoqing.yang <guoqing.yang@advance.ai>
Signed-off-by: Kent Yao <yao@apache.org>
This commit is contained in:
guoqing.yang 2021-10-18 10:21:12 +08:00 committed by Kent Yao
parent 9f654dbd80
commit 3e199ccdce
No known key found for this signature in database
GPG Key ID: F7051850A0AF904D
4 changed files with 34 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/deployment/settings.md
View File

@ -186,7 +186,7 @@ kyuubi\.engine\.pool<br>\.size\.threshold|<div style='width: 65pt;word-wrap: bre
kyuubi\.engine\.session<br>\.initialize\.sql|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>SHOW DATABASES</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>SemiColon-separated list of SQL statements to be initialized in the newly created engine session before queries. This configuration can not be used in JDBC url due to the limitation of Beeline/JDBC driver.</div>|<div style='width: 30pt'>seq</div>|<div style='width: 20pt'>1.3.0</div>
kyuubi\.engine\.share<br>\.level|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>USER</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>Engines will be shared in different levels, available configs are: <ul> <li>CONNECTION: engine will not be shared but only used by the current client connection</li> <li>USER: engine will be shared by all sessions created by a unique username, see also kyuubi.engine.share.level.subdomain</li> <li>GROUP: engine will be shared by all sessions created by all users belong to the same primary group name. The engine will be launched by the group name as the effective username, so here the group name is kind of special user who is able to visit the compute resources/data of a team. It follows the [Hadoop GroupsMapping](https://reurl.cc/xE61Y5) to map user to a primary group. If the primary group is not found, it fallback to the USER level. <li>SERVER: the App will be shared by Kyuubi servers</li></ul></div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.2.0</div>
kyuubi\.engine\.share<br>\.level\.sub\.domain|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>&lt;undefined&gt;</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>(deprecated) - Using kyuubi.engine.share.level.subdomain instead</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.2.0</div>
kyuubi\.engine\.share<br>\.level\.subdomain|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>&lt;undefined&gt;</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>Allow end-users to create a subdomain for the share level of an engine. A subdomain is a case-insensitive string values in `^[a-zA-Z_-]{1,14}$` form. For example, for `USER` share level, an end-user can share a certain engine within a subdomain, not for all of its clients. End-users are free to create multiple engines in the `USER` share level</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.4.0</div>
kyuubi\.engine\.share<br>\.level\.subdomain|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>&lt;undefined&gt;</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>Allow end-users to create a subdomain for the share level of an engine. A subdomain is a case-insensitive string values that must be a valid zookeeper sub path. For example, for `USER` share level, an end-user can share a certain engine within a subdomain, not for all of its clients. End-users are free to create multiple engines in the `USER` share level</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.4.0</div>
kyuubi\.engine\.single<br>\.spark\.session|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>false</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>When set to true, this engine is running in a single session mode. All the JDBC/ODBC connections share the temporary views, function registries, SQL configuration and the current database.</div>|<div style='width: 30pt'>boolean</div>|<div style='width: 20pt'>1.3.0</div>
kyuubi\.engine\.ui<br>\.retainedSessions|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>200</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>The number of SQL client sessions kept in the Kyuubi Query Engine web UI.</div>|<div style='width: 30pt'>int</div>|<div style='width: 20pt'>1.4.0</div>
kyuubi\.engine\.ui\.stop<br>\.enabled|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>true</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>When true, allows Kyuubi engine to be killed from the Spark Web UI.</div>|<div style='width: 30pt'>boolean</div>|<div style='width: 20pt'>1.3.0</div>

12
kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
View File

@ -714,6 +714,11 @@ object KyuubiConf {
private val validEngineSubDomain: Pattern = "^[a-zA-Z_-]{1,14}$".r.pattern
// [ZooKeeper Data Model]
// (http://zookeeper.apache.org/doc/r3.7.0/zookeeperProgrammers.html#ch_zkDataModel)
private val validEngineSubdomain: Pattern = ("(?!^[\\u002e]{1,2}$)" +
"(^[\\u0020-\\u002e\\u0030-\\u007e\\u00a0-\\ud7ff\\uf900-\\uffef]{1,}$)").r.pattern
@deprecated(s"using kyuubi.engine.share.level.subdomain instead", "1.4.0")
val ENGINE_SHARE_LEVEL_SUB_DOMAIN: OptionalConfigEntry[String] =
buildConf("engine.share.level.sub.domain")
@ -721,14 +726,15 @@ object KyuubiConf {
.version("1.2.0")
.stringConf
.transform(_.toLowerCase(Locale.ROOT))
.checkValue(validEngineSubDomain.matcher(_).matches(),
"must be [1, 14] length alphabet string, e.g. 'abc', 'apache'")
.checkValue(validEngineSubdomain.matcher(_).matches(),
"must be valid zookeeper sub path."
)
.createOptional
val ENGINE_SHARE_LEVEL_SUBDOMAIN: ConfigEntry[Option[String]] =
buildConf("engine.share.level.subdomain")
.doc("Allow end-users to create a subdomain for the share level of an engine. A" +
" subdomain is a case-insensitive string values in `^[a-zA-Z_-]{1,14}$` form." +
" subdomain is a case-insensitive string values that must be a valid zookeeper sub path." +
" For example, for `USER` share level, an end-user can share a certain engine within" +
" a subdomain, not for all of its clients. End-users are free to create multiple" +
" engines in the `USER` share level")

23
kyuubi-common/src/test/scala/org/apache/kyuubi/config/KyuubiConfSuite.scala
View File

@ -150,4 +150,27 @@ class KyuubiConfSuite extends KyuubiFunSuite {
val e1 = intercept[IllegalArgumentException](kyuubiConf.get(OPERATION_QUERY_TIMEOUT))
assert(e1.getMessage.contains("must >= 1s if set"))
}
test("kyuubi conf engine.share.level.subdomain valid path test") {
val kyuubiConf = KyuubiConf()
kyuubiConf.set(ENGINE_SHARE_LEVEL_SUBDOMAIN.key, "")
assertThrows[IllegalArgumentException](kyuubiConf.get(ENGINE_SHARE_LEVEL_SUBDOMAIN))
kyuubiConf.set(ENGINE_SHARE_LEVEL_SUBDOMAIN.key, ".")
assertThrows[IllegalArgumentException](kyuubiConf.get(ENGINE_SHARE_LEVEL_SUBDOMAIN))
kyuubiConf.set(ENGINE_SHARE_LEVEL_SUBDOMAIN.key, "..")
assertThrows[IllegalArgumentException](kyuubiConf.get(ENGINE_SHARE_LEVEL_SUBDOMAIN))
kyuubiConf.set(ENGINE_SHARE_LEVEL_SUBDOMAIN.key, "/")
assertThrows[IllegalArgumentException](kyuubiConf.get(ENGINE_SHARE_LEVEL_SUBDOMAIN))
kyuubiConf.set(ENGINE_SHARE_LEVEL_SUBDOMAIN.key, "/tmp/")
assertThrows[IllegalArgumentException](kyuubiConf.get(ENGINE_SHARE_LEVEL_SUBDOMAIN))
kyuubiConf.set(ENGINE_SHARE_LEVEL_SUBDOMAIN.key, "tmp/")
assertThrows[IllegalArgumentException](kyuubiConf.get(ENGINE_SHARE_LEVEL_SUBDOMAIN))
kyuubiConf.set(ENGINE_SHARE_LEVEL_SUBDOMAIN.key, "/tmp")
assertThrows[IllegalArgumentException](kyuubiConf.get(ENGINE_SHARE_LEVEL_SUBDOMAIN))
kyuubiConf.set(ENGINE_SHARE_LEVEL_SUBDOMAIN.key, "abc/efg")
assertThrows[IllegalArgumentException](kyuubiConf.get(ENGINE_SHARE_LEVEL_SUBDOMAIN))
val path = "kyuubi!@#$%^&*()_+-=[]{};:,.<>?"
kyuubiConf.set(ENGINE_SHARE_LEVEL_SUBDOMAIN.key, path)
assert(kyuubiConf.get(ENGINE_SHARE_LEVEL_SUBDOMAIN).get == path)
}
}

2
kyuubi-server/src/test/scala/org/apache/kyuubi/engine/spark/SparkSqlEngineSuite.scala
View File

@ -72,7 +72,7 @@ class SparkSqlEngineSuite extends WithKyuubiServer with JDBCTestUtils {
test("Fail connections on invalid sub domains") {
Seq("1", ",", "", "a" * 15, "abc.xyz").foreach { invalid =>
Seq("/", "/tmp", "", "abc/efg", ".", "..").foreach { invalid =>
val sparkHiveConfigs = Map(
ENGINE_SHARE_LEVEL.key -> "USER",
ENGINE_SHARE_LEVEL_SUBDOMAIN.key -> invalid)