[KYUUBI #839]Fix load user specific defaults from properties file
Some checks failed
Kyuubi / Check License (push) Has been cancelled
Kyuubi / Build () (push) Has been cancelled
Kyuubi / Build (-DwildcardSuites=org.apache.kyuubi.operation.tpcds.TPCDSOutputSchemaSuite,org.apache.kyuubi.operation.tpcds.TPCDSDDLSuite -Dmaven.plugin.scalatest.exclude.tags="") (push) Has been cancelled
Kyuubi / Build (-Pspark-3.0 -Dspark.archive.mirror=https://archive.apache.org/dist/spark/spark-3.1.2 -Dspark.archive.name=spark-3.1.2-bin-hadoop2.7.tgz -Dmaven.plugin.scalatest.exclude.tags=org.apache.kyuubi.tags.ExtendedSQLTest,org.apache.kyuubi.tags.DataLakeT… (push) Has been cancelled
Kyuubi / Build (-Pspark-3.1 -Pkyuubi-extension-spark_3.1 -Pspark-hadoop-3.2) (push) Has been cancelled
Kyuubi / Build (-Pspark-3.1 -Pkyuubi-extension-spark_3.1) (push) Has been cancelled
Kyuubi / Build (-Pspark-3.2-snapshot -pl :kyuubi-spark-sql-engine,:kyuubi-common,:kyuubi-ha,:kyuubi-zookeeper -Dmaven.plugin.scalatest.exclude.tags=org.apache.kyuubi.tags.ExtendedSQLTest,org.apache.kyuubi.tags.DataLakeTest) (push) Has been cancelled
SL Scan / Scan-Build (push) Has been cancelled

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
loadFromMap ignored user specific defaults that start with '___'.

### _How was this patch tested?_
- [X] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #840 from hddong/kyuubi-839.

Closes #839

ba3683e3 [hongdongdong] fix comments
a7753849 [hongdongdong] fix test
07f82667 [hongdongdong] fix test
b2a4c3cb [hongdongdong] [KYUUBI#839]Fix load user specific defaults from properties file

Authored-by: hongdongdong <hongdongdong@cmss.chinamobile.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
(cherry picked from commit 8cef5c809a)
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
This commit is contained in:
hongdongdong 2021-07-21 12:13:46 +08:00 committed by ulysses-you
parent eb27abc9f2
commit e0b8495de0
3 changed files with 10 additions and 11 deletions

View File

@ -39,7 +39,9 @@ case class KyuubiConf(loadSysDefault: Boolean = true) extends Logging {
}
private def loadFromMap(props: Map[String, String] = Utils.getSystemProperties): KyuubiConf = {
for ((key, value) <- props if key.startsWith("kyuubi.") || key.startsWith("spark.")) {
for ((key, value) <- props if key.startsWith("kyuubi.") || key.startsWith("spark.") ||
// for user specific defaults
key.startsWith("___")) {
set(key, value)
}
this

View File

@ -18,3 +18,7 @@
kyuubi.yes yes
spark.kyuubi.yes no
# kyuubi.no no
spark.user.test a
___userb___.spark.user.test b
___userc___.spark.user.test c

View File

@ -97,17 +97,10 @@ class KyuubiConfSuite extends KyuubiFunSuite {
test("get user specific defaults") {
val conf = KyuubiConf(false)
.set("spark.user.test", "a")
.set("___userb___.spark.user.test", "b")
.set("___userc___.spark.user.test", "c")
val conf = KyuubiConf().loadFileDefaults()
val all1 = conf.getUserDefaults("kyuubi").getAll
assert(all1.size === 1)
assert(all1("spark.user.test") === "a")
val all2 = conf.getUserDefaults("userb").getAll
assert(all2.size === 1)
assert(all2("spark.user.test") === "b")
assert(conf.getUserDefaults("kyuubi").getOption("spark.user.test").get === "a")
assert(conf.getUserDefaults("userb").getOption("spark.user.test").get === "b")
assert(conf.getUserDefaults("userc").getOption("spark.user.test").get === "c")
}