From 693d8a22f8a0a314a1028ec1a4095c9612e31cdf Mon Sep 17 00:00:00 2001 From: jiaoqingbo <1178404354@qq.com> Date: Mon, 1 Jul 2024 11:15:28 +0800 Subject: [PATCH] [KYUUBI #6508] Add the key-value pairs in optimizedConf to session conf MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit # :mag: Description ## Issue References 🔗 This pull request fixes #6508 ## Describe Your Solution 🔧 Add the key-value pairs in optimizedConf to session conf，to make the Kyuubi.env.SPARK_HOME take effect ## Types of changes :bookmark: - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request :coffin: When I configure the following parameters in kyuubi-env.sh ![image](https://github.com/apache/kyuubi/assets/14961757/98e8ac2b-ccdd-493c-9dff-171667b58b53) Then submit a spark batch task through the restful interface and pass the kyuubi.engineEnv.SPARK_HOME parameter to change the default value ```curl -H "Content-Type: application/json" -X POST -d '{"batchType": "SPARK", "resource":"xxxx/spark-examples_2.12-3.3.2.jar", "name": "Spark-PI", "conf": {"spark.master":"yarn","hive.server2.proxy.user":"XXXX","kyuubi.engineEnv.SPARK_HOME":"XXXXX/cluster114/spark","kyuubi.engineEnv.HADOOP_CONF_DIR":"XXXXX/conf"}, "args": [10],"className": "org.apache.spark.examples.SparkPi"}' http://XXXXX:XXXX/api/v1/batches``` Observe the log and find that SPARK_HOME is not set and takes effect. It still uses the value in kyuubi-env.sh. ![image](https://github.com/apache/kyuubi/assets/14961757/37a60b10-1f7f-4c69-8495-c96596d9bfb1) #### Behavior With This Pull Request :tada: ![image](https://github.com/apache/kyuubi/assets/14961757/c86cae46-55c6-4e7d-ac1e-c04eb600d932) After this PR, the update was successful #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6510 from jiaoqingbo/6508. Closes #6508 e89268e4b [jiaoqingbo] [KYUUBI #6508] Add the key-value pairs in optimizedConf to session conf Authored-by: jiaoqingbo <1178404354@qq.com> Signed-off-by: Cheng Pan --- .../scala/org/apache/kyuubi/session/KyuubiBatchSession.scala | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/kyuubi-server/src/main/scala/org/apache/kyuubi/session/KyuubiBatchSession.scala b/kyuubi-server/src/main/scala/org/apache/kyuubi/session/KyuubiBatchSession.scala index 4ac84c1d0..f648c39cb 100644 --- a/kyuubi-server/src/main/scala/org/apache/kyuubi/session/KyuubiBatchSession.scala +++ b/kyuubi-server/src/main/scala/org/apache/kyuubi/session/KyuubiBatchSession.scala @@ -93,6 +93,10 @@ class KyuubiBatchSession( } } + optimizedConf.foreach { + case (key, value) => sessionConf.set(key, value) + } + override lazy val name: Option[String] = batchName.filterNot(_.trim.isEmpty).orElse(optimizedConf.get(KyuubiConf.SESSION_NAME.key))