[KYUUBI #6508] Add the key-value pairs in optimizedConf to session conf

# 🔍 Description
## Issue References 🔗

This pull request fixes #6508

## Describe Your Solution 🔧

Add the key-value pairs in optimizedConf to session conf,to make the Kyuubi.env.SPARK_HOME take effect

## Types of changes 🔖

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️
When I configure the following parameters in kyuubi-env.sh
![image](https://github.com/apache/kyuubi/assets/14961757/98e8ac2b-ccdd-493c-9dff-171667b58b53)
Then submit a spark batch task through the restful interface and pass the kyuubi.engineEnv.SPARK_HOME parameter to change the default value
```curl -H "Content-Type: application/json" -X POST -d '{"batchType": "SPARK", "resource":"xxxx/spark-examples_2.12-3.3.2.jar",  "name": "Spark-PI", "conf": {"spark.master":"yarn","hive.server2.proxy.user":"XXXX","kyuubi.engineEnv.SPARK_HOME":"XXXXX/cluster114/spark","kyuubi.engineEnv.HADOOP_CONF_DIR":"XXXXX/conf"}, "args": [10],"className": "org.apache.spark.examples.SparkPi"}' http://XXXXX:XXXX/api/v1/batches```
Observe the log and find that SPARK_HOME is not set and takes effect. It still uses the value in kyuubi-env.sh.
![image](https://github.com/apache/kyuubi/assets/14961757/37a60b10-1f7f-4c69-8495-c96596d9bfb1)

#### Behavior With This Pull Request 🎉
![image](https://github.com/apache/kyuubi/assets/14961757/c86cae46-55c6-4e7d-ac1e-c04eb600d932)
After this PR, the update was successful

#### Related Unit Tests

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6510 from jiaoqingbo/6508.

Closes #6508

e89268e4b [jiaoqingbo] [KYUUBI #6508] Add the key-value pairs in optimizedConf to session conf

Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
This commit is contained in:
jiaoqingbo 2024-07-01 11:15:28 +08:00 committed by Cheng Pan
parent f66216b43c
commit 693d8a22f8
No known key found for this signature in database
GPG Key ID: 8001952629BCC75D

View File

@ -93,6 +93,10 @@ class KyuubiBatchSession(
}
}
optimizedConf.foreach {
case (key, value) => sessionConf.set(key, value)
}
override lazy val name: Option[String] =
batchName.filterNot(_.trim.isEmpty).orElse(optimizedConf.get(KyuubiConf.SESSION_NAME.key))