apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

[KYUUBI #5680] Optimize Spark engine pod name generation

Browse Source 
### _Why are the changes needed?_

Close #5680

**Background**:
1) In default case, when kyuubi submit spark sql engine on kubernetes, will generate spark driver pod name with "kyuubi-${app_name}-${engine_ref_id}-driver".
2) And app_name will be "kyuubi_${shareLevel}_${engineType}_${appUser}_${engineRefId}" if not set by user.
3) In result, we may get spark driver pod name with "kyuubi-kyuubi-${shareLevel}-${engineType}-${appUser}-${engineRefId}-${engineRefId}-driver".
4) We were hoping for a more concise and readable name, such as "kyuubi-${shareLevel}-${engineType}-${appUser}-${app_name}-${engineRefId}-driver".

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate
**1) Before modification：**
Left is unset `spark.app.name`, and right is set to “ocean”
![image](https://github.com/apache/kyuubi/assets/45907917/350aa09f-a9cb-4cc4-bdc6-71b085ba0824)

**2) Modify the code**
![image](https://github.com/apache/kyuubi/assets/45907917/f9cbef91-9e1a-482a-82bb-a78caf669438)

**3) Build module and get the jar**
![image](https://github.com/apache/kyuubi/assets/45907917/b2cf7595-dd54-46eb-bb79-354f3301499f)

**4) Build a new images**
![image](https://github.com/apache/kyuubi/assets/45907917/c0418f2d-dcde-4845-a568-84e920892359)

**5) After modification:**
![image](https://github.com/apache/kyuubi/assets/45907917/5db95d5b-2550-41b7-a11c-d8e4a6c0f09f)

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No

Closes #5695 from Ocean22/master.

Closes #5680

6dbac57b3 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/util/KubernetesUtils.scala
5ddc9f6ad [no 会 English] Update KubernetesUtils.scala
b685b60aa [no 会 English] Update KubernetesUtils.scala
6a64e17ea [no 会 English] Update KubernetesUtils.scala
1335bbd21 [no 会 English] Update EngineRef.scala
f2af95535 [no 会 English] Update KubernetesUtils.scala
891f1720d [no 会 English] Update KubernetesUtils.scala
c20b75549 [no 会 English] Update EngineRef.scala
09f3ed190 [no 会 English] Update EngineRef.scala
64bd2c156 [no 会 English] Update EngineRef.scala

Lead-authored-by: Ocean22 <1058853680@qq.com>
Co-authored-by: no 会 English <45907917+Ocean22@users.noreply.github.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
This commit is contained in:
Ocean22 2023-12-04 12:33:09 +08:00 committed by Cheng Pan
parent 2a39d697c0
commit 25eb53dcdf
No known key found for this signature in database
GPG Key ID: 8001952629BCC75D
1 changed files with 22 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

26
kyuubi-server/src/main/scala/org/apache/kyuubi/util/KubernetesUtils.scala
View File

@ -120,8 +120,8 @@ object KubernetesUtils extends Logging {
opt2.foreach { _ => require(opt1.isEmpty, errMessage) }
}
private def getResourceNamePrefix(appName: String, engineRefId: String): String = {
s"$appName-$engineRefId"
private def getResourceNamePrefix(appName: String, engineRefId: Option[String]): String = {
engineRefId.map(refId => s"$appName-$refId").getOrElse(appName)
.trim
.toLowerCase(Locale.ROOT)
.replaceAll("[^a-z0-9\\-]", "-")
@ -134,7 +134,16 @@ object KubernetesUtils extends Logging {
appName: String,
engineRefId: String,
forciblyRewrite: Boolean): String = {
lazy val resolvedResourceName = s"kyuubi-${getResourceNamePrefix(appName, engineRefId)}-driver"
val resourceNamePrefix = if (appName.contains(engineRefId)) {
getResourceNamePrefix(appName, None)
} else {
getResourceNamePrefix(appName, Some(engineRefId))
}
val resolvedResourceName = if (resourceNamePrefix.startsWith("kyuubi-")) {
s"$resourceNamePrefix-driver"
} else {
s"kyuubi-$resourceNamePrefix-driver"
}
if (forciblyRewrite || resolvedResourceName.length > DRIVER_POD_NAME_MAX_LENGTH) {
s"kyuubi-$engineRefId-driver"
} else {
@ -146,7 +155,16 @@ object KubernetesUtils extends Logging {
appName: String,
engineRefId: String,
forciblyRewrite: Boolean): String = {
val resolvedResourceName = s"kyuubi-${getResourceNamePrefix(appName, engineRefId)}"
val resourceNamePrefix = if (appName.contains(engineRefId)) {
getResourceNamePrefix(appName, None)
} else {
getResourceNamePrefix(appName, Some(engineRefId))
}
val resolvedResourceName = if (resourceNamePrefix.startsWith("kyuubi-")) {
s"$resourceNamePrefix"
} else {
s"kyuubi-$resourceNamePrefix"
}
if (forciblyRewrite || resolvedResourceName.length > EXECUTOR_POD_NAME_PREFIX_MAX_LENGTH) {
s"kyuubi-$engineRefId"
} else {