apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

[KYUUBI #3385] Set spark.kubernetes.executor.podNamePrefix if missing

Browse Source 
### _Why are the changes needed?_
In Spark, if Kubernetes executor pod is not conforming to Kubernetes naming conventions, a user gets the next error:
```
org.apache.kyuubi.KyuubiSQLException: org.apache.kyuubi.KyuubiSQLException: java.lang.IllegalArgumentException: 'kyuubi-connection-spark-sql-USER-8fc2847d-b65d-4a46-bb41-4a86385cb61c-988995839f29e15f' in spark.kubernetes.executor.podNamePrefix is invalid.
```
It can be fixed by manually setting 'spark.kubernetes.executor.podNamePrefix' in configurations, but since you cannot dynamically set a value for every user to contain their username, it will miss this essential information that helps to distinguish executors one from another. I suggest Kyuubi server adds this config automatically if it's not coming from user.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3590 from hanna-liashchuk/master.

Closes #3385

94613145 [Hanna Liashchuk] Fixed style check
4b2281ce [Hanna Liashchuk] Merged if clauses
2c52988d [hanna-liashchuk] Merge branch 'apache:master' into master
f397cdc7 [Hanna Liashchuk] Added if clause
8b91109c [Hanna Liashchuk] Added EpochMilli to the podNamePrefix
13e8fb53 [hanna-liashchuk] Merge branch 'apache:master' into master
dcef3e10 [Hanna Liashchuk] Set executor.podNamePrefix for k8s if missing

Lead-authored-by: Hanna Liashchuk <g.liashchuk@temabit.com>
Co-authored-by: hanna-liashchuk <47921651+hanna-liashchuk@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
This commit is contained in:
Hanna Liashchuk 2022-10-20 17:30:39 +08:00 committed by Cheng Pan
parent abadb05cc4
commit c454bfdcbe
No known key found for this signature in database
GPG Key ID: 8001952629BCC75D
1 changed files with 2 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala vendored
View File

@ -168,6 +168,8 @@ object SparkSQLEngine extends Logging {
if (Utils.isOnK8s) {
kyuubiConf.setIfMissing(FRONTEND_CONNECTION_URL_USE_HOSTNAME, false)
val podNamePrefix = s"kyuubi-${user}-${Instant.now().toEpochMilli}"
_sparkConf.setIfMissing("spark.kubernetes.executor.podNamePrefix", podNamePrefix)
}
// Set web ui port 0 to avoid port conflicts during non-k8s cluster mode