[KYUUBI #6756] [REST] Check max file size of uploaded resource and extra resources in batch creation
# 🔍 Description ## Issue References 🔗 This pull request fixes # ## Describe Your Solution 🔧 Check the uploaded resource files when creating batch via REST API - add config `kyuubi.batch.resource.file.max.size` for resource file's max size in bytes - add config `kyuubi.batch.extra.resource.file.max.size` for each extra resource file's max size in bytes ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6756 from bowenliang123/resource-maxsize. Closes #6756 5c409c425 [Bowen Liang] nit 4b16bcfc4 [Bowen Liang] nit 743920d25 [Bowen Liang] check resource file size max size Authored-by: Bowen Liang <liangbowen@gf.com.cn> Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
This commit is contained in:
Bowen Liang
parent
fcf0405397
commit
fb65a12936
@ -77,12 +77,14 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
|
||||
|
||||
### Batch
|
||||
|
||||
| Key | Default | Meaning | Type | Since |
|
||||
|---------------------------------------------|---------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
|
||||
| kyuubi.batch.application.check.interval | PT5S | The interval to check batch job application information. | duration | 1.6.0 |
|
||||
| kyuubi.batch.application.starvation.timeout | PT3M | Threshold above which to warn batch application may be starved. | duration | 1.7.0 |
|
||||
| kyuubi.batch.conf.ignore.list || A comma-separated list of ignored keys for batch conf. If the batch conf contains any of them, the key and the corresponding value will be removed silently during batch job submission. Note that this rule is for server-side protection defined via administrators to prevent some essential configs from tampering. You can also pre-define some config for batch job submission with the prefix: kyuubi.batchConf.[batchType]. For example, you can pre-define `spark.master` for the Spark batch job with key `kyuubi.batchConf.spark.spark.master`. | set | 1.6.0 |
|
||||
| kyuubi.batch.session.idle.timeout | PT6H | Batch session idle timeout, it will be closed when it's not accessed for this duration | duration | 1.6.2 |
|
||||
| Key | Default | Meaning | Type | Since |
|
||||
|---------------------------------------------|---------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|--------|
|
||||
| kyuubi.batch.application.check.interval | PT5S | The interval to check batch job application information. | duration | 1.6.0 |
|
||||
| kyuubi.batch.application.starvation.timeout | PT3M | Threshold above which to warn batch application may be starved. | duration | 1.7.0 |
|
||||
| kyuubi.batch.conf.ignore.list || A comma-separated list of ignored keys for batch conf. If the batch conf contains any of them, the key and the corresponding value will be removed silently during batch job submission. Note that this rule is for server-side protection defined via administrators to prevent some essential configs from tampering. You can also pre-define some config for batch job submission with the prefix: kyuubi.batchConf.[batchType]. For example, you can pre-define `spark.master` for the Spark batch job with key `kyuubi.batchConf.spark.spark.master`. | set | 1.6.0 |
|
||||
| kyuubi.batch.extra.resource.file.max.size | 0 | The maximum size in bytes of each uploaded extra resource file when creating batch. 0 or negative value means no limit. | long | 1.10.0 |
|
||||
| kyuubi.batch.resource.file.max.size | 0 | The maximum size in bytes of the uploaded resource file when creating batch. 0 or negative value means no limit. | long | 1.10.0 |
|
||||
| kyuubi.batch.session.idle.timeout | PT6H | Batch session idle timeout, it will be closed when it's not accessed for this duration | duration | 1.6.2 |
|
||||
|
||||
### Credentials
|
||||
|
||||
|
||||
@ -1876,6 +1876,24 @@ object KyuubiConf {
|
||||
.booleanConf
|
||||
.createWithDefault(true)
|
||||
|
||||
val BATCH_RESOURCE_FILE_MAX_SIZE: ConfigEntry[Long] =
|
||||
buildConf("kyuubi.batch.resource.file.max.size")
|
||||
.doc("The maximum size in bytes of the uploaded resource file" +
|
||||
" when creating batch. 0 or negative value means no limit.")
|
||||
.version("1.10.0")
|
||||
.serverOnly
|
||||
.longConf
|
||||
.createWithDefault(0)
|
||||
|
||||
val BATCH_EXTRA_RESOURCE_FILE_MAX_SIZE: ConfigEntry[Long] =
|
||||
buildConf("kyuubi.batch.extra.resource.file.max.size")
|
||||
.doc("The maximum size in bytes of each uploaded extra resource file" +
|
||||
" when creating batch. 0 or negative value means no limit.")
|
||||
.version("1.10.0")
|
||||
.serverOnly
|
||||
.longConf
|
||||
.createWithDefault(0)
|
||||
|
||||
val BATCH_SUBMITTER_ENABLED: ConfigEntry[Boolean] =
|
||||
buildConf("kyuubi.batch.submitter.enabled")
|
||||
.internal
|
||||
|
||||
@ -18,7 +18,7 @@
|
||||
package org.apache.kyuubi.server.api.v1
|
||||
|
||||
import java.io.InputStream
|
||||
import java.nio.file.{Path => JPath}
|
||||
import java.nio.file.{Files, Path => JPath}
|
||||
import java.util
|
||||
import java.util.{Collections, Locale, UUID}
|
||||
import java.util.concurrent.ConcurrentHashMap
|
||||
@ -42,7 +42,7 @@ import org.apache.kyuubi.client.exception.KyuubiRestException
|
||||
import org.apache.kyuubi.client.util.BatchUtils._
|
||||
import org.apache.kyuubi.config.KyuubiConf._
|
||||
import org.apache.kyuubi.config.KyuubiReservedKeys._
|
||||
import org.apache.kyuubi.engine.{ApplicationInfo, ApplicationManagerInfo, ApplicationState, KillResponse, KyuubiApplicationManager}
|
||||
import org.apache.kyuubi.engine._
|
||||
import org.apache.kyuubi.operation.{BatchJobSubmission, FetchOrientation, OperationState}
|
||||
import org.apache.kyuubi.server.api.ApiRequestContext
|
||||
import org.apache.kyuubi.server.api.v1.BatchesResource._
|
||||
@ -65,6 +65,8 @@ private[v1] class BatchesResource extends ApiRequestContext with Logging {
|
||||
fe.getConf.get(BATCH_INTERNAL_REST_CLIENT_REQUEST_ATTEMPT_WAIT).toInt
|
||||
private lazy val internalSecurityEnabled =
|
||||
fe.getConf.get(ENGINE_SECURITY_ENABLED)
|
||||
private lazy val resourceFileMaxSize = fe.getConf.get(BATCH_RESOURCE_FILE_MAX_SIZE)
|
||||
private lazy val extraResourceFileMaxSize = fe.getConf.get(BATCH_EXTRA_RESOURCE_FILE_MAX_SIZE)
|
||||
|
||||
private def batchV2Enabled(reqConf: Map[String, String]): Boolean = {
|
||||
fe.getConf.get(BATCH_SUBMITTER_ENABLED) &&
|
||||
@ -585,6 +587,10 @@ private[v1] class BatchesResource extends ApiRequestContext with Logging {
|
||||
uploadFileFolderPath: JPath): Unit = {
|
||||
try {
|
||||
val tempFile = Utils.writeToTempFile(inputStream, uploadFileFolderPath, fileName)
|
||||
if (resourceFileMaxSize > 0 && Files.size(tempFile.toPath) > resourceFileMaxSize) {
|
||||
throw new RuntimeException(
|
||||
s"Resource file $fileName exceeds the maximum size limit $resourceFileMaxSize bytes")
|
||||
}
|
||||
fe.sessionManager.tempFileService.addPathToExpiration(tempFile.toPath)
|
||||
request.setResource(tempFile.getPath)
|
||||
} catch {
|
||||
@ -621,6 +627,12 @@ private[v1] class BatchesResource extends ApiRequestContext with Logging {
|
||||
filePart.getValueAs(classOf[InputStream]),
|
||||
uploadFileFolderPath,
|
||||
fileName)
|
||||
if (extraResourceFileMaxSize > 0
|
||||
&& Files.size(tempFile.toPath) > extraResourceFileMaxSize) {
|
||||
throw new RuntimeException(
|
||||
s"Extra resource file $fileName exceeds the maximum size limit " +
|
||||
s"$extraResourceFileMaxSize bytes")
|
||||
}
|
||||
fe.sessionManager.tempFileService.addPathToExpiration(tempFile.toPath)
|
||||
tempFile.getPath
|
||||
} catch {
|
||||
|
||||
Loading…
Reference in New Issue
Block a user