db57e9365d
# 🔍 Description ## Issue References 🔗 - ## Describe Your Solution 🔧 Fix the memory leak on server caused by `Files.deleteOnExit`. For long-running Kyuubi server instances, some operation log files and batch job upload files are marked for deletion at exit using `Files.deleteOnExit`. However, the `files` list within the `DeleteOnExitHook` by `Files.deleteOnExit` method continuously accumulates file paths without being cleaned up, leading to a memory leak issue. This PR fix this issue by: 1. introduce a new util `FileExpirationUtils` for similar use of `Files.deleteOnExit`, with exposed method for evict file path from the list to prevent accumulative path list 2. adding a service `TempFileService ` in server module, periodical clean-up the files for operation logging path, uploaded resources and etc. And it evict the paths in `TempFileCleanupUtils` instance after cleanup. 3. add the new config `kyuubi.server.tempFile.expireTime` with a default value of 7 days, to control How often to trigger a file expiration clean-up for stale files ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6587 from bowenliang123/file-expiration. Closes #6587 e23b72e08 [liangbowen] change to P14D acaf370e7 [liangbowen] change config name to kyuubi.server.tempFile.expireTime 6c7ddd527 [liangbowen] import ed1e4d76f [liangbowen] comment: ConcurrentHashMap.newKeySet fbf73ccb4 [liangbowen] update 34d3fc71c [liangbowen] add guava to common module's dep 49c10e5ef [Bowen Liang] file expiration Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn> Co-authored-by: liangbowen <liangbowen@gf.com.cn> Co-authored-by: Bowen Liang <liangbowen@gf.com.cn> Signed-off-by: liangbowen <liangbowen@gf.com.cn>
90 lines
1.9 KiB
Plaintext
90 lines
1.9 KiB
Plaintext
#
|
|
# Licensed to the Apache Software Foundation (ASF) under one or more
|
|
# contributor license agreements. See the NOTICE file distributed with
|
|
# this work for additional information regarding copyright ownership.
|
|
# The ASF licenses this file to You under the Apache License, Version 2.0
|
|
# (the "License"); you may not use this file except in compliance with
|
|
# the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
#
|
|
|
|
*#*#
|
|
*.#*
|
|
*.db
|
|
*.iml
|
|
*.ipr
|
|
*.iws
|
|
*.pyc
|
|
*.pyo
|
|
*.swp
|
|
*~
|
|
.DS_Store
|
|
.cache
|
|
.classpath
|
|
.ensime
|
|
.ensime_cache/
|
|
.ensime_lucene
|
|
.generated-mima*
|
|
.vscode/
|
|
.idea/
|
|
.idea_modules/
|
|
.project
|
|
.pydevproject
|
|
.python-version
|
|
.scala_dependencies
|
|
.settings
|
|
build/apache-maven*
|
|
build/release/tmp
|
|
build/release/*.txt
|
|
build/scala*
|
|
build/test
|
|
target/
|
|
dist/
|
|
apache-kyuubi-*-bin*
|
|
*.gz
|
|
logs/
|
|
pid/
|
|
local/
|
|
out/
|
|
hs_err_pid*
|
|
spark-warehouse/
|
|
metastore_db
|
|
derby.log
|
|
rest-audit.log
|
|
**/.flattened-pom.xml
|
|
**/dependency-reduced-pom.xml
|
|
metrics/
|
|
/kyuubi-ha/embedded_zookeeper/
|
|
embedded_zookeeper/
|
|
/externals/kyuubi-spark-sql-engine/operation_logs/
|
|
/externals/kyuubi-spark-sql-engine/engine_operation_logs/
|
|
/externals/kyuubi-spark-sql-engine/spark-warehouse/
|
|
/work/
|
|
/upload/
|
|
/docs/_build/
|
|
/kyuubi-common/metrics/
|
|
/kyuubi-server/metrics/
|
|
**/operation_logs/
|
|
**/server_operation_logs/
|
|
**/engine_operation_logs/
|
|
conf/log4j2.xml
|
|
conf/kyuubi-defaults.conf
|
|
conf/kyuubi-env.sh
|
|
.java-version
|
|
|
|
# For Antlr
|
|
kyuubi-server/gen
|
|
kyuubi-server/src/main/antlr4/org/apache/kyuubi/sql/*.tokens
|
|
kyuubi-server/src/main/antlr4/org/apache/kyuubi/sql/gen/
|
|
|
|
# For draw.io
|
|
.$*.bkp
|
|
.$*.dtmp
|