kyuubi/docs
Bowen Liang db57e9365d [KYUUBI #6587] Periodically expire temp files and operation logs on server to avoid memeory leak by Files.deleteOnExit
# 🔍 Description
## Issue References 🔗

-

## Describe Your Solution 🔧
Fix the memory leak on server caused by `Files.deleteOnExit`.
For long-running Kyuubi server instances, some operation log files and batch job upload files are marked for deletion at exit using `Files.deleteOnExit`. However, the `files` list within the `DeleteOnExitHook`  by `Files.deleteOnExit` method continuously accumulates file paths without being cleaned up, leading to a memory leak issue.

This PR fix this issue by:
1. introduce a new util `FileExpirationUtils` for similar use of `Files.deleteOnExit`, with exposed method for evict file path from the list to prevent accumulative path list
2. adding a service `TempFileService ` in server module, periodical clean-up the files for operation logging path, uploaded resources and etc. And it evict the paths in `TempFileCleanupUtils` instance after cleanup.
3. add the new config `kyuubi.server.tempFile.expireTime` with a default value of 7 days, to control How often to trigger a file expiration clean-up for stale files

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6587 from bowenliang123/file-expiration.

Closes #6587

e23b72e08 [liangbowen] change to P14D
acaf370e7 [liangbowen] change config name to kyuubi.server.tempFile.expireTime
6c7ddd527 [liangbowen] import
ed1e4d76f [liangbowen] comment: ConcurrentHashMap.newKeySet
fbf73ccb4 [liangbowen] update
34d3fc71c [liangbowen] add guava to common module's dep
49c10e5ef [Bowen Liang] file expiration

Lead-authored-by: Bowen Liang  <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2024-08-28 17:13:27 +08:00
..
_static/css Revert "[KYUUBI #5908] [DOCS] Remove workaround for malformed table" 2023-12-24 01:53:05 +08:00
appendix [KYUUBI #4655] [DOCS] Enrich docs for Kyuubi Hive JDBC driver 2023-04-03 18:51:27 +08:00
client [KYUUBI #6008] RESTful API supports killing engine forcibly 2024-07-01 15:28:09 +08:00
configuration [KYUUBI #6587] Periodically expire temp files and operation logs on server to avoid memeory leak by Files.deleteOnExit 2024-08-28 17:13:27 +08:00
connector [KYUUBI #6558] Bump Iceberg 1.6.0 2024-07-25 22:37:21 +08:00
contributing [KYUUBI #6545] Deprecate and remove building support for Spark 3.2 2024-07-22 11:59:34 +08:00
deployment [KYUUBI #6628] [DOCS] Improve docs for GROUP Share Level 2024-08-21 14:34:15 +08:00
extensions [KYUUBI #6618] Support http bearer token authentication for REST protocol 2024-08-16 11:06:16 +00:00
imgs [KYUUBI #5914] Update layer diagram on welcome page 2023-12-25 16:13:48 +08:00
monitor [KYUUBI #6239] Rename beeline to kyuubi-beeline 2024-04-03 18:35:38 +08:00
overview [KYUUBI #6596] Fix typos in architecture page 2024-08-08 12:12:30 +00:00
quick_start [KYUUBI #6557] Support Flink 1.20 2024-08-05 22:57:39 +08:00
security [KYUUBI #6512] Improve docs for KSHC 2024-07-01 10:51:12 +08:00
tools [KYUUBI #6242] Remove block cleaner docs 2024-04-03 13:39:34 +08:00
conf.py Revert "[KYUUBI #5908] [DOCS] Remove workaround for malformed table" 2023-12-24 01:53:05 +08:00
index.rst [KYUUBI #6068] Remove community section from user docs 2024-02-21 05:20:42 +00:00
make.bat [KYUUBI #4235] [DOCS] Prefer https:// URLs in docs 2023-02-03 14:01:11 +08:00
Makefile
requirements.txt [KYUUBI #5902] Bump doc build dependencies 2023-12-21 18:37:43 -08:00