kyuubi/kyuubi-common
Bowen Liang db57e9365d [KYUUBI #6587] Periodically expire temp files and operation logs on server to avoid memeory leak by Files.deleteOnExit
# 🔍 Description
## Issue References 🔗

-

## Describe Your Solution 🔧
Fix the memory leak on server caused by `Files.deleteOnExit`.
For long-running Kyuubi server instances, some operation log files and batch job upload files are marked for deletion at exit using `Files.deleteOnExit`. However, the `files` list within the `DeleteOnExitHook`  by `Files.deleteOnExit` method continuously accumulates file paths without being cleaned up, leading to a memory leak issue.

This PR fix this issue by:
1. introduce a new util `FileExpirationUtils` for similar use of `Files.deleteOnExit`, with exposed method for evict file path from the list to prevent accumulative path list
2. adding a service `TempFileService ` in server module, periodical clean-up the files for operation logging path, uploaded resources and etc. And it evict the paths in `TempFileCleanupUtils` instance after cleanup.
3. add the new config `kyuubi.server.tempFile.expireTime` with a default value of 7 days, to control How often to trigger a file expiration clean-up for stale files

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6587 from bowenliang123/file-expiration.

Closes #6587

e23b72e08 [liangbowen] change to P14D
acaf370e7 [liangbowen] change config name to kyuubi.server.tempFile.expireTime
6c7ddd527 [liangbowen] import
ed1e4d76f [liangbowen] comment: ConcurrentHashMap.newKeySet
fbf73ccb4 [liangbowen] update
34d3fc71c [liangbowen] add guava to common module's dep
49c10e5ef [Bowen Liang] file expiration

Lead-authored-by: Bowen Liang  <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2024-08-28 17:13:27 +08:00
..
src [KYUUBI #6587] Periodically expire temp files and operation logs on server to avoid memeory leak by Files.deleteOnExit 2024-08-28 17:13:27 +08:00
pom.xml [KYUUBI #6587] Periodically expire temp files and operation logs on server to avoid memeory leak by Files.deleteOnExit 2024-08-28 17:13:27 +08:00