### Why are the changes needed?
It's a common use case that the user may want to send the service logs in a structured format to Kafka and then collect them into centralized log services for further analysis, fortunately, the Kyuubi used logging frameworks Log4j2 has built-in [KafkaAppender](https://logging.apache.org/log4j/2.x/manual/appenders/message-queue.html#KafkaAppender) and [JSON Template Layout](https://logging.apache.org/log4j/2.x/manual/json-template-layout.html), thus the goal could be achieved by just a few configurations.
To simplify the user setup steps, this PR adds `log4j-layout-template-json-<version>.jar` into Kyuubi binary tarball.
PS: I also plan to support sending engine bootstrap process(e.g. `spark-submit`) logs into Kafka with specific labels in the follow-up PRs.
### How was this patch tested?
Manually test.
Configuration in `$KYUUBI_HOME/conf/log4j2.xml`
```xml
<Configuration status="INFO">
<Appenders>
<Kafka name="kafka" topic="ecs-json-logs" syncSend="false">
<JsonTemplateLayout>
<EventTemplateAdditionalField key="app" value="kyuubi"/>
<EventTemplateAdditionalField key="cluster" value="hadoop-testing"/>
<EventTemplateAdditionalField key="host" value="${hostName}"/>
</JsonTemplateLayout>
<Property name="bootstrap.servers" value="kafka-1:9092,kafka-2:9092,kafka-3:9092"/>
<Property name="compression.type" value="gzip"/>
</Kafka>
</Appenders>
<Loggers>
<Root level="INFO">
<AppenderRef ref="kafka"/>
</Root>
</Loggers>
</Configuration>
```
Check that Kafka receives the expected structured logging message in the Elastic Common Schema(ECS) layout.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes#6861 from pan3793/structured-logging.
Closes#6861
9556da2a7 [Cheng Pan] Structured Logs
7dc6dda86 [Cheng Pan] Add log4j-layout-template-json
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
We should specify `sys:logDir` for RollingFile filePattern, otherwise the log will be rolled to kyuubi work directory.
## Types of changes 🔖
- [X] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6268 from wForget/hotfix2.
Closes#6268
b599583be [wforget] Specify logDir for RollingFile filePattern
Authored-by: wforget <643348094@qq.com>
Signed-off-by: wforget <643348094@qq.com>
# 🔍 Description
## Issue References 🔗
We meet some issue, I want to check the operation sessionHandle and the operation timeline.
I found that, it is difficult for me to check it from kyuubi log.
So, In this pr, I log the operation change log into separate file.
## Describe Your Solution 🔧
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6185 from turboFei/audit_operation.
Closes#6185
38b01921c [Wang, Fei] op audit
Lead-authored-by: Fei Wang <fwang12@ebay.com>
Co-authored-by: Wang, Fei <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
# 🔍 Description
## Issue References 🔗
This pull request fixes #
## Describe Your Solution 🔧
Add log4j2 template for kyuubi repl command.
Since I want to add rolling logs for kyuubi server, we need to add independent log4j properties for kyuubi repl command.
## Types of changes 🔖
- [ ] Bugfix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Test Plan 🧪
#### Behavior Without This Pull Request ⚰️
#### Behavior With This Pull Request 🎉
#### Related Unit Tests
---
# Checklist 📝
- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes#6196 from wForget/log4j_repl.
Closes#6196
d8d64644d [wforget] newline
10a5b9fff [wforget] remove filter
a775ee2d5 [wforget] Add log4j2 template for kyuubi repl command
Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add `%tn` to `log4j2-defaults.xml` for more informations.
This close#5425
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [X] Add screenshots for manual tests if appropriate
- [X] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes#5428 from minyk/kyuubi-5425.
Closes#5425
e42a4b286 [minyk] add thread name and id to log4j2.xml.template
705e86e49 [minyk] add thread name and id to log pattern
Authored-by: minyk <minykreva@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
The Apache Spark Community found a performance regression with log4j2. See https://github.com/apache/spark/pull/36747.
This PR to fix the performance issue on our side.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No.
Closes#5400 from ITzhangqiang/KYUUBI_5365.
Closes#5365
dbb9d8b32 [ITzhangqiang] [KYUUBI #5365] Don't use Log4j2's extended throwable conversion pattern in default logging configurations
Authored-by: ITzhangqiang <itzhangqiang@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
To provide more insights for kyuubi admin.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
Closes#5064 from turboFei/k8s_log.
Closes#5064
0c072d08e [fwang12] k8s audit
d5f2b44a5 [fwang12] save
c07306283 [fwang12] audit pod
a98234aba [fwang12] log more info
Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
### _Why are the changes needed?_
Remove `conf/log4j2.properties.template` as it has been replaced by `conf/log4j2.xml.template`
Remove `<ThresholdFilter level="FATAL"/>` in default stdout appender.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3000 from pan3793/log4j-followup.
Closes#2850
ccb9dd6e [Cheng Pan] [KYUUBI #2850][FOLLOWUP] Fix default log4j2 configuration
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This is a follow up of #2850
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#2971 from turboFei/remove_on_match_from_default.
Closes#2850
10e6067e [Fei Wang] refactor
929fde7d [Fei Wang] comments
4ad12f11 [Fei Wang] add log4j1
91168b6c [Fei Wang] fix
e5ac285e [Fei Wang] fix
efe789eb [Fei Wang] [KYUUBI #2850][FOLLOWUP] Use log4j2 xml to take replace of log4j2 properties
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
