14e12e9aa4
### Why are the changes needed? It's a common use case that the user may want to send the service logs in a structured format to Kafka and then collect them into centralized log services for further analysis, fortunately, the Kyuubi used logging frameworks Log4j2 has built-in [KafkaAppender](https://logging.apache.org/log4j/2.x/manual/appenders/message-queue.html#KafkaAppender) and [JSON Template Layout](https://logging.apache.org/log4j/2.x/manual/json-template-layout.html), thus the goal could be achieved by just a few configurations. To simplify the user setup steps, this PR adds `log4j-layout-template-json-<version>.jar` into Kyuubi binary tarball. PS: I also plan to support sending engine bootstrap process(e.g. `spark-submit`) logs into Kafka with specific labels in the follow-up PRs. ### How was this patch tested? Manually test. Configuration in `$KYUUBI_HOME/conf/log4j2.xml` ```xml <Configuration status="INFO"> <Appenders> <Kafka name="kafka" topic="ecs-json-logs" syncSend="false"> <JsonTemplateLayout> <EventTemplateAdditionalField key="app" value="kyuubi"/> <EventTemplateAdditionalField key="cluster" value="hadoop-testing"/> <EventTemplateAdditionalField key="host" value="${hostName}"/> </JsonTemplateLayout> <Property name="bootstrap.servers" value="kafka-1:9092,kafka-2:9092,kafka-3:9092"/> <Property name="compression.type" value="gzip"/> </Kafka> </Appenders> <Loggers> <Root level="INFO"> <AppenderRef ref="kafka"/> </Root> </Loggers> </Configuration> ``` Check that Kafka receives the expected structured logging message in the Elastic Common Schema(ECS) layout.  ### Was this patch authored or co-authored using generative AI tooling? No Closes #6861 from pan3793/structured-logging. Closes #6861 9556da2a7 [Cheng Pan] Structured Logs 7dc6dda86 [Cheng Pan] Add log4j-layout-template-json Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|---|---|---|
| .. | ||
| gen | ||
| kyuubi-codecov | ||
| kyuubi-tpcds | ||
| checkout_pr.sh | ||
| dependencyList | ||
| merge_kyuubi_pr.py | ||
| reformat | ||