kyuubi/conf
Cheng Pan 14e12e9aa4
[KYUUBI #6861] Configuration guide of structured logging for Kyuubi server
### Why are the changes needed?

It's a common use case that the user may want to send the service logs in a structured format to Kafka and then collect them into centralized log services for further analysis, fortunately, the Kyuubi used logging frameworks Log4j2 has built-in [KafkaAppender](https://logging.apache.org/log4j/2.x/manual/appenders/message-queue.html#KafkaAppender) and [JSON Template Layout](https://logging.apache.org/log4j/2.x/manual/json-template-layout.html), thus the goal could be achieved by just a few configurations.

To simplify the user setup steps, this PR adds `log4j-layout-template-json-<version>.jar` into Kyuubi binary tarball.

PS: I also plan to support sending engine bootstrap process(e.g. `spark-submit`) logs into Kafka with specific labels in the follow-up PRs.

### How was this patch tested?

Manually test.

Configuration in `$KYUUBI_HOME/conf/log4j2.xml`
```xml
<Configuration status="INFO">
  <Appenders>
    <Kafka name="kafka" topic="ecs-json-logs" syncSend="false">
      <JsonTemplateLayout>
        <EventTemplateAdditionalField key="app" value="kyuubi"/>
        <EventTemplateAdditionalField key="cluster" value="hadoop-testing"/>
        <EventTemplateAdditionalField key="host" value="${hostName}"/>
      </JsonTemplateLayout>
      <Property name="bootstrap.servers" value="kafka-1:9092,kafka-2:9092,kafka-3:9092"/>
      <Property name="compression.type" value="gzip"/>
    </Kafka>
  </Appenders>
  <Loggers>
    <Root level="INFO">
      <AppenderRef ref="kafka"/>
    </Root>
  </Loggers>
</Configuration>
```

Check that Kafka receives the expected structured logging message in the Elastic Common Schema(ECS) layout.
![Xnip2024-12-25_03-18-52](https://github.com/user-attachments/assets/e1b5853a-3800-4363-8ce4-7e78d0928c6a)

### Was this patch authored or co-authored using generative AI tooling?

No

Closes #6861 from pan3793/structured-logging.

Closes #6861

9556da2a7 [Cheng Pan] Structured Logs
7dc6dda86 [Cheng Pan] Add log4j-layout-template-json

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-12-25 17:22:53 +08:00
..
kyuubi-defaults.conf.template [KYUUBI #5154] [Doc] Move configuration docs to the top level 2023-08-11 18:23:08 +08:00
kyuubi-env.sh.template [KYUUBI #5729] Use G1GC as Java option example in kyuubi-env template 2023-11-22 15:44:14 +08:00
log4j2-repl.xml.template [KYUUBI #6196] Add log4j2 template for kyuubi repl commands 2024-03-19 14:35:53 +08:00
log4j2.xml.template [KYUUBI #6861] Configuration guide of structured logging for Kyuubi server 2024-12-25 17:22:53 +08:00