kyuubi/docs
Cheng Pan 14e12e9aa4
[KYUUBI #6861] Configuration guide of structured logging for Kyuubi server
### Why are the changes needed?

It's a common use case that the user may want to send the service logs in a structured format to Kafka and then collect them into centralized log services for further analysis, fortunately, the Kyuubi used logging frameworks Log4j2 has built-in [KafkaAppender](https://logging.apache.org/log4j/2.x/manual/appenders/message-queue.html#KafkaAppender) and [JSON Template Layout](https://logging.apache.org/log4j/2.x/manual/json-template-layout.html), thus the goal could be achieved by just a few configurations.

To simplify the user setup steps, this PR adds `log4j-layout-template-json-<version>.jar` into Kyuubi binary tarball.

PS: I also plan to support sending engine bootstrap process(e.g. `spark-submit`) logs into Kafka with specific labels in the follow-up PRs.

### How was this patch tested?

Manually test.

Configuration in `$KYUUBI_HOME/conf/log4j2.xml`
```xml
<Configuration status="INFO">
  <Appenders>
    <Kafka name="kafka" topic="ecs-json-logs" syncSend="false">
      <JsonTemplateLayout>
        <EventTemplateAdditionalField key="app" value="kyuubi"/>
        <EventTemplateAdditionalField key="cluster" value="hadoop-testing"/>
        <EventTemplateAdditionalField key="host" value="${hostName}"/>
      </JsonTemplateLayout>
      <Property name="bootstrap.servers" value="kafka-1:9092,kafka-2:9092,kafka-3:9092"/>
      <Property name="compression.type" value="gzip"/>
    </Kafka>
  </Appenders>
  <Loggers>
    <Root level="INFO">
      <AppenderRef ref="kafka"/>
    </Root>
  </Loggers>
</Configuration>
```

Check that Kafka receives the expected structured logging message in the Elastic Common Schema(ECS) layout.
![Xnip2024-12-25_03-18-52](https://github.com/user-attachments/assets/e1b5853a-3800-4363-8ce4-7e78d0928c6a)

### Was this patch authored or co-authored using generative AI tooling?

No

Closes #6861 from pan3793/structured-logging.

Closes #6861

9556da2a7 [Cheng Pan] Structured Logs
7dc6dda86 [Cheng Pan] Add log4j-layout-template-json

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-12-25 17:22:53 +08:00
..
_static/css Revert "[KYUUBI #5908] [DOCS] Remove workaround for malformed table" 2023-12-24 01:53:05 +08:00
appendix [KYUUBI #4655] [DOCS] Enrich docs for Kyuubi Hive JDBC driver 2023-04-03 18:51:27 +08:00
client [KYUUBI #6734] [DOC] add authentication example in REST API docs 2024-10-16 13:12:00 +08:00
configuration [KYUUBI #6829] Add metrics for batch pending max elapse time 2024-12-05 18:12:39 +08:00
connector [KYUUBI #6804] Bump Iceberg from 1.6.1 to 1.7.0 2024-11-14 18:25:09 +08:00
contributing [KYUUBI #6545] Deprecate and remove building support for Spark 3.2 2024-07-22 11:59:34 +08:00
deployment [KYUUBI #6842] Bump Spark 3.5.4 2024-12-23 11:21:45 +08:00
extensions [KYUUBI #6842] Bump Spark 3.5.4 2024-12-23 11:21:45 +08:00
imgs [KYUUBI #5914] Update layer diagram on welcome page 2023-12-25 16:13:48 +08:00
monitor [KYUUBI #6861] Configuration guide of structured logging for Kyuubi server 2024-12-25 17:22:53 +08:00
overview [KYUUBI #6596] Fix typos in architecture page 2024-08-08 12:12:30 +00:00
quick_start [KYUUBI #6557] Support Flink 1.20 2024-08-05 22:57:39 +08:00
security [KYUUBI #6728] [DOC] update Authz plugin docs of build command with -am option 2024-10-16 13:31:14 +08:00
tools [KYUUBI #6242] Remove block cleaner docs 2024-04-03 13:39:34 +08:00
conf.py Revert "[KYUUBI #5908] [DOCS] Remove workaround for malformed table" 2023-12-24 01:53:05 +08:00
index.rst [KYUUBI #6068] Remove community section from user docs 2024-02-21 05:20:42 +00:00
make.bat [KYUUBI #4235] [DOCS] Prefer https:// URLs in docs 2023-02-03 14:01:11 +08:00
Makefile [KYUUBI #874] [ASF] ASF Publish 2021-08-16 11:48:21 +08:00
requirements.txt [KYUUBI #6752] [DOC] Bump doc build requirements 2024-10-18 10:39:02 +08:00