liangbowen 69d625a1be [KYUUBI #4200 ] [Improvement] [Docs] Introduce Markdown formatting with spotless-maven-plugin and flexmark for docs

### _Why are the changes needed?_

- to consolidate styles in markdown files from manual written or auto-generated
- apply markdown formatting rules with flexmark from [spotless-maven-plugin](https://github.com/diffplug/spotless/tree/main/plugin-maven#markdown) to *.md files in `/docs`
- use `flexmark` to format markdown generation in `TestUtils` of common module used by `AllKyuubiConfiguration` and `KyuubiDefinedFunctionSuite`, as the same way in `FlexmarkFormatterFunc ` of `spotless-maven-plugin` using with `COMMONMARK` as `FORMATTER_EMULATION_PROFILE` (https://github.com/diffplug/spotless/blob/maven/2.30.0/lib/src/flexmark/java/com/diffplug/spotless/glue/markdown/FlexmarkFormatterFunc.java)
- using `flexmark` of` 0.62.2`, as the last version requiring Java 8+ (checked from pom file and bytecode version)

```
<markdown>
    <includes>
        <include>docs/**/*.md</include>
    </includes>
    <flexmark></flexmark>
</markdown>
```

- Changes applied to markdown doc files,
  -  no style change or breakings in built docs by `make html`
  - removal all the first blank in licences and comments to conform markdown style rules
  - tables regenerated by flexmark following as in [GitHub Flavored Markdown](https://help.github.com/articles/organizing-information-with-tables/) (https://github.com/vsch/flexmark-java/wiki/Extensions#tables)

### _How was this patch tested?_
- [x] regenerate docs using `make html` successfully and check all the markdown pages available
- [x] regenerate `settings.md` and `functions.md` by `AllKyuubiConfiguration` and `KyuubiDefinedFunctionSuite`, and pass the checks by both themselves and spotless check via `dev/reformat`
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4200 from bowenliang123/markdown-formatting.

Closes #4200

1eeafce4 [liangbowen] revert minor changes in AllKyuubiConfiguration
4f892857 [liangbowen] use flexmark in markdown doc generation
8c978abd [liangbowen] changes on markdown files
a9190556 [liangbowen] apply markdown formatting rules with `spotless-maven-plugin` to markdown files with in `/docs`

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>

2023-01-30 11:14:41 +08:00

8.6 KiB

Raw Blame History

Hadoop Credentials Manager

In order to pass the authentication of a kerberos secured hadoop cluster, kyuubi currently submits engines in two ways:

Submits with current kerberos user and extra SparkSubmit argument --proxy-user.
Submits with spark.kerberos.principal and spark.kerberos.keytab specified.

If engine is submitted with --proxy-user specified, its delegation tokens of hadoop cluster services are obtained by current kerberos user and can not be renewed by itself.
Thus, engine's lifetime is limited by the lifetime of delegation tokens.
To remove this limitation, kyuubi renews delegation tokens at server side in Hadoop Credentials Manager.

Engine submitted with principal and keytab can renew delegation tokens by itself. But for implementation simplicity, kyuubi server will also renew delegation tokens for it.

Configurations

Cluster Services

Kyuubi currently supports renew delegation tokens of Hadoop filesystems and Hive metastore servers.

Hadoop client configurations

Set HADOOP_CONF_DIR in $KYUUBI_HOME/conf/kyuubi-env.sh if it hasn't been set yet, e.g.

$ echo "export HADOOP_CONF_DIR=/path/to/hadoop/conf" >> $KYUUBI_HOME/conf/kyuubi-env.sh

Extra Hadoop filesystems can be specified in $KYUUBI_HOME/conf/kyuubi-defaults.conf by kyuubi.credentials.hadoopfs.uris in comma separated list.

Hive metastore configurations

Via kyuubi-defaults.conf

Specify Hive metastore configurations In $KYUUBI_HOME/conf/kyuubi-defaults.conf. Hadoop Credentials Manager will load the configurations when initialized.

Via hive-site.xml

Place your copy of hive-site.xml into $KYUUBI_HOME/conf, Kyuubi will load this config file to its classpath.

This version of configuration has lower priority than those in $KYUUBI_HOME/conf/kyuubi-defaults.conf.

Via JDBC Connection URL

Hive configurations specified in JDBC connection URL are ignored by Hadoop Credentials Manager as Hadoop Credentials Manager is initialized when Kyuubi server starts.

Credentials Renewal

Key	Default	Meaning	Type	Since
`kyuubi.credentials.hadoopfs.enabled`	true	Whether to renew Hadoop filesystem delegation tokens	boolean	1.4.0
`kyuubi.credentials.hadoopfs.uris`		Extra Hadoop filesystem URIs for which to request delegation tokens. The filesystem that hosts fs.defaultFS does not need to be listed here.	seq	1.4.0
`kyuubi.credentials.hive.enabled`	true	Whether to renew Hive metastore delegation token	boolean	1.4.0
`kyuubi.credentials.renewal.interval`	PT1H	How often Kyuubi renews one user's delegation tokens	duration	1.4.0
`kyuubi.credentials.renewal.retry.wait`	PT1M	How long to wait before retrying to fetch new credentials after a failure.	duration	1.4.0

Required Security Configs

The necessary configurations for hdfs and hive to obtain delegation token are as follows:

Key	Meaning	value
`hadoop.security.authentication`	Set the authentication for the cluster	kerberos
`hive.metastore.uris`	URI for client to contact metastore server	thrift://{metastoreHost}:{metastorePort}}
`hive.metastore.sasl.enabled`	If true, the metastore thrift interface will be secured with SASL.Clients must authenticate with Kerberos.	true
`hive.metastore.kerberos.principal`	The service principal for the metastore thrift server. The special string _HOST will be replaced automatically with the correct host name.	for example hive/_HOST@${realm}
`hive.metastore.kerberos.keytab.file`	The path to the Kerberos Keytab file containing the metastore thrift server's service principal.	for example /etc/security/keytabs/hive.service.keytab

8.6 KiB Raw Blame History