apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

[KYUUBI #4312] [DOCS] Include **/README.md in markdown style check

Browse Source 
### _Why are the changes needed?_

- Include `**/README.md` markdown files in spotless style check

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4312 from bowenliang123/reformat-readme.

Closes #4312

1fda1bdeb [Bowen Liang] Merge branch 'master' into reformat-readme
2ca8b4c81 [liangbowen] merge master
876f52a4c [liangbowen] include `**/README.md` in spotless style check

Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
This commit is contained in:
liangbowen 2023-02-14 02:23:32 +08:00 committed by Cheng Pan
parent 10adbdc185
commit bb3e06a035
No known key found for this signature in database
GPG Key ID: 8001952629BCC75D
10 changed files with 101 additions and 103 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

41
README.md
View File

@ -1,19 +1,19 @@
<!-- <!--
- Licensed to the Apache Software Foundation (ASF) under one or more - Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with - contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership. - this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0 - The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with - (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at - the License. You may obtain a copy of the License at
- -
- http://www.apache.org/licenses/LICENSE-2.0 - http://www.apache.org/licenses/LICENSE-2.0
- -
- Unless required by applicable law or agreed to in writing, software - Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS, - distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and - See the License for the specific language governing permissions and
- limitations under the License. - limitations under the License.
--> -->
<p align="center"> <p align="center">
<img src="https://svn.apache.org/repos/asf/comdev/project-logos/originals/kyuubi-1.svg" alt="Kyuubi logo" height="120px"/> <img src="https://svn.apache.org/repos/asf/comdev/project-logos/originals/kyuubi-1.svg" alt="Kyuubi logo" height="120px"/>
@ -45,6 +45,7 @@
</p> </p>
# Apache Kyuubi # Apache Kyuubi
Apache Kyuubi™ is a distributed and multi-tenant gateway to provide serverless Apache Kyuubi™ is a distributed and multi-tenant gateway to provide serverless
SQL on data warehouses and lakehouses. SQL on data warehouses and lakehouses.
@ -60,12 +61,10 @@ Kyuubi provides a pure SQL gateway through Thrift JDBC/ODBC interface for end-us
- [x] Multi-tenant Spark Support - [x] Multi-tenant Spark Support
- [x] Running Spark in a serverless way - [x] Running Spark in a serverless way
### Target Users ### Target Users
Kyuubi's goal is to make it easy and efficient for `anyone` to use Spark(maybe other engines soon) and facilitate users to handle big data like ordinary data. Here, `anyone` means that users do not need to have a Spark technical background but a human language, SQL only. Sometimes, SQL skills are unnecessary when integrating Kyuubi with Apache Superset, which supports rich visualizations and dashboards. Kyuubi's goal is to make it easy and efficient for `anyone` to use Spark(maybe other engines soon) and facilitate users to handle big data like ordinary data. Here, `anyone` means that users do not need to have a Spark technical background but a human language, SQL only. Sometimes, SQL skills are unnecessary when integrating Kyuubi with Apache Superset, which supports rich visualizations and dashboards.
In typical big data production environments with Kyuubi, there should be system administrators and end-users. In typical big data production environments with Kyuubi, there should be system administrators and end-users.
- System administrators: A small group consists of Spark experts responsible for Kyuubi deployment, configuration, and tuning. - System administrators: A small group consists of Spark experts responsible for Kyuubi deployment, configuration, and tuning.
@ -73,7 +72,6 @@ In typical big data production environments with Kyuubi, there should be system
Additionally, the Kyuubi community will continuously optimize the whole system with various features, such as History-Based Optimizer, Auto-tuning, Materialized View, SQL Dialects, Functions, e.t.c. Additionally, the Kyuubi community will continuously optimize the whole system with various features, such as History-Based Optimizer, Auto-tuning, Materialized View, SQL Dialects, Functions, e.t.c.
### Usage scenarios ### Usage scenarios
#### Port workloads from HiveServer2 to Spark SQL #### Port workloads from HiveServer2 to Spark SQL
@ -86,7 +84,6 @@ HiveServer2 can identify and authenticate a caller, and then if the caller also
Kyuubi extends the use of STS in a multi-tenant model based on a unified interface and relies on the concept of multi-tenancy to interact with cluster managers to finally gain the ability of resources sharing/isolation and data security. The loosely coupled architecture of the Kyuubi server and engine dramatically improves the client concurrency and service stability of the service itself. Kyuubi extends the use of STS in a multi-tenant model based on a unified interface and relies on the concept of multi-tenancy to interact with cluster managers to finally gain the ability of resources sharing/isolation and data security. The loosely coupled architecture of the Kyuubi server and engine dramatically improves the client concurrency and service stability of the service itself.
#### DataLake/LakeHouse Support #### DataLake/LakeHouse Support
The vision of Kyuubi is to unify the portal and become an easy-to-use data lake management platform. Different kinds of workloads, such as ETL processing and BI analytics, can be supported by one platform, using one copy of data, with one SQL interface. The vision of Kyuubi is to unify the portal and become an easy-to-use data lake management platform. Different kinds of workloads, such as ETL processing and BI analytics, can be supported by one platform, using one copy of data, with one SQL interface.
@ -95,25 +92,19 @@ The vision of Kyuubi is to unify the portal and become an easy-to-use data lake
- Multiple Catalogs support - Multiple Catalogs support
- SQL Standard Authorization support for DataLake(coming) - SQL Standard Authorization support for DataLake(coming)
#### Cloud Native Support #### Cloud Native Support
Kyuubi can deploy its engines on different kinds of Cluster Managers, such as, Hadoop YARN, Kubernetes, etc. Kyuubi can deploy its engines on different kinds of Cluster Managers, such as, Hadoop YARN, Kubernetes, etc.
![](./docs/imgs/kyuubi_migrating_yarn_to_k8s.png) ![](./docs/imgs/kyuubi_migrating_yarn_to_k8s.png)
### The Kyuubi Ecosystem(present and future) ### The Kyuubi Ecosystem(present and future)
The figure below shows our vision for the Kyuubi Ecosystem. Some of them have been realized, some in development, The figure below shows our vision for the Kyuubi Ecosystem. Some of them have been realized, some in development,
and others would not be possible without your help. and others would not be possible without your help.
![](./docs/imgs/kyuubi_ecosystem.drawio.png) ![](./docs/imgs/kyuubi_ecosystem.drawio.png)
## Online Documentation ## Online Documentation
Since Kyuubi 1.3.0-incubating, the Kyuubi online documentation is hosted by [https://kyuubi.apache.org/](https://kyuubi.apache.org/). Since Kyuubi 1.3.0-incubating, the Kyuubi online documentation is hosted by [https://kyuubi.apache.org/](https://kyuubi.apache.org/).

53
dev/kyuubi-tpcds/README.md
View File

@ -1,21 +1,22 @@
<!-- <!--
- Licensed to the Apache Software Foundation (ASF) under one or more - Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with - contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership. - this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0 - The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with - (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at - the License. You may obtain a copy of the License at
- -
- http://www.apache.org/licenses/LICENSE-2.0 - http://www.apache.org/licenses/LICENSE-2.0
- -
- Unless required by applicable law or agreed to in writing, software - Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS, - distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and - See the License for the specific language governing permissions and
- limitations under the License. - limitations under the License.
--> -->
# Introduction # Introduction
This module includes TPC-DS data generator and benchmark tool. This module includes TPC-DS data generator and benchmark tool.
# How to use # How to use
@ -27,12 +28,12 @@ package jar with following command:
Support options: Support options:
| key | default | description | | key | default | description |
|--------------|-----------------|-----------------------------------| |-------------|-----------------|-----------------------------------|
| db | default | the database to write data | | db | default | the database to write data |
| scaleFactor | 1 | the scale factor of TPC-DS | | scaleFactor | 1 | the scale factor of TPC-DS |
| format | parquet | the format of table to store data | | format | parquet | the format of table to store data |
| parallel | scaleFactor * 2 | the parallelism of Spark job | | parallel | scaleFactor * 2 | the parallelism of Spark job |
Example: the following command to generate 10GB data with new database `tpcds_sf10`. Example: the following command to generate 10GB data with new database `tpcds_sf10`.
@ -47,7 +48,7 @@ $SPARK_HOME/bin/spark-submit \
Support options: Support options:
| key | default | description | | key | default | description |
|-------------|------------------------|---------------------------------------------------------------| |-------------|------------------------|---------------------------------------------------------------|
| db | none(required) | the TPC-DS database | | db | none(required) | the TPC-DS database |
| benchmark | tpcds-v2.4-benchmark | the name of application | | benchmark | tpcds-v2.4-benchmark | the name of application |
@ -65,6 +66,7 @@ $SPARK_HOME/bin/spark-submit \
``` ```
We also support run one of the TPC-DS query: We also support run one of the TPC-DS query:
```shell ```shell
$SPARK_HOME/bin/spark-submit \ $SPARK_HOME/bin/spark-submit \
--class org.apache.kyuubi.tpcds.benchmark.RunBenchmark \ --class org.apache.kyuubi.tpcds.benchmark.RunBenchmark \
@ -73,6 +75,7 @@ $SPARK_HOME/bin/spark-submit \
The result of TPC-DS benchmark like: The result of TPC-DS benchmark like:
| name | minTimeMs | maxTimeMs | avgTimeMs | stdDev | stdDevPercent | | name | minTimeMs | maxTimeMs | avgTimeMs | stdDev | stdDevPercent |
|---------|-----------|-------------|------------|----------|----------------| |---------|-----------|------------|------------|----------|----------------|
| q1-v2.4 | 50.522384 | 868.010383 | 323.398267 | 471.6482 | 145.8413108576 | | q1-v2.4 | 50.522384 | 868.010383 | 323.398267 | 471.6482 | 145.8413108576 |

3
docker/playground/README.md
View File

@ -1,5 +1,5 @@
Playground Playground
=== ==========
## For Users ## For Users
@ -45,3 +45,4 @@ Kyuubi supply some built-in dataset, after Kyuubi started, you can run the follo
1. Build images `docker/playground/build-image.sh`; 1. Build images `docker/playground/build-image.sh`;
2. Optional to use `buildx` to build and publish cross-platform images `BUILDX=1 docker/playground/build-image.sh`; 2. Optional to use `buildx` to build and publish cross-platform images `BUILDX=1 docker/playground/build-image.sh`;

36
extensions/README.md
View File

@ -1,25 +1,24 @@
<!-- <!--
- Licensed to the Apache Software Foundation (ASF) under one or more - Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with - contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership. - this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0 - The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with - (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at - the License. You may obtain a copy of the License at
- -
- http://www.apache.org/licenses/LICENSE-2.0 - http://www.apache.org/licenses/LICENSE-2.0
- -
- Unless required by applicable law or agreed to in writing, software - Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS, - distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and - See the License for the specific language governing permissions and
- limitations under the License. - limitations under the License.
--> -->
# For developers # For developers
This folder contains plugins/extension for kyuubi server and different engine types. This folder contains plugins/extension for kyuubi server and different engine types.
- ext - ext
- kyuubi-server - kyuubi-server
- spark - spark
@ -27,4 +26,5 @@ This folder contains plugins/extension for kyuubi server and different engine ty
- trino - trino
- hive - hive
- others - others
- ... - ...

32
extensions/spark/kyuubi-spark-authz/README.md
View File

@ -1,19 +1,19 @@
<!-- <!--
- Licensed to the Apache Software Foundation (ASF) under one or more - Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with - contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership. - this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0 - The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with - (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at - the License. You may obtain a copy of the License at
- -
- http://www.apache.org/licenses/LICENSE-2.0 - http://www.apache.org/licenses/LICENSE-2.0
- -
- Unless required by applicable law or agreed to in writing, software - Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS, - distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and - See the License for the specific language governing permissions and
- limitations under the License. - limitations under the License.
--> -->
# Kyuubi Spark AuthZ Extension # Kyuubi Spark AuthZ Extension
@ -29,7 +29,6 @@
build/mvn clean package -pl :kyuubi-spark-authz_2.12 -Dspark.version=3.2.1 -Dranger.version=2.3.0 build/mvn clean package -pl :kyuubi-spark-authz_2.12 -Dspark.version=3.2.1 -Dranger.version=2.3.0
``` ```
### Supported Apache Spark Versions ### Supported Apache Spark Versions
`-Dspark.version=` `-Dspark.version=`
@ -54,3 +53,4 @@ build/mvn clean package -pl :kyuubi-spark-authz_2.12 -Dspark.version=3.2.1 -Dran
- [x] 1.0.x - [x] 1.0.x
- [x] 0.7.x - [x] 0.7.x
- [x] 0.6.x - [x] 0.6.x

33
extensions/spark/kyuubi-spark-lineage/README.md
View File

@ -1,26 +1,26 @@
<!-- <!--
- Licensed to the Apache Software Foundation (ASF) under one or more - Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with - contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership. - this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0 - The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with - (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at - the License. You may obtain a copy of the License at
- -
- http://www.apache.org/licenses/LICENSE-2.0 - http://www.apache.org/licenses/LICENSE-2.0
- -
- Unless required by applicable law or agreed to in writing, software - Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS, - distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and - See the License for the specific language governing permissions and
- limitations under the License. - limitations under the License.
--> -->
# Kyuubi Spark Listener Extension # Kyuubi Spark Listener Extension
## Functions ## Functions
- [x] All `listener` extensions can be implemented in this module, like `QueryExecutionListener` and `ExtraListener` - [x] All `listener` extensions can be implemented in this module, like `QueryExecutionListener` and `ExtraListener`
- [x] Add `SparkOperationLineageQueryExecutionListener` to extends spark `QueryExecutionListener` - [x] Add `SparkOperationLineageQueryExecutionListener` to extends spark `QueryExecutionListener`
- [x] SQL lineage parsing will be triggered after SQL execution and will be written to the json logger file - [x] SQL lineage parsing will be triggered after SQL execution and will be written to the json logger file
## Build ## Build
@ -37,3 +37,4 @@ build/mvn clean package -pl :kyuubi-spark-lineage_2.12 -Dspark.version=3.2.1
- [x] 3.3.x (default) - [x] 3.3.x (default)
- [x] 3.2.x - [x] 3.2.x
- [x] 3.1.x - [x] 3.1.x

1
kyuubi-hive-beeline/README.md
View File

@ -3,3 +3,4 @@
Aiming to make a better supported beeline for Kyuubi Aiming to make a better supported beeline for Kyuubi
- Support to show launch engine log when getting KyuubiConnection(Done, available since v1.4.0-incubating) - Support to show launch engine log when getting KyuubiConnection(Done, available since v1.4.0-incubating)

2
kyuubi-hive-jdbc/README.md
View File

@ -1,9 +1,9 @@
# Kyuubi Hive JDBC Module # Kyuubi Hive JDBC Module
Aiming to make a better supported client for Kyuubi and Spark Aiming to make a better supported client for Kyuubi and Spark
- Add catalog to getTables meta function for DataLakes (DONE, broken in v1.3.0-incubating, fixed in v1.3.1-incubating) - Add catalog to getTables meta function for DataLakes (DONE, broken in v1.3.0-incubating, fixed in v1.3.1-incubating)
- Deploy to maven central (DONE, available since v1.3.0-incubating) - Deploy to maven central (DONE, available since v1.3.0-incubating)
- Create shaded jar (DONE, available since v1.4.0-incubating) - Create shaded jar (DONE, available since v1.4.0-incubating)
- Remove Hive dependencies (DONE, available since v1.6.0-incubating) - Remove Hive dependencies (DONE, available since v1.6.0-incubating)

2
kyuubi-server/web-ui/README.md
View File

@ -17,7 +17,6 @@ npm install
To do this you can change the VITE_APP_DEV_WEB_URL parameter variable as the service url in `.env.development` in the project root directory, such as http://127.0. 0.1:8090 To do this you can change the VITE_APP_DEV_WEB_URL parameter variable as the service url in `.env.development` in the project root directory, such as http://127.0. 0.1:8090
```shell ```shell
npm run dev npm run dev
``` ```
@ -56,3 +55,4 @@ pnpm run build
# Code Format # Code Format
pnpm run prettier pnpm run prettier
``` ```

1
pom.xml
View File

@ -2076,6 +2076,7 @@
</python> </python>
<markdown> <markdown>
<includes> <includes>
<include>**/README.md</include>
<include>docs/**/*.md</include> <include>docs/**/*.md</include>
</includes> </includes>
<flexmark> <flexmark>