a97b843303
### _Why are the changes needed?_ to close #3794 This PR introduces the Apache Arrow library to improve the performance of data interchange. data flow diagram with saprk engine ``` 1. Spark Executor -> 2. Spark Driver -> 3. Kyuubi Server -> 4. Beeline/KyuubiHiveDriver ``` 1. for simple types(Numeric/String/Binary/Boolean/Datetime/Interval), we serialize the columns on the Spark executor side and deserialize them on the client side. 2. for complex types(Array/Map/Struct), we first convert these columns to hive string and serialize the string columns on the executor side, and finally deserialize them as string type on the client side. ## Local Test ### Input dataset ```scala spark.sql( """ |select * from tpcds.sf10.catalog_sales limit 2000000 |""".stripMargin) .write .mode("overwrite") .save("/tmp/parquet/tpcds/sf10/catalog_sales_2000000") ``` ### Query cat /tmp/b.sql ```sql select * from parquet.`/tmp/parquet/tpcds/sf10/catalog_sales_2000000`; ``` ### Beeline command ``` bin/beeline -u 'jdbc:hive2://0.0.0.0:10009/;' --hiveconf kyuubi.session.result.codec=simple/arrow --hiveconf kyuubi.operation.incremental.collect=false/true -f /tmp/b.sql > /dev/null ``` ### Spark ``` /Users/fchen/Software/spark-3.3.0-bin-hadoop3/bin/spark-submit \ --class org.apache.kyuubi.engine.spark.SparkSQLEngine \ --driver-memory 6g \ --master local[*] \ ...... ``` ### Result | kyuubi.session.result.codec | kyuubi.operation.incremental.collect | time spent| | --- | --- | --- | | arrow | false | 272.178 seconds | | arrow | true | 146.76 seconds | | simple | true | 607.523 seconds | | simple | false | 429.374 seconds | Closes #3803 from cfmcgrady/arrow. Closes #3803 05b2b6f5 [Fu Chen] address comment 37e54e93 [Fu Chen] rename `kyuubi.session.result.codec` to `kyuubi.operation.result.codec` fa44b102 [Fu Chen] address comment e66b3bbd [Fu Chen] rebase master 08e2a97b [Fu Chen] fix shade ab796bc6 [Fu Chen] address comment 1c979318 [Fu Chen] only consider spark engine major/minor version dd61c015 [Fu Chen] add missing relocate dep `netty` 02f1d2f7 [Fu Chen] relocate arrow/jackson deps 4a2f7f09 [Fu Chen] minor refactor SparkDataTypeTests with SemanticVersion 627cc0a2 [Fu Chen] update kyuubi-jdbc-high-shaded NOTICE 815c68c2 [Fu Chen] reduce kyuubi-hive-jdbc/pom.xml a7e2fcd9 [Fu Chen] fix style 825fda67 [Fu Chen] skip some tests when spark version < 3.3 039dbba1 [Fu Chen] minor refactor 95ad84c4 [Fu Chen] address comment e1e81f01 [Fu Chen] revert unnecessary changes 36bce1d9 [Fu Chen] clean up c789327c [Fu Chen] make sure SparkArrowbasedOperationSuite run with arrow enabled d60c3120 [Fu Chen] clean up array-related implementations 8ca1fcd6 [Fu Chen] clean up unused arrow array implementations 8f96fdf3 [Fu Chen] fix cdb83cf0 [Fu Chen] rename conf "kyuubi.operation.result.codec" to "kyuubi.session.result.codec" f30c2c9d [Fu Chen] rename conf "kyuubi.beeline.arrow.enabled" to "kyuubi.operation.result.codec" 109a4f32 [Fu Chen] Fork Apache Spark-3.3.1 org.apache.spark.sql.catalyst.util.quoteIfNeeded to adapt to Spark-3.1.x 9ec23de3 [Fu Chen] revert kyuubi-server/src/main/scala/org/apache/kyuubi/server/BackendServiceMetric.scala f2b7f2c3 [Fu Chen] revert kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiCallableStatement.java 33e820c7 [Fu Chen] revert temporary `pom.xml` changes daa259e5 [Fu Chen] add SparkArrowbasedOperationSuite eb5ed344 [Fu Chen] make kyuubi.beeline.arrow.enabled default to false and enabled via url support 5c3a48c7 [Fu Chen] fix ut SparkOperationSuite: execute statement - select column name with dots 0dfd442b [Fu Chen] complex type(array/map/array) support 86b0a485 [Fu Chen] INTERVAL_YEAR_MONTH_TYPE type support 10da4fe9 [Fu Chen] INTERVAL_DAY_TIME_TYPE type support 3a8b0da3 [Fu Chen] implement KyuubiArrowBasedResultSet.getDate a9fd48c1 [Fu Chen] implement KyuubiArrowBasedResultSet.getInt a37b5564 [Fu Chen] only ExecuteStatement support arrow fe47fdc4 [Fu Chen] add java version ArrowUtils d162a594 [Fu Chen] lisence 5414ccc7 [Fu Chen] remove debug println 335b4e8f [Fu Chen] drop scala dependency for module kyuubi-hive-jdbc f826e262 [Fu Chen] fix ut bcf9744d [Fu Chen] license 3ebca27c [Fu Chen] fix style 540423d5 [Fu Chen] timestamp support 0965b8f4 [Fu Chen] revert extensions/spark/kyuubi-spark-connector-tpcds/src/test/scala/org/apache/kyuubi/spark/connector/tpcds/TPCDSQuerySuite.scala ad1a7092 [Fu Chen] revert 5b4ee403 [Fu Chen] BackendServiceMetric 21b76d29 [Fu Chen] add more types support ace05a24 [Fu Chen] arrow-based collect Authored-by: Fu Chen <cfmcgrady@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|---|---|---|
| .github | ||
| .idea | ||
| bin | ||
| build | ||
| conf | ||
| dev | ||
| docker | ||
| docs | ||
| extensions | ||
| externals | ||
| integration-tests | ||
| kyuubi-assembly | ||
| kyuubi-common | ||
| kyuubi-ctl | ||
| kyuubi-events | ||
| kyuubi-ha | ||
| kyuubi-hive-beeline | ||
| kyuubi-hive-jdbc | ||
| kyuubi-hive-jdbc-shaded | ||
| kyuubi-metrics | ||
| kyuubi-rest-client | ||
| kyuubi-server | ||
| kyuubi-zookeeper | ||
| licenses | ||
| licenses-binary | ||
| tools/spark-block-cleaner | ||
| .asf.yaml | ||
| .dockerignore | ||
| .gitattributes | ||
| .gitignore | ||
| .rat-excludes | ||
| .readthedocs.yml | ||
| .scalafmt.conf | ||
| .travis.yml | ||
| codecov.yml | ||
| CONTRIBUTING.md | ||
| DISCLAIMER | ||
| LICENSE | ||
| LICENSE-binary | ||
| MATURITY.md | ||
| NOTICE | ||
| NOTICE-binary | ||
| pom.xml | ||
| README.md | ||
| scalastyle-config.xml | ||
Apache Kyuubi (Incubating)
What is Kyuubi?
Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC server for large-scale data management, processing, and analytics, built on top of Apache Spark and designed to support more engines (i.e., Flink). It has been open-sourced by NetEase since 2018. We are aiming to make Kyuubi an "out-of-the-box" tool for data warehouses and data lakes.
Kyuubi provides a pure SQL gateway through Thrift JDBC/ODBC interface for end-users to manipulate large-scale data with pre-programmed and extensible Spark SQL engines. This "out-of-the-box" model minimizes the barriers and costs for end-users to use Spark at the client side. At the server-side, Kyuubi server and engines' multi-tenant architecture provides the administrators a way to achieve computing resource isolation, data security, high availability, high client concurrency, etc.
- A HiveServer2-like API
- Multi-tenant Spark Support
- Running Spark in a serverless way
Target Users
Kyuubi's goal is to make it easy and efficient for anyone to use Spark(maybe other engines soon) and facilitate users to handle big data like ordinary data. Here, anyone means that users do not need to have a Spark technical background but a human language, SQL only. Sometimes, SQL skills are unnecessary when integrating Kyuubi with Apache Superset, which supports rich visualizations and dashboards.
In typical big data production environments with Kyuubi, there should be system administrators and end-users.
- System administrators: A small group consists of Spark experts responsible for Kyuubi deployment, configuration, and tuning.
- End-users: Focus on business data of their own, not where it stores, how it computes.
Additionally, the Kyuubi community will continuously optimize the whole system with various features, such as History-Based Optimizer, Auto-tuning, Materialized View, SQL Dialects, Functions, e.t.c.
Usage scenarios
Port workloads from HiveServer2 to Spark SQL
In typical big data production environments, especially secured ones, all bundled services manage access control lists to restricting access to authorized users. For example, Hadoop YARN divides compute resources into queues. With Queue ACLs, it can identify and control which users/groups can take actions on particular queues. Similarly, HDFS ACLs control access of HDFS files by providing a way to set different permissions for specific users/groups.
Apache Spark is a unified analytics engine for large-scale data processing. It provides a Distributed SQL Engine, a.k.a, the Spark Thrift Server(STS), designed to be seamlessly compatible with HiveServer2 and get even better performance.
HiveServer2 can identify and authenticate a caller, and then if the caller also has permissions for the YARN queue and HDFS files, it succeeds. Otherwise, it fails. However, on the one hand, STS is a single Spark application. The user and queue to which STS belongs are uniquely determined at startup. Consequently, STS cannot leverage cluster managers such as YARN and Kubernetes for resource isolation and sharing or control the access for callers by the single user inside the whole system. On the other hand, the Thrift Server is coupled in the Spark driver's JVM process. This coupled architect puts a high risk on server stability and makes it unable to handle high client concurrency or apply high availability such as load balancing as it is stateful.
Kyuubi extends the use of STS in a multi-tenant model based on a unified interface and relies on the concept of multi-tenancy to interact with cluster managers to finally gain the ability of resources sharing/isolation and data security. The loosely coupled architecture of the Kyuubi server and engine dramatically improves the client concurrency and service stability of the service itself.
DataLake/LakeHouse Support
The vision of Kyuubi is to unify the portal and become an easy-to-use data lake management platform. Different kinds of workloads, such as ETL processing and BI analytics, can be supported by one platform, using one copy of data, with one SQL interface.
- Logical View support via Kyuubi DataLake Metadata APIs
- Multiple Catalogs support
- SQL Standard Authorization support for DataLake(coming)
Cloud Native Support
Kyuubi can deploy its engines on different kinds of Cluster Managers, such as, Hadoop YARN, Kubernetes, etc.
The Kyuubi Ecosystem(present and future)
The figure below shows our vision for the Kyuubi Ecosystem. Some of them have been realized, some in development, and others would not be possible without your help.
Online Documentation
Since Kyuubi 1.3.0-incubating, the Kyuubi online documentation is hosted by https://kyuubi.apache.org/. You can find the latest Kyuubi documentation on this web page. For 1.2 and earlier versions, please check the Readthedocs directly.
Quick Start
Ready? Getting Started with Kyuubi.
Contributing
Contributor over time
Aside
The project took its name from a character of a popular Japanese manga - Naruto.
The character is named Kyuubi Kitsune/Kurama, which is a nine-tailed fox in mythology.
Kyuubi spread the power and spirit of fire, which is used here to represent the powerful Apache Spark.
Its nine tails stand for end-to-end multi-tenancy support of this project.
License
This project is licensed under the Apache 2.0 License. See the LICENSE file for details.