e7e77f9277
### _Why are the changes needed?_ Store all jobInfos for each statement in mem. - First, according to jobInfo, you can get how long did it run, and get which stage took the longest time by stageIds. - Second, if this job failed that you can get from jobResult field, you can look up which stage cause this situation by stageIds. Some interfaces: - KyuubiJobInfo: job's summary info, contains statementId, startTime, endTime, jobResult, stageIds. - KyuubiStatementListener: singleton pattern, used for getting metrics about job, stage, executor and so on. - KyuubiStatementMonitor: for storing data in mem and dumpping them to a file when reached threshold. Test case: externals/kyuubi-spark-sql-engine/src/test/scala/org/apache/kyuubi/engine/spark/KyuubiStatementMonitorSuite.scala ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request Closes #814 from zhang1002/branch-1.2_add-jobinfo. Closes #814 7d8ece23 [张宇翔] 1. Change the log level from INFO to DEBUG 2. Code modify: avoid producing NullPointerException 0ddd83a7 [张宇翔] 1. Remove waitForOperationToComplete(client, opHandle) bbc36bb3 [张宇翔] 1. Merge statementListener to sqlEngineListener 741fcd0f [张宇翔] Add interval b61b9406 [张宇翔] Code changes cb6f92c4 [张宇翔] 1. Remove object-lock 2. Add eventually code c1a1d732 [张宇翔] 1. Remove volatile for jobInfo 9547d49e [张宇翔] 1. Remove volatile for jobInfo 970c0002 [张宇翔] 1. Add volatile for jobInfo 62ecb53d [张宇翔] 1. Remove kyuubiJobInfoQueue 2. Modify some unit test 3. Add some code annotations db1de381 [张宇翔] 1. Add some log 2. Add some annotations 8deb61f2 [张宇翔] change some test 87d9c102 [张宇翔] 1. Event tracking: for jonInfo 2. Add unit test 86c49ebc [张宇翔] Merge master branch a0a99b3a [张宇翔] Merge master branch f248bef7 [张宇翔] Merge remote-tracking branch 'upstream/master' 5d3b9afb [张宇翔] Code optimization db3a0b6f [张宇翔] Format changes 08d6d1fc [张宇翔] Modify some annotations f62f00e9 [张宇翔] Event Tracking: For job 85025193 [张宇翔] Merge branch 'branch-1.2_spark-monitor' into branch-1.2_add-jobinfo 3c8d9af1 [张宇翔] 1. change directory structure 2. code optimizing b4290bfa [张宇翔] Event tracking: For jobInfo fe1f7cce [张宇翔] Change directory structure 5ffb54f3 [张宇翔] Add kyuubi-spark-monitor module for nightly.yml 71d33b9a [张宇翔] Add some comment 6b87dff0 [张宇翔] 1. Remove some unused code 2. Add some comment f364d433 [张宇翔] Format change: Add newline in KyuubiStatementInfo 8bddb202 [张宇翔] 1. Remove the relationship between executionId and operationId 2. Get each state by the function: setState 3. Get this statement's physicalPlan in ExecuteStatement 4. Add sparkUser item 5. Remove java code f43b3c8f [张宇翔] merge master branch 55522613 [张宇翔] Merge branch 'master' into branch-1.2_spark-monitor 6f0be547 [张宇翔] format change bbfba274 [张宇翔] Even tracking 5a62f586 [张宇翔] remove some unused conf b26345ac [张宇翔] Merge master branch and resolve conflict fdc12bbe [张宇翔] Event tracking: for kStatement 1. Store the relationship between executionId and operationId 2. Store the relationship between operationId and statement 3. Store the relationship between executionId and physicalPlan 4. Store each state and its happen time for this statement: initialized, running, finished d1676268 [张宇翔] event tracking: for SQLExecutionStart Authored-by: 张宇翔 <zhang1002@126.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|---|---|---|
| .github | ||
| .idea | ||
| bin | ||
| build | ||
| conf | ||
| dev | ||
| docker | ||
| docs | ||
| externals | ||
| kyuubi-assembly | ||
| kyuubi-common | ||
| kyuubi-ctl | ||
| kyuubi-ha | ||
| kyuubi-hive-jdbc | ||
| kyuubi-main | ||
| kyuubi-metrics | ||
| kyuubi-zookeeper | ||
| licenses-binary | ||
| tools/spark-block-cleaner | ||
| _config.yml | ||
| .asf.yaml | ||
| .gitignore | ||
| .readthedocs.yml | ||
| .travis.yml | ||
| codecov.yml | ||
| LICENSE | ||
| LICENSE-binary | ||
| NOTICE | ||
| pom.xml | ||
| README.md | ||
| scalastyle-config.xml | ||
What is Kyuubi?
Kyuubi is a distributed multi-tenant Thrift JDBC/ODBC server for large-scale data management, processing, and analytics, built on top of Apache Spark and designed to support more engines (i.e., Flink). It has been open-sourced by NetEase since 2018. We are aiming to make Kyuubi an "out-of-the-box" tool for data warehouses and data lakes.
Kyuubi provides a pure SQL gateway through Thrift JDBC/ODBC interface for end-users to manipulate large-scale data with pre-programmed and extensible Spark SQL engines. This "out-of-the-box" model minimizes the barriers and costs for end-users to use Spark at the client side. At the server-side, Kyuubi server and engines' multi-tenant architecture provides the administrators a way to achieve computing resource isolation, data security, high availability, high client concurrency, etc.
- A HiveServer2-like API
- Multi-tenant Spark Support
- Running Spark in a serverless way
Target Users
Kyuubi's goal is to make it easy and efficient for anyone to use Spark(maybe other engines soon) and facilitate users to handle big data like ordinary data. Here, anyone means that users do not need to have a Spark technical background but a human language, SQL only. Sometimes, SQL skills are unnecessary when integrating Kyuubi with Apache Superset, which supports rich visualizations and dashboards.
In typical big data production environments with Kyuubi, there should be system administrators and end-users.
- System administrators: A small group consists of Spark experts responsible for Kyuubi deployment, configuration, and tuning.
- End-users: Focus on business data of their own, not where it stores, how it computes.
Additionally, the Kyuubi community will continuously optimize the whole system with various features, such as History-Based Optimizer, Auto-tuning, Materialized View, SQL Dialects, Functions, e.t.c.
Usage scenarios
Port workloads from HiveServer2 to Spark SQL
In typical big data production environments, especially secured ones, all bundled services manage access control lists to restricting access to authorized users. For example, Hadoop YARN divides compute resources into queues. With Queue ACLs, it can identify and control which users/groups can take actions on particular queues. Similarly, HDFS ACLs control access of HDFS files by providing a way to set different permissions for specific users/groups.
Apache Spark is a unified analytics engine for large-scale data processing. It provides a Distributed SQL Engine, a.k.a, the Spark Thrift Server(STS), designed to be seamlessly compatible with HiveServer2 and get even better performance.
HiveServer2 can identify and authenticate a caller, and then if the caller also has permissions for the YARN queue and HDFS files, it succeeds. Otherwise, it fails. However, on the one hand, STS is a single Spark application. The user and queue to which STS belongs are uniquely determined at startup. Consequently, STS cannot leverage cluster managers such as YARN and Kubernetes for resource isolation and sharing or control the access for callers by the single user inside the whole system. On the other hand, the Thrift Server is coupled in the Spark driver's JVM process. This coupled architect puts a high risk on server stability and makes it unable to handle high client concurrency or apply high availability such as load balancing as it is stateful.
Kyuubi extends the use of STS in a multi-tenant model based on a unified interface and relies on the concept of multi-tenancy to interact with cluster managers to finally gain the ability of resources sharing/isolation and data security. The loosely coupled architecture of the Kyuubi server and engine dramatically improves the client concurrency and service stability of the service itself.
DataLake/LakeHouse Support
The vision of Kyuubi is to unify the portal and become an easy-to-use data lake management platform. Different kinds of workloads, such as ETL processing and BI analytics, can be supported by one platform, using one copy of data, with one SQL interface.
- Logical View support via Kyuubi DataLake Metadata APIs
- Multiple Catalogs support
- SQL Standard Authorization support for DataLake(coming)
Cloud Native Support
Kyuubi can deploy its engines on different kinds of Cluster Managers, such as, Hadoop YARN, Kubernetes, etc.
The Kyuubi Ecosystem(present and future)
The figure below shows our vision for the Kyuubi Ecosystem. Some of them have been realized, some in development, and others would not be possible without your help.
Online Documentation
Since Kyuubi 1.0.0, the Kyuubi online documentation is hosted by https://readthedocs.org/. You can find the specific version of Kyuubi documentation as listed below.
For 0.8 and earlier versions, please check the Github Pages directly.
Quick Start
Ready? Getting Started with Kyuubi.
Contributing
All bits of help are welcome. You can make various types of contributions to Kyuubi, including the following but not limited to,
- Help new users in chat channel or share your success stories with us -
- Improve Documentation -
- Test releases -
- Improve test coverage -
- Report bugs and better help developers to reproduce
- Review changes
- Make a pull request
- Promote to others
- Click the star button if you like this project
Before you start, we recommend that you check the Contribution Guidelines first.
Aside
The project took its name from a character of a popular Japanese manga - Naruto.
The character is named Kyuubi Kitsune/Kurama, which is a nine-tailed fox in mythology.
Kyuubi spread the power and spirit of fire, which is used here to represent the powerful Apache Spark.
Its nine tails stand for end-to-end multi-tenancy support of this project.
License
This project is licensed under the Apache 2.0 License. See the LICENSE file for details.