apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity
1,275 Commits 23 Branches 0 Tags 65 MiB
2675e9a80d
Commit Graph

205 Commits

Author SHA1 Message Date
Kent Yao
6292b51ec8
Support launch SparkProcessBuilder with keytab and principal (#269) 
* Support launch SparkProcessBuilder with keytab and principal

* fix tests
 2021-01-08 23:43:14 +08:00
Cheng Pan
90c120759e
Improve pom.xml 
fixes #268
Squashed commit of the following:

commit d40d0831d578af23559ea2a2544ceebc47f6df50
Author: Cheng Pan <379377944@qq.com>
Date:   Fri Jan 8 18:30:51 2021 +0800

    remove unused mockito

commit 386e7e4eb1c93006e302c41ffbc04d0acfaf9a9b
Author: Cheng Pan <379377944@qq.com>
Date:   Fri Jan 8 17:35:44 2021 +0800

    extract versions to properties
 2021-01-08 19:04:48 +08:00
Kent Yao
4c433fc59a
Add SparkSQLEngineListener 2021-01-03 02:09:25 +08:00
Kent Yao
ca1b452e9a
Prepare Kyuubi v1.1.0 2020-12-31 15:51:12 +08:00
Kent Yao
17fe83778e
ServiceDiscovery for kyuubi servver 2020-12-28 13:05:35 +08:00
Kent Yao
a0f900a629
Reformat ServiceDiscovery 2020-12-28 00:10:31 +08:00
Kent Yao
3ec0db6a40
Improve Session Share Level 2020-12-27 00:05:07 +08:00
Zen
7a79b16cdf
fix #259 engine scope 
Squashed commit of the following:

commit bb598b101da43b145bd4c95976e8decd39a02210
Merge: 54e730c c4cf523
Author: zen <xinjingziranchan@gmail.com>
Date:   Thu Dec 24 10:53:42 2020 +0800

    Merge remote-tracking branch 'origin/pr2-engine-scope' into pr2-engine-scope

    # Conflicts:
    #	externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala

commit 54e730c5b26ffb147e5c1266bdd036460cc20d82
Merge: 62d1575 512fbc5
Author: zen <xinjingziranchan@gmail.com>
Date:   Thu Dec 24 10:51:21 2020 +0800

    Merge branch 'master' into pr2-engine-scope

    # Conflicts:
    #	externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala

commit c4cf523dc5bc409ea7ac742292d5e6e2d6421b8a
Author: zen <xinjingziranchan@gmail.com>
Date:   Mon Dec 21 23:10:12 2020 +0800

    merge master and fix the conflicts

commit c53b166f1eb3e8a698f498e1d123c815cef5d665
Merge: 62d1575 e8bc44c
Author: zen <xinjingziranchan@gmail.com>
Date:   Mon Dec 21 23:02:26 2020 +0800

    Merge branch 'master' into pr2-engine-scope

    # Conflicts:
    #	externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala

commit 62d15751ea6b213319501bcfac442d21d0cd5ca6
Author: zen <xinjingziranchan@gmail.com>
Date:   Thu Dec 10 15:40:48 2020 +0800

    fix zkNamespace

commit 7e3e432318ed931b1ad16c839c4a027e00bc6ef9
Author: zen <xinjingziranchan@gmail.com>
Date:   Thu Dec 10 14:34:34 2020 +0800

    Support for configuring the session scope of SparkSQLEngine.
 2020-12-24 19:02:40 +08:00
Zen
512fbc50cf
Support Spark YARN cluster deployment mode fix #264 
Squashed commit of the following:

commit 3c875db5dd245b4f978afec92e1d02ac831c94bf
Author: zen <xinjingziranchan@gmail.com>
Date:   Wed Dec 23 19:51:22 2020 +0800

    remove unnecessary code

commit c2487ee7ca4bb4271c7ae6a6263e32e040c4c159
Merge: 67bf250 d8e5c5a
Author: zen <xinjingziranchan@gmail.com>
Date:   Wed Dec 23 10:58:16 2020 +0800

    Merge branch 'master' into pr4-deploy-mode-with-cluster

    # Conflicts:
    #	externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala

commit 67bf25062d3cb38ccf3831b2e73561e16c4f8576
Author: zen <xinjingziranchan@gmail.com>
Date:   Wed Dec 23 10:51:46 2020 +0800

    delete unit test

commit 053834808f37c18a777a923d989a9b4d62d8419c
Author: zen <xinjingziranchan@gmail.com>
Date:   Wed Dec 23 10:34:05 2020 +0800

    Support Spark YARN cluster deployment mode

commit 61b24d0e3246f319b8d2cf70932a5b71cbf4a28a
Author: zen <xinjingziranchan@gmail.com>
Date:   Tue Dec 15 15:10:31 2020 +0800

    fix null exception and add unit test

commit 363cf93a77294485e7093716b6cb1cd58b4dba45
Author: zen <xinjingziranchan@gmail.com>
Date:   Tue Dec 15 14:21:19 2020 +0800

    Support Spark YARN cluster deployment mode
 2020-12-23 20:45:09 +08:00
Zen
e8bc44c2ad
[#258] delete the engine's timeoutChecker scheduler and merge it to sessionManager's scheduler 
fix #258
Squashed commit of the following:

commit 5b0a4d1f77b32cef00f55b80b8562752c1f462d7
Author: zen <xinjingziranchan@gmail.com>
Date:   Wed Dec 16 10:15:16 2020 +0800

    fix closing session log

commit c0fe24d06638f9be18d91a4ad05943a023967c62
Author: zen <xinjingziranchan@gmail.com>
Date:   Sat Dec 12 22:08:25 2020 +0800

    move this check task to SparkSQLSessionManager.

commit 1968913702872f0ba433679596596f114f2e16d1
Author: zen <xinjingziranchan@gmail.com>
Date:   Thu Dec 10 15:55:32 2020 +0800

    add log when close session

commit 26a944787c63666813a0d64e67a36b671640d493
Author: zen <xinjingziranchan@gmail.com>
Date:   Thu Dec 10 11:32:45 2020 +0800

    delete the engine's timeoutChecker scheduler and merge it to  sessionManager's scheduler.
 2020-12-21 11:01:31 +08:00
Kent Yao
5374398050 Stringlify exception for better client debugging 2020-11-27 22:20:51 +08:00
Kent Yao
87e243c596 support query log for async exec 2020-11-27 16:40:07 +08:00
Kent Yao
f8ef7c8997 Support ansynchronous exec for Kyuubi Server 2020-11-24 10:55:29 +08:00
Kent Yao
9ca23bcf20 Support ansynchronous exec for Spark SQL engine 2020-11-20 19:52:57 +08:00
Zen
6637c58273 followup [add SparkSQLEngine idle check] fix #251 
Squashed commit of the following:

commit 808ccb738f8774e465d12dfb04a6d37256545e78
Author: Zen <xinjingziranchan@gmail.com>
Date:   Wed Nov 18 14:18:46 2020 +0800

    Fixed duplicate issues with the Jars directory when packaging Kyuubi

commit 91ca0a0f8a5a260b1ae29a31c323f69d37a14bfe
Author: zen <xinjingziranchan@gmail.com>
Date:   Wed Nov 18 14:01:56 2020 +0800

    fix log print and SparkSQLEngine timeoutChecker concurrent issue.

commit 15343d9cd07b48d3909c1085836687a4cc7f05c3
Author: zen <xinjingziranchan@gmail.com>
Date:   Tue Nov 17 21:21:03 2020 +0800

    消除打印dehug日志时，重复判断是否开启debug模式

commit eecc0bf01144b3f2601e6a821bedf786e5aafb22
Author: zen <xinjingziranchan@gmail.com>
Date:   Tue Nov 17 15:08:13 2020 +0800

    Revert "修复打包时kyuubi依赖jars目录重复，导致启动时找不到org.apache.kyuubi.server.KyuubiServer启动类"

    This reverts commit 29e9dd45

commit 29e9dd45e90b1549b68e96024b1ee866429bf40d
Author: zen <xinjingziranchan@gmail.com>
Date:   Tue Nov 17 14:36:23 2020 +0800

    修复打包时kyuubi依赖jars目录重复，导致启动时找不到org.apache.kyuubi.server.KyuubiServer启动类

commit 44364ecd289ca40181bd150998cce76762b145be
Author: zen <xinjingziranchan@gmail.com>
Date:   Tue Nov 17 13:48:21 2020 +0800

    修复SparkSQLEngine停止时没有关闭timeoutChecker线程池，以及增加timeoutChecker打印日志

commit 63a67e2c487a5418a056ff4f797b46185ed4c17b
Merge: 6028f55 64b83a4
Author: zen <xinjingziranchan@gmail.com>
Date:   Tue Nov 17 13:37:41 2020 +0800

    Merge branch 'yaooqinn-kyuubi-master'

commit 6028f55504cbc3d522bf061fde71608019243d3e
Author: 金晶(000538) <jinjing@fcbox.com>
Date:   Mon Nov 16 16:26:09 2020 +0800

    测试代码SparkOperationSuite格式调整

commit 0b14c69183691e3601aef3a9d56c545edef44499
Author: 金晶(000538) <jinjing@fcbox.com>
Date:   Mon Nov 16 14:49:42 2020 +0800

    1.新增SparkSQLEngine空闲定时检测任务
    2.修复SparkSQLEngine关闭session时未移除SparkSQLOperationManager中缓存的sparkSession

commit 8fdb6b7cec70f32dc05bfeeedcc233d0f6fb4017
Merge: fb4bace 7bfc470
Author: zen <xinjingziranchan@gmail.com>
Date:   Mon Nov 16 11:36:35 2020 +0800

    Merge pull request #1 from yaooqinn/master

    同步更新
 2020-11-18 16:41:43 +08:00
zen
7fd225b098 add SparkSQLEngine idle check 
fix #250
Squashed commit of the following:

commit 6028f55504cbc3d522bf061fde71608019243d3e
Author: 金晶(000538) <jinjing@fcbox.com>
Date:   Mon Nov 16 16:26:09 2020 +0800

    测试代码SparkOperationSuite格式调整

commit 0b14c69183691e3601aef3a9d56c545edef44499
Author: 金晶(000538) <jinjing@fcbox.com>
Date:   Mon Nov 16 14:49:42 2020 +0800

    1.新增SparkSQLEngine空闲定时检测任务
    2.修复SparkSQLEngine关闭session时未移除SparkSQLOperationManager中缓存的sparkSession

commit 8fdb6b7cec70f32dc05bfeeedcc233d0f6fb4017
Merge: fb4bace 7bfc470
Author: zen <xinjingziranchan@gmail.com>
Date:   Mon Nov 16 11:36:35 2020 +0800

    Merge pull request #1 from yaooqinn/master

    同步更新
 2020-11-16 19:28:33 +08:00
Kent Yao
b3d996917d fix static and core conf via hiveconf or hivevar 2020-11-13 16:08:47 +08:00
Kent Yao
fb4bace6a5 Ignore static and other unmodifiable settting in session 2020-11-11 11:15:09 +08:00
Kent Yao
574c119712 JDBCTests added and some related bugs fixed 2020-11-10 21:24:02 +08:00
Kent Yao
a67d485951 Missing version in readMessageBegin, old client 
2020-11-10 10:15:42,001 [61208289] - ERROR [FrontendServiceHandler-Pool: Thread-449:TThreadPoolServer$WorkerProcess@315] - Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client?
	at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:234)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
	at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
 2020-11-10 17:21:17 +08:00
Kent Yao
f8dc5a9ca3 Using ha.zookeeper.acl.enabled to control ZK acl 2020-11-09 17:34:55 +08:00
Kent Yao
7157370ef6 Add Building Doc 2020-11-05 19:05:26 +08:00
Kent Yao
642b52e965 nit 2020-10-30 16:04:20 +08:00
Kent Yao
6efa589a74 FrontendServiceSuite 2020-10-21 20:22:42 +08:00
Kent Yao
53b3fdb616 OperationStatus 2020-10-21 16:06:38 +08:00
Kent Yao
0849678453 Hive JDBC Database MetaData API Auditing 2020-10-20 18:08:28 +08:00
Kent Yao
6ad06f66e9 UtilsSuite 2020-10-16 15:21:35 +08:00
Kent Yao
e7a1054fd2 fix work dir bug 2020-10-14 14:59:49 +08:00
Kent Yao
5d1ea8547a Make SparkSQLEngine be able to self teminate 2020-10-13 20:49:54 +08:00
Kent Yao
396e06a3bb nit 2020-09-29 10:11:34 +08:00
Kent Yao
f60a6d37a7 log time cost for operation 2020-09-24 16:47:22 +08:00
Kent Yao
ff7a1221e0 handle client ip properly 2020-09-24 11:59:55 +08:00
Kent Yao
f8b8c6c172 Print spark engine dignosis at client side 2020-09-23 17:41:51 +08:00
Kent Yao
c673235a42 finish cancel operation test 2020-09-22 18:00:58 +08:00
Kent Yao
efa78a92f1 improve get schemas test 2020-09-22 17:43:41 +08:00
Kent Yao
470d7ca90d Make man service work 2020-09-21 14:05:34 +08:00
Kent Yao
07204dbef5 add test for close and cancel operation 2020-09-15 14:36:02 +08:00
Kent Yao
d8a27c914f spark-sql-engine: enable session configuration 2020-09-14 18:05:08 +08:00
Kent Yao
07c9d9578d style 2020-09-14 10:59:47 +08:00
Kent Yao
31cbee72ef add some test for execute statement operation 2020-09-11 19:28:38 +08:00
Kent Yao
f7a5549f6c refine SparkProcessBuilder 2020-09-11 16:22:39 +08:00
Kent Yao
bd536cd9c5 Remove KyuubiThriftBinaryCliService 2020-09-10 21:05:21 +08:00
Kent Yao
4ee3577944 improve test for operation log 2020-09-10 18:22:48 +08:00
Kent Yao
13706c3c3a improve test for operation log 2020-09-10 16:59:57 +08:00
Kent Yao
ee315c6963 fix test 2020-09-10 14:09:36 +08:00
Kent Yao
59b7c3a09e add more tests 2020-09-10 12:01:58 +08:00
Kent Yao
83d1ab7450 bump up spark to 3.0.1 2020-09-10 10:58:52 +08:00
Kent Yao
01d50d3ec7 add tests for meta operations 2020-09-09 16:15:45 +08:00
Kent Yao
42e92e41b6 add test for get null column 2020-09-09 11:25:18 +08:00
Kent Yao
0a43b82400 fix GetColumns 2020-09-09 10:06:15 +08:00
Kent Yao
46d6dda3f3 Improve spark sql engine test 2020-09-08 21:19:21 +08:00
Kent Yao
06178e52fe KyuubiServer Part 1 2020-09-08 14:45:40 +08:00
Kent Yao
e8ab5ab4a0 Some build staff 2020-08-21 14:39:33 +08:00
Kent Yao
0468fd4d4e Implementaton of KyuubiOperationManager 2020-08-21 10:51:55 +08:00
Kent Yao
86e1a7aac3 SparkProcessBuilder 2020-08-18 17:22:08 +08:00