apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity
1,103 Commits 23 Branches 0 Tags 65 MiB
6020b831a7
Commit Graph

191 Commits

Author SHA1 Message Date
Kent Yao
934a9d3158 Support RowSet 2020-07-08 11:38:48 +08:00
Kent Yao
e3e1a8f00b Add OperationState 2020-06-30 15:29:59 +08:00
Kent Yao
1a3e886627 drop module kyuubi-hive-thrift 2020-06-23 17:32:43 +08:00
Kent Yao
c2271ac05b Bumpup Spark to 3.0.0 from 3.0.0-preview2 2020-06-18 17:41:43 +08:00
Kent Yao
e1d55f8351 Introduce ServiceDiscovery 2020-06-17 16:14:20 +08:00
Kent Yao
5100d8a519 Support EmbeddedZkServer 2020-06-12 17:18:46 +08:00
Kent Yao
15d179507f Create Kyuubi Project Spark SQL Engine 2020-06-09 10:34:47 +08:00
konradqcharczyk
85689b6b64 Squashed commit of the following: 
commit cd39627e46666c80bcf155fed647d2a84328af99
Merge: aae0cef 9b03db7
Author: konradqcharczyk <konradqcharczyk@gmail.com>
Date:   Sun Jun 7 16:16:00 2020 +0200

    apache mirror and spark version changed

commit aae0cef4347c55a68f74f426b98378b39d0b8bdb
Author: konradqcharczyk <konradqcharczyk@gmail.com>
Date:   Sun Jun 7 14:51:16 2020 +0200

    apache mirror and spark version changed

commit 9b03db77aebb73fe3587478c36c282a750116164
Author: konradqcharczyk <konradqcharczyk@gmail.com>
Date:   Sun Jun 7 14:51:16 2020 +0200

    spark 2.4.3 changes
 2020-06-07 22:30:14 +08:00
Kent Yao
651db99866
HIVE_CLI_SERVICE_PROTOCOL_V11 (#235) 
* HIVE_CLI_SERVICE_PROTOCOL_V11

* pom

* style

* tests

* travis

* code cover
 2020-05-26 16:07:41 +08:00
Kent Yao
b9a50d62ee new scripts for new framework 2020-05-22 19:16:04 +08:00
Kent Yao
17c3c054b7
[KYUUBI-233] Make master branch target apache spark 2.4 #233 (#234) 2020-05-18 11:54:20 +08:00
Kent Yao
af9801beb8
[KYUUBI-230] Re-implement OperationLog (#231) 2020-05-18 10:42:00 +08:00
QianChao
d227e616b2 [KYUUBI-221] Optimise FetchResults speed when maxRows is large fix #221 
Squashed commit of the following:

commit f103b8d5dcbdb838b9dc1d46faec8e93c10af865
Author: QianChao <chaoqian@advance.ai>
Date:   Fri Oct 25 10:14:30 2019 +0800

    UPD: add more test cases for ColumnBaseSet

commit eabd5d4341a0ceabce9f68e20fe173415ecae981
Author: QianChao <chaoqian@advance.ai>
Date:   Thu Oct 17 15:16:40 2019 +0800

    FIX: ColumnBaseSet nulls with stream issue

commit 14c04820db2c6a22413e4b775ca71bae55d51aaf
Author: QianChao <chaoqian@advance.ai>
Date:   Wed Oct 16 15:04:22 2019 +0800

    FIX: stream access issue
 2019-10-25 11:34:29 +08:00
Kent Yao
c6ef20089e
[KYUUBI-219]Fix NPE in HiveTokenCollector (#220) 2019-10-23 16:02:23 +08:00
Kent Yao
191b8171d6
[KYUUBI-216]Make thrift IDL a separate sub module fix #216 (#217) 
* [KYUUBI-216]Make thrift IDL a separate sub module fix #216

* typo
 2019-09-25 11:38:35 +08:00
Kent Yao
947394350f
[KYUUBI-213] Check and wait the old spark context completely stopped before create a new one (#214) 
* [KYUUBI-213] Check and wait the old spark context completely stopped before create a new one #213

* stop to not stop
 2019-08-26 15:04:04 +08:00
Kent Yao
6621d361af
add user info and statement id in executing log (#212) 
* add user info and statement id in executing log

* add ut

* update conf

* long max

* modify log

* modify log

* unpersit

* typo

* add ut

* fix ut

* typo

* code cov

* add log

* ex order

* rm some ex

* none get

* Revert "none get"

This reverts commit 6792e33db11619ec4fa7c075706da1500fb229e3.

* rm debug
 2019-08-16 16:51:12 +08:00
Kent Yao
552bb100c1
[KYUUBI-208] add delegation tokens to hive session state's dfs client (#209) 
* [KYUUBI-208] add delegation tokens to hive session state's dfs client

* add ut

* handle exception

* should wrap in do as

fix #208
 2019-07-25 10:39:45 +08:00
Kent Yao
7729ff0ba5
[KYUUBI-206] Initialize external catalog ahead during open session for a user fix #206 (#207) 
* [KYUUBI-206] Initialize external catalog ahead during open session for a user fix #206

* code cov

* code cov
 2019-07-15 11:42:12 +08:00
Kent Yao
5edf961a44
[KYUUBI-204]Apache Kudu Integration Guide fix #204 (#205) 
* [KYUUBI-204]Apache Kudu Integration Guide fix #204

* references
 2019-07-09 16:47:20 +08:00
Kent Yao
abc58c4ee3
[KYUUBI-201]Add some user friendly spark configurations (#202) 
fix #201
 2019-06-25 18:40:03 +08:00
Kent Yao
f4c08c6f52
[KUUBI-198][FOLLOWUP]Fix SparkSessionCache unit test occasionally fails (#200) 
* fix travis occasionally fails

* ad ut
 2019-06-20 21:46:55 +08:00
Kent Yao
b6591d3314
[KYUUBI-198] Failover mode ha service disconnects from zk after lost leader ship (#199) 
* fxi #198 Failover mode ha service disconnects from zk after losts leader ship

* reset leader latch

* add ut

* add some ut back
 2019-06-20 17:56:22 +08:00
Kent Yao
63245dd691
[KYUUBI-196]Refactoring SparkSessionWithUGI‘s get and create SparkContext logic (#197) 
* fxit 194 Refactoring SparkSessionWithUGI‘s get and create sc logic

* upgrade scala-maven-plugin to 3.4.6

* fix travis

* fix travis again

* fix travis open jdk8

* fix travis open jdk8

* fix travis open jdk8

* fix travis open jdk8

* fix travis open jdk8

* add state to get yarn applications

* promise complete

* fix #69 capture spark exceptions

* typo

* fix #196 add ut and log

* typo

* typo

* add ut
 2019-06-17 11:47:13 +08:00
Kent Yao
e295995959
[KYUUBI-194]Add a server side metrics system (#195) 
* fix #194 add server side metrics system

* add spark session cache size

* add ut

* add ut

* add doc
 2019-06-12 11:12:18 +08:00
Kent Yao
f8a4775365
[KYUUBI-180]add a sync execute statement operation (#192) 
* fix #180 add sync execute statement operation

* fix ut
 2019-06-04 18:51:07 +08:00
Kent Yao
cf4ee6afb6
[KYUUBI-156][KYUUBI-185] Implementing all meta operations support by spark (#191) 
* [KYUUBI-156][KYUUBI-185] Implementing all meta operations support by spark

* rm used metastore client fix #156 fix #185

* add more uts

* add more uts

* fix bugs and add show functions

* meta operation method abstraction

* readme
 2019-06-04 13:19:23 +08:00
Kent Yao
71eea2ba1f
[KYUUBI-189]Add an OOM Hook to shutdown server when OutOfMemoryError occurs (#190) 
fix #189 

* [KYUUBI-189]Add an OOM Hook to shutdown server when OutOfMemoryError occurs

* add ut

* fit ut

* fit ut

* typo
 2019-05-25 16:42:13 +08:00
hustfeiwang
5ad223556b [KYUUBI-120]define interface class IKyuubiOperation to allow different implementation of kyuubiOperation 
---

---
fix #181 fix #120
---
Squashed commit of the following:

commit 3e7f81f3b8e5c4663fa54ed3e676312139693197
Author: hustfeiwang <wangfei3@corp.netease.com>
Date:   Thu May 23 18:32:08 2019 +0800

    fix unit test

commit 73763d7ca8416156132cb9e8df3e86966bf8c6e6
Author: hustfeiwang <wangfei3@corp.netease.com>
Date:   Thu May 23 09:48:21 2019 +0800

    set statementId to a val

commit 128effa6a9a89c5bb8f9c91f219e4e257ff96372
Author: hustfeiwang <wangfei3@corp.netease.com>
Date:   Wed May 22 16:06:45 2019 +0800

    set isClosedOrCanceled to a protected method

commit 267b55f1d2497872e348fc494a3ff7132a6012f0
Author: hustfeiwang <wangfei3@corp.netease.com>
Date:   Wed May 22 10:10:48 2019 +0800

    KYUUBI-120: define interface class IKyuubiOperation to allow different implementation of kyuubiOperation
 2019-05-24 11:55:32 +08:00
Kent Yao
416122675d
[KYUUBI-186][FOLLOWUP]fix ut occasionally failure fix #186 (#188) 
* [KYUUBI-186][FOLLOWUP]fix ut occasionally failure fix #186

* fix ut

* code cover
 2019-05-22 20:29:33 +08:00
Kent Yao
505e01d7e2 [KYUUBI-186]Add a max cache time to clean SparkSessions that may have token expiry issue 
---

---

fix #186 fix #187

---
Squashed commit of the following:

commit 4b8297f8d76b40f22e5419c72f0ddf1eadb3c7d9
Author: Kent Yao <yaooqinn@hotmail.com>
Date:   Mon May 20 19:36:34 2019 +0800

    add log

commit 7b811f8a0ccb65fb0d0832ba1313e3c099f74dad
Author: Kent Yao <yaooqinn@hotmail.com>
Date:   Thu May 16 16:46:04 2019 +0800

    add ut

commit 19f67fa8909c4a19f49fbe3b30a75d09c0736350
Author: Kent Yao <yaooqinn@hotmail.com>
Date:   Thu May 16 10:57:24 2019 +0800

    fix ut

commit 7ad7c20f6536d3bbc9d789c71af238331a3e03c5
Author: Kent Yao <yaooqinn@hotmail.com>
Date:   Thu May 16 10:40:42 2019 +0800

    fix ut

commit 9a114abb8be3083160ea01dab188333f675677e4
Author: Kent Yao <yaooqinn@hotmail.com>
Date:   Thu May 16 00:10:57 2019 +0800

    mv cache validating in its own class

commit 0afba5a5ab7a005948acde545b4a68c8d26bfce9
Author: Kent Yao <yaooqinn@hotmail.com>
Date:   Wed May 15 11:47:42 2019 +0800

    fix ut

commit eff3f41f91b4bfc2bfc84efc6af29fa4365d9dae
Author: Kent Yao <yaooqinn@hotmail.com>
Date:   Wed May 15 11:12:51 2019 +0800

    add ut

commit 9bbbea78bcb8f5d409819628974628f0bf59f69c
Author: Kent Yao <yaooqinn@hotmail.com>
Date:   Tue May 14 23:17:00 2019 +0800

    add ut

commit 0e0e59eedc055ddb6788a9dd6f07d4308772145d
Author: Kent Yao <yaooqinn@hotmail.com>
Date:   Tue May 14 19:39:09 2019 +0800

    mv init time to spark session cache

commit a4a1c69e29ec7d5d51a0e5026cf8f3c2aaf6b2ef
Author: Kent Yao <yaooqinn@hotmail.com>
Date:   Tue May 14 18:36:06 2019 +0800

    Add a max cache time to clean SparkSessions that may have token expiry issue fix #186
 2019-05-21 22:07:18 +08:00
Kent Yao
d7f95775de [BUILD]Bump up to v0.7.0-SNAPSHOT 2019-05-10 11:58:22 +08:00
Kent Yao
1f0bc742e4
[KYUUBI-167][FOLLOWUP]populate tokens via spark session cache mgr (#183) 
* Prepare releasing v0.6.0

* fix #KYUUBI-167 populate token via spark session cache mgr

* typo

* fix ut

* code cov
 2019-05-07 10:32:04 +08:00
Kent Yao
32c95eacf1
[KYUUBI-170]persist and coalesce before to local iterator while using incremental result collect (#171) 
* fix #170 persist and coalesce before to local iterator while using incremental result collect

*  add log

* use df.rdd.getPartitionNum as upper limit

* log & ut

* bypass coalesce if df.rdd.partition.size is small

* add ut

* code cov

* code cov
 2019-03-26 17:54:40 +08:00
Kent Yao
7790bbd9df
[KYUUBI-169]Let spark prefer heap buffer by default while using netty rpc #172 
fix #169 set spark.shuffle.io.preferDirectBufs = false by default (#172)
 2019-03-21 16:33:47 +08:00
Kent Yao
ba9b424110
[KYUUBI-150][FOLLOWUP]using the classLoader in IsolatedClassLoader (#166) 
* reset class loader

* do not add jar twice in one create function cmd

* the classloader should be closeable

* the classloader should be closeable revert

* Revert "the classloader should be closeable revert"

This reverts commit 35602a0b689da0224a4fe3dcd778afabc3f7a803.

* add ut

* add ut

* add ut

* fix ut

fix #150
 2019-03-19 15:09:36 +08:00
Kent Yao
13c68b5550
[KYUUBI-167]Handling ApplicationMaster token expiry to support long caching sparkcontext with dynamic executor allocation (#168) 
* [KYUUBI-167]Handling ApplicationMaster token expiry to support long caching sparkcontext with dynamic executor allocation - fix #167

* fix ut

* add ut

* fix ut

* add doc

* fix https://github.com/apache/spark/pull/24120

* logger

* typo

* comment/dist jar

* fix ut

* typo
 2019-03-19 13:51:08 +08:00
Kent Yao
56fa56a880
[KYUUBI-164]display root casue of statement failure on client size (#165) 
* [KYUUBI-164]display root casue of statement failure on client size

* fix ut

* fix ut
 2019-03-12 17:05:50 +08:00
Kent Yao
0ab29e5d64
[KYUUBI-162]use login user to impersonation fix #162 (#163) 2019-03-11 15:35:27 +08:00
Kent Yao
70c5c6d97a
[KYUUBI-160]create spark context thread-safely (#161) 
* fix #160 create spark context thread-safely

* using object wait
 2019-03-06 23:13:19 +08:00
Kent Yao
7631ac3a8b
fix#158 do not delete session resouces dir in close session (#159) 2019-03-05 20:14:13 +08:00
Kent Yao
7401ca3c6e
[KYUUBI-135][KYUUBI-142]add support for spark 2.4.0 release (#157) 
* fix #135 spark 2.4.0 support

* fix #142

* import

* add ut

* fix ut

* fix ut

* fix ut

* fix ut
 2019-03-02 01:16:50 +08:00
Kent Yao
185d60a273
fix #153 hive serde orc file reader has a thread pool for parallel splits gathering which case token issue in kerberized clusters (#155) 
* fix #153 hive serde orc file reader has a threadpool for parallel splits gathering which case token issue in kerberized clusters

* ut

*  fix ut
 2019-02-28 17:51:14 +08:00
Kent Yao
51e7a5d480
Prepare for developing v0.6.0 (#154) 2019-02-27 13:40:29 +08:00
Kent Yao
ae85c245d2
Prepare releasing v0.5.0 (#152) 
* Prepare releasing 0.5.0

* update documentations
 2019-02-25 12:33:48 +08:00
Kent Yao
9dd22c7216
Update log4j.xml 2019-02-23 02:10:00 +08:00
Kent Yao
72e664fced
[KYUUBI-116][Experimental] Support long cache spark session in kerberized cluster (#117) 
* fixes @116 Support long caching SparkSession/SparkContext for secured hadoop cluster

* handle sub classes of coarse grained scheduler backend

* fix ut for spark 2.2

* updating doc
 2019-02-23 01:10:05 +08:00
Kent Yao
a0a21f4eb3 fix ut 2019-02-21 18:46:16 +08:00
Kent Yao
128928b2fc typo 2019-02-21 17:46:19 +08:00
Kent Yao
443ef11fce add more uts 2019-02-21 16:53:29 +08:00
Kent Yao
0495a55f9b ut 2019-02-21 11:15:58 +08:00
Kent Yao
bf309017c9 dectect associate plans in parse phase 2019-02-20 11:56:20 +08:00
Kent Yao
2cfb2ae180 fix resource local path 2019-02-20 11:15:33 +08:00
Kent Yao
2f3c582369 set hdfs to be downloadable 2019-02-20 10:57:18 +08:00
Kent Yao
694e3ef791 localize remote jars to add them into classpath 2019-02-20 10:21:18 +08:00
Kent Yao
05653d91f6 fixes #150 un-recognized class while using add jar & create functions 2019-02-18 15:34:05 +08:00
Kent Yao
df773045cb fix ci 2019-01-25 11:47:20 +08:00
Kent Yao
69276defa3 improve code cov 2019-01-25 11:30:03 +08:00
Kent Yao
bc48b02666 fix #147 User Latest Logout information should be cleaned after backend session is stopped by session cache manager 2019-01-23 16:44:44 +08:00
maxxie
003da4d8dd 修复可能存在并发opensession的时候，realUser 可能会获取另一个session 的user 2019-01-20 23:31:08 +08:00
Kent Yao
4f4480b07e fix ut 2019-01-14 18:37:14 +08:00
Kent Yao
a80d69a655 fix #143 Set active SparkContext in KyuubiOperation 2019-01-14 17:56:18 +08:00
Kent Yao
48655f89d1 typo 2019-01-11 16:38:11 +08:00
Kent Yao
7171f77d7a fix #140 Add a server side configuration to limit the query result size 2019-01-11 15:37:17 +08:00
Kent Yao
d028049b5d asc ii map for fun 2018-12-28 18:17:20 +08:00
Kent Yao
5fe0a34ea4 Preparing v0.5.0 developing 2018-12-21 14:48:05 +08:00
Kent Yao
203f61589e rm unrelated change 2018-12-20 16:03:53 +08:00
Kent Yao
40ac044500 turn off union rdd parallel lising 2018-12-20 15:56:00 +08:00
Kent Yao
4b7f2c5b1e [KYUUBI-133]fix #133 token expiration in HadoopRDD getPartitions 2018-12-13 22:02:39 +08:00
Kent Yao
e960540c32
Merge pull request #131 from yaooqinn/KYUUBI-128 
[KYUUBI-128]add ldap authenticate support
 2018-12-12 10:42:24 +08:00
Kent Yao
942762b929 minor doc 2018-12-12 10:00:24 +08:00
Kent Yao
672fba544a add ui 2018-12-11 18:10:13 +08:00
Kent Yao
48a15c620e fix ut 2018-12-11 16:44:52 +08:00
Kent Yao
be33573e3f reviewed by dockerzhang 2018-12-11 16:25:42 +08:00
Kent Yao
9556edd9b4 add ut 2018-12-11 15:52:26 +08:00
Kent Yao
2b172a2cba fix #125 hotfix session ugi not wrapping sparkcontext initializing 2018-12-11 11:59:10 +08:00
Kent Yao
a4f5910138 fixes #128 add ldap authenticate support 2018-12-11 10:46:52 +08:00
Kent Yao
df1abd4702
Merge pull request #127 from hustfeiwang/KYUUBI-125 
KYUUBI-125 fix the error when create session in SparkSessionWithUGI fix #125
 2018-12-06 17:52:28 +08:00
Kent Yao
989068ba24
Merge pull request #123 from yaooqinn/KYUUBI-122 
[KYUUBI-122]obtain delegation tokens from possible kerberized services
 2018-12-06 14:51:23 +08:00
Kent Yao
0a166e71ae fix ut 2018-12-06 11:02:15 +08:00
hustfeiwang
6de108ee9a KYUUBI-129 kyuubi throws java.lang.UnsupportedOperationException when starting 2018-12-06 10:56:20 +08:00
Kent Yao
cf617ee5a6 fix ut 2018-12-06 10:26:20 +08:00
Kent Yao
5f5f85d6b6 add it 2018-12-05 18:33:54 +08:00
Kent Yao
3779e55e65 hdfs no host 2018-12-05 15:53:10 +08:00
hustfeiwang
1e204ba35d KYUUBI-125 fix code style 2018-11-28 09:45:52 +08:00
hustfeiwang
e60f10b927 KYUUBI-125 fix the error when create session in SparkSessionWithUGI 2018-11-27 19:45:55 +08:00
Kent Yao
bea9ebd0e7 add log 2018-11-16 17:03:58 +08:00
Kent Yao
188bb69147 add creds to current ugi directly 2018-11-16 14:33:02 +08:00
Kent Yao
e5ed088181 use session ugi to collect tokens 2018-11-16 14:02:31 +08:00
Kent Yao
6368cb1ebe fix #122 Obtain tokens for kerberized Hive/HDFS services 2018-11-16 11:30:14 +08:00
Kent Yao
d1d34e5856
Merge pull request #119 from yaooqinn/KYUUBI-118 
[KYUUBI-118]fix npe bug with incremental result collection
 2018-11-07 17:37:24 +08:00
Kent Yao
5a957ef5d8 ignore SparkEnv code cov 2018-11-07 17:13:39 +08:00
Kent Yao
d4b9d61ea4 fix #118 fix npe with incremental result collection 2018-11-07 16:54:09 +08:00
Kent Yao
e39c56fb45 typo 2018-11-05 18:25:56 +08:00
Kent Yao
65448dda04 fix #114 kyuubi should work well with spark local mode 2018-11-05 17:40:46 +08:00
Kent Yao
a4cb010feb add more uts 2018-10-30 17:47:55 +08:00
Kent Yao
b593760f62 add more uts 2018-10-30 10:43:29 +08:00
Kent Yao
ec43bde651 fix compile err 2018-10-26 17:46:21 +08:00
Kent Yao
d8a4509405 fix #107 kill yarn application fast 2018-10-26 17:38:30 +08:00
Kent Yao
2862325408 fix #108 2018-10-25 16:53:37 +08:00