Commit Graph

145 Commits

Author SHA1 Message Date
Kent Yao
7682225145
Add github action for pull request and commit (#240)
* Add github action for pull request and commit

* fix

* ut
2020-08-11 10:23:53 +08:00
Kent Yao
a3891b3c53 Implementaion of Spark SQL engine 2020-08-10 17:52:10 +08:00
Kent Yao
eea65272e2 Add Operation for spark engine 2020-07-31 11:09:11 +08:00
Kent Yao
4d2ec8a00a Add new FrontendService 2020-07-21 15:26:48 +08:00
Kent Yao
934a9d3158 Support RowSet 2020-07-08 11:38:48 +08:00
Kent Yao
e3e1a8f00b Add OperationState 2020-06-30 15:29:59 +08:00
Kent Yao
1a3e886627 drop module kyuubi-hive-thrift 2020-06-23 17:32:43 +08:00
Kent Yao
c2271ac05b Bumpup Spark to 3.0.0 from 3.0.0-preview2 2020-06-18 17:41:43 +08:00
Kent Yao
e1d55f8351 Introduce ServiceDiscovery 2020-06-17 16:14:20 +08:00
Kent Yao
5100d8a519 Support EmbeddedZkServer 2020-06-12 17:18:46 +08:00
Kent Yao
15d179507f Create Kyuubi Project Spark SQL Engine 2020-06-09 10:34:47 +08:00
konradqcharczyk
85689b6b64 Squashed commit of the following:
commit cd39627e46666c80bcf155fed647d2a84328af99
Merge: aae0cef 9b03db7
Author: konradqcharczyk <konradqcharczyk@gmail.com>
Date:   Sun Jun 7 16:16:00 2020 +0200

    apache mirror and spark version changed

commit aae0cef4347c55a68f74f426b98378b39d0b8bdb
Author: konradqcharczyk <konradqcharczyk@gmail.com>
Date:   Sun Jun 7 14:51:16 2020 +0200

    apache mirror and spark version changed

commit 9b03db77aebb73fe3587478c36c282a750116164
Author: konradqcharczyk <konradqcharczyk@gmail.com>
Date:   Sun Jun 7 14:51:16 2020 +0200

    spark 2.4.3 changes
2020-06-07 22:30:14 +08:00
Kent Yao
651db99866
HIVE_CLI_SERVICE_PROTOCOL_V11 (#235)
* HIVE_CLI_SERVICE_PROTOCOL_V11

* pom

* style

* tests

* travis

* code cover
2020-05-26 16:07:41 +08:00
Kent Yao
b9a50d62ee new scripts for new framework 2020-05-22 19:16:04 +08:00
Kent Yao
17c3c054b7
[KYUUBI-233] Make master branch target apache spark 2.4 #233 (#234) 2020-05-18 11:54:20 +08:00
Kent Yao
af9801beb8
[KYUUBI-230] Re-implement OperationLog (#231) 2020-05-18 10:42:00 +08:00
QianChao
d227e616b2 [KYUUBI-221] Optimise FetchResults speed when maxRows is large fix #221
Squashed commit of the following:

commit f103b8d5dcbdb838b9dc1d46faec8e93c10af865
Author: QianChao <chaoqian@advance.ai>
Date:   Fri Oct 25 10:14:30 2019 +0800

    UPD: add more test cases for ColumnBaseSet

commit eabd5d4341a0ceabce9f68e20fe173415ecae981
Author: QianChao <chaoqian@advance.ai>
Date:   Thu Oct 17 15:16:40 2019 +0800

    FIX: ColumnBaseSet nulls with stream issue

commit 14c04820db2c6a22413e4b775ca71bae55d51aaf
Author: QianChao <chaoqian@advance.ai>
Date:   Wed Oct 16 15:04:22 2019 +0800

    FIX: stream access issue
2019-10-25 11:34:29 +08:00
Kent Yao
c6ef20089e
[KYUUBI-219]Fix NPE in HiveTokenCollector (#220) 2019-10-23 16:02:23 +08:00
Kent Yao
191b8171d6
[KYUUBI-216]Make thrift IDL a separate sub module fix #216 (#217)
* [KYUUBI-216]Make thrift IDL a separate sub module fix #216

* typo
2019-09-25 11:38:35 +08:00
Kent Yao
947394350f
[KYUUBI-213] Check and wait the old spark context completely stopped before create a new one (#214)
* [KYUUBI-213] Check and wait the old spark context completely stopped before create a new one #213

* stop to not stop
2019-08-26 15:04:04 +08:00
Kent Yao
6621d361af
add user info and statement id in executing log (#212)
* add user info and statement id in executing log

* add ut

* update conf

* long max

* modify log

* modify log

* unpersit

* typo

* add ut

* fix ut

* typo

* code cov

* add log

* ex order

* rm some ex

* none get

* Revert "none get"

This reverts commit 6792e33db11619ec4fa7c075706da1500fb229e3.

* rm debug
2019-08-16 16:51:12 +08:00
Kent Yao
552bb100c1
[KYUUBI-208] add delegation tokens to hive session state's dfs client (#209)
* [KYUUBI-208] add delegation tokens to hive session state's dfs client

* add ut

* handle exception

* should wrap in do as

fix #208
2019-07-25 10:39:45 +08:00
Kent Yao
7729ff0ba5
[KYUUBI-206] Initialize external catalog ahead during open session for a user fix #206 (#207)
* [KYUUBI-206] Initialize external catalog ahead during open session for a user fix #206

* code cov

* code cov
2019-07-15 11:42:12 +08:00
Kent Yao
5edf961a44
[KYUUBI-204]Apache Kudu Integration Guide fix #204 (#205)
* [KYUUBI-204]Apache Kudu Integration Guide fix #204

* references
2019-07-09 16:47:20 +08:00
Kent Yao
abc58c4ee3
[KYUUBI-201]Add some user friendly spark configurations (#202)
fix #201
2019-06-25 18:40:03 +08:00
Kent Yao
f4c08c6f52
[KUUBI-198][FOLLOWUP]Fix SparkSessionCache unit test occasionally fails (#200)
* fix travis occasionally fails

* ad ut
2019-06-20 21:46:55 +08:00
Kent Yao
b6591d3314
[KYUUBI-198] Failover mode ha service disconnects from zk after lost leader ship (#199)
* fxi #198 Failover mode ha service disconnects from zk after losts leader ship

* reset leader latch

* add ut

* add some ut back
2019-06-20 17:56:22 +08:00
Kent Yao
63245dd691
[KYUUBI-196]Refactoring SparkSessionWithUGI‘s get and create SparkContext logic (#197)
* fxit 194 Refactoring SparkSessionWithUGI‘s get and create sc logic

* upgrade scala-maven-plugin to 3.4.6

* fix travis

* fix travis again

* fix travis open jdk8

* fix travis open jdk8

* fix travis open jdk8

* fix travis open jdk8

* fix travis open jdk8

* add state to get yarn applications

* promise complete

* fix #69 capture spark exceptions

* typo

* fix #196 add ut and log

* typo

* typo

* add ut
2019-06-17 11:47:13 +08:00
Kent Yao
e295995959
[KYUUBI-194]Add a server side metrics system (#195)
* fix #194 add server side metrics system

* add spark session cache size

* add ut

* add ut

* add doc
2019-06-12 11:12:18 +08:00
Kent Yao
f8a4775365
[KYUUBI-180]add a sync execute statement operation (#192)
* fix #180 add sync execute statement operation

* fix ut
2019-06-04 18:51:07 +08:00
Kent Yao
cf4ee6afb6
[KYUUBI-156][KYUUBI-185] Implementing all meta operations support by spark (#191)
* [KYUUBI-156][KYUUBI-185] Implementing all meta operations support by spark

* rm used metastore client fix #156 fix #185

* add more uts

* add more uts

* fix bugs and add show functions

* meta operation method abstraction

* readme
2019-06-04 13:19:23 +08:00
Kent Yao
71eea2ba1f
[KYUUBI-189]Add an OOM Hook to shutdown server when OutOfMemoryError occurs (#190)
fix #189 

* [KYUUBI-189]Add an OOM Hook to shutdown server when OutOfMemoryError occurs

* add ut

* fit ut

* fit ut

* typo
2019-05-25 16:42:13 +08:00
hustfeiwang
5ad223556b [KYUUBI-120]define interface class IKyuubiOperation to allow different implementation of kyuubiOperation
---

---
fix #181 fix #120
---
Squashed commit of the following:

commit 3e7f81f3b8e5c4663fa54ed3e676312139693197
Author: hustfeiwang <wangfei3@corp.netease.com>
Date:   Thu May 23 18:32:08 2019 +0800

    fix unit test

commit 73763d7ca8416156132cb9e8df3e86966bf8c6e6
Author: hustfeiwang <wangfei3@corp.netease.com>
Date:   Thu May 23 09:48:21 2019 +0800

    set statementId to a val

commit 128effa6a9a89c5bb8f9c91f219e4e257ff96372
Author: hustfeiwang <wangfei3@corp.netease.com>
Date:   Wed May 22 16:06:45 2019 +0800

    set isClosedOrCanceled to a protected method

commit 267b55f1d2497872e348fc494a3ff7132a6012f0
Author: hustfeiwang <wangfei3@corp.netease.com>
Date:   Wed May 22 10:10:48 2019 +0800

    KYUUBI-120: define interface class IKyuubiOperation to allow different implementation of kyuubiOperation
2019-05-24 11:55:32 +08:00
Kent Yao
416122675d
[KYUUBI-186][FOLLOWUP]fix ut occasionally failure fix #186 (#188)
* [KYUUBI-186][FOLLOWUP]fix ut occasionally failure fix #186

* fix ut

* code cover
2019-05-22 20:29:33 +08:00
Kent Yao
505e01d7e2 [KYUUBI-186]Add a max cache time to clean SparkSessions that may have token expiry issue
---

---

fix #186 fix #187

---
Squashed commit of the following:

commit 4b8297f8d76b40f22e5419c72f0ddf1eadb3c7d9
Author: Kent Yao <yaooqinn@hotmail.com>
Date:   Mon May 20 19:36:34 2019 +0800

    add log

commit 7b811f8a0ccb65fb0d0832ba1313e3c099f74dad
Author: Kent Yao <yaooqinn@hotmail.com>
Date:   Thu May 16 16:46:04 2019 +0800

    add ut

commit 19f67fa8909c4a19f49fbe3b30a75d09c0736350
Author: Kent Yao <yaooqinn@hotmail.com>
Date:   Thu May 16 10:57:24 2019 +0800

    fix ut

commit 7ad7c20f6536d3bbc9d789c71af238331a3e03c5
Author: Kent Yao <yaooqinn@hotmail.com>
Date:   Thu May 16 10:40:42 2019 +0800

    fix ut

commit 9a114abb8be3083160ea01dab188333f675677e4
Author: Kent Yao <yaooqinn@hotmail.com>
Date:   Thu May 16 00:10:57 2019 +0800

    mv cache validating in its own class

commit 0afba5a5ab7a005948acde545b4a68c8d26bfce9
Author: Kent Yao <yaooqinn@hotmail.com>
Date:   Wed May 15 11:47:42 2019 +0800

    fix ut

commit eff3f41f91b4bfc2bfc84efc6af29fa4365d9dae
Author: Kent Yao <yaooqinn@hotmail.com>
Date:   Wed May 15 11:12:51 2019 +0800

    add ut

commit 9bbbea78bcb8f5d409819628974628f0bf59f69c
Author: Kent Yao <yaooqinn@hotmail.com>
Date:   Tue May 14 23:17:00 2019 +0800

    add ut

commit 0e0e59eedc055ddb6788a9dd6f07d4308772145d
Author: Kent Yao <yaooqinn@hotmail.com>
Date:   Tue May 14 19:39:09 2019 +0800

    mv init time to spark session cache

commit a4a1c69e29ec7d5d51a0e5026cf8f3c2aaf6b2ef
Author: Kent Yao <yaooqinn@hotmail.com>
Date:   Tue May 14 18:36:06 2019 +0800

    Add a max cache time to clean SparkSessions that may have token expiry issue fix #186
2019-05-21 22:07:18 +08:00
Kent Yao
d7f95775de [BUILD]Bump up to v0.7.0-SNAPSHOT 2019-05-10 11:58:22 +08:00
Kent Yao
1f0bc742e4
[KYUUBI-167][FOLLOWUP]populate tokens via spark session cache mgr (#183)
* Prepare releasing v0.6.0

* fix #KYUUBI-167 populate token via spark session cache mgr

* typo

* fix ut

* code cov
2019-05-07 10:32:04 +08:00
Kent Yao
32c95eacf1
[KYUUBI-170]persist and coalesce before to local iterator while using incremental result collect (#171)
* fix #170 persist and coalesce before to local iterator while using incremental result collect

*  add log

* use df.rdd.getPartitionNum as upper limit

* log & ut

* bypass coalesce if df.rdd.partition.size is small

* add ut

* code cov

* code cov
2019-03-26 17:54:40 +08:00
Kent Yao
7790bbd9df
[KYUUBI-169]Let spark prefer heap buffer by default while using netty rpc #172
fix #169 set spark.shuffle.io.preferDirectBufs = false by default (#172)
2019-03-21 16:33:47 +08:00
Kent Yao
ba9b424110
[KYUUBI-150][FOLLOWUP]using the classLoader in IsolatedClassLoader (#166)
* reset class loader

* do not add jar twice in one create function cmd

* the classloader should be closeable

* the classloader should be closeable revert

* Revert "the classloader should be closeable revert"

This reverts commit 35602a0b689da0224a4fe3dcd778afabc3f7a803.

* add ut

* add ut

* add ut

* fix ut

fix #150
2019-03-19 15:09:36 +08:00
Kent Yao
13c68b5550
[KYUUBI-167]Handling ApplicationMaster token expiry to support long caching sparkcontext with dynamic executor allocation (#168)
* [KYUUBI-167]Handling ApplicationMaster token expiry to support long caching sparkcontext with dynamic executor allocation - fix #167

* fix ut

* add ut

* fix ut

* add doc

* fix https://github.com/apache/spark/pull/24120

* logger

* typo

* comment/dist jar

* fix ut

* typo
2019-03-19 13:51:08 +08:00
Kent Yao
56fa56a880
[KYUUBI-164]display root casue of statement failure on client size (#165)
* [KYUUBI-164]display root casue of statement failure on client size

* fix ut

* fix ut
2019-03-12 17:05:50 +08:00
Kent Yao
0ab29e5d64
[KYUUBI-162]use login user to impersonation fix #162 (#163) 2019-03-11 15:35:27 +08:00
Kent Yao
70c5c6d97a
[KYUUBI-160]create spark context thread-safely (#161)
* fix #160 create spark context thread-safely

* using object wait
2019-03-06 23:13:19 +08:00
Kent Yao
7631ac3a8b
fix#158 do not delete session resouces dir in close session (#159) 2019-03-05 20:14:13 +08:00
Kent Yao
7401ca3c6e
[KYUUBI-135][KYUUBI-142]add support for spark 2.4.0 release (#157)
* fix #135 spark 2.4.0 support

* fix #142

* import

* add ut

* fix ut

* fix ut

* fix ut

* fix ut
2019-03-02 01:16:50 +08:00
Kent Yao
185d60a273
fix #153 hive serde orc file reader has a thread pool for parallel splits gathering which case token issue in kerberized clusters (#155)
* fix #153 hive serde orc file reader has a threadpool for parallel splits gathering which case token issue in kerberized clusters

* ut

*  fix ut
2019-02-28 17:51:14 +08:00
Kent Yao
51e7a5d480
Prepare for developing v0.6.0 (#154) 2019-02-27 13:40:29 +08:00
Kent Yao
ae85c245d2
Prepare releasing v0.5.0 (#152)
* Prepare releasing 0.5.0

* update documentations
2019-02-25 12:33:48 +08:00
Kent Yao
9dd22c7216
Update log4j.xml 2019-02-23 02:10:00 +08:00