apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity
1,150 Commits 23 Branches 0 Tags 65 MiB
53addb62c5
Commit Graph

208 Commits

Author SHA1 Message Date
Kent Yao
32c95eacf1
[KYUUBI-170]persist and coalesce before to local iterator while using incremental result collect (#171) 
* fix #170 persist and coalesce before to local iterator while using incremental result collect

*  add log

* use df.rdd.getPartitionNum as upper limit

* log & ut

* bypass coalesce if df.rdd.partition.size is small

* add ut

* code cov

* code cov
 2019-03-26 17:54:40 +08:00
Kent Yao
7790bbd9df
[KYUUBI-169]Let spark prefer heap buffer by default while using netty rpc #172 
fix #169 set spark.shuffle.io.preferDirectBufs = false by default (#172)
 2019-03-21 16:33:47 +08:00
Kent Yao
ba9b424110
[KYUUBI-150][FOLLOWUP]using the classLoader in IsolatedClassLoader (#166) 
* reset class loader

* do not add jar twice in one create function cmd

* the classloader should be closeable

* the classloader should be closeable revert

* Revert "the classloader should be closeable revert"

This reverts commit 35602a0b689da0224a4fe3dcd778afabc3f7a803.

* add ut

* add ut

* add ut

* fix ut

fix #150
 2019-03-19 15:09:36 +08:00
Kent Yao
13c68b5550
[KYUUBI-167]Handling ApplicationMaster token expiry to support long caching sparkcontext with dynamic executor allocation (#168) 
* [KYUUBI-167]Handling ApplicationMaster token expiry to support long caching sparkcontext with dynamic executor allocation - fix #167

* fix ut

* add ut

* fix ut

* add doc

* fix https://github.com/apache/spark/pull/24120

* logger

* typo

* comment/dist jar

* fix ut

* typo
 2019-03-19 13:51:08 +08:00
Kent Yao
56fa56a880
[KYUUBI-164]display root casue of statement failure on client size (#165) 
* [KYUUBI-164]display root casue of statement failure on client size

* fix ut

* fix ut
 2019-03-12 17:05:50 +08:00
Kent Yao
0ab29e5d64
[KYUUBI-162]use login user to impersonation fix #162 (#163) 2019-03-11 15:35:27 +08:00
Kent Yao
70c5c6d97a
[KYUUBI-160]create spark context thread-safely (#161) 
* fix #160 create spark context thread-safely

* using object wait
 2019-03-06 23:13:19 +08:00
Kent Yao
7631ac3a8b
fix#158 do not delete session resouces dir in close session (#159) 2019-03-05 20:14:13 +08:00
Kent Yao
7401ca3c6e
[KYUUBI-135][KYUUBI-142]add support for spark 2.4.0 release (#157) 
* fix #135 spark 2.4.0 support

* fix #142

* import

* add ut

* fix ut

* fix ut

* fix ut

* fix ut
 2019-03-02 01:16:50 +08:00
Kent Yao
185d60a273
fix #153 hive serde orc file reader has a thread pool for parallel splits gathering which case token issue in kerberized clusters (#155) 
* fix #153 hive serde orc file reader has a threadpool for parallel splits gathering which case token issue in kerberized clusters

* ut

*  fix ut
 2019-02-28 17:51:14 +08:00
Kent Yao
51e7a5d480
Prepare for developing v0.6.0 (#154) 2019-02-27 13:40:29 +08:00
Kent Yao
ae85c245d2
Prepare releasing v0.5.0 (#152) 
* Prepare releasing 0.5.0

* update documentations
 2019-02-25 12:33:48 +08:00
Kent Yao
9dd22c7216
Update log4j.xml 2019-02-23 02:10:00 +08:00
Kent Yao
72e664fced
[KYUUBI-116][Experimental] Support long cache spark session in kerberized cluster (#117) 
* fixes @116 Support long caching SparkSession/SparkContext for secured hadoop cluster

* handle sub classes of coarse grained scheduler backend

* fix ut for spark 2.2

* updating doc
 2019-02-23 01:10:05 +08:00
Kent Yao
a0a21f4eb3 fix ut 2019-02-21 18:46:16 +08:00
Kent Yao
128928b2fc typo 2019-02-21 17:46:19 +08:00
Kent Yao
443ef11fce add more uts 2019-02-21 16:53:29 +08:00
Kent Yao
0495a55f9b ut 2019-02-21 11:15:58 +08:00
Kent Yao
bf309017c9 dectect associate plans in parse phase 2019-02-20 11:56:20 +08:00
Kent Yao
2cfb2ae180 fix resource local path 2019-02-20 11:15:33 +08:00
Kent Yao
2f3c582369 set hdfs to be downloadable 2019-02-20 10:57:18 +08:00
Kent Yao
694e3ef791 localize remote jars to add them into classpath 2019-02-20 10:21:18 +08:00
Kent Yao
05653d91f6 fixes #150 un-recognized class while using add jar & create functions 2019-02-18 15:34:05 +08:00
Kent Yao
df773045cb fix ci 2019-01-25 11:47:20 +08:00
Kent Yao
69276defa3 improve code cov 2019-01-25 11:30:03 +08:00
Kent Yao
bc48b02666 fix #147 User Latest Logout information should be cleaned after backend session is stopped by session cache manager 2019-01-23 16:44:44 +08:00
maxxie
003da4d8dd 修复可能存在并发opensession的时候，realUser 可能会获取另一个session 的user 2019-01-20 23:31:08 +08:00
Kent Yao
4f4480b07e fix ut 2019-01-14 18:37:14 +08:00
Kent Yao
a80d69a655 fix #143 Set active SparkContext in KyuubiOperation 2019-01-14 17:56:18 +08:00
Kent Yao
48655f89d1 typo 2019-01-11 16:38:11 +08:00
Kent Yao
7171f77d7a fix #140 Add a server side configuration to limit the query result size 2019-01-11 15:37:17 +08:00
Kent Yao
d028049b5d asc ii map for fun 2018-12-28 18:17:20 +08:00
Kent Yao
5fe0a34ea4 Preparing v0.5.0 developing 2018-12-21 14:48:05 +08:00
Kent Yao
203f61589e rm unrelated change 2018-12-20 16:03:53 +08:00
Kent Yao
40ac044500 turn off union rdd parallel lising 2018-12-20 15:56:00 +08:00
Kent Yao
4b7f2c5b1e [KYUUBI-133]fix #133 token expiration in HadoopRDD getPartitions 2018-12-13 22:02:39 +08:00
Kent Yao
e960540c32
Merge pull request #131 from yaooqinn/KYUUBI-128 
[KYUUBI-128]add ldap authenticate support
 2018-12-12 10:42:24 +08:00
Kent Yao
942762b929 minor doc 2018-12-12 10:00:24 +08:00
Kent Yao
672fba544a add ui 2018-12-11 18:10:13 +08:00
Kent Yao
48a15c620e fix ut 2018-12-11 16:44:52 +08:00
Kent Yao
be33573e3f reviewed by dockerzhang 2018-12-11 16:25:42 +08:00
Kent Yao
9556edd9b4 add ut 2018-12-11 15:52:26 +08:00
Kent Yao
2b172a2cba fix #125 hotfix session ugi not wrapping sparkcontext initializing 2018-12-11 11:59:10 +08:00
Kent Yao
a4f5910138 fixes #128 add ldap authenticate support 2018-12-11 10:46:52 +08:00
Kent Yao
df1abd4702
Merge pull request #127 from hustfeiwang/KYUUBI-125 
KYUUBI-125 fix the error when create session in SparkSessionWithUGI fix #125
 2018-12-06 17:52:28 +08:00
Kent Yao
989068ba24
Merge pull request #123 from yaooqinn/KYUUBI-122 
[KYUUBI-122]obtain delegation tokens from possible kerberized services
 2018-12-06 14:51:23 +08:00
Kent Yao
0a166e71ae fix ut 2018-12-06 11:02:15 +08:00
hustfeiwang
6de108ee9a KYUUBI-129 kyuubi throws java.lang.UnsupportedOperationException when starting 2018-12-06 10:56:20 +08:00
Kent Yao
cf617ee5a6 fix ut 2018-12-06 10:26:20 +08:00
Kent Yao
5f5f85d6b6 add it 2018-12-05 18:33:54 +08:00
Kent Yao
3779e55e65 hdfs no host 2018-12-05 15:53:10 +08:00
hustfeiwang
1e204ba35d KYUUBI-125 fix code style 2018-11-28 09:45:52 +08:00
hustfeiwang
e60f10b927 KYUUBI-125 fix the error when create session in SparkSessionWithUGI 2018-11-27 19:45:55 +08:00
Kent Yao
bea9ebd0e7 add log 2018-11-16 17:03:58 +08:00
Kent Yao
188bb69147 add creds to current ugi directly 2018-11-16 14:33:02 +08:00
Kent Yao
e5ed088181 use session ugi to collect tokens 2018-11-16 14:02:31 +08:00
Kent Yao
6368cb1ebe fix #122 Obtain tokens for kerberized Hive/HDFS services 2018-11-16 11:30:14 +08:00
Kent Yao
d1d34e5856
Merge pull request #119 from yaooqinn/KYUUBI-118 
[KYUUBI-118]fix npe bug with incremental result collection
 2018-11-07 17:37:24 +08:00
Kent Yao
5a957ef5d8 ignore SparkEnv code cov 2018-11-07 17:13:39 +08:00
Kent Yao
d4b9d61ea4 fix #118 fix npe with incremental result collection 2018-11-07 16:54:09 +08:00
Kent Yao
e39c56fb45 typo 2018-11-05 18:25:56 +08:00
Kent Yao
65448dda04 fix #114 kyuubi should work well with spark local mode 2018-11-05 17:40:46 +08:00
Kent Yao
a4cb010feb add more uts 2018-10-30 17:47:55 +08:00
Kent Yao
b593760f62 add more uts 2018-10-30 10:43:29 +08:00
Kent Yao
ec43bde651 fix compile err 2018-10-26 17:46:21 +08:00
Kent Yao
d8a4509405 fix #107 kill yarn application fast 2018-10-26 17:38:30 +08:00
Kent Yao
2862325408 fix #108 2018-10-25 16:53:37 +08:00
Kent Yao
c1a39552a7 add ut 2018-10-25 11:26:41 +08:00
Kent Yao
f297be1214 fix travis ut 2018-10-25 10:43:18 +08:00
Kent Yao
48ef2f0401 fix #109 add meaningful msg for front service port confliction 2018-10-25 10:29:20 +08:00
Kent Yao
b591f05e4f modify project version 2018-09-29 16:22:36 +08:00
Kent Yao
60a9de2791 add documentation 2018-09-29 15:55:06 +08:00
Kent Yao
eb608a42e3 add uts 2018-09-28 15:18:56 +08:00
Kent Yao
5f84f71803 add a failover mode for kyuubi ha 2018-09-27 17:37:30 +08:00
Kent Yao
1302ebe3ce fix ut 2018-09-05 16:52:39 +08:00
Kent Yao
5204e53e72 add ut 2018-09-05 16:52:39 +08:00
Kent Yao
e78d275b6b fix #99 delay implementing spark authorizer 2018-09-05 16:52:39 +08:00
Kent Yao
8bb14f9f22 fix #99 add an configuration to skip switching db 2018-09-05 16:52:39 +08:00
Kent Yao
0e39ed2796 fix #96 spark session cache manager as abstract service 2018-08-22 10:54:36 +08:00
Kent Yao
0b3d2e9ba4 fix wrong bind port when 0 is set 2018-07-28 16:25:48 +08:00
Kent Yao
3b98cdd063 fix travis; fix port error 2018-07-27 20:02:52 +08:00
Kent Yao
53eb12f60f add state monitor 2018-07-27 16:36:51 +08:00
Kent Yao
9121a4f61b acls and print am container information 2018-07-26 18:05:06 +08:00
Kent Yao
248259d84f verify resource setting and add some comments 2018-07-26 16:36:16 +08:00
Kent Yao
94ff3f65f3 fixes #46 containerize kyuubi server with yarn 2018-07-25 11:59:19 +08:00
Kent Yao
b0ffd785ea add ut plain sasl 2018-06-30 18:05:47 +08:00
Kent Yao
c36188a246 add ut plain sasl 2018-06-30 12:12:11 +08:00
Kent Yao
3e2869be34 add more ut 2018-06-30 00:27:31 +08:00
Kent Yao
7c121a3748 fix ut again 2018-06-29 22:59:04 +08:00
Kent Yao
6f7f7c89ed fix ut again 2018-06-29 22:25:13 +08:00
Kent Yao
90553f2852 fix ut again 2018-06-29 20:19:45 +08:00
Kent Yao
6b3fc767e3 fix ut failure by sercurity 2018-06-29 20:01:35 +08:00
Kent Yao
07de3c0d00 catch ConnectionLossException tooo 2018-06-29 19:41:49 +08:00
Kent Yao
2204994ebf Kyuubi-92: Kyuubi server fails to start in ha mode for NodeExistsException fix #92 2018-06-29 16:42:56 +08:00
Kent Yao
8a7cdf4121 add ut 2018-06-25 10:31:06 +08:00
Kent Yao
921b481b4d add ut 2018-06-24 04:06:35 +08:00
Kent Yao
f165165254 add ut 2018-06-24 01:47:41 +08:00
Kent Yao
0a5de5ba65 add ut 2018-06-24 01:39:42 +08:00
Kent Yao
5345687d23 add ut 2018-06-23 00:07:40 +08:00
Kent Yao
24a7381c7f add ut 2018-06-22 23:44:05 +08:00
Kent Yao
d60b535007 add ut 2018-06-22 23:20:31 +08:00
Kent Yao
889cb37f53 add ut 2018-06-22 17:56:11 +08:00
Kent Yao
6dde5a1eb2 add ut 2018-06-22 17:53:56 +08:00
Kent Yao
5468bd9d2a add ut 2018-06-22 17:10:11 +08:00
Kent Yao
b3ab9b14e3 ServiceExceptionSuite test added 2018-06-22 16:30:11 +08:00
Kent Yao
747df3d414 ServiceUtils test added 2018-06-22 16:02:48 +08:00
Kent Yao
d7b0b83c85 delete kyuubi server exception 2018-06-22 14:50:02 +08:00
Kent Yao
6676ebfc01 init commit for making kyuubi server a sub-project fix #90 2018-06-22 14:20:04 +08:00