### _Why are the changes needed?_
Close#3064
Refer the comments by cxzl25
Spark uses the code
`URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory(hadoopConf))` supports the class path from HDFS.
But because the scala compiler only supports adding file schema urls to the class path, non-file schema urls will cause NPE.
```java
Error: Error operating ExecuteScala: java.lang.NullPointerException
at scala.tools.nsc.classpath.FileUtils$AbstractFileOps$.isJarOrZip$extension(FileUtils.scala:32)
at scala.tools.nsc.classpath.ClassPathFactory$.newClassPath(ClassPathFactory.scala:90)
at scala.tools.nsc.Global.$anonfun$extendCompilerClassPath$1(Global.scala:832)
```
scala.tools.nsc.Global#extendCompilerClassPath
```scala
AbstractFile.getURL(u)
```
scala.reflect.io.AbstractFile#getURL
```scala
def getURL(url: URL): AbstractFile =
if (url.getProtocol == "file") {
val f = new java.io.File(url.toURI)
if (f.isDirectory) getDirectory(f)
else getFile(f)
} else null
```
spark-shell supports --jars hdfs jar. At this time, submit will download the remote jar to the local and pass it to spark-shell through the `spark.repl.local.jars` configuration.
In this pr, I localize the remote jar url at first, and then add it into repl class path.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3292 from turboFei/scala_npe.
Closes#3064
095c40d8 [Fei Wang] filter
37637f76 [Fei Wang] add non-exist
ec87ea85 [Fei Wang] add ut
5a823d3b [Fei Wang] save
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
close#3411.
skip checking create privilege for output table of cache table sql
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3416 from bowenliang123/3411-authz-cache-table.
Closes#3411
95f08fbb [Bowen Liang] revert to default config value in previous ut
685e1f45 [Bowen Liang] style
70cd7a74 [Bowen Liang] update test 'CacheTableAsSelect'
19f09489 [Bowen Liang] nit
389e1ef7 [Bowen Liang] skip checking create privilege for output table of cache sql
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3414
Currently, Kyuubi supports output of planOnly in different modes in different styles, As more and more modes and styles are supported, defining enumerations in kyuubiConf is a bit messy and not easy to extend, so this pr aims to tidy up these
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3415 from Yikf/tidy-up.
Closes#3414
e81142e5 [yikf] Tidy up plan only related modes and styles
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Redirect logs to `target/unit-tests.log` as other modules did.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3399 from pan3793/lineage-log.
Closes#3399
531b8dbb [Cheng Pan] nit
543577cd [Cheng Pan] [TEST] Add log4j conf for Spark lineage plugin module
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3405
The test case is unstable, the result depends on whether we pre-build `kyuubi-flink-sql-engine` first, so we just remove it to avoid confusing users.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3408 from pan3793/test.
Closes#3408
0a7e430a [Cheng Pan] Remove the unstable flink process builder test case
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
close https://github.com/apache/incubator-kyuubi/issues/3311
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3324 from lsm1/fix/operation_language_incorrect.
Closes#3311
9192f61f [senmiaoliu] revert delete PlanOnlyOperationSuite.scala
9d0079b1 [senmiaoliu] move ut in SparkQueryTests
e8b160da [senmiaoliu] add unknown for operation language
Authored-by: senmiaoliu <senmiaoliu@trip.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3376
This pr aims support planOnly output in different styles (Users can perform SQL scheduling based on the JSON description of TreeNode)
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3377 from Yikf/planOnly-support-json.
Closes#3376
b63ae718 [yikf] Support json_description for PlanOnlyStatement
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
close#3395.
Add client docs for PyHive.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3394 from bowenliang123/3309-pyhive-doc.
Closes#3395
31a37b5c [liangbowen] add pyhive docs
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3378
This pr aims to improve hive-connector module tests and make CI perform tests for the hive connector
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3379 from Yikf/hive-connector-test.
Closes#3378
72ad050c [yikf] CI test for hive-connector
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
to close#3371
Adding an ranger plugin config for enabling full access privileges, Authz checks all access privileges and throw exception with message of unsatisfied privileges on different resources in natural order in execution plan.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3380 from bowenliang123/3371-authz-full-check.
Closes#3371
9326196d [Bowen Liang] add RangerConfigProvider for getting plugin config of different Ranger versions. add invokeStatic in AuthZUtils.
65d765dd [Bowen Liang] use ArrayBuffer for value of accessTypeToResource map. import scala LinkedHashMap.
959cd7dc [liangbowen] nit: update ut
dbfd3479 [liangbowen] 1.use LinkedHashMap and LinkedHashSet for keeping in natual order for consistent error msg output, 2. update ut
3e0a5feb [liangbowen] revert to not enforcing output error message in alphabet order
1df4130d [Bowen Liang] add ut with multiple input tables and one output table
5241fd6a [Bowen Liang] refactor verify method in more scala style
d7bc6841 [Bowen Liang] nit
32b2dd6d [Bowen Liang] nit: docs
d9787be4 [Bowen Liang] nit: docs
87b095cf [Bowen Liang] simplify verify method
a0d27e0c [Bowen Liang] simplify verify method and update docs
c9b4e065 [Bowen Liang] rename config to ranger.plugin.plugin.authorize.in.single.call, simplify access checking, and related docs
f6ccc1ac [liangbowen] move verify to SparkRangerAdminPlugin and change config name to ranger.plugin.spark.enable.full.access.violation.msg
c531cadb [Bowen Liang] add config doc in Authz
041c49b1 [Bowen Liang] move config to ranger plugin conf
f4fbeb23 [liangbowen] support for full access checking
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
For code review and check pr, we need Github CI help check web ui.
Run `npm run coverage && npm run build-check` to find syntax issues and build errors.
Actions used node from https://github.com/marketplace/actions/setup-node-js-environmentClose#3335
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3340 from zwangsheng/feature/web_ci.
Closes#3340
037953b7 [zwangsheng] fix symble |
07f6dc09 [zwangsheng] use npm install pnpm
c643333e [zwangsheng] use pnpm instead
a57bc311 [zwangsheng] install
f2a94ca2 [zwangsheng] fix runner
1d85336d [zwangsheng] fix runner
7c5c58c2 [zwangsheng] fix runner
adb94e46 [zwangsheng] fix
33359d99 [zwangsheng] fix
95a34cbb [zwangsheng] fix
7d3207a9 [zwangsheng] add ci
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Close#3208
Fix flaky test:
```
MetadataManagerSuite:
- metadata request metrics *** FAILED ***
2 did not equal 1 (MetadataManagerSuite.scala:141)
```
Cleanup the retrying metadata to prevent flaky test.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3331 from turboFei/flaky_test.
Closes#3208
5b16a9a2 [Fei Wang] use eventually
9cb8edb8 [Fei Wang] save
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Ranger adapter Arctic update sql
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3375 from hellojinsilei/feature-ranger-arctic.
Closes#3375
c60d72f9 [jinsilei] fix arctic ranger
53458f4c [jinsilei] add arctic rabger
Authored-by: jinsilei <jinsilei@corp.netease.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Some users encounter errors on building, enable -x to help user identify the failure reason
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3372 from pan3793/dist.
Closes#3372
7bdfb39f [Cheng Pan] Enable -x on build/dist
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
fix#3325
Permanent views are generally registed globally and then used for unifing, masking, joining tables for different practical purposes. The data manager would like to authorize views to users , but not all the source tables.
Authz plugin is not satisfiying this senario as it checks privileges all the source table instead of the permanent views.
Suggesting chaning behaviour of PrivilegesBuilder
- check privileges for permanent view
- skip privileges check for shadowed source view of permanent views
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3326 from bowenliang123/feature-authz-perm-view.
Closes#3325
468f47ae [Bowen Liang] fix ut
b9d2cdb6 [Bowen Liang] fix ut
82c7ae63 [Bowen Liang] fix case for #3326. add "view" type for withCleanTmpResources.
bcd35d33 [Bowen Liang] add ut
3cfb4320 [Bowen Liang] Merge branch 'master' into feature-authz-perm-view
81e8b992 [Bowen Liang] rename to hasResolvedPermanentView and check if plan resolved
4903f0a9 [Bowen Liang] change to use mergeProjection in PivilegesBuilder , checking column level privilege.
78ff9d30 [Bowen Liang] view casting (+1 squashed commit) Squashed commits: [98f523d1] fix error in view casting
3706dda0 [Bowen Liang] rename ViewAccessAnalysis to RuleApplyPermanentViewMarker (+2 squashed commits) Squashed commits: [207ed7b5] rename ViewAccessAnalysis to RuleApplyPermanentViewMarker [f58be005] nit
6ef81319 [Bowen Liang] use isSparkVersionAtLeast 3.1.0 instead of try-catch RuntimeException non-existed isTempView field
76ec6b42 [Bowen Liang] simplify and remove applyPermanentViewMarker (+2 squashed commits) Squashed commits: [802e5223] nit [ba1beb49] nit
aea561c5 [Bowen Liang] ut for spark30 and spark31greater
41c67ac9 [Bowen Liang] use reflection for isTempView of View
94a837fc [liangbowen] add ut for permanent view privilege checks
a0072c0c [liangbowen] add PermanentViewMarker for marking perm view catalogTable and check PrivilegesBuilder with PermanentViewMarker, finally maker cleanup in RuleEliminateViewMarker
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
We met the case that the LaunchEngine operation is succeed.
But the spark engine met exception and terminated with SUCCESS state.
And then the kyuubi connection stuck until reach the `kyuubi.session.engine.initialize.timeout`.
The engine application terminated in 1.5 minutes and the engine init timeout is 10m.
FYI:
So, we shall check the engine application state when waiting the engine initialization, and fast fail the connection if the application has been in terminated state.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3315 from turboFei/fast_fail_on_terminate.
Closes#3315
03333623 [Fei Wang] add comments
f1632dbb [Fei Wang] fix ut
4059a4fe [Fei Wang] combine the state
33dc4a88 [Fei Wang] only check it for exit 0
d5746af1 [Fei Wang] fix ut
854f3e3b [Fei Wang] add metrics
62001610 [Fei Wang] revert
3fb1b630 [Fei Wang] remove engine ref check, meaning less
d2c2c794 [Fei Wang] add ut
ddafa24a [Fei Wang] fast fail the connection on engine termiante
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
fix#3360.
- support batch verifying calling to Ranger plugin, keep emitting exception for first unallowed request
- batch verifying privilege of columns for single resource
Befor PR,
```
case ObjectType.COLUMN if resource.getColumns.nonEmpty =>
resource.getColumns.foreach { col =>
...
verify(req, auditHandler)
}
```
After PR,
```scala
case ObjectType.COLUMN if resource.getColumns.nonEmpty =>
val reqs = ArrayBuffer[AccessRequest]()
resource.getColumns.foreach { col =>
...
reqs += req
}
batchVerify(reqs.toList, auditHandler)
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3361 from bowenliang123/feature-authz-batch-verify.
Closes#3360
154ff781 [liangbowen] minor update with common-collections util
2617fabc [Bowen Liang] refactor verify method for batching and remove batchVerify method
dc56611e [liangbowen] minor update
9da5ce0b [liangbowen] nit
e99f5bae [Bowen Liang] implements batchVerify with SparkRangerAdminPlugin.isAccessAllowed
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
This is a minor change, aims to remove authZ-module from the profile tag, this is due to authz supports all Spark versions currently supported by Kyuubi, and authz-module is defined in the common Module
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3370 from Yikf/authz-pom-minor.
Closes#3370
3c58a095 [yikf] remove authz-module for profile
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Support to get engine id, name, url in KyuubiConnection.
It is helpful and friendly for BI.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3354 from turboFei/lanch_engine_result.
Closes#3354
4966de28 [Fei Wang] comments
984b5d35 [Fei Wang] fix ut
963217af [Fei Wang] filter null
a2c0ce2c [Fei Wang] fix
4f2a8b18 [Fei Wang] get from engine side
61607c7c [Fei Wang] return
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3357
This pr aims to improve doc description for configuration kyuubi.operation.plan.only.mode
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3358 from Yikf/planOnly-doc.
Closes#3357
571f4e0c [yikf] Improve doc description for configuration `kyuubi.operation.plan.only.mode`
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
fix#3343
Privilege checks of CreateViewCommand retained for PersistedViews only, and skipped for LocalTempView or GlobalTempView.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3344 from bowenliang123/improve-skipauth-tempviews.
Closes#3343
eff4f732 [Bowen Liang] CreateViewCommand tablePrivileges checks for only persisted views, and adding ut
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
When I testing v1.6.0-incubating-rc0 w/ `kyuubi.operation.progress.enabled=true`, the beeline broken on rendering progress bar.
```
STAGES ATTEMPT STATUS TOTAL COMPLETED RUNNING PENDING FAILED
----------------------------------------------------------------------------------------------
org/apache/commons/lang/StringUtils
Closing: 0: jdbc:hive2://0.0.0.0:10009/default
```
And got fixed after adding `commons-lang-2.6.jar`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
<img width="671" alt="image" src="https://user-images.githubusercontent.com/26535726/187076711-d060d522-c7a5-46e5-b47f-fa4786407e09.png">
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3363 from pan3793/beeline.
Closes#3363
2edf69e5 [Cheng Pan] add commons-lang dep
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
fix#3349
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3350 from jiaoqingbo/kyuubi3349.
Closes#3349
85ca155b [jiaoqingbo] [KYUUBI #3349] Bump unboundid-ldapsdk version from 5.1.4 to 6.0.5
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3352
It's a regression issue, that came from upgrading from Hive 2.3 to Hive 3.1.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3353 from pan3793/HIVE-21899.
Closes#3353
85a25c8e [Cheng Pan] Backport HIVE-21899 - Utils.getCanonicalHostName() may return IP address depending on DNS infra
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix#3352
[HIVE-17218](https://issues.apache.org/jira/browse/HIVE-17218): Canonical-ize hostnames for Hive metastore, and HS2 servers
This may have the problem mentioned by JIRA
[HIVE-20583](https://issues.apache.org/jira/browse/HIVE-20583): Use canonical hostname only for kerberos auth in HiveConnection
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3355 from pan3793/HIVE-20583.
Closes#3355
6e2ffa49 [Cheng Pan] Fix NPE
f09b8e6c [Cheng Pan] Backport HIVE-20583
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Skip maven javadoc plugin for release publish.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3348 from SteNicholas/maven-javadoc-plugin.
Closes#3348
4ba825fc [SteNicholas] Specifies none checks to be performed on Javadoc comments for maven-javadoc-plugin
Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This is a minor change, aims to relocation `AccessControlException` from `org.apache.kyuubi.plugin.spark.authz.utils` to `org.apache.kyuubi.plugin.spark.authz`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3347 from Yikf/minor-relocation.
Closes#3347
05d95a7a [yikf] Relocation `AccessControlException` from org.apache.kyuubi.plugin.spark.authz.utils to org.apache.kyuubi.plugin.spark.authz
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Now `activeStages` only has the stageid, and does not record and output the number of stage retries, which may cause errors when the stage fails to retry.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3336 from cxzl25/spark_listener_stage_attempt.
Closes#3336
a6da472a [sychen] use StageAttempt instead of StageId
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
1. `principal` supports `X/_HOSTEXAMPLE.COM`
2. `kyuubiClientPrincipal` supports headless keytab, `XEXAMPLE.COM`
https://github.com/apache/incubator-kyuubi/pull/3023
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3346 from cxzl25/3023_followup.
Closes#3023
15309298 [sychen] support principal _HOST and kyuubiClientPrincipal headless keytab
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3217#issuecomment-1219271537
This pr aims to bump ranger version from 2.2.0 to 2.3.0
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3333 from Yikf/pr/3282.
Closes#3333
07b35b9f [Cheng Pan] Update log4j2-test.xml
45a403b4 [yikf] [KYUUBI #3217][FOLLOWUP] Bump ranger version to 2.3.0
Lead-authored-by: yikf <yikaifei1@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add v1.6.0-incubating changelog.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3342 from SteNicholas/changelog-1.6.0-incubating.
Closes#3342
76bf8c6e [SteNicholas] Add v1.6.0-incubating changelog
Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
To close#3337
excludes `node_modules` folder.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3338 from zwangsheng/fix/linese_check_exclude_node_modules.
Closes#3338
80f45440 [zwangsheng] fix
4f310f70 [zwangsheng] fix
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
`npm run build` fail with following log
```
error TS6504: File '/Users/yangbinjie01/IdeaProjects/incubator-kyuubi/kyuubi-server/web-ui/src/views/operation/completedJobs/index.vue.__VLS_script.js' is a JavaScript file. Did you mean to enable the 'allowJs' option?
The file is in the program because:
Root file specified for compilation
...
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3334 from zwangsheng/fix/npm_build.
Closes#3334
4142f2ad [zwangsheng] fix
a1370a8f [zwangsheng] add
e86648ec [zwangsheng] fix npm run build
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
fix: #3327
### _Why are the changes needed?_
The contributing link in the PULL_REQUEST_TEMPLATE is out-of-date, so this patch replaces it with the new contributing link.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3329 from YesOrNo828/Kyuubi-3327.
Closes#3327
badf37b0 [yexianxun] [KYUUBI #3327] Replace the contributing link in the PR template
Authored-by: yexianxun <yexianxun@corp.netease.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Now the connection path is:
`client`-> `kyuubiServer` -> `kyuubiEngine`
Maybe the connection between `client -> kyuubiServer` is valid, but the connection between `kyuubiServer -> kyuubiEngine` is not.
So we need check the whole path.
BTW, before, if customer invoke KyuubiConnection::isValid, it can make the connection between client and kyuubiServer keep alive.
Now, it can make the whole connection path keep alive.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3318 from turboFei/is_valid.
Closes#3318
af4d6b7b [Fei Wang] Transfer the TGetInfoReq to kyuubi engine side to keep the kyuubi connection alive
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
close https://github.com/apache/incubator-kyuubi/issues/3128
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3304 from Kwafoor/master.
Closes#3128
88a5ebe9 [wangjunbo] [kyuubi-3128] Support CostMode for PlanOnlyStatement
Authored-by: wangjunbo <wangjunbo@qiyi.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Currently, Apache Kyuubi support Spark 3.0.0 and above, but Kyuubi-AuthZ support Spark 2.4, Spark 3.0 and above, to avoid subsequent maintenance costs, we plan to move out support for Spark 2.4.
This pr aims to modify the documentation to indicate that version 2.4 is not supported
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3305 from Yikf/authz-spark2.4-notsupport.
Closes#3226
5c756f51 [yikf] authz not support spark2.4
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3306
Currently, Kyuubi-AuthZ support Apache Spark 3.0 and above, and there are no CI workflow to check compatibility with Apache Spark 3.0, this pr aim to add a CI workflow for AuthZ module
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3307 from Yikf/GA.
Closes#3306
15ab4b76 [yikf] Add Authz module GA for mutiple spark versions
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
This pr is a follow-up, aims to fix CI failure due to https://github.com/apache/incubator-kyuubi/pull/3299#discussion_r953265471
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3319 from Yikf/master.
Closes#3298
d0cba33f [yikf] [KYUUBI #3298][FOLLOWUP] Fix CI failure
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
We should enable spark-3.3 profile as well to check spark-3.3 specific modules, like kyuubi-spark-hive-connector and kyuubi-spark-kudu-connector.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3316 from pan3793/build.
Closes#3316
e8e1036f [Cheng Pan] nit
bbb30fc7 [Cheng Pan] Build: enable spark-3.3 profile on GA
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Publish daily snapshot on branch-1.6 and stop to publish on branch-1.4
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3317 from pan3793/daily.
Closes#3317
6d54fe8e [Cheng Pan] [BUILD] Publish daily snapshot on branch-1.6
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
close https://github.com/apache/incubator-kyuubi/issues/3089#issue-1306657936
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3261 from zhaomin1423/custom_event.
Closes#3089
a329f4d9 [Min] fix test
165834be [Min] add comment
e216022d [Min] catch exception when provider create event handler
3d452bc2 [Min] fix review
4e7c4b4b [Min] [KYUUBI #3089] [FEATURE] Support to load custom event handlers by ServiceLoader
Authored-by: Min <zhaomin1423@163.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Supplement the licenses.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3254 from hddong/supplement-license.
Closes#3254
b9dfef64 [hongdongdong] remove netty-transport-native-kqueue from license-binary
2185c964 [hongdongdong] Fix
adb8f479 [hongdongdong] Fix
0eeeb0f5 [hongdongdong] Update notice
c296f78e [hongdd] Update license and notice
88482efe [hongdongdong] Supplement the licenses of support etcd discovery
Lead-authored-by: hongdongdong <hongdd@apache.org>
Co-authored-by: hongdd <hongdd@hongdddeMac-mini.local>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/pull/3283#discussion_r950108009
This pr aims to make a helper method for try-finally block to code simplification
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3303 from yikf/refactor-try-finally.
Closes#3303
7273a95a [yikf] [MINOR] Refactor `RangerSparkExtensionSuite` to simple try-finally
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
This is a pr about init frontend framework using vue3.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3250 from tongwl/KYUUBI-3248.
Closes#3250
2c0d7744 [William Tong] add licenses for files
4ab298fb [William Tong] transfer element-plus auto import to full import
4a9478c2 [William Tong] code change from as commented
a1aaba03 [William Tong] Init frontend framework
Authored-by: William Tong <weitong@cisco.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
fix https://github.com/apache/incubator-kyuubi/issues/3298
Curently, a serval snippets use `getClass.getClassLoader` to get the classLoader in Apache Kyuubi, this pr aims to unify this behavior in `Utils.getContextOrKyuubiClassLoader`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3299 from yikf/unify-calssloader.
Closes#3298
ae081ad9 [yikf] Unify the approach of get classLoader in Kyuubi
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
In a modern database architecture, users may have a strong need for federated queries. Since there are a large number of Hive warehouse in the history database, we tried to implement the Hive V2 Datasource based on Spark Datasource V2 to meet this need. for the discussion, see :https://lists.apache.org/thread/fq8ywr58rzf9bycflj1q4fl1xyz2rq2w
This PR is the first step in fixing https://github.com/apache/incubator-kyuubi/issues/3259, having
- initialization implementation
- support read code path
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3260 from yikf/hive-v2-connector.
Closes#3259
753aca30 [yikf] Initial implementation of the Hive Connector based on the Spark datasource V2
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Construct `lifetimeTerminatingChecker` only when needed.
The default value of `kyuubi.session.engine.spark.max.lifetime` is 0, so the `lifetimeTerminatingChecker` is not enabled, there is no need to construct a `ThreadScheduledExecutor`, and `shutdown` can also be avoided when stopping.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3301 from cxzl25/lazy_lifetime_checker.
Closes#3301
3a79bf74 [sychen] add @volatile
a8c9beea [sychen] construct lifetimeTerminatingChecker only when needed
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
