### _Why are the changes needed?_
Some users encounter errors on building, enable -x to help user identify the failure reason
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3372 from pan3793/dist.
Closes#3372
7bdfb39f [Cheng Pan] Enable -x on build/dist
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
fix#3325
Permanent views are generally registed globally and then used for unifing, masking, joining tables for different practical purposes. The data manager would like to authorize views to users , but not all the source tables.
Authz plugin is not satisfiying this senario as it checks privileges all the source table instead of the permanent views.
Suggesting chaning behaviour of PrivilegesBuilder
- check privileges for permanent view
- skip privileges check for shadowed source view of permanent views
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3326 from bowenliang123/feature-authz-perm-view.
Closes#3325
468f47ae [Bowen Liang] fix ut
b9d2cdb6 [Bowen Liang] fix ut
82c7ae63 [Bowen Liang] fix case for #3326. add "view" type for withCleanTmpResources.
bcd35d33 [Bowen Liang] add ut
3cfb4320 [Bowen Liang] Merge branch 'master' into feature-authz-perm-view
81e8b992 [Bowen Liang] rename to hasResolvedPermanentView and check if plan resolved
4903f0a9 [Bowen Liang] change to use mergeProjection in PivilegesBuilder , checking column level privilege.
78ff9d30 [Bowen Liang] view casting (+1 squashed commit) Squashed commits: [98f523d1] fix error in view casting
3706dda0 [Bowen Liang] rename ViewAccessAnalysis to RuleApplyPermanentViewMarker (+2 squashed commits) Squashed commits: [207ed7b5] rename ViewAccessAnalysis to RuleApplyPermanentViewMarker [f58be005] nit
6ef81319 [Bowen Liang] use isSparkVersionAtLeast 3.1.0 instead of try-catch RuntimeException non-existed isTempView field
76ec6b42 [Bowen Liang] simplify and remove applyPermanentViewMarker (+2 squashed commits) Squashed commits: [802e5223] nit [ba1beb49] nit
aea561c5 [Bowen Liang] ut for spark30 and spark31greater
41c67ac9 [Bowen Liang] use reflection for isTempView of View
94a837fc [liangbowen] add ut for permanent view privilege checks
a0072c0c [liangbowen] add PermanentViewMarker for marking perm view catalogTable and check PrivilegesBuilder with PermanentViewMarker, finally maker cleanup in RuleEliminateViewMarker
Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
We met the case that the LaunchEngine operation is succeed.
But the spark engine met exception and terminated with SUCCESS state.
And then the kyuubi connection stuck until reach the `kyuubi.session.engine.initialize.timeout`.
The engine application terminated in 1.5 minutes and the engine init timeout is 10m.
FYI:
So, we shall check the engine application state when waiting the engine initialization, and fast fail the connection if the application has been in terminated state.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3315 from turboFei/fast_fail_on_terminate.
Closes#3315
03333623 [Fei Wang] add comments
f1632dbb [Fei Wang] fix ut
4059a4fe [Fei Wang] combine the state
33dc4a88 [Fei Wang] only check it for exit 0
d5746af1 [Fei Wang] fix ut
854f3e3b [Fei Wang] add metrics
62001610 [Fei Wang] revert
3fb1b630 [Fei Wang] remove engine ref check, meaning less
d2c2c794 [Fei Wang] add ut
ddafa24a [Fei Wang] fast fail the connection on engine termiante
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
fix#3360.
- support batch verifying calling to Ranger plugin, keep emitting exception for first unallowed request
- batch verifying privilege of columns for single resource
Befor PR,
```
case ObjectType.COLUMN if resource.getColumns.nonEmpty =>
resource.getColumns.foreach { col =>
...
verify(req, auditHandler)
}
```
After PR,
```scala
case ObjectType.COLUMN if resource.getColumns.nonEmpty =>
val reqs = ArrayBuffer[AccessRequest]()
resource.getColumns.foreach { col =>
...
reqs += req
}
batchVerify(reqs.toList, auditHandler)
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3361 from bowenliang123/feature-authz-batch-verify.
Closes#3360
154ff781 [liangbowen] minor update with common-collections util
2617fabc [Bowen Liang] refactor verify method for batching and remove batchVerify method
dc56611e [liangbowen] minor update
9da5ce0b [liangbowen] nit
e99f5bae [Bowen Liang] implements batchVerify with SparkRangerAdminPlugin.isAccessAllowed
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
This is a minor change, aims to remove authZ-module from the profile tag, this is due to authz supports all Spark versions currently supported by Kyuubi, and authz-module is defined in the common Module
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3370 from Yikf/authz-pom-minor.
Closes#3370
3c58a095 [yikf] remove authz-module for profile
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Support to get engine id, name, url in KyuubiConnection.
It is helpful and friendly for BI.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3354 from turboFei/lanch_engine_result.
Closes#3354
4966de28 [Fei Wang] comments
984b5d35 [Fei Wang] fix ut
963217af [Fei Wang] filter null
a2c0ce2c [Fei Wang] fix
4f2a8b18 [Fei Wang] get from engine side
61607c7c [Fei Wang] return
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3357
This pr aims to improve doc description for configuration kyuubi.operation.plan.only.mode
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3358 from Yikf/planOnly-doc.
Closes#3357
571f4e0c [yikf] Improve doc description for configuration `kyuubi.operation.plan.only.mode`
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
fix#3343
Privilege checks of CreateViewCommand retained for PersistedViews only, and skipped for LocalTempView or GlobalTempView.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3344 from bowenliang123/improve-skipauth-tempviews.
Closes#3343
eff4f732 [Bowen Liang] CreateViewCommand tablePrivileges checks for only persisted views, and adding ut
Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
When I testing v1.6.0-incubating-rc0 w/ `kyuubi.operation.progress.enabled=true`, the beeline broken on rendering progress bar.
```
STAGES ATTEMPT STATUS TOTAL COMPLETED RUNNING PENDING FAILED
----------------------------------------------------------------------------------------------
org/apache/commons/lang/StringUtils
Closing: 0: jdbc:hive2://0.0.0.0:10009/default
```
And got fixed after adding `commons-lang-2.6.jar`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
<img width="671" alt="image" src="https://user-images.githubusercontent.com/26535726/187076711-d060d522-c7a5-46e5-b47f-fa4786407e09.png">
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3363 from pan3793/beeline.
Closes#3363
2edf69e5 [Cheng Pan] add commons-lang dep
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
fix#3349
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3350 from jiaoqingbo/kyuubi3349.
Closes#3349
85ca155b [jiaoqingbo] [KYUUBI #3349] Bump unboundid-ldapsdk version from 5.1.4 to 6.0.5
Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3352
It's a regression issue, that came from upgrading from Hive 2.3 to Hive 3.1.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3353 from pan3793/HIVE-21899.
Closes#3353
85a25c8e [Cheng Pan] Backport HIVE-21899 - Utils.getCanonicalHostName() may return IP address depending on DNS infra
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix#3352
[HIVE-17218](https://issues.apache.org/jira/browse/HIVE-17218): Canonical-ize hostnames for Hive metastore, and HS2 servers
This may have the problem mentioned by JIRA
[HIVE-20583](https://issues.apache.org/jira/browse/HIVE-20583): Use canonical hostname only for kerberos auth in HiveConnection
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3355 from pan3793/HIVE-20583.
Closes#3355
6e2ffa49 [Cheng Pan] Fix NPE
f09b8e6c [Cheng Pan] Backport HIVE-20583
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Skip maven javadoc plugin for release publish.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3348 from SteNicholas/maven-javadoc-plugin.
Closes#3348
4ba825fc [SteNicholas] Specifies none checks to be performed on Javadoc comments for maven-javadoc-plugin
Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This is a minor change, aims to relocation `AccessControlException` from `org.apache.kyuubi.plugin.spark.authz.utils` to `org.apache.kyuubi.plugin.spark.authz`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3347 from Yikf/minor-relocation.
Closes#3347
05d95a7a [yikf] Relocation `AccessControlException` from org.apache.kyuubi.plugin.spark.authz.utils to org.apache.kyuubi.plugin.spark.authz
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Now `activeStages` only has the stageid, and does not record and output the number of stage retries, which may cause errors when the stage fails to retry.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3336 from cxzl25/spark_listener_stage_attempt.
Closes#3336
a6da472a [sychen] use StageAttempt instead of StageId
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
1. `principal` supports `X/_HOSTEXAMPLE.COM`
2. `kyuubiClientPrincipal` supports headless keytab, `XEXAMPLE.COM`
https://github.com/apache/incubator-kyuubi/pull/3023
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3346 from cxzl25/3023_followup.
Closes#3023
15309298 [sychen] support principal _HOST and kyuubiClientPrincipal headless keytab
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3217#issuecomment-1219271537
This pr aims to bump ranger version from 2.2.0 to 2.3.0
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3333 from Yikf/pr/3282.
Closes#3333
07b35b9f [Cheng Pan] Update log4j2-test.xml
45a403b4 [yikf] [KYUUBI #3217][FOLLOWUP] Bump ranger version to 2.3.0
Lead-authored-by: yikf <yikaifei1@gmail.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Add v1.6.0-incubating changelog.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3342 from SteNicholas/changelog-1.6.0-incubating.
Closes#3342
76bf8c6e [SteNicholas] Add v1.6.0-incubating changelog
Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
To close#3337
excludes `node_modules` folder.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3338 from zwangsheng/fix/linese_check_exclude_node_modules.
Closes#3338
80f45440 [zwangsheng] fix
4f310f70 [zwangsheng] fix
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
`npm run build` fail with following log
```
error TS6504: File '/Users/yangbinjie01/IdeaProjects/incubator-kyuubi/kyuubi-server/web-ui/src/views/operation/completedJobs/index.vue.__VLS_script.js' is a JavaScript file. Did you mean to enable the 'allowJs' option?
The file is in the program because:
Root file specified for compilation
...
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3334 from zwangsheng/fix/npm_build.
Closes#3334
4142f2ad [zwangsheng] fix
a1370a8f [zwangsheng] add
e86648ec [zwangsheng] fix npm run build
Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
fix: #3327
### _Why are the changes needed?_
The contributing link in the PULL_REQUEST_TEMPLATE is out-of-date, so this patch replaces it with the new contributing link.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3329 from YesOrNo828/Kyuubi-3327.
Closes#3327
badf37b0 [yexianxun] [KYUUBI #3327] Replace the contributing link in the PR template
Authored-by: yexianxun <yexianxun@corp.netease.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Now the connection path is:
`client`-> `kyuubiServer` -> `kyuubiEngine`
Maybe the connection between `client -> kyuubiServer` is valid, but the connection between `kyuubiServer -> kyuubiEngine` is not.
So we need check the whole path.
BTW, before, if customer invoke KyuubiConnection::isValid, it can make the connection between client and kyuubiServer keep alive.
Now, it can make the whole connection path keep alive.
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3318 from turboFei/is_valid.
Closes#3318
af4d6b7b [Fei Wang] Transfer the TGetInfoReq to kyuubi engine side to keep the kyuubi connection alive
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
close https://github.com/apache/incubator-kyuubi/issues/3128
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3304 from Kwafoor/master.
Closes#3128
88a5ebe9 [wangjunbo] [kyuubi-3128] Support CostMode for PlanOnlyStatement
Authored-by: wangjunbo <wangjunbo@qiyi.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Currently, Apache Kyuubi support Spark 3.0.0 and above, but Kyuubi-AuthZ support Spark 2.4, Spark 3.0 and above, to avoid subsequent maintenance costs, we plan to move out support for Spark 2.4.
This pr aims to modify the documentation to indicate that version 2.4 is not supported
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3305 from Yikf/authz-spark2.4-notsupport.
Closes#3226
5c756f51 [yikf] authz not support spark2.4
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/issues/3306
Currently, Kyuubi-AuthZ support Apache Spark 3.0 and above, and there are no CI workflow to check compatibility with Apache Spark 3.0, this pr aim to add a CI workflow for AuthZ module
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3307 from Yikf/GA.
Closes#3306
15ab4b76 [yikf] Add Authz module GA for mutiple spark versions
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
This pr is a follow-up, aims to fix CI failure due to https://github.com/apache/incubator-kyuubi/pull/3299#discussion_r953265471
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3319 from Yikf/master.
Closes#3298
d0cba33f [yikf] [KYUUBI #3298][FOLLOWUP] Fix CI failure
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
We should enable spark-3.3 profile as well to check spark-3.3 specific modules, like kyuubi-spark-hive-connector and kyuubi-spark-kudu-connector.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3316 from pan3793/build.
Closes#3316
e8e1036f [Cheng Pan] nit
bbb30fc7 [Cheng Pan] Build: enable spark-3.3 profile on GA
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Publish daily snapshot on branch-1.6 and stop to publish on branch-1.4
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3317 from pan3793/daily.
Closes#3317
6d54fe8e [Cheng Pan] [BUILD] Publish daily snapshot on branch-1.6
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
close https://github.com/apache/incubator-kyuubi/issues/3089#issue-1306657936
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3261 from zhaomin1423/custom_event.
Closes#3089
a329f4d9 [Min] fix test
165834be [Min] add comment
e216022d [Min] catch exception when provider create event handler
3d452bc2 [Min] fix review
4e7c4b4b [Min] [KYUUBI #3089] [FEATURE] Support to load custom event handlers by ServiceLoader
Authored-by: Min <zhaomin1423@163.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
### _Why are the changes needed?_
Supplement the licenses.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3254 from hddong/supplement-license.
Closes#3254
b9dfef64 [hongdongdong] remove netty-transport-native-kqueue from license-binary
2185c964 [hongdongdong] Fix
adb8f479 [hongdongdong] Fix
0eeeb0f5 [hongdongdong] Update notice
c296f78e [hongdd] Update license and notice
88482efe [hongdongdong] Supplement the licenses of support etcd discovery
Lead-authored-by: hongdongdong <hongdd@apache.org>
Co-authored-by: hongdd <hongdd@hongdddeMac-mini.local>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Fix https://github.com/apache/incubator-kyuubi/pull/3283#discussion_r950108009
This pr aims to make a helper method for try-finally block to code simplification
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3303 from yikf/refactor-try-finally.
Closes#3303
7273a95a [yikf] [MINOR] Refactor `RangerSparkExtensionSuite` to simple try-finally
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
This is a pr about init frontend framework using vue3.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3250 from tongwl/KYUUBI-3248.
Closes#3250
2c0d7744 [William Tong] add licenses for files
4ab298fb [William Tong] transfer element-plus auto import to full import
4a9478c2 [William Tong] code change from as commented
a1aaba03 [William Tong] Init frontend framework
Authored-by: William Tong <weitong@cisco.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
fix https://github.com/apache/incubator-kyuubi/issues/3298
Curently, a serval snippets use `getClass.getClassLoader` to get the classLoader in Apache Kyuubi, this pr aims to unify this behavior in `Utils.getContextOrKyuubiClassLoader`
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3299 from yikf/unify-calssloader.
Closes#3298
ae081ad9 [yikf] Unify the approach of get classLoader in Kyuubi
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
In a modern database architecture, users may have a strong need for federated queries. Since there are a large number of Hive warehouse in the history database, we tried to implement the Hive V2 Datasource based on Spark Datasource V2 to meet this need. for the discussion, see :https://lists.apache.org/thread/fq8ywr58rzf9bycflj1q4fl1xyz2rq2w
This PR is the first step in fixing https://github.com/apache/incubator-kyuubi/issues/3259, having
- initialization implementation
- support read code path
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3260 from yikf/hive-v2-connector.
Closes#3259
753aca30 [yikf] Initial implementation of the Hive Connector based on the Spark datasource V2
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Construct `lifetimeTerminatingChecker` only when needed.
The default value of `kyuubi.session.engine.spark.max.lifetime` is 0, so the `lifetimeTerminatingChecker` is not enabled, there is no need to construct a `ThreadScheduledExecutor`, and `shutdown` can also be avoided when stopping.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3301 from cxzl25/lazy_lifetime_checker.
Closes#3301
3a79bf74 [sychen] add @volatile
a8c9beea [sychen] construct lifetimeTerminatingChecker only when needed
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
### _Why are the changes needed?_
Fixes#3272
Kyuubi server stop sequence doesn't respect graceful shutdown triggered in the discovery clients' `DeRegisterWatcher` loops. This leads to frontend service components being stopped before sessions close, resetting connections and leading to client disruption.
PR synchronizes these threads to ensure that service components wait for client sessions to close before continuing the stop sequence.
* Existing service discovery deregistration functionality is maintained to ensure that new connections are established with healthy instances once the stop sequence has begun.
* Existing Hadoop shutdown hooks ensure that the watcher can be timed out, and are configurable via `hadoop.service.shutdown.timeout`. The Kyuubi server will always shut down within 30 seconds when using default config.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3291 from bgrams/fix/graceful-shutdown.
Closes#3272
ffec2a2a [Brandon Grams] Synchronize graceful session shutdown
Authored-by: Brandon Grams <brandongr@zillowgroup.com>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
This is a minor fix and aims to use `AccessControlException` instead of `RuntimeException`
Subsequent code added to the AuthZ module should also use this exception if it is intended to represent an access exception
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3281 from yikf/auth-e.
Closes#3281
9033be48 [yikf] [MINOR] Use AccessControlException instead of RuntimeException if check privilege failed
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
To fix the config name and placeholder with `username` introduced in https://github.com/apache/incubator-kyuubi/pull/3235 violate this convention as in JDBC driver use `user` keyword used for connection user rather than `username`,
1. change config name from `kyuubi.authentication.jdbc.username` to `kyuubi.authentication.jdbc.user`
2. change placeholder from `${username}` to `${user}`
3. update docs and config description related to above changes, and sync the update in jdbc auth docs statement details to config docs.
4. fix error in throwing AuthenticationException with auth db password. ut added for the fix.
5. other minor update in docs of custom auth
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3288 from bowenliang123/jdbc-auth-config-update.
Closes#3222
502703fb [Bowen Liang] skip map for placeholder value lookup
3733be41 [liangbowen] nit
ab00525b [liangbowen] nit
2301c4f0 [liangbowen] fix ut of jdbc auth with wrong_password
06f0c1bb [liangbowen] remove redundant docs
ec4565b3 [liangbowen] remove redundant docs
ae1cce2e [liangbowen] fix compilation error of configLog
5d14103b [liangbowen] simplify configLog
6678e657 [liangbowen] reformat
52c1038e [liangbowen] simplify placeholder checking
21c2d5ea [liangbowen] check whether placeholders in supported list before conn establishment or authenticate
7db0adf5 [liangbowen] ut for unknown placeholder
657de6af [liangbowen] nit
736b3f22 [liangbowen] refactoring placeholder value lookup, for preventing setString multiple times with "i+1"
86c89125 [liangbowen] setMaxRows after prepare placeholder, to postpone operation on jdbc conn
115fae50 [liangbowen] increase test code coverage
b45b28cb [liangbowen] resultSet returned by executeQuery is never null
e1c07274 [liangbowen] update ut for redactPassword in JdbcUtils
b4a52e29 [liangbowen] fix typo in docs of custom auth
371c2c6e [liangbowen] move redactPassword method to JdbcUtils and add ut.
a4973c59 [liangbowen] reformat code
486e1503 [liangbowen] fix error in throwing AuthenticationException with auth db password. add ut for the fix.
efced90a [liangbowen] update settings.md
ef97e354 [liangbowen] add SELECT prefix hint for doc of kyuubi.authentication.jdbc.query
025f94c7 [liangbowen] fix username to user in JdbcAuthenticationProviderImpl by 1. use config name `kyuubi.authentication.jdbc.user`, 2. use ${user} placeholder instead of ${username}
Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3297 from zhaomin1423/minor_null.
Closes#3297
9008737b [Min] [MINOR] Null is replaced by KyuubiSQLException.featureNotSupported()
Authored-by: Min <zhaomin1423@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
HADOOP-18088 replaces log4j 1.x with reload4j, we should exclude it becuase we are using log4j2, otherwise slf4j complains
```
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/chengpan/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.18.0/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/chengpan/.m2/repository/org/slf4j/slf4j-reload4j/1.7.36/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3287 from pan3793/reload4j.
Closes#3287
60930d7f [Cheng Pan] Exclude reload4j from hadoop-minikdc
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
This is the followup of #3235, the main change is introdude `JdbcUtils` to simplify code, and allow empty password for Jdbc auth.
Jdbc connection pool has been removed because `JdbcAuthenticationProviderImpl` will be created on each connection, we can improve to use singleton in the future
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3278 from pan3793/jdbc-followup.
Closes#3222
2863cae7 [Cheng Pan] Update kyuubi-common/src/test/scala/org/apache/kyuubi/service/authentication/JdbcAuthenticationProviderImplSuite.scala
51a9c458 [Cheng Pan] Update kyuubi-common/src/test/scala/org/apache/kyuubi/service/authentication/JdbcAuthenticationProviderImplSuite.scala
eee3c55e [Cheng Pan] Update kyuubi-common/src/test/scala/org/apache/kyuubi/util/JdbcUtilsSuite.scala
d02bb999 [Cheng Pan] nit
e001b5b5 [Cheng Pan] nit
8cf5cd67 [Cheng Pan] nit
032f2df8 [Cheng Pan] nit
8a42f184 [Cheng Pan] nit
c7893fdf [Cheng Pan] JdbcUtilsSuite
f97f2d90 [Cheng Pan] remove pool
a8812d0f [Cheng Pan] move render result set to test
83d7d4cf [Cheng Pan] fix ut
db787a4b [Cheng Pan] nit
864f9ddc [Cheng Pan] nit
b60decf5 [Cheng Pan] nit
8c66e0be [Cheng Pan] nit
2063c438 [Cheng Pan] [KYUUBI #3222][FOLLOWUP] Introdude JdbcUtils to simplify code
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
Followup of #3020
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3271 from turboFei/refine_ldap.
Closes#3020
caa35d84 [Fei Wang] comment
65597cb4 [Fei Wang] refine
Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
fix https://github.com/apache/incubator-kyuubi/issues/3279
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3280 from yikf/bump-spotless.
Closes#3279
7a8adb8d [yikf] bump spotless to 2.24.1
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
### _Why are the changes needed?_
fix https://github.com/apache/incubator-kyuubi/issues/3269
Add docs for JDBC auth provider from https://github.com/apache/incubator-kyuubi/issues/3222 shipping within 1.6.0, add jdbc.rst in /docs/security path
simplify the config docs for kyuubi.authentication.jdbc.query in /docs/deployment/settings.md by moving examples to the docs of jdbc auth provider.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3275 from bowenliang123/doc-jdbc-auth.
Closes#3275
e393f6db [liangbowen] fix config example mistakes
ab70cc8d [liangbowen] update doc for auth query sql
ebd5e194 [liangbowen] update doc description
0b15af59 [liangbowen] typo
953000fe [liangbowen] typo
ab944e73 [liangbowen] typo
e3c3d259 [liangbowen] regenerate conf docs. udpate jdbc.md.
45429ada [liangbowen] typo
3b05338e [liangbowen] add docs for jdbc authentication provider
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Support macros in Row-filter condition expression, introduced in Ranger 2.3 ([release notes](https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+2.3.0+-+Release+Notes)), is an major feature to significantly simplify the row-filter condition expression in practice by replacing explicit condition query by using user/group's attributes.
- [RANGER-3605](https://issues.apache.org/jira/browse/RANGER-3605) : Support macros in row-filter/condition expressions
- [RANGER-3550](https://issues.apache.org/jira/browse/RANGER-3550) : support for using user/tag attributes in row-filter expressions and conditions
Consider user liangtiancheng with attribute born_city = guangzhou, we can define the row filter condition with city='${{USER.born_city}}' with the macro feature.
However, This feature implicit relies on an config named `ranger.plugin.spark.enable.implicit.userstore.enricher` and the default value false will prevent RangerUserStoreEnricher fetching user/group and their attributes. Macros in row-filter condition will fallback to null value (as lack of user attributes value in UserStore of auth context) in script transformation unexpectedly and imperceptibly.
Improving doc of ranger-spark-security.xml to aware of this feature and related config.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3267 from bowenliang123/doc-ranger-macros.
Closes#3217
aee4b201 [liangbowen] plain text
a7ec3bc9 [liangbowen] update docs and clearify difference between ranger.plugin.hive.policy.cache.dir and ranger.plugin.spark.policy.cache.dir configs
4887bd1f [liangbowen] simplify with "row filter expressions"
fa62402f [liangbowen] skip list items
9dd1cd4c [liangbowen] h5. Using Marcos in Row Level Filters
849bed5b [liangbowen] add docs for Additional configs for using Marcos in row-level filter
Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Kent Yao <yao@apache.org>
### _Why are the changes needed?_
Among the kyuubi ldap configuration parameters, there is currently "kyuubi.authentication.ldap.base.dn=", but there is no place corresponding to the CN password setting. similar to hive
"hive.server2.authentication.ldap.binddn",
"hive.server2.authentication.ldap.bindpw".
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3213 from dev-lpq/branch-ldap-1.6.
Closes#3020
31817cc6 [pengqli] upgrade ldap to support ldap before 1.6.0
31094030 [pengqli] upgrade ldap to support ldap before 1.6.0
f62e4c96 [pengqli] upgrade ldap to support ldap before 1.6.0
e03e036e [pengqli] add AUTHENTICATION_LDAP_GUIDKEY deprecated
6f680dfc [pengqli] add ldap config AUTHENTICATION_LDAP_BINDDN
19f8e6ce [pengqli] add ldap config AUTHENTICATION_LDAP_BINDDN
07df56b8 [pengqli] upgrade LdapAuthenticationProviderImpl
1a48f418 [pengqli] upgrade settings.md
8e9b5721 [pengqli] upgrade ldap ldapBindpw
5793a811 [pengqli] upgrade ldap ldapBindpw
822d4c30 [pengqli] upgrade docs/deployment/settings.md
c2b20418 [pengqli] upgrade ldap WithLdapServer UTs
5d7a301c [pengqli] add AUTHENTICATION_LDAP_PASSWORD into serverOnlyConfEntries
10b42b1b [pengqli] upgrade AuthenticationProviderFactorySuite UTs
490bfae8 [pengqli] kyuubi ldap add new config property kyuubi.authentication.ldap.bindnpw and kyuubi.authentication.ldap.attrs
5e071252 [pengqli] upgrafe config kyuubi.authentication.ldap.attrs
4e771ec6 [pengqli] kyuubi ldap add new config property kyuubi.authentication.ldap.bindnpw and kyuubi.authentication.ldap.attrs
Authored-by: pengqli <pengqli@cisco.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
### _Why are the changes needed?_
fix https://github.com/apache/incubator-kyuubi/issues/3226
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#3262 from yikf/auth-once.
Closes#3226
59c1321e [yikf] fix
dad11aa8 [yikf] [KYUUBI #3226] check privileges once only
Authored-by: yikf <yikaifei1@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
