724ae93989
### _Why are the changes needed?_
Before this PR, Kyuubi skips authentication for the call command in iceberg. After this PR, it needs `alter` permission to use this command.
This PR supports checking iceberg call rewrite_data_files command:
- CALL ice.system.rewrite_data_files(table => 'ice.lcl.test_ice', options => map('min-input-files','2'))
- some other call statements in iceberg ( https://iceberg.apache.org/docs/latest/spark-procedures ) may be supported but needs to be further determined. The next phase is planned to support all commands as documented.
When the `rewrite_data_files` command is called, there are two situations:
1. When the triggering conditions are not be met(such as input-files < min-input-files) , there is only one logical plan. We need to check one logical plan permission:
```
* == Physical Plan 1 ==
* Call (1)
```
2. When the triggering conditions are met, two logical plan will be triggered. We should check two logical plans permission.
```
* == Physical Plan 1 ==
* Call (1)
*
* == Physical Plan 2 ==
* AppendData (3)
* +- * ColumnarToRow (2)
* +- BatchScan local.iceberg_ns.call_command_table (1)
```
The second plan (`AppendData`) is a bit special, causing us to have to modify the logic of extracting table names in `DataSourceV2RelationTableExtractor`. The `DataSourceV2Relation#identifier` in `AppendData` will be rewrite task id, so it is a more appropriate way to parse the table name from `DataSourceV2Relation#table`
<img width="1318" alt="image" src="https://github.com/apache/kyuubi/assets/31469905/efc815cb-8dfe-4a9f-8e24-d62883ef6eff">
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?
No.
Closes #5248 from yabola/master.
Closes #5248
f3a2d736e [liangbowen] initDataFilesCount
6c0da35f7 [Bowen Liang] skip input checks
4a5ab6da4 [Bowen Liang] comment
850100b6b [Bowen Liang] update
c3158d25a [Bowen Liang] change operation type to ALTERTABLE_PROPERTIES
8d5c8b1f8 [Bowen Liang] rewriteDataFiles2
2d79e1730 [Bowen Liang] nit
36f1d9d60 [chenliang.lu] update description for ut
bd99e0dfa [chenliang.lu] update description for ut
a95e1bb4f [Bowen Liang] nit
7a0089907 [Bowen Liang] nit
06566815b [Bowen Liang] revert opType
1dd211b59 [Bowen Liang] withSingleCallEnabled
657c9dcd3 [Bowen Liang] Merge branch 'master' into yabola/master
cc7dda1f7 [Bowen Liang] update policy
aecfd50cf [Bowen Liang] update table_spec
66ddd89b1 [Bowen Liang] update
b1e12789f [Bowen Liang] remove defaultBob
b8f4fabaf [Bowen Liang] change operation type
465fa48a2 [Bowen Liang] change ut to iceberg namespace
0cfe285ea [Bowen Liang] comment
2b835c943 [Bowen Liang] remove implicit casting in StringTableExtractor
280ad029b [Bowen Liang] removed snapshotCommand tests
83be1d0fd [Bowen Liang] removed snapshotCommand tests
49bd8e14c [Bowen Liang] assertion on all required checks
d65915eb9 [Bowen Liang] separate cases
7513e055f [chenliang.lu] update ut
2ac0e1141 [chenliang.lu] add more ut
31a85a9a2 [Bowen Liang] rename ut
a9773de2f [Bowen Liang] introduce TableTableExtractor
da23d838f [Bowen Liang] input table
45129bc3f [Bowen Liang] update
adbe7535b [Bowen Liang] interceptContains
f24e07d9a [Bowen Liang] update table_command_spec.json
8a71a3c17 [Bowen Liang] rename to CallProcedure
5c0fd5689 [Bowen Liang] rename uts
c0c03176e [Bowen Liang] remove blanks
da55dc597 [Bowen Liang] fix imports
450f0779b [Bowen Liang] rename IcebergCallArgsTableExtractor to ExpressionSeqTableExtractor
0deeffd57 [chenliang.lu] Support iceberg call statement in ranger authentication
414ff2a42 [ChenliangLu] Merge branch 'apache:master' into master
c85059a0b [chenliang.lu] fix comments
85ffd0f88 [chenliang.lu] Support iceberg call statement in ranger authentication
66dbfcc35 [chenliang.lu] optimize codes for table extractor
f1e93fcd7 [chenliang.lu] Support iceberg call statement in ranger authentication
Lead-authored-by: yabola <chenlianglu@tencent.com>
Co-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: chenliang.lu <marssss2929@gmail.com>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: ChenliangLu <31469905+yabola@users.noreply.github.com>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
|
||
|---|---|---|
| .. | ||
| server/kyuubi-server-plugin | ||
| spark | ||
| README.md | ||
For developers
This folder contains plugins/extension for kyuubi server and different engine types.
- ext
- kyuubi-server
- spark
- flink
- trino
- hive
- others
- ...