3c2291ef5c
### _Why are the changes needed?_ To close #5677 Typeof expression miss column information, in this pr we add a place holder to support this. For sql ``` SELECT typeof(id), typeof(age) FROM default.t1 LIMIT 1 ``` Without this pr, typeof after optimizer passed to PrivilegeBuild is just an attribute, miss the origin child expression info. ``` GlobalLimit 21 +- LocalLimit 21 +- Project [int AS typeof(id)#27, int AS typeof(age)#28] +- Relation default.t1[id#18,age#19,part#20] parquet ``` When handle this plan's project list, it's an attribute and miss expression when expression result is a constant value, then we can't extract the `TypeOf`'s child expression.  This is caused by `TypeOf` expression is foldable <img width="778" alt="image" src="https://github.com/apache/kyuubi/assets/46485123/770a36fc-235f-4f26-bca7-a5058e120919"> It will be convert to a constant value after spark optimizer, then cause we miss the child expression, then can't extract columns by `collectLeaves` In this pr we wrap the `TypeOf` by a non-foldable expression placeholder then we can get the expression contains column of table when `mergeProjection` After this pr, the plan is like below ``` GlobalLimit 21 +- LocalLimit 21 +- Project [typeofplaceholder(id#21) AS typeof(id)#30, typeofplaceholder(day#23) AS typeof(day)#31] +- HiveTableRelation [`default`.`table1`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, Data Cols: [id#21, scope#22, day#23], Partition Cols: []] ``` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes #5678 from AngersZhuuuu/KYUUBO-5677. Closes #5677 273cd61bf [Angerszhuuuu] Update TypeOfPlaceHolder.scala 8b4383db5 [Angerszhuuuu] done af5dae541 [Angerszhuuuu] Update RangerSparkExtensionSuite.scala 37edcdf6d [Angerszhuuuu] fix for spark-3.1 df15599a3 [Angerszhuuuu] update a7dd3c453 [Angerszhuuuu] Update RangerSparkExtension.scala 6b76dac0d [Angerszhuuuu] [KYUUBI #5677][AUTHZ] Typeof expression miss column information f8a94c72d [Angerszhuuuu] [KYUUBI #5677][AUTHZ] Typeof expression miss column information Authored-by: Angerszhuuuu <angers.zhu@gmail.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|---|---|---|
| .. | ||
| server/kyuubi-server-plugin | ||
| spark | ||
| README.md | ||
For developers
This folder contains plugins/extension for kyuubi server and different engine types.
- ext
- kyuubi-server
- spark
- flink
- trino
- hive
- others
- ...