f67140e650
# 🔍 Description ## Issue References 🔗 This pull request fixes #5594 ## Describe Your Solution 🔧 For case ``` def filter_func(iterator): for pdf in iterator: yield pdf[pdf.id == 1] df = spark.read.table("test_mapinpandas") execute_result = df.mapInPandas(filter_func, df.schema).show() ``` The logical plan is ``` GlobalLimit 21 +- LocalLimit 21 +- Project [cast(id#5 as string) AS id#11, name#6] +- MapInPandas filter_func(id#0, name#1), [id#5, name#6] +- HiveTableRelation [`default`.`test_mapinpandas`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, Data Cols: [id#0, name#1], Partition Cols: []] ``` When handle `MapInPandas`, we didn't match its input with `HiveTableRelation`, cause we miss input table's columns. This pr fix this In this pr, we remove the branch of each project such as `Project`, `Aggregate` etc, handle it together. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ For case ``` def filter_func(iterator): for pdf in iterator: yield pdf[pdf.id == 1] df = spark.read.table("test_mapinpandas") execute_result = df.mapInPandas(filter_func, df.schema).show() ``` We miss column info of table `test_mapinpandas` #### Behavior With This Pull Request 🎉 We got privilege object of table `test_mapinpandas` with it's column info. #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [x] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5787 from AngersZhuuuu/KYUUBI-5594-approach2. Closes #5594 e08545599 [Angerszhuuuu] Update RangerSparkExtensionSuite.scala 49f09fb0a [Angerszhuuuu] Update RangerSparkExtensionSuite.scala 4781f75b9 [Angerszhuuuu] Update PrivilegesBuilderSuite.scala 9e9208d38 [Angerszhuuuu] Update V2JdbcTableCatalogRangerSparkExtensionSuite.scala 626d3dd88 [Angerszhuuuu] Update RangerSparkExtensionSuite.scala 3d69997de [Angerszhuuuu] Update PrivilegesBuilderSuite.scala 6eb4b8e1a [Angerszhuuuu] Update RangerSparkExtensionSuite.scala 61efb8ae3 [Angerszhuuuu] update 794ebb7be [Angerszhuuuu] Merge branch 'master' into KYUUBI-5594-approach2 a236da86b [Angerszhuuuu] Update PrivilegesBuilderSuite.scala 74bd3f4d5 [Angerszhuuuu] Update RangerSparkExtensionSuite.scala 4acbc4276 [Angerszhuuuu] Merge branch 'KYUUBI-5594-approach2' of https://github.com/AngersZhuuuu/incubator-kyuubi into KYUUBI-5594-approach2 266f7e877 [Angerszhuuuu] update a6c784546 [Angerszhuuuu] Update PrivilegesBuilder.scala d785d5fdf [Angerszhuuuu] Merge branch 'master' into KYUUBI-5594-approach2 014ef3b84 [Angerszhuuuu] Update PrivilegesBuilder.scala 7e1cd37a1 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5594-approach2 71d266162 [Angerszhuuuu] update db9594170 [Angerszhuuuu] update 490eb95c2 [Angerszhuuuu] update 70d110e89 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5594-approach2 e6a587718 [Angerszhuuuu] Update PrivilegesBuilder.scala 5ff22b103 [Angerszhuuuu] Update PrivilegesBuilder.scala e6843014b [Angerszhuuuu] Update PrivilegesBuilder.scala 594b202f7 [Angerszhuuuu] Update PrivilegesBuilder.scala 2f87c61e1 [Angerszhuuuu] Update RangerSparkExtensionSuite.scala 1de8c1c68 [Angerszhuuuu] Update PrivilegesBuilder.scala ad17255d7 [Angerszhuuuu] Update PrivilegesBuilderSuite.scala 4f5e8505f [Angerszhuuuu] update 64349ed97 [Angerszhuuuu] Update PrivilegesBuilder.scala 11b7a4c13 [Angerszhuuuu] Update PrivilegesBuilder.scala 9a58fb0c4 [Angerszhuuuu] update d0b022ec9 [Angerszhuuuu] Update RuleApplyPermanentViewMarker.scala e0f28a640 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5594 0ebdd5de5 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5594 8e53236ac [Angerszhuuuu] update 3bafa7ca5 [Angerszhuuuu] update d6e984e07 [Angerszhuuuu] update b00bf5e20 [Angerszhuuuu] Update PrivilegesBuilder.scala 821422852 [Angerszhuuuu] update 93fc6892b [Angerszhuuuu] Merge branch 'master' into KYUUBI-5594 04184e39d [Angerszhuuuu] update 0bb762467 [Angerszhuuuu] Revert "Revert "Update PrivilegesBuilder.scala"" f481283ae [Angerszhuuuu] Revert "Update PrivilegesBuilder.scala" 9f871822f [Angerszhuuuu] Revert "Update PrivilegesBuilder.scala" 29b67c457 [Angerszhuuuu] Update PrivilegesBuilder.scala 8785ad1ab [Angerszhuuuu] Update PrivilegesBuilder.scala 270f21dcc [Angerszhuuuu] Update RangerSparkExtensionSuite.scala 60872efcb [Angerszhuuuu] Update RangerSparkExtensionSuite.scala c34f32ea2 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5594 86fc4756a [Angerszhuuuu] Update PrivilegesBuilder.scala 404f1ea4c [Angerszhuuuu] Update PrivilegesBuilder.scala dcca394e0 [Angerszhuuuu] Update PrivilegesBuilder.scala c2c6fa447 [Angerszhuuuu] Update PrivilegesBuilder.scala 6f6a36e5b [Angerszhuuuu] Merge branch 'master' into KYUUBI-5594]-AUTH]BuildQuery-should-respect-normal-node's-input 4dd47a124 [Angerszhuuuu] update c549b6a1a [Angerszhuuuu] update 80013b981 [Angerszhuuuu] Update PrivilegesBuilder.scala 3cbba422a [Angerszhuuuu] Update PrivilegesBuilder.scala Authored-by: Angerszhuuuu <angers.zhu@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|---|---|---|
| .. | ||
| server/kyuubi-server-plugin | ||
| spark | ||
| README.md | ||
For developers
This folder contains plugins/extension for kyuubi server and different engine types.
- ext
- kyuubi-server
- spark
- flink
- trino
- hive
- others
- ...