04f443792b
# 🔍 Description ## Issue References 🔗 This pull request fixes #6754 ## Describe Your Solution 🔧 Right now in RuleAuthorization we use an ArrayBuffer to collect access requests, which is very slow because each new PrivilegeObject needs to be compared with all access requests. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ Add benchmark Before ```sh Java HotSpot(TM) 64-Bit Server VM 17.0.12+8-LTS-286 on Mac OS X 14.6 Apple M3 Collecting files ranger access request: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------ 50000 files benchmark 181863 189434 NaN -0.0 -181863368958.0 1.0X ```` #### Behavior With This Pull Request 🎉 After ```sh Java HotSpot(TM) 64-Bit Server VM 17.0.12+8-LTS-286 on Mac OS X 14.6 Apple M3 Collecting files ranger access request: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------ 50000 files benchmark 1281 1310 33 -0.0 -1280563000.0 1.0X ``` #### Related Unit Tests Exists UT --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6758 from wankunde/ranger2. Closes #6754 9d7d1964b [wankunde] [KYUUBI #6754] Improve the performance of ranger access requests 88b9c049b [wankun] Update extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/spark/sql/RuleAuthorizationBenchmark.scala 20c55fbeb [wankun] Update extensions/spark/kyuubi-spark-authz/pom.xml f5a3b6ca5 [wankunde] [KYUUBI #6754] Improve the performance of ranger access requests 9793249de [wankunde] [KYUUBI #6754] Improve the performance of ranger access requests d86b01f9c [wankunde] [KYUUBI #6754] Improve the performance of ranger access requests b904b491b [wankunde] [KYUUBI #6754] Improve the performance of ranger access requests aad08a6bb [wankunde] [KYUUBI #6754] Improve the performance of ranger access requests 1374604bc [wankunde] [KYUUBI #6754] Improve the performance of ranger access requests 01e15c149 [wankun] Update extensions/spark/kyuubi-spark-authz/pom.xml 805e8a9c0 [wankun] Update extensions/spark/kyuubi-spark-authz/pom.xml e19817943 [wankunde] [KYUUBI #6754] Improve the performance of ranger access requests Lead-authored-by: wankunde <wankunde@163.com> Co-authored-by: wankun <wankun@apache.org> Signed-off-by: Bowen Liang <liangbowen@gf.com.cn> |
||
|---|---|---|
| .. | ||
| kyuubi-extension-spark-3-3 | ||
| kyuubi-extension-spark-3-4 | ||
| kyuubi-extension-spark-3-5 | ||
| kyuubi-extension-spark-jdbc-dialect | ||
| kyuubi-spark-authz | ||
| kyuubi-spark-authz-shaded | ||
| kyuubi-spark-connector-common | ||
| kyuubi-spark-connector-hive | ||
| kyuubi-spark-connector-tpcds | ||
| kyuubi-spark-connector-tpch | ||
| kyuubi-spark-jvm-quake | ||
| kyuubi-spark-lineage | ||