apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

[KYUUBI #7077] Spark 3.5: Enhance MaxScanStrategy for DSv2

Browse Source 
### Why are the changes needed?

To enhance the MaxScanStrategy in Spark's DSv2 to ensure it only works for relations that support statistics reporting. This prevents Spark from returning a default value of Long.MaxValue, which, leads to some queries failing or behaving unexpectedly.
### How was this patch tested?

It tested out locally.

### Was this patch authored or co-authored using generative AI tooling?

No

Closes #7077 from zhaohehuhu/dev-0527.

Closes #7077

64001c94e [zhaohehuhu] fix MaxScanStrategy for datasource v2

Authored-by: zhaohehuhu <luoyedeyi459@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
This commit is contained in:
zhaohehuhu 2025-05-29 13:25:55 +08:00 committed by Cheng Pan
parent 1af6647132
commit bcaff5a3f1
No known key found for this signature in database
GPG Key ID: 8001952629BCC75D
1 changed files with 2 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
extensions/spark/kyuubi-extension-spark-3-5/src/main/scala/org/apache/kyuubi/sql/watchdog/MaxScanStrategy.scala
View File

@ -23,6 +23,7 @@ import org.apache.spark.sql.catalyst.SQLConfHelper
import org.apache.spark.sql.catalyst.catalog.{CatalogTable, HiveTableRelation} import org.apache.spark.sql.catalyst.catalog.{CatalogTable, HiveTableRelation}
import org.apache.spark.sql.catalyst.planning.ScanOperation import org.apache.spark.sql.catalyst.planning.ScanOperation
import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
import org.apache.spark.sql.connector.read.SupportsReportStatistics
import org.apache.spark.sql.execution.SparkPlan import org.apache.spark.sql.execution.SparkPlan
import org.apache.spark.sql.execution.datasources.{CatalogFileIndex, HadoopFsRelation, InMemoryFileIndex, LogicalRelation} import org.apache.spark.sql.execution.datasources.{CatalogFileIndex, HadoopFsRelation, InMemoryFileIndex, LogicalRelation}
import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation
@ -237,7 +238,7 @@ case class MaxScanStrategy(session: SparkSession)
_, _,
_, _,
_, _,
relation @ DataSourceV2ScanRelation(_, _, _, _, _)) => relation @ DataSourceV2ScanRelation(_, _: SupportsReportStatistics, _, _, _)) =>
val table = relation.relation.table val table = relation.relation.table
if (table.partitioning().nonEmpty) { if (table.partitioning().nonEmpty) {
val partitionColumnNames = table.partitioning().map(_.describe()) val partitionColumnNames = table.partitioning().map(_.describe())