Make ExecutionMode.HashResults handle null value

In Spark 1.6, if a value is null, `getLong` will throw an exception. Before 1.6, it will return 0. With this PR, we will check if the result is null. If it is null, null will be returned instead of 0.

Author: Yin Huai <yhuai@databricks.com>

Closes #41 from yhuai/fixSumHash.
This commit is contained in:
Yin Huai 2015-12-08 15:28:48 -08:00 committed by Michael Armbrust
parent 43c2f23bb9
commit 3af656defa

View File

@ -607,14 +607,13 @@ abstract class Benchmark(
case ExecutionMode.HashResults =>
val columnStr = dataFrame.schema.map(_.name).mkString(",")
// SELECT SUM(HASH(col1, col2, ...)) FROM (benchmark query)
val ret =
val row =
dataFrame
.selectExpr(s"hash($columnStr) as hashValue")
.groupBy()
.sum("hashValue")
.head()
.getLong(0)
result = Some(ret)
result = if (row.isNullAt(0)) None else Some(row.getLong(0))
}
}