Coalesce non-partitioned tables. (#118)

In #109 coalescing of non-partitioned tables into 1 file seems to have gotten accidentally removed.
Put it back, but only when clusterByPartitionedColumns == true
Considering that we coalesce partitions only when that setting is true, it seems to be consistent to use it also for non-partitioned tables.

It may be better to change the name of the parameter, but that changes the interface, and possibly should be left for some future clean up.
This commit is contained in:
Juliusz Sompolski 2017-09-04 18:05:42 +02:00 committed by GitHub
parent 3e1bbd00ed
commit bcda8fc1e5

View File

@ -211,7 +211,12 @@ abstract class Tables(sqlContext: SQLContext, scaleFactor: String,
data.write
}
} else {
data.write
if (clusterByPartitionColumns) {
// treat non-partitioned tables as "one partition" that we want to coalesce
data.coalesce(1).write
} else {
data.write
}
}
writer.format(format).mode(mode)
if (partitionColumns.nonEmpty) {