github/spark-sql-perf
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Use DECIMAL and DATE in the default TPCDS notebooks. (#130)

Browse Source 
A long time ago, DECIMAL was substituted by DOUBLE and DATE by STRING to workaround some problems.
There is no reason to do it anymore.
This commit is contained in:
Juliusz Sompolski 2018-03-07 21:44:42 +01:00 committed by GitHub
parent b7ac7e55ae
commit a8acd53fdd
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 15 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
src/main/notebooks/tpcds_datagen.scala
View File

@ -22,9 +22,9 @@ val scaleFactor = "1"
// data format.
val format = "parquet"
// If false, float type will be used instead of decimal.
val useDecimal = false
val useDecimal = true
// If false, string type will be used instead of date.
val useDate = false
val useDate = true
// If true, rows with nulls in partition key will be thrown away.
val filterNull = false
// If true, partitions will be coalesced into a single file during generation.

15
src/main/notebooks/tpcds_run.scala
View File

@ -4,9 +4,20 @@
// COMMAND ----------
// Database to be used:
// TPCDS Scale factor
val scaleFactor = "1"
val databaseName = s"tpcds_sf${scaleFactor}_nodecimal_nodate_withnulls"
val iterations = 1 // how many times to run the whole set of queries.
// If false, float type will be used instead of decimal.
val useDecimal = true
// If false, string type will be used instead of date.
val useDate = true
// name of database to be used.
val databaseName = s"tpcds_sf${scaleFactor}" +
s"""_${if (useDecimal) "with" else "no"}decimal""" +
s"""_${if (useDate) "with" else "no"}date""" +
s"""_${if (filterNull) "no" else "with"}nulls"""
val iterations = 2 // how many times to run the whole set of queries.
val timeout = 60 // timeout in hours