github/spark-sql-perf
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
235 Commits 1 Branch 0 Tags 682 KiB
master
Commit Graph

235 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Michael Armbrust
f8aa93d968 Initial set of tests for Datasets 
Author: Michael Armbrust <michael@databricks.com>

Closes #42 from marmbrus/dataset-tests.
 2015-12-08 16:04:42 -08:00
Michael Armbrust
0aa2569a18 Write only one file per run 
Author: Michael Armbrust <michael@databricks.com>

Closes #35 from marmbrus/oneResultFile.
 2015-12-08 15:46:20 -08:00
Michael Armbrust
12b7537181 Update databricks plugin 
Author: Michael Armbrust <michael@databricks.com>

Closes #43 from marmbrus/updatePlugin.
 2015-12-08 15:29:47 -08:00
Yin Huai
3af656defa Make ExecutionMode.HashResults handle null value 
In Spark 1.6, if a value is null, `getLong` will throw an exception. Before 1.6, it will return 0. With this PR, we will check if the result is null. If it is null, null will be returned instead of 0.

Author: Yin Huai <yhuai@databricks.com>

Closes #41 from yhuai/fixSumHash.
 2015-12-08 15:28:48 -08:00
Nong Li
43c2f23bb9 Fixes for Q34 and Q73 to return results deterministically. 
Author: Nong Li <nong@databricks.com>

Closes #38 from nongli/tpcds.
 2015-11-25 15:03:33 -08:00
Nong
70e0dbe656 Add official TPCDS 1.4 queries. 
Author: Nong <nong@cloudera.com>

Closes #36 from nongli/tpcds.
 2015-11-24 13:12:46 -08:00
Nong Li
1aa5bfc838 Add remaining tpcds tables. 
Author: Nong Li <nongli@gmail.com>

Closes #34 from nongli/tpcds.
 2015-11-19 13:50:00 -08:00
Andrew Or
e2073129cf Setting version to 0.2.2-SNAPSHOT 2015-11-18 13:17:56 -08:00
Andrew Or
180003e4f9 Setting version to 0.2.1 2015-11-18 13:17:48 -08:00
Nong Li
8d9e8ce9a3 Add another fact table and updates to load a single table at a time. 
Author: Nong Li <nongli@gmail.com>

Closes #31 from nongli/more_tables.
 2015-11-18 11:12:01 -08:00
Andrew Or
426ae30a2e Increase integration surface area with Spark perf 
The changes in this PR are centered around making `Benchmark#runExperiment` accept things other than `Query`s. In particular, in spark-perf we don't always have a DataFrame or an RDD to work with and may want to run arbitrary code (e.g. ALS.train). This PR makes it possible to use the same code in `Benchmark` to do this.

I tested this on dogfood and it works well there.

Author: Andrew Or <andrew@databricks.com>

Closes #33 from andrewor14/spark-perf.
 2015-11-18 10:50:46 -08:00
Michael Armbrust
e516e1e7b3 Use published preview release of 1.6 
Author: Michael Armbrust <michael@databricks.com>

Closes #32 from marmbrus/spark16.
 2015-11-16 22:46:36 -08:00
Andrew Or
172ae79f8d Introduce small integration point with Spark perf 
This allows us to report Spark perf results in the same format as SQL benchmark results. marmbrus

Author: Andrew Or <andrew@databricks.com>

Closes #30 from andrewor14/spark-perf.
 2015-11-16 17:46:53 -08:00
Michael Armbrust
ebcd5db414 Setting version to 0.2.1-SNAPSHOT 2015-11-13 12:41:59 -08:00
Michael Armbrust
e547490b7c Setting version to 0.2.0 2015-11-13 12:41:51 -08:00
Michael Armbrust
344b31ed69 Update to Spark 1.6 
Some internal interfaces changed, so we need to bump the Spark version to run tests on Spark 1.6.

Author: Michael Armbrust <michael@databricks.com>

Closes #29 from marmbrus/spark16.
 2015-11-13 12:40:00 -08:00
Cheng Lian
50808c436b Fixes typos in README.md 
Author: Cheng Lian <lian@databricks.com>

Closes #25 from liancheng/readme-fix.
 2015-11-11 12:05:44 -08:00
Nong Li
dc48f2e49b Support generating the data as "text". 
This previously failed since text only supports a single column. Having the option of
text output is useful to quickly see what the generator is doing.

Author: Nong Li <nongli@gmail.com>

Closes #27 from nongli/text.
 2015-11-11 12:05:14 -08:00
bit1129
f63d40ce9f Add 2 queries 
Author: bit1129 <bit1129@gmail.com>

Closes #22 from bit1129/master.
 2015-09-16 10:10:20 -07:00
Michael Armbrust
40d085f1c7 Add dashboard notebook 
Author: Michael Armbrust <michael@databricks.com>

Closes #21 from marmbrus/master.
 2015-09-11 17:46:07 -07:00
Michael Armbrust
a30dcd3cb7 Merge pull request #20 from marmbrus/releases 
Add a plugin for making releases to Databricks
 2015-09-11 12:25:16 -07:00
Michael Armbrust
8b441c1ee2 Update build.sbt 2015-09-11 12:16:55 -07:00
Michael Armbrust
482aac583a env in travis 2015-09-09 22:47:40 -07:00
Michael Armbrust
9d8ca399db env in travis 2015-09-09 22:45:31 -07:00
Michael Armbrust
241d45817c Setting version to 0.1.3-SNAPSHOT 2015-09-09 22:34:56 -07:00
Michael Armbrust
9c196d0be5 Setting version to 0.1.2 2015-09-09 22:33:46 -07:00
Michael Armbrust
a4458af0be verison 2015-09-09 22:32:46 -07:00
Michael Armbrust
479e4081c2 Add a release process for pushing to DBC 2015-09-09 22:32:31 -07:00
Michael Armbrust
f03b3af719 Fail gracefully when invalid CPU logs are encountered 
Author: Michael Armbrust <michael@databricks.com>

Closes #18 from marmbrus/parseCpuFail.
 2015-09-09 22:02:23 -07:00
Michael Armbrust
e2dc749480 Add more tests for join performance 
Author: Michael Armbrust <michael@databricks.com>

Closes #17 from marmbrus/joinPerf.
 2015-09-09 21:56:47 -07:00
Michael Armbrust
08cb68ca20 Make it easier to write benchmarks in notebooks 
Author: Michael Armbrust <michael@databricks.com>

Closes #19 from marmbrus/notebookTests.
 2015-09-09 21:49:50 -07:00
Michael Armbrust
ddeead18ce Add compilation testing with travis 
There are no tests yet... but this at least tests compilation.

Author: Michael Armbrust <michael@databricks.com>

Closes #15 from marmbrus/travis.
 2015-09-09 21:36:26 -07:00
Michael Armbrust
a49938903d Add Merge Script 
Shamelessly copied from spark... still prompts for options that don't make sense (JIRA) but its better than messy merge trees :)

Author: Michael Armbrust <michael@databricks.com>

Closes #16 from marmbrus/mergeScript.
 2015-09-09 20:03:52 -07:00
Yin Huai
88fa2f5af2 Merge pull request #9 from yhuai/genData 
Add data generation support for TPC-DS
 2015-09-04 15:48:10 -07:00
Yin Huai
34f66a0a10 Add a option of filter rows with null partition column values. 2015-08-26 11:14:19 -07:00
Yin Huai
f4e20af107 fix typo 2015-08-25 23:31:50 -07:00
Yin Huai
06eb11f326 Fix the seed to 100 and use distribute by instead of order by. 2015-08-25 20:44:14 -07:00
Yin Huai
9936d49239 Add a option to orderBy partition columns. 2015-08-25 20:44:14 -07:00
Yin Huai
58188c6711 Allow users to use double instead of decimal for generated tables. 2015-08-25 20:44:14 -07:00
Yin Huai
88aadb45a4 Update README. 2015-08-25 20:44:14 -07:00
Yin Huai
77fbe22b7b address comments. 2015-08-25 20:44:13 -07:00
Yin Huai
97093a45cd Update readme and register temp tables. 2015-08-25 20:44:13 -07:00
Yin Huai
edb4daba80 Bug fix. 2015-08-25 20:44:13 -07:00
Yin Huai
544adce70f Add methods to genData. 2015-08-25 20:44:13 -07:00
Michael Armbrust
e046705e7f update version 2015-08-24 16:14:17 -07:00
Michael Armbrust
98dd76befd Release 0.1.1 2015-08-24 16:13:51 -07:00
Michael Armbrust
32215e05ee Block completion of cpu collection 2015-08-24 16:13:26 -07:00
Michael Armbrust
e5ac7f6b4a update version 0.1.1-SNAPSHOT 2015-08-23 13:45:01 -07:00
Michael Armbrust
cabbf7291c release 0.1 2015-08-23 13:44:23 -07:00
Yin Huai
8e46fbdb6c Merge pull request #11 from marmbrus/cpuProfile 
Add support for CPU Profiling
 2015-08-21 17:15:23 -07:00