github/spark-sql-perf
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
75 Commits 1 Branch 0 Tags 682 KiB
426ae30a2e
Commit Graph

75 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Andrew Or
426ae30a2e Increase integration surface area with Spark perf 
The changes in this PR are centered around making `Benchmark#runExperiment` accept things other than `Query`s. In particular, in spark-perf we don't always have a DataFrame or an RDD to work with and may want to run arbitrary code (e.g. ALS.train). This PR makes it possible to use the same code in `Benchmark` to do this.

I tested this on dogfood and it works well there.

Author: Andrew Or <andrew@databricks.com>

Closes #33 from andrewor14/spark-perf.
 2015-11-18 10:50:46 -08:00
Michael Armbrust
e516e1e7b3 Use published preview release of 1.6 
Author: Michael Armbrust <michael@databricks.com>

Closes #32 from marmbrus/spark16.
 2015-11-16 22:46:36 -08:00
Andrew Or
172ae79f8d Introduce small integration point with Spark perf 
This allows us to report Spark perf results in the same format as SQL benchmark results. marmbrus

Author: Andrew Or <andrew@databricks.com>

Closes #30 from andrewor14/spark-perf.
 2015-11-16 17:46:53 -08:00
Michael Armbrust
ebcd5db414 Setting version to 0.2.1-SNAPSHOT 2015-11-13 12:41:59 -08:00
Michael Armbrust
e547490b7c Setting version to 0.2.0 2015-11-13 12:41:51 -08:00
Michael Armbrust
344b31ed69 Update to Spark 1.6 
Some internal interfaces changed, so we need to bump the Spark version to run tests on Spark 1.6.

Author: Michael Armbrust <michael@databricks.com>

Closes #29 from marmbrus/spark16.
 2015-11-13 12:40:00 -08:00
Cheng Lian
50808c436b Fixes typos in README.md 
Author: Cheng Lian <lian@databricks.com>

Closes #25 from liancheng/readme-fix.
 2015-11-11 12:05:44 -08:00
Nong Li
dc48f2e49b Support generating the data as "text". 
This previously failed since text only supports a single column. Having the option of
text output is useful to quickly see what the generator is doing.

Author: Nong Li <nongli@gmail.com>

Closes #27 from nongli/text.
 2015-11-11 12:05:14 -08:00
bit1129
f63d40ce9f Add 2 queries 
Author: bit1129 <bit1129@gmail.com>

Closes #22 from bit1129/master.
 2015-09-16 10:10:20 -07:00
Michael Armbrust
40d085f1c7 Add dashboard notebook 
Author: Michael Armbrust <michael@databricks.com>

Closes #21 from marmbrus/master.
 2015-09-11 17:46:07 -07:00
Michael Armbrust
a30dcd3cb7 Merge pull request #20 from marmbrus/releases 
Add a plugin for making releases to Databricks
 2015-09-11 12:25:16 -07:00
Michael Armbrust
8b441c1ee2 Update build.sbt 2015-09-11 12:16:55 -07:00
Michael Armbrust
482aac583a env in travis 2015-09-09 22:47:40 -07:00
Michael Armbrust
9d8ca399db env in travis 2015-09-09 22:45:31 -07:00
Michael Armbrust
241d45817c Setting version to 0.1.3-SNAPSHOT 2015-09-09 22:34:56 -07:00
Michael Armbrust
9c196d0be5 Setting version to 0.1.2 2015-09-09 22:33:46 -07:00
Michael Armbrust
a4458af0be verison 2015-09-09 22:32:46 -07:00
Michael Armbrust
479e4081c2 Add a release process for pushing to DBC 2015-09-09 22:32:31 -07:00
Michael Armbrust
f03b3af719 Fail gracefully when invalid CPU logs are encountered 
Author: Michael Armbrust <michael@databricks.com>

Closes #18 from marmbrus/parseCpuFail.
 2015-09-09 22:02:23 -07:00
Michael Armbrust
e2dc749480 Add more tests for join performance 
Author: Michael Armbrust <michael@databricks.com>

Closes #17 from marmbrus/joinPerf.
 2015-09-09 21:56:47 -07:00
Michael Armbrust
08cb68ca20 Make it easier to write benchmarks in notebooks 
Author: Michael Armbrust <michael@databricks.com>

Closes #19 from marmbrus/notebookTests.
 2015-09-09 21:49:50 -07:00
Michael Armbrust
ddeead18ce Add compilation testing with travis 
There are no tests yet... but this at least tests compilation.

Author: Michael Armbrust <michael@databricks.com>

Closes #15 from marmbrus/travis.
 2015-09-09 21:36:26 -07:00
Michael Armbrust
a49938903d Add Merge Script 
Shamelessly copied from spark... still prompts for options that don't make sense (JIRA) but its better than messy merge trees :)

Author: Michael Armbrust <michael@databricks.com>

Closes #16 from marmbrus/mergeScript.
 2015-09-09 20:03:52 -07:00
Yin Huai
88fa2f5af2 Merge pull request #9 from yhuai/genData 
Add data generation support for TPC-DS
 2015-09-04 15:48:10 -07:00
Yin Huai
34f66a0a10 Add a option of filter rows with null partition column values. 2015-08-26 11:14:19 -07:00
Yin Huai
f4e20af107 fix typo 2015-08-25 23:31:50 -07:00
Yin Huai
06eb11f326 Fix the seed to 100 and use distribute by instead of order by. 2015-08-25 20:44:14 -07:00
Yin Huai
9936d49239 Add a option to orderBy partition columns. 2015-08-25 20:44:14 -07:00
Yin Huai
58188c6711 Allow users to use double instead of decimal for generated tables. 2015-08-25 20:44:14 -07:00
Yin Huai
88aadb45a4 Update README. 2015-08-25 20:44:14 -07:00
Yin Huai
77fbe22b7b address comments. 2015-08-25 20:44:13 -07:00
Yin Huai
97093a45cd Update readme and register temp tables. 2015-08-25 20:44:13 -07:00
Yin Huai
edb4daba80 Bug fix. 2015-08-25 20:44:13 -07:00
Yin Huai
544adce70f Add methods to genData. 2015-08-25 20:44:13 -07:00
Michael Armbrust
e046705e7f update version 2015-08-24 16:14:17 -07:00
Michael Armbrust
98dd76befd Release 0.1.1 2015-08-24 16:13:51 -07:00
Michael Armbrust
32215e05ee Block completion of cpu collection 2015-08-24 16:13:26 -07:00
Michael Armbrust
e5ac7f6b4a update version 0.1.1-SNAPSHOT 2015-08-23 13:45:01 -07:00
Michael Armbrust
cabbf7291c release 0.1 2015-08-23 13:44:23 -07:00
Yin Huai
8e46fbdb6c Merge pull request #11 from marmbrus/cpuProfile 
Add support for CPU Profiling
 2015-08-21 17:15:23 -07:00
Yin Huai
8674c153b7 Merge pull request #10 from marmbrus/updateSBT 
Update SBT
 2015-08-20 16:52:29 -07:00
Michael Armbrust
3f33db31c0 Update SBT 2015-08-20 16:46:45 -07:00
Michael Armbrust
00aa49e8e4 Add support for CPU Profiling. 2015-08-20 16:46:12 -07:00
Yin Huai
249157f6a6 Fix typo. 2015-08-17 12:56:35 -07:00
Michael Armbrust
d6f89c862d Merge pull request #8 from yhuai/hashSum 
Add a new ExecutionMode to calculate the sum of hash values of result rows.
 2015-08-14 12:39:51 -07:00
Yin Huai
d5c3104ec6 address comments. 2015-08-14 11:39:06 -07:00
Yin Huai
51546868f4 You can specific perf result location. 2015-08-13 18:43:50 -07:00
Yin Huai
11bfdc7c5a Add an ExecutionMode to check query results. 2015-08-13 18:43:49 -07:00
Yin Huai
1fe1729331 Merge pull request #7 from marmbrus/fixBreakdown 
Fixes to breakdown calculation and table creation.
 2015-08-13 18:39:53 -07:00
Michael Armbrust
ed8ddfedcd yins comments 2015-08-13 17:54:00 -07:00