Commit Graph

2 Commits

Author SHA1 Message Date
Cheng Pan
01f0ea8609
[KYUUBI #1743] Fix parallelism of DataGenerator and other enhancements
### _Why are the changes needed?_

The parallelism of DataGenerator always is `spark.sparkContext.defaultParallelism`, it does not make sense for generating large scale data.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1743 from pan3793/tpcds.

Closes #1743

62f7c866 [Cheng Pan] nit
fdcf8329 [Cheng Pan] nit
a52ff489 [Cheng Pan] Fix parallelism of DataGenerator and other enhancements

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-01-13 11:41:22 +08:00
ulysses-you
37a4e5c0da
[KYUUBI #1496] Support tpcds benchmark
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Support tpcds benchmark in `dev/kyuubi-tpcds` module.

Add a `README.md` in `dev/kyuubi-tpcds` module to show how to use.

The mian code is from [databricks-spark-sql-perf](https://github.com/databricks/spark-sql-perf)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1496 from ulysses-you/tpcds-benchmark.

Closes #1496

d4afe2d1 [ulysses-you] comment
54a146ef [ulysses-you] pom
91e71692 [ulysses-you] docs
20eadc49 [ulysses-you] benchmark

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2021-12-06 20:08:08 +08:00