[KYUUBI #1018] Set spark.sql.execution.topKSortFallbackThreshold to 10000
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> To avoid performance issues in the topK scenario and close #1018 . ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1054 from byyue/feature-1018. Closes #1018 21de1fcb [Brian Yue] Set spark.sql.execution.topKSortFallbackThreshold to 10000 Authored-by: Brian Yue <code.byyue@gmail.com> Signed-off-by: Kent Yao <yao@apache.org>
This commit is contained in:
parent
d83674248e
commit
033784c691
@ -96,6 +96,7 @@ object SparkSQLEngine extends Logging {
|
||||
|
||||
def createSpark(): SparkSession = {
|
||||
val sparkConf = new SparkConf()
|
||||
sparkConf.setIfMissing("spark.sql.execution.topKSortFallbackThreshold", "10000")
|
||||
sparkConf.setIfMissing("spark.sql.legacy.castComplexTypesToString.enabled", "true")
|
||||
sparkConf.setIfMissing("spark.master", "local")
|
||||
sparkConf.setIfMissing("spark.ui.port", "0")
|
||||
|
||||
Loading…
Reference in New Issue
Block a user