celeborn/client-spark
Fu Chen 2bd1d86d41
[CELEBORN-775] Fix executorCores calculation in SparkShuffleManager for Spark local mode
### What changes were proposed in this pull request?

As title

### Why are the changes needed?

```shell
$ bin/spark-shell --master local[2]
23/07/06 16:11:30 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/07/06 16:11:40 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context available as 'sc' (master = local[2], app id = local-1688631101733).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.3.1
      /_/

Using Scala version 2.12.15 (OpenJDK 64-Bit Server VM, Java 1.8.0_292)
Type in expressions to have them evaluated.
Type :help for more information.

scala> spark.sparkContext.getConf.get("spark.executor.cores")
java.util.NoSuchElementException: spark.executor.cores
  at org.apache.spark.SparkConf.$anonfun$get$1(SparkConf.scala:245)
  at scala.Option.getOrElse(Option.scala:189)
  at org.apache.spark.SparkConf.get(SparkConf.scala:245)
  ... 47 elided

scala>
```

### Does this PR introduce _any_ user-facing change?

No

### How was this patch tested?

CelebornPipelineSortSuite should cover this change

Closes #1685 from cfmcgrady/local-core-number.

Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-07-06 16:29:59 +08:00
..
common [CELEBORN-751] Rename remain rss related class name and filenames etc 2023-07-04 10:20:08 +08:00
spark-2 [CELEBORN-775] Fix executorCores calculation in SparkShuffleManager for Spark local mode 2023-07-06 16:29:59 +08:00
spark-2-shaded [CELEBORN-716][BUILD] Correct the to name when renaming the Netty native library 2023-06-26 21:57:06 +08:00
spark-3 [CELEBORN-775] Fix executorCores calculation in SparkShuffleManager for Spark local mode 2023-07-06 16:29:59 +08:00
spark-3-shaded [CELEBORN-716][BUILD] Correct the to name when renaming the Netty native library 2023-06-26 21:57:06 +08:00