apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

[KYUUBI #361] Remove Hive Suffix for built in

Browse Source 
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #361](https://badgen.net/badge/Preview/Closes%20%23361/blue)](https://github.com/yaooqinn/kyuubi/pull/361) ![5](https://badgen.net/badge/%2B/5/red) ![11](https://badgen.net/badge/-/11/green) ![1](https://badgen.net/badge/commits/1/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

the builtin hive1.2 is no need for builtin anymore

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #361 from yaooqinn/hivesuffix.

99d49fc [Kent Yao] Remove Hive Suffix for built in

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
This commit is contained in:
Kent Yao 2021-02-24 00:13:20 +08:00
parent ee50890b00
commit 28db93cc75
No known key found for this signature in database
GPG Key ID: F7051850A0AF904D
2 changed files with 5 additions and 11 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

13
bin/load-kyuubi-env.sh
View File

@ -53,18 +53,13 @@ if [[ -z ${JAVA_HOME} ]]; then
fi
export KYUUBI_SCALA_VERSION="${KYUUBI_SCALA_VERSION:-"2.12"}"
SPARK_VERSION_BUILD="$(grep "Spark " "$KYUUBI_HOME/RELEASE" | awk -F ' ' '{print $2}')"
HADOOP_VERSION_BUILD="$(grep "Hadoop " "$KYUUBI_HOME/RELEASE" | awk -F ' ' '{print $2}')"
HIVE_VERSION_BUILD="$(grep "Hive " "$KYUUBI_HOME/RELEASE" | awk -F ' ' '{print $2}')"
if [[ ${HIVE_VERSION_BUILD:0:3} == "2.3" ]]; then
HIVE_VERSION_SUFFIX=""
else
HIVE_VERSION_SUFFIX="-hive1.2"
if [[ -f ${KYUUBI_HOME}/RELEASE ]]; then
SPARK_VERSION_BUILD="$(grep "Spark " "$KYUUBI_HOME/RELEASE" | awk -F ' ' '{print $2}')"
HADOOP_VERSION_BUILD="$(grep "Hadoop " "$KYUUBI_HOME/RELEASE" | awk -F ' ' '{print $2}')"
SPARK_BUILTIN="${KYUUBI_HOME}/externals/spark-$SPARK_VERSION_BUILD-bin-hadoop${HADOOP_VERSION_BUILD:0:3}"
fi
SPARK_BUILTIN="${KYUUBI_HOME}/externals/spark-$SPARK_VERSION_BUILD-bin-hadoop${HADOOP_VERSION_BUILD:0:3}$HIVE_VERSION_SUFFIX"
if [[ ! -d ${SPARK_BUILTIN} ]]; then
MVN="${MVN:-"${KYUUBI_HOME}/build/mvn"}"
SPARK_VERSION_BUILD=$("$MVN" help:evaluate -Dexpression=spark.version 2>/dev/null\

3
kyuubi-main/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala
View File

@ -49,7 +49,6 @@ class SparkProcessBuilder(
} getOrElse {
val sparkVer = SPARK_COMPILE_VERSION
val hadoopVer = HADOOP_COMPILE_VERSION.take(3)
val hiveVer = if (HIVE_COMPILE_VERSION.take(3).toDouble < 2.3) "-hive1.2" else ""
val kyuubiPattern = "/kyuubi/"
val cwd = getClass.getProtectionDomain.getCodeSource.getLocation.getPath
val idx = kyuubiPattern.length + cwd.lastIndexOf(kyuubiPattern)
@ -59,7 +58,7 @@ class SparkProcessBuilder(
"externals",
"kyuubi-download",
"target",
s"spark-$sparkVer-bin-hadoop$hadoopVer$hiveVer",
s"spark-$sparkVer-bin-hadoop$hadoopVer",
"bin", SPARK_SUBMIT_FILE)
}
path.toAbsolutePath.toFile.getCanonicalPath