apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

[KYUUBI #1382] Sync the flink-provided profile in docs and testing suite comments

Browse Source 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Sync the flink-provided profile in docs and testing suite comments.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1382 from turboFei/flink_provided.

Closes #1382

3ae96963 [Fei Wang] address nit
ef5bba7f [fwang12] reorder
e9c9c527 [fwang12] save

Lead-authored-by: fwang12 <fwang12@ebay.com>
Co-authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
This commit is contained in:
fwang12 2021-12-01 20:33:47 +08:00 committed by Fei Wang
parent 712749050e
commit a6d23238cb
8 changed files with 21 additions and 17 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.github/workflows/master.yml vendored
View File

@ -54,7 +54,7 @@ jobs:
- java: 11
spark: '3.1'
spark-hadoop: '3.2'
profiles: '-DskipTests -Pspark-provided'
profiles: '-DskipTests -Pflink-provided,spark-provided'
env:
SPARK_LOCAL_IP: localhost

8
.github/workflows/publish-snapshot.yml vendored
View File

@ -37,9 +37,9 @@ jobs:
ASF_PASSWORD: ${{ secrets.NEXUS_PW }}
run: |
rm -rf ~/.m2/repository/org/apache/kyuubi
./build/mvn clean deploy -s ./build/release/asf-settings.xml -DskipTests -Pspark-provided
./build/mvn clean deploy -s ./build/release/asf-settings.xml -DskipTests -Pspark-provided,spark-3.1 -pl dev/kyuubi-extension-spark-3-1 -am
./build/mvn clean deploy -s ./build/release/asf-settings.xml -DskipTests -Pspark-provided,spark-3.2 -pl dev/kyuubi-extension-spark-3-2 -am
./build/mvn clean deploy -s ./build/release/asf-settings.xml -DskipTests -Pflink-provided,spark-provided
./build/mvn clean deploy -s ./build/release/asf-settings.xml -DskipTests -Pflink-provided,spark-provided,spark-3.1 -pl dev/kyuubi-extension-spark-3-1 -am
./build/mvn clean deploy -s ./build/release/asf-settings.xml -DskipTests -Pflink-provided,spark-provided,spark-3.2 -pl dev/kyuubi-extension-spark-3-2 -am
- name: Publish snapshot - branch-1.3
if: ${{ matrix.branch == 'branch-1.3' }}
env:
@ -47,4 +47,4 @@ jobs:
ASF_PASSWORD: ${{ secrets.NEXUS_PW }}
run: |
rm -rf ~/.m2/repository/org/apache/kyuubi
./build/mvn clean deploy -s ./build/release/asf-settings.xml -DskipTests -Pspark-provided,kyuubi-extension-spark-3-1
./build/mvn clean deploy -s ./build/release/asf-settings.xml -DskipTests -Pflink-provided,spark-provided,kyuubi-extension-spark-3-1

4
build/dist
View File

@ -30,6 +30,7 @@ set -e
KYUUBI_HOME="$(cd "`dirname "$0"`/.."; pwd)"
DISTDIR="$KYUUBI_HOME/dist"
MAKE_TGZ=false
# TODO: add FLINK_PROVIDED option
SPARK_PROVIDED=false
NAME=none
MVN="$KYUUBI_HOME/build/mvn"
@ -40,10 +41,11 @@ function usage {
echo ""
echo "Usage:"
echo "+--------------------------------------------------------------------------------------+"
echo "| ./build/dist [--name <custom_name>] [--tgz] [--spark-provided] <maven build options> |"
echo "| ./build/dist [--name <custom_name>] [--tgz] [--flink-provided] [--spark-provided] <maven build options> |"
echo "+--------------------------------------------------------------------------------------+"
echo "name: - custom binary name, using project version if undefined"
echo "tgz: - whether to make a whole bundled package"
echo "flink-provided: - whether to make a package without Flink binary"
echo "spark-provided: - whether to make a package without Spark binary"
echo ""
}

6
build/release/release.sh
View File

@ -94,12 +94,12 @@ upload_svn_staging() {
}
upload_nexus_staging() {
${KYUUBI_DIR}/build/mvn clean deploy -DskipTests -Papache-release,spark-provided \
${KYUUBI_DIR}/build/mvn clean deploy -DskipTests -Papache-release,flink-provided,spark-provided \
-s "${KYUUBI_DIR}/build/release/asf-settings.xml"
${KYUUBI_DIR}/build/mvn clean deploy -DskipTests -Papache-release,spark-provided,spark-3.1 \
${KYUUBI_DIR}/build/mvn clean deploy -DskipTests -Papache-release,flink-provided,spark-provided,spark-3.1 \
-s "${KYUUBI_DIR}/build/release/asf-settings.xml" \
-pl dev/kyuubi-extension-spark-3-1 -am
${KYUUBI_DIR}/build/mvn clean deploy -DskipTests -Papache-release,spark-provided,spark-3.2 \
${KYUUBI_DIR}/build/mvn clean deploy -DskipTests -Papache-release,flink-provided,spark-provided,spark-3.2 \
-s "${KYUUBI_DIR}/build/release/asf-settings.xml" \
-pl dev/kyuubi-extension-spark-3-2 -am
}

7
docs/develop_tools/distribution.md
View File

@ -33,10 +33,11 @@ For more information on usage, run `./build/dist --help`
Usage:
+--------------------------------------------------------------------------------------+
| ./build/dist [--name <custom_name>] [--tgz] [--spark-provided] <maven build options> |
| ./build/dist [--name <custom_name>] [--tgz] [--flink-provided] [--spark-provided] <maven build options> |
+--------------------------------------------------------------------------------------+
name: - custom binary name, using project version if undefined
tgz: - whether to make a whole bundled package
flink-provided: - whether to make a package without Flink binary
spark-provided: - whether to make a package without Spark binary
```
@ -48,10 +49,10 @@ For instance,
This results in a Kyuubi distribution named `kyuubi-{version}-bin-custom-name.tgz` for you.
If you are planing to deploy Kyuubi where `spark` is provided, in other word, it's not required to bundle spark binary, use
If you are planing to deploy Kyuubi where `spark`/`flink` is provided, in other word, it's not required to bundle spark/flink binary, use
```bash
./build/dist --tgz --spark-provided
./build/dist --tgz --spark-provided --flink-provided
```
Then you will get a Kyuubi distribution without spark binary named `kyuubi-{version}-bin.tgz`.

4
externals/kyuubi-spark-sql-engine/src/test/scala/org/apache/kyuubi/engine/spark/udf/KyuubiDefinedFunctionSuite.scala vendored
View File

@ -30,12 +30,12 @@ import org.apache.kyuubi.{KyuubiFunSuite, TestUtils}
*
* To run the entire test suite:
* {{{
* build/mvn clean install -Pspark-provided -DwildcardSuites=org.apache.kyuubi.engine.spark.udf.KyuubiDefinedFunctionSuite
* build/mvn clean install -Pflink-provided,spark-provided -DwildcardSuites=org.apache.kyuubi.engine.spark.udf.KyuubiDefinedFunctionSuite
* }}}
*
* To re-generate golden files for entire suite, run:
* {{{
* KYUUBI_UPDATE=1 build/mvn clean install -Pspark-provided -DwildcardSuites=org.apache.kyuubi.engine.spark.udf.KyuubiDefinedFunctionSuite
* KYUUBI_UPDATE=1 build/mvn clean install -Pflink-provided,spark-provided -DwildcardSuites=org.apache.kyuubi.engine.spark.udf.KyuubiDefinedFunctionSuite
* }}}
*/
// scalastyle:on line.size.limit

3
kyuubi-common/src/test/scala/org/apache/kyuubi/TestUtils.scala
View File

@ -52,7 +52,8 @@ object TestUtils {
}
reader.close()
val hint = s"$markdown out of date, please update doc with " +
s"KYUUBI_UPDATE=1 build/mvn clean install -Pspark-provided -DwildcardSuites=$agent"
s"KYUUBI_UPDATE=1 build/mvn clean install -Pflink-provided,spark-provided " +
s"-DwildcardSuites=$agent"
assert(newOutput.size === expected.size, hint)
newOutput.zip(expected).foreach { case (out, in) => assert(out === in, hint) }

4
kyuubi-server/src/test/scala/org/apache/kyuubi/config/AllKyuubiConfiguration.scala
View File

@ -35,12 +35,12 @@ import org.apache.kyuubi.zookeeper.ZookeeperConf
*
* To run the entire test suite:
* {{{
* build/mvn clean install -Pspark-provided -DwildcardSuites=org.apache.kyuubi.config.AllKyuubiConfiguration
* build/mvn clean install -Pflink-provided,spark-provided -DwildcardSuites=org.apache.kyuubi.config.AllKyuubiConfiguration
* }}}
*
* To re-generate golden files for entire suite, run:
* {{{
* KYUUBI_UPDATE=1 build/mvn clean install -Pspark-provided -DwildcardSuites=org.apache.kyuubi.config.AllKyuubiConfiguration
* KYUUBI_UPDATE=1 build/mvn clean install -Pflink-provided,spark-provided -DwildcardSuites=org.apache.kyuubi.config.AllKyuubiConfiguration
* }}}
*/
// scalastyle:on line.size.limit