apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity
2675e9a80d
kyuubi/.github/workflows/master.yml
Cheng Pan 589b23f578
[KYUUBI #1257] [BUILD] Reduce unnecessary maven profile 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
The changes based on https://github.com/apache/incubator-kyuubi/pull/1226#issuecomment-942979767

In this PR, I'm going to remove profiles `kyuubi-extension-spark-3-1` and `kyuubi-extension-spark-3-2`, and keep `spark-3.0`, `spark-3.1`, `spark-3.2`.

After changes, when `spark-3.1` is active, set `spark.version` to 3.1.2 and enable `kyuubi-extension-spark-common` module and `kyuubi-extension-spark-3-1` module, `spark-3.2` does the same thing

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1257 from pan3793/pom.

Closes #1257

1d871b6c [Cheng Pan] nit
874f30c1 [Cheng Pan] Address comments
00a32b25 [Cheng Pan] nit
445666ee [Cheng Pan] typo
bd96097e [Cheng Pan] nit
e85116cd [Cheng Pan] nit
98347c8c [Cheng Pan] Update doc
6615d7cc [Cheng Pan] Exclude HudiTest in cross test
087d3fde [Cheng Pan] Add spark 3.2 binary test
ccff63c1 [Cheng Pan] update GA
23b81dcd [Cheng Pan] Address comments
3055c2a8 [Cheng Pan] nit
49a41e86 [Cheng Pan] nit
8d400798 [Cheng Pan] Reduce unnecessary maven profile
ced8d987 [Cheng Pan] Reduce unnecessary maven profile

Lead-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: Cheng Pan <chengpan@ChengdeMac-mini.local>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2021-10-22 14:51:42 +08:00

202 lines
6.6 KiB
YAML
Raw Blame History

name: Kyuubi
on:
push:
branches:
- master
- branch-*
pull_request:
branches:
- master
- branch-*
jobs:
rat:
name: Check License
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout@v2
- uses: actions/setup-java@v1
with:
java-version: '1.8'
- run: build/mvn org.apache.rat:apache-rat-plugin:check
- name: Upload rat report
if: failure()
uses: actions/upload-artifact@v2
with:
name: rat
path: "**/target/rat*.txt"
build:
name: Build
runs-on: ubuntu-20.04
strategy:
matrix:
include:
- java: 8
spark: '3.0'
spark-hadoop: '2.7'
- java: 8
spark: '3.1'
spark-hadoop: '2.7'
profiles: '-Dspark.archive.mirror=https://archive.apache.org/dist/spark/spark-3.0.3 -Dspark.archive.name=spark-3.0.3-bin-hadoop2.7.tgz -Dmaven.plugin.scalatest.exclude.tags=org.apache.kyuubi.tags.ExtendedSQLTest,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.HudiTest,org.apache.kyuubi.tags.IcebergTest'
- java: 8
spark: '3.1'
spark-hadoop: '3.2'
profiles: '-Dspark.archive.mirror=https://archive.apache.org/dist/spark/spark-3.2.0 -Dspark.archive.name=spark-3.2.0-bin-hadoop3.2.tgz -Dmaven.plugin.scalatest.exclude.tags=org.apache.kyuubi.tags.ExtendedSQLTest,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.HudiTest,org.apache.kyuubi.tags.IcebergTest'
- java: 8
spark: '3.1'
spark-hadoop: '3.2'
codecov: 'true'
- java: 8
spark: '3.2'
spark-hadoop: '3.2'
- java: 11
spark: '3.1'
spark-hadoop: '3.2'
profiles: '-DskipTests -Pspark-provided'
env:
SPARK_LOCAL_IP: localhost
steps:
- uses: actions/checkout@v2
- name: Setup JDK ${{ matrix.java }}
uses: actions/setup-java@v1
with:
java-version: ${{ matrix.java }}
- uses: actions/cache@v1
with:
path: ~/.m2/repository/com
key: ${{ runner.os }}-maven-com-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-com-
- uses: actions/cache@v1
with:
path: ~/.m2/repository/org
key: ${{ runner.os }}-maven-org-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-org-
- uses: actions/cache@v1
with:
path: ~/.m2/repository/net
key: ${{ runner.os }}-maven-net-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-net-
- uses: actions/cache@v1
with:
path: ~/.m2/repository/io
key: ${{ runner.os }}-maven-io-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-io-
- name: Build with Maven
run: >-
./build/mvn clean install -Dmaven.javadoc.skip=true -V
-Pspark-${{ matrix.spark }} -Pspark-hadoop-${{ matrix.spark-hadoop }} ${{ matrix.profiles }}
- name: Code coverage
if: ${{ matrix.codecov == 'true' }}
uses: codecov/codecov-action@v2
with:
verbose: true
- name: Detected Dependency List Change
if: ${{ matrix.java == 8 }}
run: build/dependency.sh
- name: Upload test logs
if: failure()
uses: actions/upload-artifact@v2
with:
name: unit-tests-log
path: |
**/target/unit-tests.log
**/kyuubi-spark-sql-engine.log*
**/target/scalastyle-output.xml
tpcds:
name: TPC-DS Tests
runs-on: ubuntu-20.04
env:
SPARK_LOCAL_IP: localhost
steps:
- uses: actions/checkout@v2
- name: Setup JDK 1.8
uses: actions/setup-java@v1
with:
java-version: '1.8'
- uses: actions/cache@v1
with:
path: ~/.m2/repository/com
key: ${{ runner.os }}-maven-com-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-com-
- uses: actions/cache@v1
with:
path: ~/.m2/repository/org
key: ${{ runner.os }}-maven-org-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-org-
- uses: actions/cache@v1
with:
path: ~/.m2/repository/net
key: ${{ runner.os }}-maven-net-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-net-
- uses: actions/cache@v1
with:
path: ~/.m2/repository/io
key: ${{ runner.os }}-maven-io-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-io-
- name: Run TPC-DS Tests
run: >-
./build/mvn clean install -Dmaven.javadoc.skip=true -V
-pl kyuubi-server -am
-Pspark-3.1
-Dmaven.plugin.scalatest.exclude.tags=''
-Dtest=none -DwildcardSuites=org.apache.kyuubi.operation.tpcds
minikube-it:
name: Minikube Integration Test
runs-on: ubuntu-20.04
steps:
- name: Checkout
uses: actions/checkout@v2
# https://github.com/docker/build-push-action
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
- name: Build Kyuubi Docker Image
uses: docker/build-push-action@v2
with:
# passthrough CI into build container
build-args: CI=${CI}
context: .
file: docker/Dockerfile
load: true
tags: apache/kyuubi:latest
# from https://github.com/marketplace/actions/setup-minikube-kubernetes-cluster
- name: Setup Minikube
uses: manusa/actions-setup-minikube@v2.4.2
with:
minikube version: 'v1.16.0'
kubernetes version: 'v1.19.2'
- name: kubectl pre-check
run: |
kubectl get serviceaccount
kubectl create serviceaccount kyuubi
kubectl get serviceaccount
- name: start kyuubi
run: kubectl apply -f kubernetes/integration-tests/test-k8s.yaml
- name: kyuubi pod check
run: kubectl get pods
- name: integration tests
run: >-
./build/mvn clean install -Dmaven.javadoc.skip=true -V
-pl kubernetes/integration-tests -am
-Pkubernetes
-Dtest=none -DwildcardSuites=org.apache.kyuubi.kubernetes.test
- name: Upload test logs
if: failure()
uses: actions/upload-artifact@v2
with:
name: unit-tests-log
path: |
**/target/unit-tests.log
Reference in New Issue View Git Blame Copy Permalink