Commit Graph

55 Commits

Author SHA1 Message Date
Cheng Pan
2c64e4e5fc
[KYUUBI #7165] Remove banned action-shellcheck
### Why are the changes needed?

The plugin is banned by ASF.

> ludeeus/action-shellcheck1.1.0 is not allowed to be used in apache/kyuubi.

https://github.com/apache/kyuubi/actions/runs/16745477309

### How was this patch tested?

Pass GHA.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #7165 from pan3793/rm-shellcheck.

Closes #7165

dfda2314c [Cheng Pan] Remove banned action-shellcheck

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2025-08-07 15:05:00 +08:00
Bowen Liang
1d668ea096 [KYUUBI #6768] [CI] skip explicitly setting check-latest to false
# 🔍 Description
## Issue References 🔗

This pull request fixes #

## Describe Your Solution 🔧

- skip explicitly set the parameter `check-latest` of Github action `actions/setup-java` to false, which is its default value.
- docs: https://github.com/actions/setup-java?tab=readme-ov-file#check-latest

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6768 from bowenliang123/action-java-latest-false.

Closes #6768

a0ecd4688 [Bowen Liang] update

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2024-10-22 16:09:35 +08:00
Bowen Liang
fcf0405397 [KYUUBI #6741] [CI] update GHA actions/upload-artifact from v3 to v4
# 🔍 Description
## Issue References 🔗

This pull request fixes #

## Describe Your Solution 🔧

- as titled

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6741 from bowenliang123/gha-upload-artifact-v4.

Closes #6741

83a0ee90e [Bowen Liang] update GHA actions/upload-artifact from v3 to v4

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2024-10-21 15:48:19 +08:00
Bowen Liang
a3edd24188 [KYUUBI #6739] [CI] Bump runner image from Ubuntu 22.04 to 24.04
# 🔍 Description
## Issue References 🔗

This pull request fixes #

## Describe Your Solution 🔧

- The Ubuntu 24.04 image for Actions is now generally available since Sep 2024 : https://github.blog/changelog/2024-09-25-actions-new-images-and-ubuntu-latest-changes/

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6739 from bowenliang123/ubuntu-2404.

Closes #6739

2d120d1f6 [Bowen Liang] bump ci image from ubuntu 22.04 to 24.04

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2024-10-21 15:45:53 +08:00
Bowen Liang
e663cc35d4 [KYUUBI #6687] [BUILD] Use MAVEN_ARGS for passing Maven CLI arguments
# 🔍 Description
## Issue References 🔗

This pull request fixes #

## Describe Your Solution 🔧

- inspired by comments from #6664 : https://github.com/apache/kyuubi/pull/6664#issuecomment-2344351119

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6687 from bowenliang123/mvn-args.

Closes #6687

6a9170b60 [Cheng Pan] Update .github/workflows/master.yml
3573131d8 [liangbowen] export MAVEN_ARGS
36ced0532 [liangbowen] update
a93fa28ee [liangbowen] MAVEN_ARGS

Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2024-10-21 10:18:29 +08:00
Bowen Liang
9ef38b9582 [KYUUBI #6712] [UI] Bump pnpm to v9 and nodejs to v20 LTS
# 🔍 Description
## Issue References 🔗

This pull request fixes #

## Describe Your Solution 🔧

- bump `pnpm` to v9.11, release note: https://github.com/pnpm/pnpm/releases/tag/v9.11.0
  - updated pnpm lockfile by running `pnpm install --fix-lockfile`
- bump `nodejs` version from v18 to v20 (LTS version, released on 18 Apr 2023)
- update related github actions, aligning the pnpm and nodejs versions

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6712 from bowenliang123/pnpm9.

Closes #6712

ca6e425f4 [liangbowen] nit
a9ea92216 [liangbowen] bump pnpm to v9

Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2024-09-26 16:26:20 +08:00
Bowen Liang
98b3929230 [KYUUBI #6682] Bump super-linter to v7
# 🔍 Description
## Issue References 🔗

This pull request fixes #

## Describe Your Solution 🔧

- bump superlinter Github Action to v7
- skip linting `.env` files as the rules are too strict in unorder keys
- remove deprecated `ERROR_ON_MISSING_EXEC_BIT` option
- no major change

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

- pass the style CI workflow

#### Related Unit Tests

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6682 from bowenliang123/superlinter-7.

Closes #6682

0ff321127 [Bowen Liang] skip linting env files
cab7aa810 [liangbowen] Remove deprecated ERROR_ON_MISSING_EXEC_BIT
67515df5b [liangbowen] bump superlinter to v7

Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn>
Co-authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2024-09-18 10:37:55 +08:00
Cheng Pan
063a192c7a
[KYUUBI #6545] Deprecate and remove building support for Spark 3.2
# 🔍 Description

This pull request aims to remove building support for Spark 3.2, while still keeping the engine support for Spark 3.2.

Mailing list discussion: https://lists.apache.org/thread/l74n5zl1w7s0bmr5ovxmxq58yqy8hqzc

- Remove Maven profile `spark-3.2`, and references on docs, release scripts, etc.
- Keep the cross-version verification to ensure that the Spark SQL engine built on the default Spark version (3.5) still works well on Spark 3.2 runtime.
- Merge `kyuubi-extension-spark-common` into `kyuubi-extension-spark-3-3`
- Remove `log4j.properties` as Spark moves to Log4j2 since 3.3 (SPARK-37814)

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Pass GHA.

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6545 from pan3793/deprecate-spark-3.2.

Closes #6545

54c172528 [Cheng Pan] fix
f4602e805 [Cheng Pan] Deprecate and remove building support for Spark 3.2
2e083f89f [Cheng Pan] fix style
458a92c53 [Cheng Pan] nit
929e1df36 [Cheng Pan] Deprecate and remove building support for Spark 3.2

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-07-22 11:59:34 +08:00
Cheng Pan
da2f0799e7
[KYUUBI #6170] Merge JDBC Trino TPC jobs and Style Check jobs in GitHub Actions workflow
# 🔍 Description

We are asked by ASF Infra to limit the max-parallel up to 10 for each workflow.

Merge JDBC Trino TPC jobs in GitHub Actions CI to reduce concurrency of workflow, also increase max-parallel of Spark jobs from 1 to 3

Also merge "shell check" job into "style check" job

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Pass GA.

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6170 from pan3793/ci-2.

Closes #6170

9bf505321 [Cheng Pan] merge shell check
d0e7ece54 [Cheng Pan] nit
b17169573 [Cheng Pan] Tune CI

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-03-12 15:22:22 +08:00
Cheng Pan
e0d706e696
[KYUUBI #6091] Deprecate and remove building support for Spark 3.1
# 🔍 Description
## Issue References 🔗

This pull request aims to remove building support for Spark 3.1, while still keeping the engine support for Spark 3.1.

- VOTE: https://lists.apache.org/thread/670fx1qx7rm0vpvk8k8094q2d0fthw5b
- VOTE RESULT: https://lists.apache.org/thread/0zdxg5zjnc1wpxmw9mgtsxp1ywqt6qvb

The next step is to clean up code in Spark extensions to drop 3.1-related code.

## Describe Your Solution 🔧

- Remove Maven profile `spark-3.1`, and references on docs, release scripts, etc.
- Keep the cross-version verification to ensure that the Spark SQL engine built on the default Spark version (3.4) still works well on Spark 3.1 runtime.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Pass GA.

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6091 from pan3793/remove-spark-3.1-profile.

Closes #6091

ce2983284 [Cheng Pan] nit
5887c808b [Cheng Pan] migration guide
cf28096d3 [Cheng Pan] Log deprecation message on Spark SQL engine with 3.1
a467e618d [Cheng Pan] nit
e11c0fb31 [Cheng Pan] Remove building support for Spark 3.1

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-03-04 20:23:06 +08:00
zwangsheng
c6bba915b2
[KYUUBI #5941] Drop Kubernetes Block Cleaner Tool from Kyuubi
# 🔍 Description
## Issue References 🔗

This pull request fixes #5941

## Describe Your Solution 🔧

Originally aims to support Spark On Kubernetes Shuffle Data Clean, limitations became more and more apparent over time, so let's drop this.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #5942 from zwangsheng/KYUUBI#5941.

Closes #5941

23bf14f37 [Cheng Pan] Update docs/tools/spark_block_cleaner.md
1c3350186 [zwangsheng] fix comment
0bdbb1104 [zwangsheng] nit
0a5aa2bfa [zwangsheng] fix comments

Lead-authored-by: zwangsheng <binjieyang@apache.org>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-01-04 15:58:10 +08:00
Bowen Liang
0adb7d1b4f [KYUUBI #5842] [BUILD] Bump actions/setup-node from v3 to v4
# 🔍 Description
## Issue References 🔗

This pull request fixes #

## Describe Your Solution 🔧

- `setup-node`  Github Action v4 release note: https://github.com/actions/setup-node/releases/tag/v4.0.0. Update node version runtime for the actions from node16 to node20.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5842 from bowenliang123/actions-node-4.

Closes #5842

8610525a8 [Bowen Liang] Bump actions/setup-node from v3 to v4

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2023-12-09 23:32:26 +08:00
Bowen Liang
1f2734af02 [KYUUBI #5839] [BUILD] Bump actions/setup-java from v3 to v4
# 🔍 Description
## Issue References 🔗

As described.

## Describe Your Solution 🔧

- `setup-java`  Github Action v4 release note: https://github.com/actions/setup-java/releases/tag/v4.0.0. Update node version runtime for the actions from node16 to node20.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5839 from bowenliang123/actions-java-4.

Closes #5839

af76bf44c [Bowen Liang] Bump actions/setup-java from v3 to v4

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2023-12-09 23:30:51 +08:00
Bowen Liang
7072bda19f [KYUUBI #5838] [BUILD] Bump actions/setup-python from v4 to v5
# 🔍 Description
## Issue References 🔗

As described.

## Describe Your Solution 🔧

- `setup-python`  Github Action v5 release note: https://github.com/actions/setup-python/releases/tag/v5.0.0. Update node version runtime for the actions from node16 to node20.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5838 from bowenliang123/actions-python-5.

Closes #5838

f1703d690 [Bowen Liang] Bump actions/setup-python from v4 to v5

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2023-12-09 23:30:21 +08:00
liangbowen
f6e7ef3d50 [KYUUBI #5774] [CI] Bump actions/checkout from v3 to v4
# 🔍 Description
## Issue References 🔗

As described.

## Describe Your Solution 🔧

Bump actions/checkout from v3 to v4
- v4.1.1 release notes: https://github.com/actions/checkout/releases/tag/v4.1.1
- default Nodejs runtime for the action bumped from 16 to 20

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️
No behavior changes.

#### Behavior With This Pull Request 🎉
No behavior changes.

#### Related Unit Tests
CI workflows.

---

# Checklists
## 📝 Author Self Checklist

- [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [ ] Pull request title is okay.
- [ ] No license issues.
- [ ] Milestone correctly set?
- [ ] Test coverage is ok
- [ ] Assignees are selected.
- [ ] Minimum number of approvals
- [ ] No changes are requested

**Be nice. Be informative.**

Closes #5774 from bowenliang123/checkout-v4.

Closes #5774

c31edbffb [liangbowen] Bump actions/checkout from v3 to v4

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2023-11-27 10:33:02 +08:00
Bowen Liang
48bdc7d4cb [KYUUBI #5444] [CI] Check style with profiles of Spark 3.4 and 3.5
### _Why are the changes needed?_

- Check style with profile of Spark 3.4 and 3.5 in the style workflow
- Isolated scalastyle check for Spark 3.1 profile, as Iceberg 1.3.1 does not support Spark 3.5

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No.

Closes #5444 from bowenliang123/spotless-spark34.

Closes #5444

ca98c52e6 [Bowen Liang] update
9550a4a0a [Bowen Liang] enable spark-3.1 profile for spotless style checks
dae35290b [Bowen Liang] revert order
3e488cf48 [Bowen Liang] Check with Spark 3.1 profile separately
e879f09f8 [Bowen Liang] revert the order of spark versions in profiles
16456a7fd [Bowen Liang] check style with profile of Spark 3.4 and 3.5

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2023-10-19 14:31:20 +08:00
wforget
d2c072b7c2
[KYUUBI #5336] Spark extension supports Spark 3.5
### _Why are the changes needed?_

It is basically copied from `kyuubi-extension-spark-3-4`.

### _How was this patch tested?_

Compiled successfully:
```
build/mvn clean install -DskipTests -Pflink-provided,spark-provided,hive-provided,spark-3.5
```

- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No

Closes #5336 from wForget/dev_spark_3_5.

Closes #5336

7ba99804a [wforget] remove iceberg.version in spark-3.5 profile
a18ce166a [wforget] Regenerate KyuubiEnsureRequirements based on EnsureRequirements in spark 3.5
4725c4701 [wforget] fix iceberg version
f5a8ea934 [wforget] Bump iceberg 1.4.0
06558dcfa [wforget] make kyuubi-spark-authz plugin compatible with Spark3.5
90d0e4c70 [wforget] make kyuubi-spark-authz plugin compatible with Spark3.5
4bc8d24d6 [wforget] add ci
1b3f2d916 [wforget] Make kyuubi spark extension compatible with Spark3.5

Authored-by: wforget <643348094@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-10-09 10:51:07 +08:00
Cheng Pan
94b2b9bb33
[KYUUBI #5113] Remove dummy Spark Kudu connector module
### _Why are the changes needed?_

It was planned but actually delayed, remove this dummy module to save CI and avoid confusing users and release managers.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

Closes #5113 from pan3793/remove-kudu.

Closes #5113

ff8fd2e6a [Cheng Pan] Remove Spark Kudu connector

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-08-01 17:05:42 +08:00
ulysses-you
5512c6c60f
[KYUUBI #5018] Make kyuubi spark extension compatible with Spark3.4
### _Why are the changes needed?_

The main change is copy code from `kyuubi-extension-spark-common_2.12` and `kyuubi-extension-spark-3-3`. The reason copy `kyuubi-extension-spark-common_2.12` is that, Spark3.4 v1 writes does not have ctas...

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

Closes #5018 from ulysses-you/extension.

Closes #5018

1ca4951e1 [ulysses-you] fix
fb20bb1f0 [ulysses-you] fix
338a3120a [ulysses-you] rebase
241a436fd [ulysses-you] fix
70618f368 [ulysses-you] fix
5f5e2b7c8 [ulysses-you] fix
301ae5c04 [ulysses-you] fix
bb9416811 [ulysses-you] fix
a38595d0a [ulysses-you] copy
9dec27d6c [ulysses-you] copy

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulyssesyou <ulyssesyou@apache.org>
2023-07-06 18:22:23 +08:00
liangbowen
0875ce7066
[KYUUBI #4784] Use nodejs v18 in CI builds and maven web-ui builds
### _Why are the changes needed?_

- nodejs v18 is the current active LTS version of nodejs since October 2022, while v16 is in maintenance mode and no longer in the active status

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

This patch had conflicts when merged, resolved by
Committer: Cheng Pan <chengpan@apache.org>

Closes #4784 from bowenliang123/nodejs18.

Closes #4784

483c3ac56 [liangbowen] use nodejs v18 in ci builds and maven builds

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-05-03 18:42:23 +08:00
liangbowen
e4fe4814ec
[KYUUBI #4783] Use pnpm v8 and update pnpm lockfile to v6 schema
### _Why are the changes needed?_

- v8 is now the stable version of pnpm
- lockfile v6 is used by default since pnpm v8.0, which improves the readability of the lockfile by removing hashes from package IDs and more resistant to git merge conflicts (https://github.com/pnpm/pnpm/releases/tag/v8.0.0)
- pnpm 8 supports nodejs 16 and 18 (https://pnpm.io/installation#compatibility)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4783 from bowenliang123/pnpm8.

Closes #4783

8b3db3820 [liangbowen] use pnpm v8 and update lock file to v6

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-05-03 18:39:22 +08:00
liangbowen
83c4bed076
[KYUUBI #4747] [INFRA] Install and cache build/mvn in CI builds
### _Why are the changes needed?_

- introduce a custom action for installing and caching Maven, to prevent repeatedly downloading Maven binary from remote.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4747 from bowenliang123/cache-maven.

Closes #4747

bbd9e1f4c [liangbowen] fix cache path
dc342180e [liangbowen] add action for caching maven in ci builds

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-04-23 09:52:34 +08:00
liangbowen
3ac8df8324 [KYUUBI #4695] Bump super-linter action from v4 to v5
### _Why are the changes needed?_

- super-liner action v5.0.0 release notes: https://github.com/github/super-linter/releases/tag/v5.0.0

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4695 from bowenliang123/superlinter-v5.

Closes #4695

7cd3095a9 [liangbowen] update
7826c8043 [liangbowen] update
89e7b3d7d [liangbowen] bump superlinter to v5

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-04-17 09:42:25 +08:00
Cheng Pan
6431225b44
Revert "[KYUUBI #4274] [INFRA] Introduce mvnd to speed up CI jobs of Dependency, Licence and Style Check"
This reverts commit d862272645.
2023-04-12 20:13:38 +08:00
Cheng Pan
1c6965270a
Revert "[KYUUBI #4274] [FOLLOWUP] Increase maximum degree of concurrency for mvnd in CI jobs"
This reverts commit 5fab9b710a.
2023-04-12 20:09:25 +08:00
Cheng Pan
af82f4dfb8
Revert "[KYUUBI #4481] Setup and use cached maven in CI jobs"
This reverts commit d2a0fe2b60.
2023-04-12 20:08:49 +08:00
zwangsheng
6a7c57140d
[KYUUBI #4626] [UI][CI] Fix Web UI Ci check fail with pnpm the lockfile needs updates
### _Why are the changes needed?_

After pnpm released 8.X version

We should specify pnpm version when npm install.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
- [x] CI with Web UI Build Check & Web UI Style Check

Closes #4626 from zwangsheng/fix/web_ui_ci.

Closes #4626

f7fdc0676 [Cheng Pan] Update .github/workflows/web-ui.yml
61bc10124 [Cheng Pan] Update .github/workflows/style.yml
7d9353218 [zwangsheng] Specify pnpm version
d678c6cb6 [zwangsheng] for feature version
6d83c2b01 [zwangsheng] revert
bbc4b4c2c [zwangsheng] Specify pnpm version

Lead-authored-by: zwangsheng <2213335496@qq.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-28 18:10:14 +08:00
liangbowen
06e69ddb2b [KYUUBI #4295] Introduce super-linter action for linting JSON, XML, ENV files and bash_exec
### _Why are the changes needed?_

- introduce `action/super-linter` with multiple linters, (https://github.com/marketplace/actions/super-linter)
- using the smaller size version of image `github/super-linter/slim` for faster image pulling
- enable linting for `bash_exec` for checking executable of bash scripts, with fixed `tools/spark-block-cleaner/kubernetes/docker/entrypoint.sh`
- enable  integrity checks for JSON files in JSONC style
- enable integrity checks for XML files
- enable linting for ENV files

### _How was this patch tested?_
- [x] Pass CI jobs

Closes #4295 from bowenliang123/superlinter-action.

Closes #4295

321d24912 [liangbowen] update
497bf801c [liangbowen] use JSONC instead of JSON to support comments
43787e233 [liangbowen] use superlinter

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-03-18 21:54:45 +08:00
liangbowen
d2a0fe2b60 [KYUUBI #4481] Setup and use cached maven in CI jobs
### _Why are the changes needed?_

- Setup and use cached `build/mvn` in CI jobs, to prevent repeatedly downloading mvn from remote

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4481 from bowenliang123/setup-maven.

Closes #4481

3906430da [liangbowen] ignore occasional mvnd failure
366806256 [liangbowen] setup cached build/maven

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-03-08 22:35:40 +08:00
zwangsheng
ab52c9d4ee
[KYUUBI #4461] [CI][K8S]Add Kyuubi Kubernetes IT Module to CI style check
### _Why are the changes needed?_

Add `Kyuubi Kubernetes IT` Module to CI style check

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

- [X] Run CI

Closes #4461 from zwangsheng/ci/add_kubernetes.

Closes #4461

7ebac75b0 [zwangsheng] fix style for kubernetes-it
c1754d452 [zwangsheng] Add Kyuubi Kubernetes IT Module to CI style check

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-03-07 14:00:49 +08:00
liangbowen
5fab9b710a [KYUUBI #4274] [FOLLOWUP] Increase maximum degree of concurrency for mvnd in CI jobs
### _Why are the changes needed?_

- concurrent execution and smart builder is shipped with `mvnd`, but the CI log shows they are not activated
- increase maximum degree of concurrency and utilize `SmartBuilder` feature in `mvnd`. `-Dmvnd.minThreads` is set as fallback and it will be ignored if `--threads` or `-Dmvnd.threads` is used.

#### Before:
(https://github.com/apache/kyuubi/actions/runs/4276652363/jobs/7444896450#step:6:64)
```
[INFO] Build maximum degree of concurrency is 1
```

#### After:
(https://github.com/apache/kyuubi/actions/runs/4279322563/jobs/7449912132#step:8:64)
```
[INFO] Using the SmartBuilder implementation with a thread count of 4
[INFO] Build maximum degree of concurrency is 4
```
(https://github.com/apache/kyuubi/actions/runs/4279322563/jobs/7449912132#step:8:553)
```
[INFO] Segment walltime 13 s, segment projects service time 36 s, effective/maximum degree of concurrency 2.63/4
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4420 from bowenliang123/mvnd-smart.

Closes #4274

64f0cc53e [liangbowen] update
391da26d3 [liangbowen] quite in dep and style jobs
c776356a5 [liangbowen] use -Dmvnd.minThreads
d10bfb70c [liangbowen] use -Dmvnd.minThreads
f793d2b6b [liangbowen] warn
bbb929fe3 [liangbowen] quiet
a5e6d4914 [liangbowen] logging warn level
3cfbaad45 [liangbowen] enable quite option and cancel redirect stderr
09442ff69 [liangbowen] update
ca5f855fa [liangbowen] try to redirect stderr
0ebdd18d4 [liangbowen] set completion and print MAVEN_CLI_OPTS
d3b2c9656 [liangbowen] revert explicitly setting `mvnd.noBuffering`
e2eca0140 [liangbowen] revert explicitly setting `mvnd.noBuffering`
dba25e327 [liangbowen] explicitly set `mvnd.noBuffering` to false prevent displaying events continuously
b7583f936 [liangbowen] shell
d60ebd047 [liangbowen] change mvnd cache key
2cc576ef8 [liangbowen] dep
bbd6414c9 [liangbowen] move multithread setting to mvnd's MAVEN_CLI_OPTS
347b58c8c [liangbowen] increase concurrency for spotless check
15c5519e4 [liangbowen] increase concurrency

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
2023-03-02 18:00:19 +08:00
liangbowen
d862272645
[KYUUBI #4274] [INFRA] Introduce mvnd to speed up CI jobs of Dependency, Licence and Style Check
### _Why are the changes needed?_

#### Motivation:
- speed up maven building and testing with `mvnd` (https://github.com/apache/maven-mvnd). `mvnd` itself embeds exactly the same distribution of maven with more improvements and features on the client side and compilation server in daemon mode.
- also inspired by custom github action for mvnd installing in Apache Camel (https://github.com/apache/camel/blob/main/.github/actions/install-mvnd/action.yml)

#### Changes in this PR:
- introducing `mvnd` by adding `build/mvnd` script
- use `mvnd` version `0.9.0` which embeds  `maven` 3.8.7. The maven version embedded in `mvnd` is also check and guaranteed  the same as maven version required in pom by `build/mvnd` script.
- use mvnd in CI jobs of Depedency Check and Style Check

#### Comparision
- CI jobs
(both with maven dependencies cached in local repo)

Job | with `build/mvn` | with `build/mvnd`
--- | --- | ---
Dependency Check | 9min1sec;[see log](https://github.com/apache/kyuubi/actions/runs/3966175262/jobs/6796688187) | 6min 46 sec ; [see log](https://github.com/apache/kyuubi/actions/runs/4124518481/jobs/7123867015)
Style Check | 10min32sec ; [see log](https://github.com/apache/kyuubi/actions/runs/4122613274/jobs/7119603428)  | 7min50s ; [see log](https://github.com/apache/kyuubi/actions/runs/4125013599/jobs/7125006946)
Licence Check | 32s ; [see log](https://github.com/apache/kyuubi/actions/runs/4130799867/jobs/7137835062) | 21s ; [see log](https://github.com/apache/kyuubi/actions/runs/4130923126/jobs/7138088114)

- building entire maven project on local machine (skipping test running and style checking with `fast` profile)

`build/mvnd clean install -Pfast`
<img width="645" alt="Pasted Graphic" src="https://user-images.githubusercontent.com/1935105/217001594-52dfd19d-6c57-4bc4-8336-90bb0c030bd4.png">

`build/mvn clean install -Pfast`
<img width="643" alt="Pasted Graphic 1" src="https://user-images.githubusercontent.com/1935105/217001619-c2ebe7e3-bd14-4f5c-bb02-2ae4ae99cf7f.png">

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4274 from bowenliang123/ci-mvnd.

Closes #4274

380c1c9ea [liangbowen] update cache key
a93bba7b7 [liangbowen] style
bc8da133e [Bowen Liang] Merge branch 'master' into ci-mvnd
97becc035 [liangbowen] add `build/maven-mvnd-*/**` to rat-excludes
5b61f50b9 [liangbowen] apply mvnd to license check
7be181d0f [liangbowen] typo in build/mvnd
c52861df8 [liangbowen] update
f6d0eb28c [liangbowen] introduce mvnd

Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <bowenliang@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-10 16:19:45 +08:00
Cheng Pan
fec6ee6110
[KYUUBI #4285] Do not cancel CI jobs on stable branch
### _Why are the changes needed?_

Since https://github.com/apache/kyuubi/pull/2175, the previous running CI jobs will be canceled on the same branch or PR to save CI resources, this PR keeps the behavior for PR but doesn't cancel the running CI jobs when commit to the stable branch.

Ref: https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#example-using-a-fallback-value

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

I pushed an empty commit to verify the behavior does not change in PR, and I need to merge this to master first, then merge another PR immediately to verify the rest behaviors.

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4285 from pan3793/ci-con.

Closes #4285

3aa7f427e [Cheng Pan] empty
4156eb196 [Cheng Pan] Do not cancel CI jobs on stable branch

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-02-09 19:27:42 +08:00
Cheng Pan
17ac19127a [KYUUBI #4136] [INFRA] Pin ludeeus/action-shellcheck@1.1.0
### _Why are the changes needed?_

To recover the failed style check CI job, "ludeeus/action-shellcheckmaster is not allowed to be used in apache/kyuubi."

The ASF only allows ludeeus/action-shellcheck1.1.0, see https://issues.apache.org/jira/browse/INFRA-24032

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4136 from pan3793/shell-check.

Closes #4136

5604d7118 [Cheng Pan] scandir
2d1b0a1de [Cheng Pan] [INFRA] Pin ludeeus/action-shellcheck@1.1.0

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Fu Chen <cfmcgrady@gmail.com>
2023-01-10 12:19:54 +08:00
袁福元
2dac364177
[KYUUBI #4078] [INFRA] Enable shellcheck on CI
### _Why are the changes needed?_

Fix #2062

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4078 from xiaoyuandajian/fix-#2062.

Closes #4078

a5a01945 [Cheng Pan] Update .github/workflows/style.yml
3c08e692 [袁福元] fix the shellcheck found error file
a33cbb22 [袁福元] fix the shellcheck found error file
da195782 [袁福元] fix the shellcheck found error file
b24e6adf [袁福元] fix the shellcheck found error file
90ef0712 [袁福元] fix the shellcheck found error file
42acb31f [袁福元] fixup! test1
31c1783d [袁福元] fixup! test
c40b5f38 [袁福元] Revert "test"
d97801fa [袁福元] Merge remote-tracking branch 'origin/fix-#2062' into fix-#2062
39a9ff31 [袁福元] test
fc7198da [xiaoyuandajian] Update style.yml
62e80726 [袁福元] test
0abdef36 [xiaoyuandajian] Delete main.yml
1e252e2a [xiaoyuandajian] Create main.yml
c19728ec [xiaoyuandajian] Delete main.yml
5db61e5a [xiaoyuandajian] Delete main1.yml
9e27a363 [xiaoyuandajian] Create main1.yml
69540652 [xiaoyuandajian] Create main.yml

Lead-authored-by: 袁福元 <yuanfuyuan@mafengwo.com>
Co-authored-by: xiaoyuandajian <51512358+xiaoyuandajian@users.noreply.github.com>
Co-authored-by: Cheng Pan <pan3793@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-01-05 15:13:43 +08:00
liangbowen
00f52a2cb2
[KYUUBI #3828] [PySpark] Support Python style check with spotless in CI style workflow and reformat tool
### _Why are the changes needed?_

to close #3828.

 Python code style checking support.
1. reuse Spotless maven plugin for Python style check
2. add style check to CI style workflow
3. add python linting support to `dev/reformat`. checks whether `black` installed in PATH.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3823 from bowenliang123/spotless-python.

Closes #3828

4a4de885 [liangbowen] simplify empty tags
0bb9ec7c [liangbowen] simplify empty tag in pom
9dd39531 [liangbowen] lint python code with black via spotless
f85020fa [liangbowen] typo
4c93bce0 [liangbowen] install python 3.9 first
23fc4b96 [liangbowen] ci install black version from added `spotless.python.black.version` property
73f746b0 [Bowen Liang] Update dev/reformat
46667a00 [liangbowen] update style.yml
9c20b434 [liangbowen] update style.yml
21017e5e [liangbowen] update style.yml
8272c0bc [liangbowen] add python style to style checking for CI
e102726c [liangbowen] add profile spotless in dev/reformat if black found in path
062e9bf2 [liangbowen] add python scan for spotless. add new profile `spotless-python` for python file path.

Lead-authored-by: liangbowen <liangbowen@gf.com.cn>
Co-authored-by: Bowen Liang <bowen.liang.123@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-11-22 19:44:02 +08:00
Yikf
2b86d6dde0 [KYUUBI #3692] [INFRA] [FOLLOWUP] Upgrade Github actions version
### _Why are the changes needed?_

This pr aims to upgrade github actions version, They all include actions/core upgrades(Cleanup set-output warning), as follow:
1) Upgrade actions/checkout from v2 to v3, https://github.com/actions/checkout/releases/tag/v3.1.0
2) Upgrade docker/login-action from v1 to v2, https://github.com/docker/login-action/releases/tag/v2.1.0
3) Upgrade actions/upload-artifact from v2 to v3, https://github.com/actions/upload-artifact/releases/tag/v3.1.1
4) Upgrade actions/stale from v5 to v6, https://github.com/actions/stale/releases/tag/v6.0.1

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3700 from Yikf/upgrade-ga.

Closes #3692

68289e9e [Yikf] Upgrade GA

Authored-by: Yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-10-29 10:43:37 +00:00
Yikf
ffb9532d61
[KYUUBI #3692] [INFRA] Upgrade actions/setup-java to v3 with temurin distribution
### _Why are the changes needed?_

Close https://github.com/apache/incubator-kyuubi/issues/3692

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3694 from Yikf/upgrade-GA.

Closes #3692

3fd2a326 [Yikf] Upgrade actions/setup-java to v3

Authored-by: Yikf <yikaifei1@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-10-26 10:23:26 +08:00
Fu Chen
e8622e6bd7
Revert "[KYUUBI #3454] [CI] Check Kyuubi modules available only when **/pom.xml is changed"
This reverts commit 8b82268dc1.
2022-09-13 20:19:08 +08:00
Fu Chen
8b82268dc1
[KYUUBI #3454] [CI] Check Kyuubi modules available only when **/pom.xml is changed
### _Why are the changes needed?_

Check Kyuubi modules available only when `**/pom.xml` is changed

https://github.com/marketplace/actions/paths-changes-filter#conditional-execution

### _How was this patch tested?_

Pass CI.

Closes #3454 from cfmcgrady/style-ci-followup.

Closes #3454

f849fbd8 [Fu Chen] check on pom change

Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-09-08 16:01:42 +08:00
Fu Chen
6570d4aebb
[KYUUBI #3422] [CI] Run install before checking style
### _Why are the changes needed?_

To avoid unexpected errors on bumping version and adding new modules.

do some tests based on this PR.

- https://github.com/cfmcgrady/incubator-kyuubi/pull/22
- https://github.com/cfmcgrady/incubator-kyuubi/pull/23
- https://github.com/cfmcgrady/incubator-kyuubi/pull/24
- https://github.com/cfmcgrady/incubator-kyuubi/pull/25

### _How was this patch tested?_

Pass CI.

Closes #3422 from cfmcgrady/style-ci.

Closes #3422

f79e2d43 [Fu Chen] remove out-of-date comment
f2a87976 [Fu Chen] address comment
b7a4e487 [Fu Chen] add comment
df3c543e [Fu Chen] connector-hive
7962663b [Fu Chen] kudu
2b55eb1c [Fu Chen] spark-3.3
6120f3a2 [Fu Chen] install spark-3.3
0b13bc60 [Fu Chen] update
bb71fb24 [Fu Chen] condition
7b00c6cd [Fu Chen] continue-on-error
c978a946 [Fu Chen] install submodule
f20228e0 [Fu Chen] [CI] Run install before checking style

Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-09-07 20:03:04 +08:00
zwangsheng
29eb6914b6
[KYUUBI #3340] [Subtask][KPIP-5] Github CI check kyuubi web ui
### _Why are the changes needed?_

For code review and check pr, we need Github CI help check web ui.
Run `npm run coverage && npm run build-check` to find syntax issues and build errors.
Actions used node from https://github.com/marketplace/actions/setup-node-js-environment

Close #3335
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3340 from zwangsheng/feature/web_ci.

Closes #3340

037953b7 [zwangsheng] fix symble |
07f6dc09 [zwangsheng] use npm install pnpm
c643333e [zwangsheng] use pnpm instead
a57bc311 [zwangsheng] install
f2a94ca2 [zwangsheng] fix runner
1d85336d [zwangsheng] fix runner
7c5c58c2 [zwangsheng] fix runner
adb94e46 [zwangsheng] fix
33359d99 [zwangsheng] fix
95a34cbb [zwangsheng] fix
7d3207a9 [zwangsheng] add ci

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-09-02 14:28:10 +08:00
Ada Wang
6061098aec
[KYUUBI #2584][INFRA] Migrate CI to Ubuntu 22.04
### _Why are the changes needed?_

Close #2584

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2585 from deadwind4/bump-ubuntu.

Closes #2584

050f1b9a [Luning Wang] Bump actions-setup-minikub to 2.7.0
51a77ddb [Ada Wang] [KYUUBI #2584][INFRA] Migrate CI to Ubuntu 22.04

Lead-authored-by: Ada Wang <wang4luning@gmail.com>
Co-authored-by: Luning Wang <wang4luning@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-08-09 17:50:02 +08:00
guanhua.lgh
c1158accca
[KYUUBI #2974][FEATURE] EOL Support for Spark 3.0
### _Why are the changes needed?_
This PR is to drop support for spark 3.0 from compiling/building and test.  [#2974](https://github.com/apache/incubator-kyuubi/issues/2974)
Following the discussion on dev list https://lists.apache.org/thread/3mw8zz9r0tzg2dpj6717xy59610bhkp4

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3035 from huage1994/2974.

Closes #2974

add45cfe [guanhua.lgh] [FEATURE] EOL Support for Spark 3.0

Authored-by: guanhua.lgh <guanhua.lgh@alibaba-inc.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-07-12 11:01:50 +08:00
Fu Chen
6cd2ad9ef7
[KYUUBI #2780] Refine stylecheck
### _Why are the changes needed?_

1. reformat code should include all of the modules in the project.
2. stylecheck should include all of the modules in the project.
3. there is a bug in the stage `Upload style check report`.

Since the `grep` returns code 1 when the pattern is not found, the upload report does not work before this pr.

```
> echo "a" > /tmp/a; grep abc /tmp/a; echo $?
1
```

Before this PR
```
Run for log in `find * -name "scalastyle-output.xml"`;  do echo "=========$log========="; grep "error" $log; done
=========dev/kyuubi-tpcds/target/scalastyle-output.xml=========
Error: Process completed with exit code 1.
```

After this PR
```
Run for log in `find * -name "scalastyle-output.xml"`;
<?xml version="1.0" encoding="UTF-8"?>
<checkstyle version="5.0">
 <file name="/home/runner/work/incubator-kyuubi/incubator-kyuubi/extensions/spark/kyuubi-extension-spark-3-3/src/main/scala/org/apache/kyuubi/sql/KyuubiSparkSQLExtension.scala">
  <error column="0" line="21" source="org.scalastyle.scalariform.ImportOrderChecker" severity="error" message="There should at least one a single empty line separating groups 3rdParty and kyuubi."/>
 </file>
</checkstyle>
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2780 from cfmcgrady/refine-stylecheck.

Closes #2780

a9c4a71e [Fu Chen] update reformat
f5d4d4e4 [Fu Chen] address comment
0665e7e5 [Fu Chen] address
91d123ee [Fu Chen] fix
f5d6414f [Fu Chen] style.yml
c56fa081 [Fu Chen] dev/reformat

Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-06-01 16:49:01 +08:00
ulysses-you
ff19ae73c9
[KYUUBI #2329] [KYUUBI#2214][FOLLOWUP] Cleanup kubernetes-deployment-it
### _Why are the changes needed?_

Change `kubernetes-deployment-it` to `kubernetes-it`

### _How was this patch tested?_
Pass CI

Closes #2329 from ulysses-you/KYUUBI-2214-FOLLOWUP.

Closes #2329

eae09ddc [ulysses-you] cleanup

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-04-12 21:55:13 +08:00
Kent Yao
3eb1fa9e48
[KYUUBI #2175] Improve CI with cancel & concurrency & paths filter
### _Why are the changes needed?_

Better ASF GHA resource usage, see more https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status

- cancel-in-progress enabled. w/ this, a previous build can be canceled if new commit(s) have arrived
- style check w/o install. currently, the style check takes 8~10min to finish because an install step before it. now, it shall return within a minute. note that, this might 'fail' itself when we introduce a new module and others depend on it, but it's ok to verify by the log whether a PR ok to be merged.
- dependency check runs only when pom.xml updates. this flow now runs only pom changes as it is the only way that may cause dependency changes.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

**Canceled**: https://github.com/apache/incubator-kyuubi/runs/5604809742?check_suite_focus=true

<img width="548" alt="image" src="https://user-images.githubusercontent.com/8326978/159059722-480f4fd6-d87e-4b07-984e-eb5b08a84beb.png">

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2175 from yaooqinn/gha.

Closes #2175

33777bc5 [Kent Yao] Update .github/workflows/style.yml
ec41ddd0 [Kent Yao] Update .github/workflows/style.yml
79db981f [Kent Yao] fix
4e1fb387 [Kent Yao] fix
733df9cb [Kent Yao] Improve CI with cancel & concurrency
494c3a84 [Kent Yao] Revert "Improve CI with cancel & concurrency"
57641ac4 [Kent Yao] Improve CI with cancel & concurrency
6dafe970 [Kent Yao] Improve CI with cancel & concurrency
40160ab9 [Kent Yao] Improve CI with cancel & concurrency
377285c2 [Kent Yao] Improve CI with cancel & concurrency
dbe8999e [Kent Yao] Improve CI with cancel & concurrency
78a97260 [Kent Yao] Improve CI with cancel & concurrency
7e984ebd [Kent Yao] Improve CI with cancel & concurrency
d2538210 [Kent Yao] Improve CI with cancel & concurrency
51a7c48c [Kent Yao] Improve CI with cancel & concurrency
c5e3177a [Kent Yao] Improve CI with cancel & concurrency

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2022-03-19 21:06:08 +08:00
Kent Yao
5836db6004
[KYUUBI #1947] Revamp Github Action settings for CI
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

1. Add missing asf header
2. Disable fail-fast to let jobs run individually
3. Group defaults maven builds with java & spark version matrix
    - so this PR brings fully tests with JDK 11 back
4. Group others(now cross-version check only) in `include`
5. replace `actions/cachev2` with `actions/setup-javav2` - `cache: 'maven'`

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Need to pass all CI jobs

Closes #1947 from yaooqinn/ga.

Closes #1947

7d5d2472 [Kent Yao] address comment
d8d347b0 [Kent Yao] address comment
8c3cba99 [Kent Yao] address comment
7400cdc6 [Kent Yao] checkout
20eeb7bb [Kent Yao] test
b1c70e42 [Kent Yao] test
4ca4f233 [Kent Yao] test
b588cd32 [Kent Yao] test
da4f6aea [Kent Yao] test
057790d5 [Kent Yao] test
01b07b25 [Kent Yao] test
2906ec66 [Kent Yao] test
4d97bf19 [Kent Yao] test
cf16f169 [Kent Yao] test
8c12b8ef [Kent Yao] test
3b84e450 [Kent Yao] test
6e21d97a [Kent Yao] test
dc2d9004 [Kent Yao] test
1cff392d [Kent Yao] test
a4b47476 [Kent Yao] test
c4718022 [Kent Yao] test
d264d33d [Kent Yao] test
ee293c13 [Kent Yao] test
903cb0bb [Kent Yao] test
f74fa1a5 [Kent Yao] test

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2022-02-22 13:39:54 +08:00
ulysses-you
681153036d
[KYUUBI #1536] Add session conf advisor
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
closes https://github.com/apache/incubator-kyuubi/issues/1536

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1892 from ulysses-you/KYUUBI-1536.

Closes #1536

f0c450a9 [ulysses-you] codecov
35fac4f6 [ulysses-you] fix
f34cfbf4 [ulysses-you] comment
78a54bc1 [ulysses-you] address comment
5cb574ba [ulysses-you] fix test
1033909b [ulysses-you] fix
3c925e56 [ulysses-you] address comment
a74988cd [ulysses-you] style
649f02d0 [ulysses-you] Add session conf advisor
debd3adb [ulysses-you] Add session conf advisor

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-02-15 17:48:43 +08:00
Cheng Pan
b2bad04d75 [KYUUBI #1769][FOLLOWUP] Exclude log4j 12 deps from spark-hive and fix detect deps change workflow
### _Why are the changes needed?_

The previous PR did not handle log4j 1.2 deps of `spark-hive`, and the current detect deps change workflow has the dirty cache issue, this PR also fix it.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1856 from pan3793/log4j2.

Closes #1769

2785acf4 [Cheng Pan] nit
58affcb5 [Cheng Pan] update workflow
301acb6f [Cheng Pan] update check deps script
7611c7e0 [Cheng Pan] revert exclusion
239e7cc4 [Cheng Pan] Remove duplicated scalatest
e143fecc [Cheng Pan] dependencyList
a7254771 [Cheng Pan] [KYUUBI #1769][FOLLOWUP] Exclude log4j 12 deps from test deps

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-02-01 23:59:17 +08:00