Commit Graph

39 Commits

Author SHA1 Message Date
Bowen Liang
d3520ddbce [KYUUBI #6769] [RELEASE] Bump 1.11.0-SNAPSHOT
# 🔍 Description
## Issue References 🔗

This pull request fixes #

## Describe Your Solution 🔧

Preparing v1.11.0-SNAPSHOT after branch-1.10 cut

```shell
build/mvn versions:set -DgenerateBackupPoms=false -DnewVersion="1.11.0-SNAPSHOT"
(cd kyuubi-server/web-ui && npm version "1.11.0-SNAPSHOT")
```

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6769 from bowenliang123/bump-1.11.

Closes #6769

6db219d28 [Bowen Liang] get latest_branch by sorting version in branch name
465276204 [Bowen Liang] update package.json
81f2865e5 [Bowen Liang] bump

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
2024-10-23 17:10:56 +08:00
Cheng Pan
4ee9481674
[KYUUBI #6391] Bump Arrow from 15.0.2 to 16.0.0
# 🔍 Description

A regular dependency upgrading, additionally, Arrow 15 introduced the eclipse-collections dependencies but removed in Arrow 16.

https://github.com/apache/arrow/issues/40896

Note: This PR upgrades Arrow to 16.0.0 instead of 16.1.0 due to https://github.com/apache/arrow/issues/41717

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Pass GHA

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6391 from pan3793/arrow-16.

Closes #6391

d8ea702a3 [Cheng Pan] 16.0.0
8a8bc46ab [Cheng Pan] Bump Arrow from 15.0.2 to 16.1.0

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-05-20 19:26:15 +08:00
dupeng
3133b5911c
[KYUUBI #6293] Upgrade Arrow from 12.0.0 to 15.0.2
# 🔍 Description
## Issue References 🔗

This pull request fixes #6293

## Describe Your Solution 🔧
Here are my main modifications:

1. In the project POM file, I have updated the Apache Arrow version from 12.0.0 to 15.0.2.
2. After executing `build/dependency.sh --replace`, the `dev/dependencyList` file was automatically updated.
3. I noticed the addition of the "Eclipse Collections" dependency and checked its open-source license. Following the existing format, I have accordingly modified the `NOTICE-binary` and `LICENSE-binary` files.

However, I remain uncertain whether my changes to the LICENSE/NOTICE sections adhere to the community's standards. I kindly request guidance from the community regarding this matter.

## Types of changes 🔖

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6304 from dupen01/issue-arrow.

Closes #6293

e858a1c12 [dupeng] Merge remote-tracking branch 'origin/issue-arrow' into issue-arrow
205d905ef [Perl Du] Merge branch 'apache:master' into issue-arrow
238ef7ac6 [dupeng] Merge branch 'master' into issue-arrow
14a53f293 [dupeng] update kyuubi-hive-jdbc-shaded pom
62968d2e2 [dupeng] Update the license information for Eclipse Collections.
323136ec4 [dupeng] Upgrade Arrow to 15.0.2 and update LICENSE/NOTICE

Lead-authored-by: dupeng <dunett@163.com>
Co-authored-by: Perl Du <34719039+dupen01@users.noreply.github.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-04-15 16:20:36 +08:00
Binjie Yang
eb278c562d
[RELEASE] Bump 1.10.0-SNAPSHOT 2024-03-13 14:24:49 +08:00
Cheng Pan
4855ffeb7a
[KYUUBI #5953] [LICENSE] Update NOTICE
# 🔍 Description
## Issue References 🔗

This pull request fixes the NOTICE issue.

## Describe Your Solution 🔧

According to https://infra.apache.org/licensing-howto.html#alv2-dep, we should analyze and bubble up the relevant portions into the top-level NOTICE file, no matter whether they're ASF projects or not.

> If the dependency supplies a NOTICE file, its contents must be analyzed and the relevant portions bubbled up into the top-level NOTICE file.

Kyuubi has some code copied/derived from Apache Iceberg/Hive/Spark/Parquet MR, so we should update NOTICE to respect that.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

Non-code change, review is required.

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #5953 from pan3793/notice.

Closes #5953

51b4e114d [Cheng Pan] update notice
9850d63a6 [Cheng Pan] Update NOTICE

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2024-01-10 19:29:01 +08:00
Cheng Pan
9be0c65fe9
[KYUUBI #5783] Switch to kyuubi-relocated-hive-service-rpc
# 🔍 Description
## Issue References 🔗

TL;DR there are some issues with shading Thrift RPC classes during the engine packaging phase, see details in the PR description of https://github.com/apache/kyuubi-shaded/pull/20.

## Describe Your Solution 🔧

This PR aims to migrate from vanilla `hive-service-rpc`, `libfb303`, `libthrift` to `kyuubi-relocated-hive-service-rpc` introduced in https://github.com/apache/kyuubi-shaded/pull/20, the detailed works are:

- replace imported deps in `pom.xml` and rename the package prefix in all modules, except for
  - `kyuubi-server` there are a few places use vanilla thrift classes to access HMS to get token
  - `kyuubi-hive-sql-engine` Hive method invocation
- update relocations rules in modules that creates shaded jar
- introduce `HiveRpcUtils` in `kyuubi-hive-sql-engine` module for object conversion.

As part of the whole change, this PR upgrades from the Kyuubi Shaded 0.1.0 to 0.2.0, which changes the jars name. see https://kyuubi.apache.org/shaded-release/0.2.0.html

## Types of changes 🔖

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

Pass all Hive UT with Hive 3.1.3, and IT with Hive 3.1.3 and 2.3.9 (also tested with 2.1.1-cdh6.3.2)

---

# Checklists
## 📝 Author Self Checklist

- [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project
- [x] I have performed a self-review
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

## 📝 Committer Pre-Merge Checklist

- [x] Pull request title is okay.
- [x] No license issues.
- [x] Milestone correctly set?
- [x] Test coverage is ok
- [x] Assignees are selected.
- [x] Minimum number of approvals
- [x] No changes are requested

**Be nice. Be informative.**

Closes #5783 from pan3793/rpc-shaded.

Closes #5783

b45d4deaa [Cheng Pan] remove staging repo
890076a20 [Cheng Pan] Kyuubi Shaded 0.2.0 RC0
071945d45 [Cheng Pan] Rebase
199794ed9 [Cheng Pan] fix
fc128b170 [Cheng Pan] fix
26d313896 [Cheng Pan] fix
632984c92 [Cheng Pan] fix
428305589 [Cheng Pan] fix
6301e28fd [Cheng Pan] fix
955cdb33b [Cheng Pan] Switch to kyuubi-shaded-hive-service-rpc

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-12-07 19:55:10 +08:00
Cheng Pan
6061a05f24
Bump 1.9.0-SNAPSHOT 2023-09-04 14:23:12 +08:00
Cheng Pan
208ab3af62
[KYUUBI #4852] Switch to Kyuubi Shaded Zookeeper
### _Why are the changes needed?_

This PR aims to migrate the vanilla Zookeeper and Curator to the Kyuubi Shaded Zookeeper. It's the first step to adapting JDK 17.

There is a known issue [ZOOKEEPER-3779](https://issues.apache.org/jira/browse/ZOOKEEPER-3779) that Zookeeper 3.4 client can not run on JDK 14 and above, in https://github.com/apache/kyuubi-shaded/pull/5, we fixed this issue by a surgical.

With the above fixing, zk-3.4 and zk-3.6 clients both work well on JDK 17, we just randomly pick some cases to make sure zk-3.6 is tested

zk-3.4 client supports zk-3.4+ server, but zk-3.6 client only supports zk-3.5+ server; in the meanwhile, zk-3.4 is adopted widely, (CDH 5/6, HDP, EMR created before 2023).

We are sticky to zk-3.4 to ensure that Kyuubi can be out-of-box in the most existing Hadoop cluster but also provide zk-3.6 as an alternative(simply replace the kyuubi-shaded-zk-3.4 jar w/ kyuubi-shaded-zk-3.6, or build w/ -Pzookeeper-3.6) for users who concerns that zk-3.4 is EOL.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4852 from pan3793/shaded-zk.

Closes #4852

d960cc945 [Cheng Pan] remove staging repo
1b3622080 [Cheng Pan] Switch to Kyuubi Shaded Zookeeper

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2023-05-21 20:49:00 +08:00
Cheng Pan
4e226ac3cc
Bump 1.8.0-SNAPSHOT 2023-02-10 15:25:49 +08:00
Tianlin Liao
8e8d8be8d2
[KYUUBI #4020] remove incubating from kyuubi source code
### _Why are the changes needed?_

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4069 from lightning-L/kyuubi-4020.

Closes #4020

97406ca0 [Tianlin Liao] [KYUUBI #4020] remove incubating from kyuubi source code

Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2023-01-04 09:43:20 +08:00
Fu Chen
a97b843303
[KYUUBI #3803] Improve the data interchange performance via Arrow serialization/deserialization
### _Why are the changes needed?_

to close #3794

This PR introduces the Apache Arrow library to improve the performance of data interchange.

data flow diagram with saprk engine

```
1. Spark Executor -> 2. Spark Driver -> 3. Kyuubi Server -> 4. Beeline/KyuubiHiveDriver
```

1. for simple types(Numeric/String/Binary/Boolean/Datetime/Interval), we serialize the columns on the Spark executor side and deserialize them on the client side.

2. for complex types(Array/Map/Struct), we first convert these columns to hive string and serialize the string columns on the executor side, and finally deserialize them as string type on the client side.

## Local Test

### Input dataset

```scala
spark.sql(
  """
    |select * from tpcds.sf10.catalog_sales limit 2000000
    |""".stripMargin)
  .write
  .mode("overwrite")
  .save("/tmp/parquet/tpcds/sf10/catalog_sales_2000000")
```

### Query

cat /tmp/b.sql

```sql
select * from parquet.`/tmp/parquet/tpcds/sf10/catalog_sales_2000000`;
```

### Beeline command
```
bin/beeline -u 'jdbc:hive2://0.0.0.0:10009/;' --hiveconf kyuubi.session.result.codec=simple/arrow --hiveconf kyuubi.operation.incremental.collect=false/true -f /tmp/b.sql > /dev/null
```

### Spark

```
/Users/fchen/Software/spark-3.3.0-bin-hadoop3/bin/spark-submit \
    --class org.apache.kyuubi.engine.spark.SparkSQLEngine \
    --driver-memory 6g \
    --master local[*] \
    ......
```

### Result

| kyuubi.session.result.codec | kyuubi.operation.incremental.collect | time spent|
| --- | --- | --- |
| arrow | false | 272.178 seconds |
| arrow | true | 146.76 seconds |
| simple | true | 607.523 seconds |
| simple | false | 429.374 seconds |

Closes #3803 from cfmcgrady/arrow.

Closes #3803

05b2b6f5 [Fu Chen] address comment
37e54e93 [Fu Chen] rename `kyuubi.session.result.codec` to `kyuubi.operation.result.codec`
fa44b102 [Fu Chen] address comment
e66b3bbd [Fu Chen] rebase master
08e2a97b [Fu Chen] fix shade
ab796bc6 [Fu Chen] address comment
1c979318 [Fu Chen] only consider spark engine major/minor version
dd61c015 [Fu Chen] add missing relocate dep `netty`
02f1d2f7 [Fu Chen] relocate arrow/jackson deps
4a2f7f09 [Fu Chen] minor refactor SparkDataTypeTests with SemanticVersion
627cc0a2 [Fu Chen] update kyuubi-jdbc-high-shaded NOTICE
815c68c2 [Fu Chen] reduce kyuubi-hive-jdbc/pom.xml
a7e2fcd9 [Fu Chen] fix style
825fda67 [Fu Chen] skip some tests when spark version < 3.3
039dbba1 [Fu Chen] minor refactor
95ad84c4 [Fu Chen] address comment
e1e81f01 [Fu Chen] revert unnecessary changes
36bce1d9 [Fu Chen] clean up
c789327c [Fu Chen] make sure SparkArrowbasedOperationSuite run with arrow enabled
d60c3120 [Fu Chen] clean up array-related implementations
8ca1fcd6 [Fu Chen] clean up unused arrow array implementations
8f96fdf3 [Fu Chen] fix
cdb83cf0 [Fu Chen] rename conf "kyuubi.operation.result.codec" to "kyuubi.session.result.codec"
f30c2c9d [Fu Chen] rename conf "kyuubi.beeline.arrow.enabled" to "kyuubi.operation.result.codec"
109a4f32 [Fu Chen] Fork Apache Spark-3.3.1 org.apache.spark.sql.catalyst.util.quoteIfNeeded to adapt to Spark-3.1.x
9ec23de3 [Fu Chen] revert kyuubi-server/src/main/scala/org/apache/kyuubi/server/BackendServiceMetric.scala
f2b7f2c3 [Fu Chen] revert kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiCallableStatement.java
33e820c7 [Fu Chen] revert temporary `pom.xml` changes
daa259e5 [Fu Chen] add SparkArrowbasedOperationSuite
eb5ed344 [Fu Chen] make kyuubi.beeline.arrow.enabled default to false and enabled via url support
5c3a48c7 [Fu Chen] fix ut SparkOperationSuite: execute statement - select column name with dots
0dfd442b [Fu Chen] complex type(array/map/array) support
86b0a485 [Fu Chen] INTERVAL_YEAR_MONTH_TYPE type support
10da4fe9 [Fu Chen] INTERVAL_DAY_TIME_TYPE type support
3a8b0da3 [Fu Chen] implement KyuubiArrowBasedResultSet.getDate
a9fd48c1 [Fu Chen] implement KyuubiArrowBasedResultSet.getInt
a37b5564 [Fu Chen] only ExecuteStatement support arrow
fe47fdc4 [Fu Chen] add java version ArrowUtils
d162a594 [Fu Chen] lisence
5414ccc7 [Fu Chen] remove debug println
335b4e8f [Fu Chen] drop scala dependency for module kyuubi-hive-jdbc
f826e262 [Fu Chen] fix ut
bcf9744d [Fu Chen] license
3ebca27c [Fu Chen] fix style
540423d5 [Fu Chen] timestamp support
0965b8f4 [Fu Chen] revert extensions/spark/kyuubi-spark-connector-tpcds/src/test/scala/org/apache/kyuubi/spark/connector/tpcds/TPCDSQuerySuite.scala
ad1a7092 [Fu Chen] revert
5b4ee403 [Fu Chen] BackendServiceMetric
21b76d29 [Fu Chen] add more types support
ace05a24 [Fu Chen] arrow-based collect

Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-11-28 16:07:37 +08:00
liangbowen
2ac10f91d5
[KYUUBI #3842] [Improvement] Support maven pom.xml code style check with spotless plugin
### _Why are the changes needed?_

Introduce code style check support for Maven's pom.xml with sortPom in spotless maven plugin.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3843 from bowenliang123/spotless-pom.

Closes #3842

3c654597 [liangbowen] apply to pom.xml
fd1536f7 [liangbowen] set expandEmptyElements to true
e498423f [liangbowen] apply spotless:apply to all pom.xml
e46bcfec [liangbowen] add pom style check support in spotless

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-11-23 22:08:00 +08:00
SteNicholas
77b036f3a8
[KYUUBI #3264] [RELEASE] Bump 1.7.0-SNAPSHOT
### _Why are the changes needed?_

Preparing v1.7.0-SNAPSHOT with branch-1.6 cut

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3264 from SteNicholas/prepare-1.7.0-snapshot.

Closes #3264

374d56bf [SteNicholas] preparing v1.7.0-SNAPSHOT with branch-1.6 cut

Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-08-18 11:23:54 +08:00
Cheng Pan
8778209797
[KYUUBI #3023] Kyuubi Hive JDBC: Replace UGI-based Kerberos authentication w/ JAAS
### _Why are the changes needed?_

The current JDBC Kerberos authentication and Hadoop UGI classes are strongly coupled, it's not friendly for downstream projects which do not have Hadoop dependencies, and users who are not familiar w/ Hadoop security mechanism.

This PR proposes to replace the UGI-based Kerberos authentication with JAAS-based.

The main logic is

```
if (param.auth == 'noSasl') {
    // no-sasl
    raw socket/http requrest
} else if (!param.principal) {
    // sasl/plain
    auth by param.username, param.password
} else if (param.principal && param.kerberosAuthType == 'fromSubject') {
    [0]
} else if (param.principal && param.kyuubiClientPrincipal && param.kyuubiClientKeytab) {
    [1]
} else if (param.principal && !param.kyuubiClientPrincipal && !param.kyuubiClientKeytab) {
    [3]
}

- [0][JDK] subject from context
- [1][JDK] principle + keytab
- [2][UGI] principle + keytab (no required, use [1] instead)
- [3][JDK] principle + tgt cache
- [4][UGI] principle + tgt cache (no required, use [3] instead)
```

To achieve this, we need to introduce new JDBC parameters `kyuubiClientPrincipal` and `kyuubiClientKeytab`.

The above description is pure JDK-based and totally removed Hadoop dependencies from Kyuubi Hive JDBC driver.

To minimize the scope of this PR, I separate the support of delegation token to a follow-up PR #3096, and it still needs UGI.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3023 from pan3793/ugi.

Closes #3023

7b45f471 [Cheng Pan] Remove UGI in JDBC driver and use JAAS-based Kerberos authentication instead
3876c3d6 [Cheng Pan] Add util method isCommandAvailable

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-07-27 04:01:46 +08:00
Cheng Pan
3e5a92ef37
[KYUUBI #3004] Clean up JDBC shaded client pom and license
### _Why are the changes needed?_

Clean up JDBC shaded client pom and license because we removed some dependencies.

This PR also disables `minimizeJar` in JDBC shaded client, because it may cause unexpected "class not found" issues.

The current JDBC shaded client jar size is 11MB.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3004 from pan3793/cleanup.

Closes #3004

1887f353 [Cheng Pan] Clean up JDBC shaded client pom and license

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-07-05 09:49:12 +08:00
jiaoqingbo
4b8dc7962d
[KYUUBI #2996] Remove Hive storage-api dependencies from Kyuubi Hive JDBC
### _Why are the changes needed?_

fix #2996

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3003 from jiaoqingbo/kyuubi-2996.

Closes #2996

bc919312 [jiaoqingbo] [KYUUBI #2996] Remove Hive storage-api dependencies from Kyuubi Hive JDBC

Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-07-04 18:46:51 +08:00
jiaoqingbo
774934fe57
[KYUUBI #2994] Remove Hive common dependencies from Kyuubi Hive JDBC
### _Why are the changes needed?_

fix #2994

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2995 from jiaoqingbo/kyuubi-2994.

Closes #2994

c9c695b1 [jiaoqingbo] [KYUUBI #2994] Remove Hive common dependencies from Kyuubi Hive JDBC

Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-07-04 16:51:03 +08:00
Cheng Pan
5eb83b4c7c
[KYUUBI #2999] Fix Kyuubi Hive Beeline dependencies
### _Why are the changes needed?_

Before https://github.com/apache/incubator-kyuubi/pull/1715, `kyuubi-hive-beeline` and `kyuubi-hive-jdbc` managed dependencies in-self, after https://github.com/apache/incubator-kyuubi/pull/1715 and https://github.com/apache/incubator-kyuubi/issues/1716, `kyuubi-hive-beeline` uses some transitive dependencies of `kyuubi-hive-jdbc`, the recent dependencies change of `kyuubi-hive-jdbc` breaks `kyuubi-hive-beeline`.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2999 from pan3793/beeline.

Closes #2999

b92b8a17 [Cheng Pan] Fix Kyuubi Hive Beeline dependencies

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-07-04 14:19:39 +08:00
jiaoqingbo
d8d6903f5d
[KYUUBI #2987] Remove Hive shims-common and shims-0.23 dependencies from Kyuubi Hive JDBC
### _Why are the changes needed?_

fix #2987

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2992 from jiaoqingbo/kyuubi-2987.

Closes #2987

97e25292 [jiaoqingbo] [KYUUBI #2987] Remove Hive shims-common and shims-0.23 dependencies from Kyuubi Hive JDBC

Authored-by: jiaoqingbo <1178404354@qq.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-07-02 22:34:09 +08:00
Cheng Pan
56c016166f
[KYUUBI #2989] Remove HS2 active-passive support in Kyuubi Hive JDBC client
### _Why are the changes needed?_

HS2 introduced active-passive in 3.0 via HIVE-18281, Kyuubi does not support that.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2989 from pan3793/2959-followup.

Closes #2989

c8ed51dd [Cheng Pan] more cleanup
d9e2e009 [Cheng Pan] Remove Kyuubi Hive JDBC client active-passive mode support

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-07-01 21:20:13 +08:00
Tianlin Liao
b817fcf76a [KYUUBI #2643] [SUB-TASK][KPIP-4] Implement kyuubi rest sdk for batch job operation
### _Why are the changes needed?_

To close #2643

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2723 from lightning-L/kyuubi-2643.

Closes #2643

b7c409bcf [Tianlin Liao] unify junit dependency
ae2caca33 [Tianlin Liao] implements AutoClosable for KyuubiRestClient; rename server in rest builder to spnego builder
19e078a32 [Tianlin Liao] reuse ObjectMapper
ef98f4c2a [Tianlin Liao] minor improvements
15d958acf [Tianlin Liao] unify httpclient version parameter
37032a6a9 [Tianlin Liao] fix dependency
6c97745aa [Tianlin Liao] support NONE and NOSASL basic auth type
aa0a1c011 [Tianlin Liao] add optional tag for hadoop-client-api/hadoop-client-runtime
275d1d042 [Tianlin Liao] add integration test in kyuubi-server
b407d0229 [Tianlin Liao] apiBasePath is not necessary for the client builder
db11f10eb [Tianlin Liao] add test case for http response code is not 200
361343fe0 [Tianlin Liao] add builder arguments validation
a2684de86 [Tianlin Liao] remove gson dependency; fix get log api typo
ed3498938 [Tianlin Liao] Define KyuubiRestException and remove kyuubi-common dependency
b46f0c4d3 [Tianlin Liao] add AuthScheme config in Builder
61796d782 [Tianlin Liao] fix sun.security does not exist/not visible error
de9bb26e2 [Tianlin Liao] [KYUUBI #2643] [SUB-TASK][KPIP-4] Implement kyuubi rest sdk for batch job operation

Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-05-29 08:40:33 +08:00
gabrywu
a6e14ac38d [KYUUBI #2560] Upgrade kyuubi-hive-jdbc hive version to 3.1.3
### Why are the changes needed?

upgrade hive to 3.1.3 in the kyuubi-hive-jdbc module

### change details

file|comment
--- | ---
kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/HttpBasicAuthInterceptor.java | add customCookies as a construct parameter
kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/HttpKerberosRequestInterceptor.java | replace assumeSubject field with `Subject loggedInSubject` ; add customCookies as a construct parameter
kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/HttpRequestInterceptorBase.java | add customCookies as a construct parameter; add customCookies initialization codes
kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/HttpTokenAuthInterceptor.java | add customCookies as a construct parameter
kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/JdbcColumn.java | support TIMESTAMP_WITH_TIMEZONE & UNION_TYPE type
kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiBaseResultSet.java | support TIMESTAMPLOCALTZ_TYPE
kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiConnection.java | add loggedInSubject,wmPool & wmApp fields and their initialization codes; add getAllUrls function; use Utils.getCanonicalHostName to extract host from connParams; clear hive conf in embedded mode; use SessionUtils.getTokenStrForm to get token string; remove isZkDynamicDiscoveryMode
kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiDatabaseMetaData.java | support getSQLKeywords; add Object class type parameter to KyuubiMetaDataResultSet
kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiMetaDataResultSet.java | optimize initialization logic of data & columnTypes field; add normalizeColumnNames function
kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/KyuubiStatement.java | split closeClientOperation into 2 functions(closeStatementIfNeeded & closeClientOperation) & add reInitState function to replace initFlags & ad error messages to exceptions in CANCELED_STATE
kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/Utils.java | add dummyAuthorityString, SERVICE_DISCOVERY_MODE_ZOOKEEPER_HA, ZOOKEEPER_ACTIVE_PASSIVE_HA_DEFAULT_NAMESPACE, WM_POOL, HTTP_COOKIE_PREFIX, APPLICATION fields; replace authorityList field with suppliedURLAuthority in JdbcConnectionParams;add a construct function to JdbcConnectionParams; replace getAuthorityList function with getSuppliedURLAuthority; split parseURL into 2 different functions(parseURL, extractURLComponents); add configureConnParamsFromZooKeeper, handleAllDeprecations, getAuthorityFromJdbcURL functions; remove configureConnParams;remove joinStringArray;add getCanonicalHostName function
kyuubi-hive-jdbc/src/main/java/org/apache/kyuubi/jdbc/hive/ZooKeeperHiveClientHelper.java | add more functions
kyuubi-server/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier | change package name of DelegationTokenIdentifier
kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HiveDelegationTokenProvider.scala | change package name of DelegationTokenIdentifier
kyuubi-server/src/test/scala/org/apache/kyuubi/credentials/HiveDelegationTokenProviderSuite.scala | add configs, METASTORE_CONNECTION_POOLING_TYPE = none,METASTORE_AUTO_CREATE_ALL = true, METASTORE_SCHEMA_VERIFICATION = false; add construct parameter clientConf to HadoopThriftAuthBridgeWithServerContextClassLoader
kyuubi-common/src/test/scala/org/apache/kyuubi/operation/SparkMetadataTests.scala | remove getSQLKeywords from unsupported list
externals/kyuubi-hive-sql-engine/pom.xml | add HikariCP to test scope
pom.xml | update hive version to 3.1.3
kyuubi-hive-jdbc-shaded/pom.xml | add hive-standalone-metastore dependency; add hive-llap-client dependency
kyuubi-hive-jdbc/pom.xml | add hive-standalone-metastore dependency;add hive-llap-client dependency
kyuubi-server/pom.xml | add hive-standalone-metastore dependency
dev/dependencyList | add hive-standalone-metastore
kyuubi-hive-jdbc-shaded/src/main/resources/META-INF/LICENSE | update content because of hive-standalone-metastore & hive-llap-client
kyuubi-hive-jdbc-shaded/src/main/resources/META-INF/NOTICE | update content because of hive-standalone-metastore & hive-llap-client

### How was this patch tested?

use existing UTs to test this patch

Closes #2560 from gabrywu/jdbc-3.1.

Closes #2560

9c5e3eda [gabry.wu] Merge branch 'master' into jdbc-3.1
c23cefc5 [gabrywu] update NOTICE rollback unnecessary change
b9394144 [gabrywu] change hikaricp.version position to satisfy alphabetical order
59ebed37 [gabrywu] add hive-storage-api explicitly
d22dc585 [gabrywu] rollback to minimize diff of this PR
33c13f0f [gabrywu] update LICENSE, NOTICE, LICENSE add hive-llap-client to kyuubi-hive-jdbc-shaded module
d1ce2c6c [gabrywu] manage hikaricp.version under dependencyManagement
1f4b7c72 [gabrywu] Merge remote-tracking branch 'origin/jdbc-3.1' into jdbc-3.1
fd309054 [gabrywu] manage hikaricp.version under dependencyManagement
39aa7ae6 [gabrywu] manage hikaricp.version under dependencyManagement
85dc983a [gabrywu] manage hikaricp.version under dependencyManagement
545578c5 [gabrywu] update dependencyList file
f479f642 [gabrywu] remove optional flag of hive-standalone-metastore in kyuubi-server
5ee8f89b [gabrywu] 1. add hive-storage-api 2. enable METASTORE_AUTO_CREATE_ALL 3. disable METASTORE_SCHEMA_VERIFICATION
f70b0354 [gabrywu] add HikariCP to test scope
c93ab167 [gabrywu] set METASTORE_CONNECTION_POOLING_TYPE to none
feb53d6e [gabrywu] support getSQLKeywords func
4601b9b9 [gabrywu] exclude package.jdo
4f4a19bf [gabrywu] upgrade hive version to 3.3.1 in dependencyList
0b88a77a [gabrywu] upgrade hive to 3.1.3

Lead-authored-by: gabrywu <gabrywu@apache.org>
Co-authored-by: gabry.wu <gabrywu@apache.org>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-05-09 22:05:22 +08:00
yangrong688
3ab2c81dce
[KYUUBI #2032][Subtask] Hive Backend Engine - new APIs with hive-service-rpc 3.1.2 - SetClientInfo
### _Why are the changes needed?_

Hive Backend Engine - new APIs with hive-service-rpc 3.1.2 - SetClientInfo

Implement SetClientInfo based on [HIVE-18240](https://issues.apache.org/jira/browse/HIVE-18240?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&focusedCommentId=16959838#comment-16959838).

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2384 from yangrong688/2032.

Closes #2032

ce63ce07 [yangrong688] fix
2264986b [yangrong688] rebase from apache/master

Authored-by: yangrong688 <yangrong.jxufe@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-04-29 18:34:30 +08:00
Cheng Pan
33424a1b47
[KYUUBI #2115] Update license and enhance collect_licenses script
### _Why are the changes needed?_

This PR updates the LICENSE and NOTICE files for both source release and binary release, and enhances the `collect_licenses.sh` script to support 1)collect `NOTICE*`(previous only `NOTICE`) files, 2) remove redundant or duplicated text from final NOTICE

This PR also fix the LICENSE and NOTICE files for shaded modules

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2115 from pan3793/license.

Closes #2115

e515eaaf [Cheng Pan] nit
53052ee6 [Cheng Pan] License for static resource
b14837fe [Cheng Pan] Remove CCO
2fe01d58 [Cheng Pan] nit
cb617e15 [Cheng Pan] log conf
7aaf2dbf [Cheng Pan] Update LICENSE
412a04f4 [Cheng Pan] Fix kyuubi-version-info.properties
0ed01fa7 [Cheng Pan] recover MANIFEST.MF
f0a5d192 [Cheng Pan] Update LICENSE for shade modules
407b9ddb [Cheng Pan] python3
830c7a92 [Cheng Pan] Revert "Setup python3"
239890da [Cheng Pan] Revert "setup-python@v3"
b0b27664 [Cheng Pan] setup-python@v3
3487cd45 [Cheng Pan] Setup python3
82ae8f01 [Cheng Pan] Revert "Update Dockerfile install python3"
4df10c7c [Cheng Pan] Update Dockerfile install python3
bbd87a43 [Cheng Pan] niy
152d2d03 [Cheng Pan] nit
85e89dcf [Cheng Pan] update license
df891a63 [Cheng Pan] Revert "Clean LICENSE"
09dcba9f [Cheng Pan] enhance collect_license.sh
acb624c9 [Cheng Pan] enhance collect-licenses.sh
7e0d5657 [Cheng Pan] Clean LICENSE
8eb5e356 [Cheng Pan] [LICENSE] Update LICENSE-binary

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-03-14 19:45:40 +08:00
Cheng Pan
f8efcb7125
[KYUUBI #2103] Revert "[KYUUBI #1948] Upgrade thrift version to 0.16.0"
### _Why are the changes needed?_

This PR reverts "[KYUUBI #1948] Upgrade thrift version to 0.16.0" in branch-1.5 and some subsequent commits due to issues caused by breaking changes in Thrift, e.g. #2101, #2093

We need more time to evaluate it and add more test cases to ensure stability before the next release.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2103 from pan3793/revert.

Closes #2103

Closes #1948

575dfe8d [Cheng Pan] properties order
4c94481e [Cheng Pan] fix deps
bfbdeb89 [Cheng Pan] Add log4j deps for kyuubi-hive-jdbc test
bad4e90e [Cheng Pan] surefire
f2ae8b9b [Cheng Pan] Revert "[KYUUBI #1948] Upgrade thrift version to 0.16.0"
4501f743 [Cheng Pan] Revert "[KYUUBI #1948][FOLLOWUP] Remove TFramedTransport in extension-spark-3-1 and extension-spark-3-2"
cc04df49 [Cheng Pan] Revert "[KYUUBI #1996] Clean up log4j2 conf and enhance surefire plugin conf"
52a20e17 [Cheng Pan] Revert "[KYUUBI #1948][FOLLOWUP] Relocate fb303 classes"

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-03-12 21:05:39 +08:00
Kent Yao
abf2026816
[KYUUBI #2045] Preparing v1.6.0-SNAPSHOT
### _Why are the changes needed?_

Preparing v1.6.0-SNAPSHOT with branch-1.5 cut

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #2045 from yaooqinn/pv.

Closes #2045

6a506a8f [Kent Yao] Preparing v1.6.0-SNAPSHOT

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2022-03-06 22:21:45 +08:00
Kent Yao
d6bbe32a01
[KYUUBI #1981] Remove unused artifacts for common and parent module
### _Why are the changes needed?_

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1981 from yaooqinn/branch-dep.

Closes #1981

07bf0e29 [Kent Yao] codec
96e41e7d [Kent Yao] codec
1da26432 [Kent Yao] guava
6cf0abb3 [Kent Yao] base64 & sha1
3b45ec2a [Kent Yao] base64
5490cfa7 [Kent Yao] http
6f4b48b8 [Kent Yao] n
2a87a4cf [Kent Yao] d
a5ad3ccc [Kent Yao] Refine Deps in common module
24d5299e [Kent Yao] Refine Deps in common module

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2022-03-01 13:35:49 +08:00
SteNicholas
770499ca54
[KYUUBI #1948] Upgrade thrift version to 0.16.0
### _Why are the changes needed?_

Upgrade libthrift to 0.16.0 due to [CVE-2020-13949](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-13949) and the coming upstream change of Spark https://github.com/apache/spark/pull/34362

### _What changed in this PR?_

- Upgrade libthrift to 0.16.0

- Shade and relocate `thrift` and `hive-service-rpc` classes in `kyuubi-spark-engine`, it's necessary to avoide conflicting with old thrift libs bundled in Spark binary releases.

- Due to thrift change the method signature, the subclasses those interfaces in Kyuubi also need to modify to pass compile.
We rely on Hive 2.3.9 jars in some components, e.g. `kyuubi-hive-jdbc`, `LocalMetaServer` in `kyuubi-server` test classes.

Some classes in Hive jars compiled against old thrift interfaces which are not compatible with thrift 0.16.0, it causes runtime link error, we found the following classes which breaks the test and copied them with necessary modification to make it work with thrift 0.16.0.

    - `TFramedTransport`
    - `TFilterTransport`
    - `TUGIAssumingTransport`
    - `TUGIContainingTransport`

- Next Steps, I think it's worth to do in separated PRs.

    - Recover the `HiveDelegationTokenProviderSuite`, one approach is use an isolate classloader to load HMS classes and thrift 0.9.3 classes from Maven, this approach can also be used for the planed Zoopkeeper upgrading to help us verficating the compatibility of Zookeeper Server 3.4.x.
    - Rewrite `kyuubi-hive-jdbc` to make it decouple with Hive jars, because there maybe other places which may not work with thrift 0.16.0 but the UTs does not cover.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1953 from SteNicholas/KYUUBI-1948.

Closes #1948

de5d1ea2 [SteNicholas] [KYUUBI #1948] Upgrade thrift version to 0.16.0
898effcd [SteNicholas] [KYUUBI #1948] Upgrade thrift version to 0.16.0
803e270c [SteNicholas] [KYUUBI #1948] Upgrade thrift version to 0.16.0

Authored-by: SteNicholas <programgeek@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
2022-02-23 20:51:59 +08:00
Kent Yao
f25e5c93b8
[KYUUBI #1949] [INFA] Add homepage for every module in maven central
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Take spark as an example.

![image](https://user-images.githubusercontent.com/8326978/154938094-87e53b38-5264-4aca-a7e2-269f9bb3c2d1.png)

[link](https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.13/3.2.1)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1949 from yaooqinn/url.

Closes #1949

f88c0598 [Kent Yao] [INFA] Add homepage for every module in maven cental
684f588f [Kent Yao] [INFA] Add homepage for every module in maven cental

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2022-02-21 21:09:29 +08:00
Fei Wang
ec719b82de
[KYUUBI #1769][FOLLOWUP] Revert the dependency change for kyuubi-hive-jdbc(shaded) modules
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Revert the dependency changes for kyuubi-hive-jdbc and kyuubi-hive-jdbc-shaded modules from #1842

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1854 from turboFei/slf4j-api.

Closes #1769

dffe6c5c [Fei Wang] reserve for kyuubi-hive-beeline
20012434 [Fei Wang] [KYUUBI #1769][FOLLOWUP] Revert the dependency change for kyuubi-hive-beeline and kyuubi-hive-jdbc(shaded) modules

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2022-02-01 22:36:30 +08:00
Fei Wang
885ace06c8 [KYUUBI #1769][FOLLOWUP] Some cleanups for log4j properties
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

This is a followup of #1769. Cleanup the log4j properties files and using log4j2.properties for UT.

### _How was this patch tested?_

Passed UT.

Closes #1842 from turboFei/remove_log4j1.

Closes #1769

b6757555 [Fei Wang] unused change
365fbec7 [Fei Wang] revert license
14c64ec9 [Fei Wang] exclude log4j 1.2.17 in slf4j-log4j12
034b04db [Fei Wang] recover jcl-over-slf4j and slf4j-api for k8s it
2beae75e [Fei Wang] remove log4j 1.2.17
b00f07b5 [Fei Wang] remove from license
e55fd2ec [Fei Wang] remove unused dependencies
ab86f023 [Fei Wang] [KYUUBI #1769][FOLLOWUP] Some cleanups for log4j properties

Authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
2022-01-28 00:58:25 +08:00
Kent Yao
c1a6cd587c
[KYUUBI #1716] Unify Hive deps for server and client
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

This PR make the hive deps be consistent between server and clients for a single release, we shall not play client-server compatibility tricky in one release

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1716 from yaooqinn/hive.

Closes #1716

2eb3bdf7 [Kent Yao] Merge branch 'master' into hive
c1bf6369 [Kent Yao] dep
c7f0f26f [Kent Yao] dep
645d76ab [Kent Yao] fix test
3ac4f55c [Kent Yao] Merge branch 'master' into hive
f1ca475a [Kent Yao] delta test
01d30291 [Kent Yao] refine
1e93f33a [Kent Yao] Merge branch 'master' of github.com:apache/incubator-kyuubi into hive
5bd96313 [Kent Yao] Unify Hive deps for server and client

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2022-01-12 16:36:40 +08:00
fwang12
119e7f2c2c [KYUUBI #1441] [BUILD] Bump 1.5.0-SNAPSHOT
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1441 from turboFei/bump_1.5.0.

Closes #1441

0c03c75d [fwang12] [BUILD] Bump 1.5.0-SNAPSHOT

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
2021-11-23 22:22:26 +08:00
fwang12
21cc217281 [KYUUBI #1419] Refactor hive client related property names
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Refactor the kyuubi client related property names from `hive.cli` to `hive.client`.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1419 from turboFei/refactor_hive_cli.

Closes #1419

de532387 [fwang12] cli -> client

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
2021-11-19 16:09:43 +08:00
fwang12
e5e49b5b1d
[KYUUBI #1414] Add kyuubi-hive-beeline module to support KyuubiConnection and KyuubiStatement
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Add kyuubi-hive-beeline module to support KyuubiConnection and KyuubiStatement.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate
<img width="1410" alt="Screen Shot 2021-11-19 at 1 56 21 PM" src="https://user-images.githubusercontent.com/6757692/142572735-409afe25-b96a-49da-8191-e9a8ced3947b.png">

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1414 from turboFei/rebase-kyuubi_beeline.

Closes #1414

4694bbca [fwang12] return engine launch handle anyway
20ca1400 [fwang12] address comments
2999c30a [fwang12] fix ut
4d3abf4f [fwang12] jars.target.dir to target
4659f1f6 [fwang12] add readme
7e74c5c8 [fwang12] integrate beeline and server
969af045 [fwang12] fix beeline shell
90395ade [fwang12] BeeLine version By Apache Kyuubi
4e63695e [fwang12] fix ut
026e41ef [fwang12] refactor properties
a49677af [fwang12] refactor build and beeline script
ca68eba1 [fwang12] use beeline mode property instead of static method
4a1932e7 [fwang12] cli -> beeline
624281e3 [fwang12] nit
d7d27a09 [fwang12] refactor kyuubi cli assembly
9d9b25a9 [fwang12] simplify the waitLaunchEngineToComplete
cc025ecf [fwang12] Rebase PR_1407

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2021-11-19 15:05:10 +08:00
fwang12
ed43391f8c [KYUUBI #1381] Rewrite kyuubi hive jdbc module based on hive branch-2.3 code base
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Hive code base:
8e7f23f34b

commit 8e7f23f34b2ce7328c9d571a13c336f0c8cdecb6 (HEAD, origin/branch-2.3)
Author: Neelesh Srinivas Salian <nssalianusers.noreply.github.com>
Date:   Tue Oct 19 12:50:31 2021 -0700

    HIVE-25616: Hive-24741 backport to 2.3 (Neelesh Srinivas Salian reviewed by Vihang Karajgaonkar)

    Closes (#2730)

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1381 from turboFei/kyuubi_hive_jdbc.

Closes #1381

f39f1a1d [fwang12] adreess comments
5612f490 [fwang12] remove kyuubi-hive-jdbc from code cov
f5a6376e [fwang12] do not fail if no tests
4e99ffcb [fwang12] check kyuubi-hive-jdbc code coverage
1c5f2e16 [fwang12] refactor the test classes
d7f9c7d7 [fwang12] copy test classes from hive branch-2.3
a28c077a [fwang12] rename classname in comments
42f7c5ea [fwang12] try to remove hive-jdbc dependency
fcf5953a [fwang12] use kyuubi hive jdbc utils
17402f4d [fwang12] replace licenses
5cbca43d [fwang12] add missing class ignored by .gitignore
b2c19334 [fwang12] revert docs change
bdc860b4 [fwang12] backport kyuubiHiveDriver
04731e72 [fwang12] make some rename
9d57076b [fwang12] remove from org.apache.hive.jdbc to org.apache.kyuubi.jdbc.hive
86813e6b [fwang12] Backport KyuubiDriver
42f748d6 [fwang12] Backport KyuubiDatabaseMetaData.getTables and for KyuubiConnection.getMetaData, it is not needed
f0d2aedb [fwang12] rename hive classes to kyuubi version
3ea8ba73 [fwang12] copy the source code from hive branch-2.3
7db4e5ac [fwang12] remove at first

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
2021-11-15 15:38:38 +08:00
Cheng Pan
5c72046b74
[KYUUBI #1149][FOLLOWUP] Exclude all Hive transitive deps
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
I see unexpected deps after #1261, because #1261 removes hive deps from root pom `<dependecyManagement>`.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1278 from pan3793/jdbc.

Closes #1149

a33eaeab [Cheng Pan] [KYUUBI #1149][FOLLOWUP] Exclude all Hive transitve deps

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-10-25 11:46:48 +08:00
Cheng Pan
eb5749d34b
[KYUUBI #1238] [BUILD] Add back Scala stuff to root POM
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
This partial revert [KYUUBI #1203], recover codecov

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1238 from pan3793/mvn.

Closes #1238

b3ba87b2 [Cheng Pan] remove unnecessary changes
c124340a [Cheng Pan] nit
39c436be [Cheng Pan] pom
28546cbd [Cheng Pan] CI
f22e9c91 [Cheng Pan] nit
72e9542d [Cheng Pan] Rebase master
7cd1cba3 [Cheng Pan] [BUILD] Add back Scala stuff to root POM

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-10-15 17:49:48 +08:00
Cheng Pan
86b8e1ce62
[KYUUBI #1236] Add module kyuubi-hive-jdbc-shaded to resolve IDEA-93855
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
The idea based on the analysis of https://github.com/apache/incubator-kyuubi/issues/1131#issuecomment-943163597

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

![image](https://user-images.githubusercontent.com/26535726/137326522-be071bc0-523c-436c-8981-01c97ada777f.png)

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1236 from pan3793/shade.

Closes #1236

7a4e3ccc [Cheng Pan] Add module kyuubi-hive-jdbc-shaded to resolve IDEA-93855

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-10-15 15:27:26 +08:00