apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

[KYUUBI #762] Add Kyuubi Hive JDBC Module

Browse Source 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Aiming to make a better supported client for Kyuubi and Spark

- Add catalog to getTables meta function for DataLakes (DONE)
- Deploy to maven central (PENDING)

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #762 from yaooqinn/kyuubi-jdbc.

Closes #762

5a881c7b [Kent Yao] address comments
6f01f8e0 [Kent Yao] Add Kyuubi Hive JDBC Module

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
This commit is contained in:
Kent Yao 2021-07-08 11:34:59 +08:00 committed by Cheng Pan
parent 819484fd57
commit 7385f01562
No known key found for this signature in database
GPG Key ID: F07E6C29ED4E2E5B
9 changed files with 380 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
externals/kyuubi-spark-sql-engine/pom.xml vendored
View File

@ -213,6 +213,20 @@
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<executions>
<execution>
<id>prepare-test-jar</id>
<phase>test-compile</phase>
<goals>
<goal>test-jar</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>

7
kyuubi-hive-jdbc/README.md Normal file
View File

@ -0,0 +1,7 @@
# Kyuubi Hive JDBC Module
Aiming to make a better supported client for Kyuubi and Spark
- Add catalog to getTables meta function for DataLakes (DONE)
- Deploy to maven central (PENDING)

113
kyuubi-hive-jdbc/pom.xml Normal file
View File

@ -0,0 +1,113 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>kyuubi</artifactId>
<groupId>org.apache.kyuubi</groupId>
<version>1.3.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>kyuubi-hive-jdbc</artifactId>
<name>Kyuubi Project Hive JDBC Client</name>
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-service-rpc</artifactId>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-service</artifactId>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-common</artifactId>
</dependency>
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-serde</artifactId>
</dependency>
<dependency>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
</dependency>
<dependency>
<groupId>org.apache.kyuubi</groupId>
<artifactId>kyuubi-common</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
</dependency>
<dependency>
<groupId>org.apache.kyuubi</groupId>
<artifactId>kyuubi-spark-sql-engine</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.kyuubi</groupId>
<artifactId>kyuubi-spark-sql-engine</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.binary.version}</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>commons-collections</groupId>
<artifactId>commons-collections</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
</build>
</project>

41
kyuubi-hive-jdbc/src/main/scala/org/apache/kyuubi/jdbc/KyuubiConnection.scala Normal file
View File

@ -0,0 +1,41 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.kyuubi.jdbc
import java.sql.{DatabaseMetaData, SQLException}
import java.util.Properties
import org.apache.hive.jdbc.HiveConnection
import org.apache.hive.service.rpc.thrift.{TCLIService, TSessionHandle}
class KyuubiConnection(url: String, info: Properties) extends HiveConnection(url, info) {
override def getMetaData: DatabaseMetaData = {
if (isClosed) {
throw new SQLException("Connection is closed")
} else {
val clientField = classOf[HiveConnection].getDeclaredField("client")
clientField.setAccessible(true)
val client = clientField.get(this).asInstanceOf[TCLIService.Iface]
val handleField = classOf[HiveConnection].getDeclaredField("sessHandle")
handleField.setAccessible(true)
val sessionHandle = handleField.get(this).asInstanceOf[TSessionHandle]
new KyuubiDatabaseMetaData(this, client, sessionHandle)
}
}
}

59
kyuubi-hive-jdbc/src/main/scala/org/apache/kyuubi/jdbc/KyuubiDatabaseMetaData.scala Normal file
View File

@ -0,0 +1,59 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.kyuubi.jdbc
import java.sql.{ResultSet, SQLException}
import scala.collection.JavaConverters._
import org.apache.hive.jdbc.{HiveDatabaseMetaData, HiveQueryResultSet}
import org.apache.hive.service.cli.HiveSQLException
import org.apache.hive.service.rpc.thrift.{TGetTablesReq, TSessionHandle, TStatusCode}
import org.apache.hive.service.rpc.thrift.TCLIService.Iface
import org.apache.thrift.TException
class KyuubiDatabaseMetaData(conn: KyuubiConnection, client: Iface, sessHandle: TSessionHandle)
extends HiveDatabaseMetaData(conn, client, sessHandle) {
override def getTables(
catalog: String,
schemaPattern: String,
tableNamePattern: String,
types: Array[String]): ResultSet = {
val getTableReq: TGetTablesReq = new TGetTablesReq(sessHandle)
getTableReq.setCatalogName(catalog)
getTableReq.setSchemaName(if (schemaPattern == null) "%" else schemaPattern)
getTableReq.setTableName(tableNamePattern)
if (types != null) {
getTableReq.setTableTypes(types.toList.asJava)
}
val getTableResp = try {
client.GetTables(getTableReq)
} catch {
case e: TException => throw new SQLException(e.getMessage, "08S01", e)
}
val tStatus = getTableResp.getStatus
if (tStatus.getStatusCode != TStatusCode.SUCCESS_STATUS) {
throw new HiveSQLException(tStatus)
}
new HiveQueryResultSet.Builder(conn)
.setClient(client)
.setSessionHandle(sessHandle)
.setStmtHandle(getTableResp.getOperationHandle)
.build
}
}

39
kyuubi-hive-jdbc/src/main/scala/org/apache/kyuubi/jdbc/KyuubiDriver.scala Normal file
View File

@ -0,0 +1,39 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.kyuubi.jdbc
import java.sql.{Connection, DriverManager, SQLException}
import java.util.Properties
import org.apache.hive.jdbc.HiveDriver
class KyuubiDriver extends HiveDriver {
override def connect(url: String, info: Properties): Connection = {
if (acceptsURL(url)) {
new KyuubiConnection(url, info)
} else null
}
}
object KyuubiDriver {
try {
DriverManager.registerDriver(new KyuubiDriver)
} catch {
case e: SQLException => throw new RuntimeException("Failed to register driver", e)
}
}

40
kyuubi-hive-jdbc/src/test/resources/log4j.properties Normal file
View File

@ -0,0 +1,40 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Set everything to be logged to the file target/unit-tests.log
log4j.rootLogger=INFO, CA, FA
#Console Appender
log4j.appender.CA=org.apache.log4j.ConsoleAppender
log4j.appender.CA.layout=org.apache.log4j.PatternLayout
log4j.appender.CA.layout.ConversionPattern=%d{HH:mm:ss.SSS} %p %c: %m%n
log4j.appender.CA.Threshold = FATAL
#File Appender
log4j.appender.FA=org.apache.log4j.FileAppender
log4j.appender.FA.append=false
log4j.appender.FA.file=target/unit-tests.log
log4j.appender.FA.layout=org.apache.log4j.PatternLayout
log4j.appender.FA.layout.ConversionPattern=%d{HH:mm:ss.SSS} %t %p %c{2}: %m%n
# Set the logger level of File Appender to WARN
log4j.appender.FA.Threshold = INFO
# SPARK-34128：Suppress undesirable TTransportException warnings involved in THRIFT-4805
log4j.appender.console.filter.1=org.apache.log4j.varia.StringMatchFilter
log4j.appender.console.filter.1.StringToMatch=Thrift error occurred during processing of message
log4j.appender.console.filter.1.AcceptOnMatch=false

51
kyuubi-hive-jdbc/src/test/scala/org/apache/kyuubi/jdbc/KyuubiDriverSuite.scala Normal file
View File

@ -0,0 +1,51 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.kyuubi.jdbc
import java.util.Properties
import org.apache.kyuubi.engine.spark.WithSparkSQLEngine
import org.apache.kyuubi.engine.spark.shim.SparkCatalogShim
class KyuubiDriverSuite extends WithSparkSQLEngine {
test("get tables with kyuubi driver") {
val kyuubiDriver = new KyuubiDriver()
val connection = kyuubiDriver.connect(getJdbcUrl, new Properties())
val statement = connection.createStatement()
val table = s"${SparkCatalogShim.SESSION_CATALOG}.default.kyuubi_hive_jdbc"
try {
statement.execute(s"CREATE TABLE ${table}(key int) using parquet")
assert(connection.isInstanceOf[KyuubiConnection])
val metaData = connection.getMetaData
assert(metaData.isInstanceOf[KyuubiDatabaseMetaData])
val resultSet = metaData.getTables(SparkCatalogShim.SESSION_CATALOG, "default", "%", null)
assert(resultSet.next())
assert(resultSet.getString(1) === SparkCatalogShim.SESSION_CATALOG)
assert(resultSet.getString(2) === "default")
assert(resultSet.getString(3) === "kyuubi_hive_jdbc")
} finally {
statement.execute(s"DROP TABLE ${table}")
statement.close()
connection.close()
}
connection
}
override def withKyuubiConf: Map[String, String] = Map.empty
}

16
pom.xml
View File

@ -33,6 +33,7 @@
<module>kyuubi-common</module>
<module>kyuubi-ctl</module>
<module>kyuubi-ha</module>
<module>kyuubi-hive-jdbc</module>
<module>kyuubi-main</module>
<module>kyuubi-metrics</module>
<module>kyuubi-zookeeper</module>
@ -66,6 +67,8 @@
<codahale.metrics.version>4.1.1</codahale.metrics.version>
<commons-codec.version>1.15</commons-codec.version>
<commons-collections.version>3.2.2</commons-collections.version>
<commons-io.version>2.8.0</commons-io.version>
<commons-lang.version>2.6</commons-lang.version>
<commons-lang3.version>3.10</commons-lang3.version>
<curator.version>2.12.0</curator.version>
<delta.version>1.0.0</delta.version>
@ -291,12 +294,25 @@
<version>${commons-collections.version}</version>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>${commons-io.version}</version>
</dependency>
<dependency>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
<version>${commons-lang.version}</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>${commons-lang3.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.binary.version}</artifactId>