7b289ae6ff
Add classification of SQL_Type: this PR is just for DDL
Now we use the simple name to judge this operation belong to which classification of SQL_Type.
The matching rule store in #{sql-classification-default.json}.
The key is the simpleClassName of this analyzed logical plan, the value is the classification of this sql.
It will store in the #{sparksession.conf}, the configuration item is #{spark.sql.classification}.
Why we use the file to store the whole rules?
If we need to change the matching rule, we only need to revise this file.
When the kyuubi started, it will load this json file only once.
When user execute a sql, it will generate a logicalPlan and get its simple name.
By the rules, the function will return the classification that belong to this sql, then it will set this value into #{sparksession.conf}.
For this function, I use SparkSessionExtension to develop it.
### _Why are the changes needed?_
By these SQL_Type, we can do something below:
1. Reduce some log print:
For Auxiliary Statements(the sql type is others), we don't need to print operation log.
2. Optimizing some configuration item.
For example, in final stage, the conf is different between DML and DQL.
Through this configuration item, we can use different conf for them.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes #1002 from zhang1002/branch-1.3_get-sql-type-for-ddl.
Closes #1002
55ef6af7 [张宇翔] some modification
f1f8b355 [张宇翔] change other to undefined
1052ae45 [张宇翔] Change some code standards
5e21dc62 [张宇翔] Change some code standards
3017b96c [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-ddl
c55652fd [张宇翔] Merge remote-tracking branch 'upstream/master'
c2572b98 [张宇翔] 1. Use RuleBuilder to develop this function 2. Use analyzed logical plan to judge this sql's classification 3. Change the matching rule: use map, the key is simpleClassName, the value is classification of this sql
93b5624a [张宇翔] Exclude license check for json
d8187ced [张宇翔] Exclude license check for json
e46bc86e [张宇翔] Add exception
3b358bf0 [张宇翔] Add licence
1125b600 [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-ddl
ef5e8c55 [张宇翔] Merge remote-tracking branch 'upstream/master'
ba8f99eb [张宇翔] Use extension to get simpleName
c0bdea7b [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-ddl
5a75384c [张宇翔] Merge remote-tracking branch 'upstream/master'
55b85849 [张宇翔] Update settings.md
ecbd8000 [张宇翔] Repair the scalastyle violations.
76edd20d [张宇翔] Repair the scalastyle violations.
d8e820ee [张宇翔] Merge branch 'master' into branch-1.3_get-sql-type-for-ddl
8da4f7ed [张宇翔] Merge remote-tracking branch 'upstream/master'
65a90958 [张宇翔] Classification for sqlType: DDL
a7ba1bfc [张宇翔] Merge remote-tracking branch 'upstream/master'
b662a989 [张宇翔] Merge remote-tracking branch 'upstream/master'
4c8f3b87 [张宇翔] Merge remote-tracking branch 'upstream/master'
8b686767 [张宇翔] Merge remote-tracking branch 'upstream/master'
cf99e309 [张宇翔] Merge remote-tracking branch 'upstream/master'
0afaa578 [张宇翔] Merge remote-tracking branch 'upstream/master'
b24fea07 [张宇翔] Merge remote-tracking branch 'upstream/master'
e517cfc5 [张宇翔] Merge remote-tracking branch 'upstream/master'
18aebe76 [张宇翔] Merge remote-tracking branch 'upstream/master'
f248bef7 [张宇翔] Merge remote-tracking branch 'upstream/master'
5ffb54f3 [张宇翔] Add kyuubi-spark-monitor module for nightly.yml
Authored-by: 张宇翔 <zhang1002@126.com>
Signed-off-by: Kent Yao <yao@apache.org>
1754 lines
73 KiB
XML
1754 lines
73 KiB
XML
<?xml version="1.0" encoding="UTF-8"?>
|
|
<!--
|
|
~ Licensed to the Apache Software Foundation (ASF) under one or more
|
|
~ contributor license agreements. See the NOTICE file distributed with
|
|
~ this work for additional information regarding copyright ownership.
|
|
~ The ASF licenses this file to You under the Apache License, Version 2.0
|
|
~ (the "License"); you may not use this file except in compliance with
|
|
~ the License. You may obtain a copy of the License at
|
|
~
|
|
~ http://www.apache.org/licenses/LICENSE-2.0
|
|
~
|
|
~ Unless required by applicable law or agreed to in writing, software
|
|
~ distributed under the License is distributed on an "AS IS" BASIS,
|
|
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
~ See the License for the specific language governing permissions and
|
|
~ limitations under the License.
|
|
-->
|
|
|
|
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
|
|
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
|
|
|
<parent>
|
|
<groupId>org.apache</groupId>
|
|
<artifactId>apache</artifactId>
|
|
<version>23</version>
|
|
</parent>
|
|
<modelVersion>4.0.0</modelVersion>
|
|
|
|
<groupId>org.apache.kyuubi</groupId>
|
|
<artifactId>kyuubi-parent</artifactId>
|
|
<version>1.4.0-SNAPSHOT</version>
|
|
<packaging>pom</packaging>
|
|
|
|
<name>Kyuubi Project Parent</name>
|
|
<url>https://kyuubi.apache.org/</url>
|
|
|
|
<scm>
|
|
<connection>scm:git:git@github.com:apache/incubator-kyuubi.git</connection>
|
|
<developerConnection>scm:git:https://gitbox.apache.org/repos/asf/incubator-kyuubi.git</developerConnection>
|
|
<url>scm:git:git@github.com:apache/incubator-kyuubi.git</url>
|
|
<tag>HEAD</tag>
|
|
</scm>
|
|
|
|
<licenses>
|
|
<license>
|
|
<name>The Apache Software License, Version 2.0</name>
|
|
<url>https://www.apache.org/licenses/LICENSE-2.0.txt</url>
|
|
<distribution>manual</distribution>
|
|
</license>
|
|
</licenses>
|
|
|
|
<issueManagement>
|
|
<system>GitHub Issues</system>
|
|
<url>https://github.com/apache/incubator-kyuubi/issues</url>
|
|
</issueManagement>
|
|
|
|
<developers>
|
|
<developer>
|
|
<id>aajisaka</id>
|
|
<name>Akira Ajisaka</name>
|
|
<email>aajisaka@apache.org</email>
|
|
<organization>Yahoo</organization>
|
|
<url>https://github.com/aajisaka</url>
|
|
</developer>
|
|
<developer>
|
|
<id>pan3793</id>
|
|
<name>Cheng Pan</name>
|
|
<email>chengpan@apache.org</email>
|
|
<url>https://github.com/pan3793</url>
|
|
</developer>
|
|
<developer>
|
|
<id>turbofei</id>
|
|
<name>Fei Wang</name>
|
|
<email>feiwang@apache.org</email>
|
|
<organization>eBay</organization>
|
|
<url>https://github.com/turbofei</url>
|
|
</developer>
|
|
<developer>
|
|
<id>jhx1008</id>
|
|
<name>Hongxiang Jiang</name>
|
|
<email>jhx1008@apache.org</email>
|
|
<organization>NetEase</organization>
|
|
<url>https://github.com/jhx1008</url>
|
|
</developer>
|
|
<developer>
|
|
<id>ningjiang</id>
|
|
<name>Willem Ning Jiang</name>
|
|
<email>ningjiang@apache.org</email>
|
|
<organization>Huawai</organization>
|
|
<url>https://github.com/ningjiang</url>
|
|
</developer>
|
|
<developer>
|
|
<id>ulysses-you</id>
|
|
<name>Xiduo You</name>
|
|
<email>ulyssesyou@apache.org</email>
|
|
<organization>NetEase</organization>
|
|
<url>https://github.com/ulysses-you</url>
|
|
</developer>
|
|
<developer>
|
|
<id>yaooqinn</id>
|
|
<name>Kent Yao</name>
|
|
<email>yao@apache.org</email>
|
|
<organization>NetEase</organization>
|
|
<url>https://github.com/yaooqinn</url>
|
|
</developer>
|
|
<developer>
|
|
<id>apache9</id>
|
|
<name>Duo Zhang</name>
|
|
<email>zhangduo@apache.org</email>
|
|
<organization>Xiaomi</organization>
|
|
<url>https://github.com/apache9</url>
|
|
</developer>
|
|
<developer>
|
|
<id>zjffdu</id>
|
|
<name>Jeff Zhang</name>
|
|
<email>zjffdu@apache.org</email>
|
|
<organization>Alibaba</organization>
|
|
<url>https://github.com/zjffdu</url>
|
|
</developer>
|
|
</developers>
|
|
|
|
<mailingLists>
|
|
<mailingList>
|
|
<name>Dev Mailing List</name>
|
|
<post>dev@kyuubi.apache.org</post>
|
|
<subscribe>dev-subscribe@kyuubi.apache.org</subscribe>
|
|
<unsubscribe>dev-unsubscribe@kyuubi.apache.org</unsubscribe>
|
|
<archive>https://mail-archives.apache.org/mod_mbox/kyuubi-dev</archive>
|
|
</mailingList>
|
|
</mailingLists>
|
|
|
|
<modules>
|
|
<module>dev/kyuubi-codecov</module>
|
|
<module>externals/kyuubi-download</module>
|
|
<module>externals/kyuubi-spark-monitor</module>
|
|
<module>externals/kyuubi-spark-sql-engine</module>
|
|
<module>kyuubi-assembly</module>
|
|
<module>kyuubi-common</module>
|
|
<module>kyuubi-ctl</module>
|
|
<module>kyuubi-ha</module>
|
|
<module>kyuubi-hive-jdbc</module>
|
|
<module>kyuubi-metrics</module>
|
|
<module>kyuubi-server</module>
|
|
<module>kyuubi-zookeeper</module>
|
|
</modules>
|
|
|
|
<properties>
|
|
<java.version>1.8</java.version>
|
|
<maven.version>3.6.3</maven.version>
|
|
<maven.compiler.source>${java.version}</maven.compiler.source>
|
|
<maven.compiler.target>${java.version}</maven.compiler.target>
|
|
<scala.version>2.12.14</scala.version>
|
|
<scala.binary.version>2.12</scala.binary.version>
|
|
|
|
<codahale.metrics.version>4.1.1</codahale.metrics.version>
|
|
<commons-codec.version>1.15</commons-codec.version>
|
|
<commons-collections.version>3.2.2</commons-collections.version>
|
|
<commons-io.version>2.8.0</commons-io.version>
|
|
<commons-lang.version>2.6</commons-lang.version>
|
|
<commons-lang3.version>3.10</commons-lang3.version>
|
|
<curator.version>2.12.0</curator.version>
|
|
<delta.version>1.0.0</delta.version>
|
|
<google.jsr305.version>3.0.2</google.jsr305.version>
|
|
<guava.version>30.1-jre</guava.version>
|
|
<hadoop.version>3.2.2</hadoop.version>
|
|
<hadoop.binary.version>3.2</hadoop.binary.version>
|
|
<hive.version>2.3.7</hive.version>
|
|
<iceberg.name>iceberg-spark3-runtime</iceberg.name>
|
|
<iceberg.version>0.12.0</iceberg.version>
|
|
<jackson.version>2.11.4</jackson.version>
|
|
<jakarta.servlet-api.version>4.0.4</jakarta.servlet-api.version>
|
|
<jaxb.version>2.2.11</jaxb.version>
|
|
<javax-activation.version>1.1.1</javax-activation.version>
|
|
<jetty.version>9.4.41.v20210516</jetty.version>
|
|
<kubernetes-client.version>5.5.0</kubernetes-client.version>
|
|
<ldapsdk.version>5.1.4</ldapsdk.version>
|
|
<prometheus.version>0.10.0</prometheus.version>
|
|
<scalatest.version>3.2.9</scalatest.version>
|
|
<scopt.version>4.0.1</scopt.version>
|
|
<slf4j.version>1.7.30</slf4j.version>
|
|
<spark.version>3.1.2</spark.version>
|
|
<spark.archive.name>spark-${spark.version}-bin-hadoop${hadoop.binary.version}.tgz</spark.archive.name>
|
|
<spark.archive.mirror>https://archive.apache.org/dist/spark/spark-${spark.version}</spark.archive.mirror>
|
|
<spark.archive.download.skip>false</spark.archive.download.skip>
|
|
<zookeeper.version>3.4.14</zookeeper.version>
|
|
|
|
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
|
|
<jars.target.dir>${project.build.directory}/scala-${scala.binary.version}/jars</jars.target.dir>
|
|
|
|
<maven.plugin.antrun.version>3.0.0</maven.plugin.antrun.version>
|
|
<maven.plugin.assembly.version>3.2.0</maven.plugin.assembly.version>
|
|
<maven.plugin.build.helper.version>3.2.0</maven.plugin.build.helper.version>
|
|
<maven.plugin.clean.version>3.1.0</maven.plugin.clean.version>
|
|
<maven.plugin.compiler.version>3.8.1</maven.plugin.compiler.version>
|
|
<maven.plugin.dependency.version>3.1.1</maven.plugin.dependency.version>
|
|
<maven.plugin.deploy.version>3.0.0-M1</maven.plugin.deploy.version>
|
|
<maven.plugin.download.version>1.6.6</maven.plugin.download.version>
|
|
<maven.plugin.gpg.version>3.0.1</maven.plugin.gpg.version>
|
|
<maven.plugin.javadoc.version>3.3.0</maven.plugin.javadoc.version>
|
|
<!-- DO NOT bump 4.4.0, see https://github.com/apache/incubator-kyuubi/pull/441 -->
|
|
<!-- DO NOT bump 4.5.3, see https://github.com/apache/incubator-kyuubi/issues/708 -->
|
|
<maven.plugin.scala.version>4.3.0</maven.plugin.scala.version>
|
|
<maven.plugin.surefire.version>2.22.0</maven.plugin.surefire.version>
|
|
<maven.plugin.scalatest.version>2.0.2</maven.plugin.scalatest.version>
|
|
<maven.plugin.scalatest.exclude.tags>org.apache.kyuubi.tags.ExtendedSQLTest</maven.plugin.scalatest.exclude.tags>
|
|
<maven.plugin.scalatest.include.tags></maven.plugin.scalatest.include.tags>
|
|
<maven.plugin.jacoco.version>0.8.6</maven.plugin.jacoco.version>
|
|
<maven.plugin.jar.version>3.2.0</maven.plugin.jar.version>
|
|
<maven.plugin.scalastyle.version>1.0.0</maven.plugin.scalastyle.version>
|
|
<maven.plugin.shade.version>3.2.4</maven.plugin.shade.version>
|
|
<maven.plugin.silencer.version>1.7.5</maven.plugin.silencer.version>
|
|
<maven.plugin.source.version>3.2.1</maven.plugin.source.version>
|
|
<maven.plugin.rat.version>0.13</maven.plugin.rat.version>
|
|
|
|
<!-- Needed for consistent times -->
|
|
<maven.build.timestamp.format>yyyy-MM-dd HH:mm:ss z</maven.build.timestamp.format>
|
|
|
|
<!-- Package to use when relocating shaded classes. -->
|
|
<kyuubi.shade.packageName>org.apache.kyuubi.shade</kyuubi.shade.packageName>
|
|
</properties>
|
|
|
|
<repositories>
|
|
|
|
<repository>
|
|
<id>gcs-maven-central-mirror</id>
|
|
<name>GCS Maven Central mirror Asia Pacific</name>
|
|
<url>https://maven-central-asia.storage-download.googleapis.com/maven2/</url>
|
|
<releases>
|
|
<enabled>true</enabled>
|
|
</releases>
|
|
<snapshots>
|
|
<enabled>false</enabled>
|
|
</snapshots>
|
|
</repository>
|
|
|
|
<repository>
|
|
<id>central</id>
|
|
<name>Maven Repository</name>
|
|
<url>https://repo.maven.apache.org/maven2</url>
|
|
<releases>
|
|
<enabled>true</enabled>
|
|
</releases>
|
|
<snapshots>
|
|
<enabled>false</enabled>
|
|
</snapshots>
|
|
</repository>
|
|
|
|
<repository>
|
|
<id>Apache Snapshots Repository</id>
|
|
<url>https://repository.apache.org/snapshots/</url>
|
|
<releases>
|
|
<enabled>true</enabled>
|
|
</releases>
|
|
<snapshots>
|
|
<enabled>true</enabled>
|
|
</snapshots>
|
|
</repository>
|
|
</repositories>
|
|
|
|
<pluginRepositories>
|
|
<pluginRepository>
|
|
<id>gcs-maven-central-mirror</id>
|
|
<name>GCS Maven Central mirror Asia Pacific</name>
|
|
<url>https://maven-central-asia.storage-download.googleapis.com/maven2/</url>
|
|
<releases>
|
|
<enabled>true</enabled>
|
|
</releases>
|
|
<snapshots>
|
|
<enabled>false</enabled>
|
|
</snapshots>
|
|
</pluginRepository>
|
|
|
|
<pluginRepository>
|
|
<id>central</id>
|
|
<url>https://repo.maven.apache.org/maven2</url>
|
|
<releases>
|
|
<enabled>true</enabled>
|
|
</releases>
|
|
<snapshots>
|
|
<enabled>false</enabled>
|
|
</snapshots>
|
|
</pluginRepository>
|
|
<pluginRepository>
|
|
<id>apache</id>
|
|
<name>Apache Repository Snapshots</name>
|
|
<url>https://repository.apache.org/snapshots</url>
|
|
<releases>
|
|
<enabled>false</enabled>
|
|
</releases>
|
|
<snapshots>
|
|
<enabled>true</enabled>
|
|
<updatePolicy>daily</updatePolicy>
|
|
<checksumPolicy>warn</checksumPolicy>
|
|
</snapshots>
|
|
</pluginRepository>
|
|
|
|
<pluginRepository>
|
|
<id>sonatype-public-repository</id>
|
|
<url>https://oss.sonatype.org/content/groups/public</url>
|
|
<snapshots>
|
|
<enabled>false</enabled>
|
|
</snapshots>
|
|
<releases>
|
|
<enabled>true</enabled>
|
|
</releases>
|
|
</pluginRepository>
|
|
</pluginRepositories>
|
|
|
|
<dependencyManagement>
|
|
<dependencies>
|
|
|
|
<dependency>
|
|
<groupId>org.scala-lang</groupId>
|
|
<artifactId>scala-library</artifactId>
|
|
<version>${scala.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.scala-lang</groupId>
|
|
<artifactId>scala-reflect</artifactId>
|
|
<version>${scala.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.apache.hadoop</groupId>
|
|
<artifactId>hadoop-client-api</artifactId>
|
|
<version>${hadoop.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.apache.hadoop</groupId>
|
|
<artifactId>hadoop-client-runtime</artifactId>
|
|
<version>${hadoop.version}</version>
|
|
<exclusions>
|
|
<exclusion>
|
|
<groupId>commons-logging</groupId>
|
|
<artifactId>commons-logging</artifactId>
|
|
</exclusion>
|
|
</exclusions>
|
|
</dependency>
|
|
|
|
<!--
|
|
Curator has a hard dependency on Guava, see detail at
|
|
https://cwiki.apache.org/confluence/display/CURATOR/TN13
|
|
-->
|
|
<dependency>
|
|
<groupId>com.google.guava</groupId>
|
|
<artifactId>guava</artifactId>
|
|
<version>${guava.version}</version>
|
|
<exclusions>
|
|
<exclusion>
|
|
<groupId>org.checkerframework</groupId>
|
|
<artifactId>checker-qual</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>com.google.errorprone</groupId>
|
|
<artifactId>error_prone_annotations</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>com.google.j2objc</groupId>
|
|
<artifactId>j2objc-annotations</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>com.google.guava</groupId>
|
|
<artifactId>listenablefuture</artifactId>
|
|
</exclusion>
|
|
</exclusions>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>com.google.code.findbugs</groupId>
|
|
<artifactId>jsr305</artifactId>
|
|
<version>${google.jsr305.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>commons-codec</groupId>
|
|
<artifactId>commons-codec</artifactId>
|
|
<version>${commons-codec.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>commons-collections</groupId>
|
|
<artifactId>commons-collections</artifactId>
|
|
<version>${commons-collections.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>commons-io</groupId>
|
|
<artifactId>commons-io</artifactId>
|
|
<version>${commons-io.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>commons-lang</groupId>
|
|
<artifactId>commons-lang</artifactId>
|
|
<version>${commons-lang.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.apache.commons</groupId>
|
|
<artifactId>commons-lang3</artifactId>
|
|
<version>${commons-lang3.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.apache.spark</groupId>
|
|
<artifactId>spark-sql_${scala.binary.version}</artifactId>
|
|
<version>${spark.version}</version>
|
|
<exclusions>
|
|
<!--
|
|
Use Hadoop Shaded Client to gain more clean transitive dependencies
|
|
-->
|
|
<exclusion>
|
|
<groupId>org.apache.hadoop</groupId>
|
|
<artifactId>hadoop-client</artifactId>
|
|
</exclusion>
|
|
</exclusions>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.apache.spark</groupId>
|
|
<artifactId>spark-hive_${scala.binary.version}</artifactId>
|
|
<version>${spark.version}</version>
|
|
<exclusions>
|
|
<!--
|
|
Use Hadoop Shaded Client to gain more clean transitive dependencies
|
|
-->
|
|
<exclusion>
|
|
<groupId>org.apache.hadoop</groupId>
|
|
<artifactId>hadoop-client</artifactId>
|
|
</exclusion>
|
|
</exclusions>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.apache.spark</groupId>
|
|
<artifactId>spark-core_${scala.binary.version}</artifactId>
|
|
<version>${spark.version}</version>
|
|
<type>test-jar</type>
|
|
<exclusions>
|
|
<!--
|
|
Use Hadoop Shaded Client to gain more clean transitive dependencies
|
|
-->
|
|
<exclusion>
|
|
<groupId>org.apache.hadoop</groupId>
|
|
<artifactId>hadoop-client</artifactId>
|
|
</exclusion>
|
|
</exclusions>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.apache.spark</groupId>
|
|
<artifactId>spark-catalyst_${scala.binary.version}</artifactId>
|
|
<version>${spark.version}</version>
|
|
<type>test-jar</type>
|
|
<exclusions>
|
|
<!--
|
|
Use Hadoop Shaded Client to gain more clean transitive dependencies
|
|
-->
|
|
<exclusion>
|
|
<groupId>org.apache.hadoop</groupId>
|
|
<artifactId>hadoop-client</artifactId>
|
|
</exclusion>
|
|
</exclusions>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.apache.spark</groupId>
|
|
<artifactId>spark-sql_${scala.binary.version}</artifactId>
|
|
<version>${spark.version}</version>
|
|
<type>test-jar</type>
|
|
<exclusions>
|
|
<!--
|
|
Use Hadoop Shaded Client to gain more clean transitive dependencies
|
|
-->
|
|
<exclusion>
|
|
<groupId>org.apache.hadoop</groupId>
|
|
<artifactId>hadoop-client</artifactId>
|
|
</exclusion>
|
|
</exclusions>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>io.fabric8</groupId>
|
|
<artifactId>kubernetes-client</artifactId>
|
|
<version>${kubernetes-client.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.apache.hive</groupId>
|
|
<artifactId>hive-common</artifactId>
|
|
<version>${hive.version}</version>
|
|
<exclusions>
|
|
<exclusion>
|
|
<groupId>org.apache.hive</groupId>
|
|
<artifactId>hive-shims</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.ant</groupId>
|
|
<artifactId>ant</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.hadoop</groupId>
|
|
<artifactId>hadoop-common</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.hadoop</groupId>
|
|
<artifactId>hadoop-auth</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.zookeeper</groupId>
|
|
<artifactId>zookeeper</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.slf4j</groupId>
|
|
<artifactId>slf4j-api</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.slf4j</groupId>
|
|
<artifactId>slf4j-log4j12</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>log4j</groupId>
|
|
<artifactId>log4j</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>commons-logging</groupId>
|
|
<artifactId>commons-logging</artifactId>
|
|
</exclusion>
|
|
<!-- Begin of Hive 2.3 exclusion -->
|
|
<exclusion>
|
|
<groupId>org.apache.orc</groupId>
|
|
<artifactId>orc-core</artifactId>
|
|
</exclusion>
|
|
<!-- jetty-all conflict with jetty 9.4.12.v20180830 -->
|
|
<exclusion>
|
|
<groupId>org.eclipse.jetty.aggregate</groupId>
|
|
<artifactId>jetty-all</artifactId>
|
|
</exclusion>
|
|
<!-- org.apache.logging.log4j:* conflict with log4j 1.2.17 -->
|
|
<exclusion>
|
|
<groupId>org.apache.logging.log4j</groupId>
|
|
<artifactId>*</artifactId>
|
|
</exclusion>
|
|
<!-- Hive includes javax.servlet to fix the Hive on Spark test failure; see HIVE-12783 -->
|
|
<exclusion>
|
|
<groupId>org.eclipse.jetty.orbit</groupId>
|
|
<artifactId>javax.servlet</artifactId>
|
|
</exclusion>
|
|
<!-- hive-storage-api is needed and must be explicitly included later -->
|
|
<exclusion>
|
|
<groupId>org.apache.hive</groupId>
|
|
<artifactId>hive-storage-api</artifactId>
|
|
</exclusion>
|
|
<!-- End of Hive 2.3 exclusion -->
|
|
</exclusions>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.apache.hive</groupId>
|
|
<artifactId>hive-jdbc</artifactId>
|
|
<version>${hive.version}</version>
|
|
<exclusions>
|
|
<exclusion>
|
|
<groupId>org.apache.hive</groupId>
|
|
<artifactId>hive-common</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.hive</groupId>
|
|
<artifactId>hive-metastore</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.hive</groupId>
|
|
<artifactId>hive-serde</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.hive</groupId>
|
|
<artifactId>hive-service</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.hive</groupId>
|
|
<artifactId>hive-service-rpc</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.hive</groupId>
|
|
<artifactId>hive-shims</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.curator</groupId>
|
|
<artifactId>curator-framework</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.thrift</groupId>
|
|
<artifactId>libthrift</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.thrift</groupId>
|
|
<artifactId>libfb303</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.zookeeper</groupId>
|
|
<artifactId>zookeeper</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.slf4j</groupId>
|
|
<artifactId>slf4j-api</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.slf4j</groupId>
|
|
<artifactId>slf4j-log4j12</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>log4j</groupId>
|
|
<artifactId>log4j</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>commons-logging</groupId>
|
|
<artifactId>commons-logging</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.codehaus.groovy</groupId>
|
|
<artifactId>groovy-all</artifactId>
|
|
</exclusion>
|
|
</exclusions>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.apache.hive</groupId>
|
|
<artifactId>hive-serde</artifactId>
|
|
<version>${hive.version}</version>
|
|
<exclusions>
|
|
<exclusion>
|
|
<groupId>org.apache.hive</groupId>
|
|
<artifactId>hive-common</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.hive</groupId>
|
|
<artifactId>hive-shims</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>commons-codec</groupId>
|
|
<artifactId>commons-codec</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>com.google.code.findbugs</groupId>
|
|
<artifactId>jsr305</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.avro</groupId>
|
|
<artifactId>avro</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.thrift</groupId>
|
|
<artifactId>libthrift</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.thrift</groupId>
|
|
<artifactId>libfb303</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.slf4j</groupId>
|
|
<artifactId>slf4j-api</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.slf4j</groupId>
|
|
<artifactId>slf4j-log4j12</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>log4j</groupId>
|
|
<artifactId>log4j</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>commons-logging</groupId>
|
|
<artifactId>commons-logging</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.codehaus.groovy</groupId>
|
|
<artifactId>groovy-all</artifactId>
|
|
</exclusion>
|
|
<!-- Begin of Hive 2.3 exclusion -->
|
|
<exclusion>
|
|
<groupId>org.apache.hive</groupId>
|
|
<artifactId>hive-service-rpc</artifactId>
|
|
</exclusion>
|
|
<!-- parquet-hadoop-bundle:1.8.1 conflict with 1.10.1 -->
|
|
<exclusion>
|
|
<groupId>org.apache.parquet</groupId>
|
|
<artifactId>parquet-hadoop-bundle</artifactId>
|
|
</exclusion>
|
|
<!-- Do not need Jasper, see HIVE-19799 -->
|
|
<exclusion>
|
|
<groupId>tomcat</groupId>
|
|
<artifactId>jasper-compiler</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>tomcat</groupId>
|
|
<artifactId>jasper-runtime</artifactId>
|
|
</exclusion>
|
|
<!-- End of Hive 2.3 exclusion -->
|
|
<exclusion>
|
|
<groupId>commons-lang</groupId>
|
|
<artifactId>commons-lang</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>net.sf.opencsv</groupId>
|
|
<artifactId>opencsv</artifactId>
|
|
</exclusion>
|
|
</exclusions>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.apache.hive</groupId>
|
|
<artifactId>hive-service</artifactId>
|
|
<version>${hive.version}</version>
|
|
<exclusions>
|
|
<exclusion>
|
|
<groupId>*</groupId>
|
|
<artifactId>*</artifactId>
|
|
</exclusion>
|
|
</exclusions>
|
|
</dependency>
|
|
|
|
<!--
|
|
because of THRIFT-4805, we don't upgrade to libthrift:0.12.0,
|
|
because of THRIFT-5274, we don't upgrade to libthrift:0.13.0,
|
|
so just keep hive-service-rpc:2.3.7 transitive dependency libthrift:0.9.3
|
|
-->
|
|
<dependency>
|
|
<groupId>org.apache.hive</groupId>
|
|
<artifactId>hive-service-rpc</artifactId>
|
|
<version>${hive.version}</version>
|
|
<exclusions>
|
|
<exclusion>
|
|
<groupId>commons-codec</groupId>
|
|
<artifactId>commons-codec</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>commons-cli</groupId>
|
|
<artifactId>commons-cli</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>commons-logging</groupId>
|
|
<artifactId>commons-logging</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>tomcat</groupId>
|
|
<artifactId>*</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.httpcomponents</groupId>
|
|
<artifactId>*</artifactId>
|
|
</exclusion>
|
|
</exclusions>
|
|
</dependency>
|
|
|
|
<!-- hive shims. Required for connecting to hive-server protected with kerberos. -->
|
|
<dependency>
|
|
<groupId>org.apache.hive</groupId>
|
|
<artifactId>hive-shims</artifactId>
|
|
<version>${hive.version}</version>
|
|
<exclusions>
|
|
<exclusion>
|
|
<groupId>com.google.guava</groupId>
|
|
<artifactId>guava</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.hadoop</groupId>
|
|
<artifactId>hadoop-yarn-server-resourcemanager</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.curator</groupId>
|
|
<artifactId>curator-framework</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.thrift</groupId>
|
|
<artifactId>libthrift</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.zookeeper</groupId>
|
|
<artifactId>zookeeper</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.slf4j</groupId>
|
|
<artifactId>slf4j-api</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.slf4j</groupId>
|
|
<artifactId>slf4j-log4j12</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>log4j</groupId>
|
|
<artifactId>log4j</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>commons-logging</groupId>
|
|
<artifactId>commons-logging</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.codehaus.groovy</groupId>
|
|
<artifactId>groovy-all</artifactId>
|
|
</exclusion>
|
|
<!-- Begin of Hive 2.3 exclusion -->
|
|
<exclusion>
|
|
<groupId>org.apache.logging.log4j</groupId>
|
|
<artifactId>log4j-slf4j-impl</artifactId>
|
|
</exclusion>
|
|
<!-- End of Hive 2.3 exclusion -->
|
|
</exclusions>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.apache.hive</groupId>
|
|
<artifactId>hive-llap-client</artifactId>
|
|
<version>${hive.version}</version>
|
|
<exclusions>
|
|
<exclusion>
|
|
<groupId>org.apache.hive</groupId>
|
|
<artifactId>hive-common</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.hive</groupId>
|
|
<artifactId>hive-llap-common</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.zookeeper</groupId>
|
|
<artifactId>zookeeper</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.curator</groupId>
|
|
<artifactId>curator-framework</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.curator</groupId>
|
|
<artifactId>apache-curator</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.commons</groupId>
|
|
<artifactId>commons-lang3</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.slf4j</groupId>
|
|
<artifactId>slf4j-api</artifactId>
|
|
</exclusion>
|
|
</exclusions>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.apache.hive</groupId>
|
|
<artifactId>hive-llap-common</artifactId>
|
|
<version>${hive.version}</version>
|
|
<exclusions>
|
|
<exclusion>
|
|
<groupId>org.apache.hive</groupId>
|
|
<artifactId>hive-common</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.hive</groupId>
|
|
<artifactId>hive-serde</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.commons</groupId>
|
|
<artifactId>commons-lang3</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.slf4j</groupId>
|
|
<artifactId>slf4j-api</artifactId>
|
|
</exclusion>
|
|
</exclusions>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.apache.curator</groupId>
|
|
<artifactId>curator-framework</artifactId>
|
|
<version>${curator.version}</version>
|
|
<exclusions>
|
|
<exclusion>
|
|
<groupId>org.apache.zookeeper</groupId>
|
|
<artifactId>zookeeper</artifactId>
|
|
</exclusion>
|
|
</exclusions>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.apache.curator</groupId>
|
|
<artifactId>curator-recipes</artifactId>
|
|
<version>${curator.version}</version>
|
|
<exclusions>
|
|
<exclusion>
|
|
<groupId>org.apache.zookeeper</groupId>
|
|
<artifactId>zookeeper</artifactId>
|
|
</exclusion>
|
|
</exclusions>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.apache.curator</groupId>
|
|
<artifactId>curator-client</artifactId>
|
|
<version>${curator.version}</version>
|
|
<exclusions>
|
|
<exclusion>
|
|
<groupId>com.google.guava</groupId>
|
|
<artifactId>guava</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.apache.zookeeper</groupId>
|
|
<artifactId>zookeeper</artifactId>
|
|
</exclusion>
|
|
</exclusions>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.slf4j</groupId>
|
|
<artifactId>slf4j-api</artifactId>
|
|
<version>${slf4j.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.slf4j</groupId>
|
|
<artifactId>jcl-over-slf4j</artifactId>
|
|
<version>${slf4j.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.slf4j</groupId>
|
|
<artifactId>slf4j-log4j12</artifactId>
|
|
<version>${slf4j.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>io.dropwizard.metrics</groupId>
|
|
<artifactId>metrics-core</artifactId>
|
|
<version>${codahale.metrics.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>io.dropwizard.metrics</groupId>
|
|
<artifactId>metrics-jvm</artifactId>
|
|
<version>${codahale.metrics.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>io.dropwizard.metrics</groupId>
|
|
<artifactId>metrics-jmx</artifactId>
|
|
<version>${codahale.metrics.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>io.dropwizard.metrics</groupId>
|
|
<artifactId>metrics-json</artifactId>
|
|
<version>${codahale.metrics.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>com.fasterxml.jackson.core</groupId>
|
|
<artifactId>jackson-databind</artifactId>
|
|
<version>${jackson.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>com.fasterxml.jackson.module</groupId>
|
|
<artifactId>jackson-module-scala_${scala.binary.version}</artifactId>
|
|
<version>${jackson.version}</version>
|
|
</dependency>
|
|
|
|
<!--
|
|
compatible with dropwizard 3.1.x, 3.2.x, 4.0.x, 4.1.x, 4.2.x
|
|
see https://github.com/prometheus/client_java/issues/662
|
|
-->
|
|
<dependency>
|
|
<groupId>io.prometheus</groupId>
|
|
<artifactId>simpleclient_dropwizard</artifactId>
|
|
<version>${prometheus.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>io.prometheus</groupId>
|
|
<artifactId>simpleclient_servlet</artifactId>
|
|
<version>${prometheus.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>jakarta.servlet</groupId>
|
|
<artifactId>jakarta.servlet-api</artifactId>
|
|
<version>${jakarta.servlet-api.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.eclipse.jetty</groupId>
|
|
<artifactId>jetty-client</artifactId>
|
|
<version>${jetty.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.eclipse.jetty</groupId>
|
|
<artifactId>jetty-server</artifactId>
|
|
<version>${jetty.version}</version>
|
|
<exclusions>
|
|
<!--
|
|
Use `jakarta.servlet-api` instead.
|
|
-->
|
|
<exclusion>
|
|
<groupId>javax.servlet</groupId>
|
|
<artifactId>javax.servlet-api</artifactId>
|
|
</exclusion>
|
|
</exclusions>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.eclipse.jetty</groupId>
|
|
<artifactId>jetty-servlet</artifactId>
|
|
<version>${jetty.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.scalatest</groupId>
|
|
<artifactId>scalatest_${scala.binary.version}</artifactId>
|
|
<version>${scalatest.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.apache.hadoop</groupId>
|
|
<artifactId>hadoop-minikdc</artifactId>
|
|
<version>${hadoop.version}</version>
|
|
<exclusions>
|
|
<exclusion>
|
|
<groupId>org.slf4j</groupId>
|
|
<artifactId>slf4j-log4j12</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>junit</groupId>
|
|
<artifactId>junit</artifactId>
|
|
</exclusion>
|
|
</exclusions>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>com.unboundid</groupId>
|
|
<artifactId>unboundid-ldapsdk</artifactId>
|
|
<version>${ldapsdk.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.apache.hadoop</groupId>
|
|
<artifactId>hadoop-client-minicluster</artifactId>
|
|
<version>${hadoop.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.apache.zookeeper</groupId>
|
|
<artifactId>zookeeper</artifactId>
|
|
<version>${zookeeper.version}</version>
|
|
<exclusions>
|
|
<exclusion>
|
|
<groupId>*</groupId>
|
|
<artifactId>*</artifactId>
|
|
</exclusion>
|
|
</exclusions>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>org.apache.iceberg</groupId>
|
|
<artifactId>${iceberg.name}</artifactId>
|
|
<version>${iceberg.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>io.delta</groupId>
|
|
<artifactId>delta-core_${scala.binary.version}</artifactId>
|
|
<version>${delta.version}</version>
|
|
<exclusions>
|
|
<!--
|
|
Apache Spark already has Antlr and Scala dependencies, and a potential compatibility
|
|
issue of Antlr version on delta 1.0.0: https://github.com/delta-io/delta/pull/676
|
|
-->
|
|
<exclusion>
|
|
<groupId>org.antlr</groupId>
|
|
<artifactId>*</artifactId>
|
|
</exclusion>
|
|
<exclusion>
|
|
<groupId>org.scala-lang</groupId>
|
|
<artifactId>scala-library</artifactId>
|
|
</exclusion>
|
|
</exclusions>
|
|
</dependency>
|
|
|
|
<!--
|
|
Required for JDK 11. Previous as a transitive dependency of hadoop-client,
|
|
since SPARK-33212(will be included in Apache Spark 3.2), Apache Spark moved
|
|
to Hadoop Shaded Client, and Kyuubi always uses Hadoop Shaded Client, thus
|
|
we need explicitly add this dependency.
|
|
-->
|
|
<dependency>
|
|
<groupId>javax.xml.bind</groupId>
|
|
<artifactId>jaxb-api</artifactId>
|
|
<version>${jaxb.version}</version>
|
|
</dependency>
|
|
|
|
<!--
|
|
Required for Yarn Mini Cluster on JDK 11, it already be a transitive dependency
|
|
of spark-core, thus only should explicitly add on Kyuubi Server Main test scope.
|
|
-->
|
|
<dependency>
|
|
<groupId>javax.activation</groupId>
|
|
<artifactId>activation</artifactId>
|
|
<version>${javax-activation.version}</version>
|
|
</dependency>
|
|
|
|
<dependency>
|
|
<groupId>com.github.scopt</groupId>
|
|
<artifactId>scopt_${scala.binary.version}</artifactId>
|
|
<version>${scopt.version}</version>
|
|
</dependency>
|
|
</dependencies>
|
|
</dependencyManagement>
|
|
|
|
<dependencies>
|
|
<dependency>
|
|
<groupId>org.scalatest</groupId>
|
|
<artifactId>scalatest_${scala.binary.version}</artifactId>
|
|
<scope>test</scope>
|
|
</dependency>
|
|
</dependencies>
|
|
|
|
<build>
|
|
<pluginManagement>
|
|
<plugins>
|
|
<plugin>
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
<artifactId>maven-antrun-plugin</artifactId>
|
|
<version>${maven.plugin.antrun.version}</version>
|
|
<executions>
|
|
<execution>
|
|
<id>create-tmp-dir</id>
|
|
<phase>generate-test-resources</phase>
|
|
<goals>
|
|
<goal>run</goal>
|
|
</goals>
|
|
<configuration>
|
|
<target>
|
|
<mkdir dir="${project.build.directory}/tmp"/>
|
|
</target>
|
|
</configuration>
|
|
</execution>
|
|
</executions>
|
|
</plugin>
|
|
|
|
<plugin>
|
|
<groupId>org.codehaus.mojo</groupId>
|
|
<artifactId>build-helper-maven-plugin</artifactId>
|
|
<version>${maven.plugin.build.helper.version}</version>
|
|
<executions>
|
|
<execution>
|
|
<id>module-timestamp-property</id>
|
|
<phase>validate</phase>
|
|
<goals>
|
|
<goal>timestamp-property</goal>
|
|
</goals>
|
|
<configuration>
|
|
<name>module.build.timestamp</name>
|
|
<pattern>${maven.build.timestamp.format}</pattern>
|
|
<timeSource>current</timeSource>
|
|
<timeZone>Asia/Shanghai</timeZone>
|
|
</configuration>
|
|
</execution>
|
|
<execution>
|
|
<id>local-timestamp-property</id>
|
|
<phase>validate</phase>
|
|
<goals>
|
|
<goal>timestamp-property</goal>
|
|
</goals>
|
|
<configuration>
|
|
<name>local.build.timestamp</name>
|
|
<pattern>${maven.build.timestamp.format}</pattern>
|
|
<timeSource>build</timeSource>
|
|
<timeZone>Asia/Shanghai</timeZone>
|
|
</configuration>
|
|
</execution>
|
|
<execution>
|
|
<id>add-scala-sources</id>
|
|
<phase>generate-sources</phase>
|
|
<goals>
|
|
<goal>add-source</goal>
|
|
</goals>
|
|
<configuration>
|
|
<sources>
|
|
<source>src/main/scala-${scala.binary.version}</source>
|
|
</sources>
|
|
</configuration>
|
|
</execution>
|
|
</executions>
|
|
</plugin>
|
|
|
|
<plugin>
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
<artifactId>maven-compiler-plugin</artifactId>
|
|
<version>${maven.plugin.compiler.version}</version>
|
|
<configuration>
|
|
<source>${java.version}</source>
|
|
<target>${java.version}</target>
|
|
<encoding>UTF-8</encoding>
|
|
<maxmem>1024m</maxmem>
|
|
<fork>true</fork>
|
|
<compilerArgs>
|
|
<arg>-Xlint:all,-serial,-path</arg>
|
|
</compilerArgs>
|
|
</configuration>
|
|
</plugin>
|
|
|
|
<plugin>
|
|
<groupId>net.alchim31.maven</groupId>
|
|
<artifactId>scala-maven-plugin</artifactId>
|
|
<version>${maven.plugin.scala.version}</version>
|
|
<executions>
|
|
<execution>
|
|
<id>eclipse-add-source</id>
|
|
<goals>
|
|
<goal>add-source</goal>
|
|
</goals>
|
|
</execution>
|
|
<execution>
|
|
<id>scala-compile-first</id>
|
|
<goals>
|
|
<goal>compile</goal>
|
|
</goals>
|
|
</execution>
|
|
<execution>
|
|
<id>scala-test-compile-first</id>
|
|
<goals>
|
|
<goal>testCompile</goal>
|
|
</goals>
|
|
</execution>
|
|
<execution>
|
|
<id>attach-scaladocs</id>
|
|
<phase>verify</phase>
|
|
<goals>
|
|
<goal>doc-jar</goal>
|
|
</goals>
|
|
</execution>
|
|
</executions>
|
|
<configuration>
|
|
<scalaVersion>${scala.version}</scalaVersion>
|
|
<recompileMode>incremental</recompileMode>
|
|
<args>
|
|
<arg>-unchecked</arg>
|
|
<arg>-deprecation</arg>
|
|
<arg>-feature</arg>
|
|
<arg>-explaintypes</arg>
|
|
<arg>-Yno-adapted-args</arg>
|
|
<arg>-P:silencer:globalFilters=.*deprecated.*</arg>
|
|
<arg>-Xfatal-warnings</arg>
|
|
<arg>-Ywarn-unused:imports</arg>
|
|
</args>
|
|
<jvmArgs>
|
|
<jvmArg>-Xms1024m</jvmArg>
|
|
<jvmArg>-Xmx1024m</jvmArg>
|
|
<jvmArg>-XX:ReservedCodeCacheSize=512M</jvmArg>
|
|
</jvmArgs>
|
|
<javacArgs>
|
|
<javacArg>-source</javacArg>
|
|
<javacArg>${java.version}</javacArg>
|
|
<javacArg>-target</javacArg>
|
|
<javacArg>${java.version}</javacArg>
|
|
<javacArg>-Xlint:all,-serial,-path,-try</javacArg>
|
|
</javacArgs>
|
|
<compilerPlugins>
|
|
<compilerPlugin>
|
|
<groupId>com.github.ghik</groupId>
|
|
<artifactId>silencer-plugin_${scala.version}</artifactId>
|
|
<version>${maven.plugin.silencer.version}</version>
|
|
</compilerPlugin>
|
|
</compilerPlugins>
|
|
</configuration>
|
|
</plugin>
|
|
|
|
<!-- disable surefire -->
|
|
<plugin>
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
<artifactId>maven-surefire-plugin</artifactId>
|
|
<version>${maven.plugin.surefire.version}</version>
|
|
<configuration>
|
|
<skipTests>true</skipTests>
|
|
</configuration>
|
|
</plugin>
|
|
<!-- enable scalatest -->
|
|
<plugin>
|
|
<groupId>org.scalatest</groupId>
|
|
<artifactId>scalatest-maven-plugin</artifactId>
|
|
<version>${maven.plugin.scalatest.version}</version>
|
|
<configuration>
|
|
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
|
|
<junitxml>.</junitxml>
|
|
<filereports>TestSuite.txt</filereports>
|
|
<environmentVariables>
|
|
<KYUUBI_WORK_DIR_ROOT>${project.build.directory}/work</KYUUBI_WORK_DIR_ROOT>
|
|
</environmentVariables>
|
|
<systemProperties>
|
|
<log4j.configuration>file:src/test/resources/log4j.properties</log4j.configuration>
|
|
<java.io.tmpdir>${project.build.directory}/tmp</java.io.tmpdir>
|
|
<spark.driver.memory>2g</spark.driver.memory>
|
|
<kyuubi.metrics.json.location>${project.build.directory}/metrics</kyuubi.metrics.json.location>
|
|
<kyuubi.frontend.bind.host>localhost</kyuubi.frontend.bind.host>
|
|
</systemProperties>
|
|
<tagsToExclude>${maven.plugin.scalatest.exclude.tags}</tagsToExclude>
|
|
<tagsToInclude>${maven.plugin.scalatest.include.tags}</tagsToInclude>
|
|
</configuration>
|
|
<executions>
|
|
<execution>
|
|
<id>test</id>
|
|
<goals>
|
|
<goal>test</goal>
|
|
</goals>
|
|
</execution>
|
|
</executions>
|
|
</plugin>
|
|
|
|
<plugin>
|
|
<groupId>org.jacoco</groupId>
|
|
<artifactId>jacoco-maven-plugin</artifactId>
|
|
<version>${maven.plugin.jacoco.version}</version>
|
|
<executions>
|
|
<execution>
|
|
<id>pre-test</id>
|
|
<goals>
|
|
<goal>prepare-agent</goal>
|
|
</goals>
|
|
</execution>
|
|
<execution>
|
|
<id>report</id>
|
|
<phase>test</phase>
|
|
<goals>
|
|
<goal>report</goal>
|
|
</goals>
|
|
<configuration>
|
|
<outputDirectory>${project.build.directory}/codecov</outputDirectory>
|
|
</configuration>
|
|
</execution>
|
|
</executions>
|
|
</plugin>
|
|
|
|
<plugin>
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
<artifactId>maven-shade-plugin</artifactId>
|
|
<version>${maven.plugin.shade.version}</version>
|
|
</plugin>
|
|
|
|
<plugin>
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
<artifactId>maven-jar-plugin</artifactId>
|
|
<version>${maven.plugin.jar.version}</version>
|
|
</plugin>
|
|
|
|
<plugin>
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
<artifactId>maven-assembly-plugin</artifactId>
|
|
<version>${maven.plugin.assembly.version}</version>
|
|
</plugin>
|
|
|
|
<plugin>
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
<artifactId>maven-dependency-plugin</artifactId>
|
|
<version>${maven.plugin.dependency.version}</version>
|
|
<executions>
|
|
<execution>
|
|
<id>default-cli</id>
|
|
<goals>
|
|
<goal>build-classpath</goal>
|
|
</goals>
|
|
<configuration>
|
|
<!-- This includes dependencies with 'runtime' and 'compile' scopes;
|
|
see the docs for includeScope for more details -->
|
|
<includeScope>runtime</includeScope>
|
|
</configuration>
|
|
</execution>
|
|
<execution>
|
|
<id>generate-test-classpath</id>
|
|
<phase>test-compile</phase>
|
|
<goals>
|
|
<goal>build-classpath</goal>
|
|
</goals>
|
|
<configuration>
|
|
<includeScope>test</includeScope>
|
|
<outputProperty>test_classpath</outputProperty>
|
|
</configuration>
|
|
</execution>
|
|
<execution>
|
|
<id>copy-module-dependencies</id>
|
|
<phase>package</phase>
|
|
<goals>
|
|
<goal>copy-dependencies</goal>
|
|
</goals>
|
|
<configuration>
|
|
<includeScope>runtime</includeScope>
|
|
<outputDirectory>${jars.target.dir}</outputDirectory>
|
|
</configuration>
|
|
</execution>
|
|
</executions>
|
|
</plugin>
|
|
|
|
<plugin>
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
<artifactId>maven-clean-plugin</artifactId>
|
|
<version>${maven.plugin.clean.version}</version>
|
|
<configuration>
|
|
<filesets>
|
|
<fileset>
|
|
<directory>${project.basedir}</directory>
|
|
<includes>
|
|
<include>**/*.log</include>
|
|
</includes>
|
|
</fileset>
|
|
<fileset>
|
|
<directory>${project.basedir}/embedded_zookeeper</directory>
|
|
</fileset>
|
|
<fileset>
|
|
<directory>${project.basedir}/metastore_db</directory>
|
|
</fileset>
|
|
<fileset>
|
|
<directory>${project.basedir}/metrics</directory>
|
|
</fileset>
|
|
<fileset>
|
|
<directory>${project.basedir}/spark-warehouse</directory>
|
|
</fileset>
|
|
</filesets>
|
|
</configuration>
|
|
</plugin>
|
|
|
|
<plugin>
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
<artifactId>maven-deploy-plugin</artifactId>
|
|
<version>${maven.plugin.deploy.version}</version>
|
|
</plugin>
|
|
|
|
<plugin>
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
<artifactId>maven-source-plugin</artifactId>
|
|
<version>${maven.plugin.source.version}</version>
|
|
</plugin>
|
|
|
|
<plugin>
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
<artifactId>maven-javadoc-plugin</artifactId>
|
|
<version>${maven.plugin.javadoc.version}</version>
|
|
</plugin>
|
|
|
|
<plugin>
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
<artifactId>maven-gpg-plugin</artifactId>
|
|
<version>${maven.plugin.gpg.version}</version>
|
|
</plugin>
|
|
|
|
<plugin>
|
|
<groupId>org.scalastyle</groupId>
|
|
<artifactId>scalastyle-maven-plugin</artifactId>
|
|
<version>${maven.plugin.scalastyle.version}</version>
|
|
<configuration>
|
|
<verbose>false</verbose>
|
|
<failOnViolation>true</failOnViolation>
|
|
<includeTestSourceDirectory>true</includeTestSourceDirectory>
|
|
<failOnWarning>false</failOnWarning>
|
|
<sourceDirectory>${project.basedir}/src/main/scala</sourceDirectory>
|
|
<testSourceDirectory>${project.basedir}/src/test/scala</testSourceDirectory>
|
|
<configLocation>scalastyle-config.xml</configLocation>
|
|
<outputFile>${project.build.directory}/scalastyle-output.xml</outputFile>
|
|
<outputEncoding>UTF-8</outputEncoding>
|
|
</configuration>
|
|
<executions>
|
|
<execution>
|
|
<goals>
|
|
<goal>check</goal>
|
|
</goals>
|
|
</execution>
|
|
</executions>
|
|
</plugin>
|
|
<plugin>
|
|
<groupId>com.googlecode.maven-download-plugin</groupId>
|
|
<artifactId>download-maven-plugin</artifactId>
|
|
<version>${maven.plugin.download.version}</version>
|
|
</plugin>
|
|
|
|
<plugin>
|
|
<groupId>org.apache.rat</groupId>
|
|
<artifactId>apache-rat-plugin</artifactId>
|
|
<version>${maven.plugin.rat.version}</version>
|
|
<inherited>true</inherited>
|
|
<executions>
|
|
<execution>
|
|
<phase>verify</phase>
|
|
<goals>
|
|
<goal>check</goal>
|
|
</goals>
|
|
</execution>
|
|
</executions>
|
|
<configuration>
|
|
<excludeSubProjects>false</excludeSubProjects>
|
|
<numUnapprovedLicenses>0</numUnapprovedLicenses>
|
|
<licenseFamilies>
|
|
<licenseFamily implementation="org.apache.rat.license.SimpleLicenseFamily">
|
|
<familyName>Apache License 2.0</familyName>
|
|
</licenseFamily>
|
|
</licenseFamilies>
|
|
<excludes>
|
|
<exclude>**/.*/**</exclude>
|
|
<exclude>**/*.json</exclude>
|
|
<exclude>**/*.prefs</exclude>
|
|
<exclude>**/*.log</exclude>
|
|
<exclude>**/*.md</exclude>
|
|
<exclude>**/*.iml</exclude>
|
|
<exclude>**/*.tmpl</exclude>
|
|
<exclude>**/target/**</exclude>
|
|
<exclude>**/out/**</exclude>
|
|
<exclude>**/spark-warehouse/**</exclude>
|
|
<exclude>**/metastore_db/**</exclude>
|
|
<exclude>**/licenses/LICENSE*</exclude>
|
|
<exclude>**/licenses-binary/LICENSE*</exclude>
|
|
<exclude>**/dependency-reduced-pom.xml</exclude>
|
|
<exclude>**/scalastyle-output.xml</exclude>
|
|
<exclude>NOTICE*</exclude>
|
|
<exclude>docs/**</exclude>
|
|
<exclude>build/apache-maven-*/**</exclude>
|
|
<exclude>build/scala-*/**</exclude>
|
|
<exclude>**/**/operation_logs/**/**</exclude>
|
|
<exclude>**/*.output.schema</exclude>
|
|
<exclude>**/kyuubi-*-bin*/**</exclude>
|
|
</excludes>
|
|
</configuration>
|
|
</plugin>
|
|
</plugins>
|
|
</pluginManagement>
|
|
|
|
<plugins>
|
|
<!-- dump test classpath into a folder -->
|
|
<plugin>
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
<artifactId>maven-dependency-plugin</artifactId>
|
|
</plugin>
|
|
|
|
<plugin>
|
|
<groupId>org.codehaus.mojo</groupId>
|
|
<artifactId>build-helper-maven-plugin</artifactId>
|
|
</plugin>
|
|
|
|
<plugin>
|
|
<groupId>net.alchim31.maven</groupId>
|
|
<artifactId>scala-maven-plugin</artifactId>
|
|
</plugin>
|
|
|
|
<plugin>
|
|
<groupId>org.scalastyle</groupId>
|
|
<artifactId>scalastyle-maven-plugin</artifactId>
|
|
</plugin>
|
|
|
|
<!-- disable surefire -->
|
|
<plugin>
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
<artifactId>maven-surefire-plugin</artifactId>
|
|
</plugin>
|
|
|
|
<!-- enable scalatest -->
|
|
<plugin>
|
|
<groupId>org.scalatest</groupId>
|
|
<artifactId>scalatest-maven-plugin</artifactId>
|
|
</plugin>
|
|
|
|
<plugin>
|
|
<groupId>org.jacoco</groupId>
|
|
<artifactId>jacoco-maven-plugin</artifactId>
|
|
</plugin>
|
|
|
|
<plugin>
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
<artifactId>maven-antrun-plugin</artifactId>
|
|
</plugin>
|
|
|
|
<plugin>
|
|
<groupId>org.apache.rat</groupId>
|
|
<artifactId>apache-rat-plugin</artifactId>
|
|
</plugin>
|
|
</plugins>
|
|
</build>
|
|
|
|
<profiles>
|
|
<profile>
|
|
<id>mirror-cn</id>
|
|
<properties>
|
|
<spark.archive.mirror>https://mirrors.bfsu.edu.cn/apache/spark/spark-${spark.version}</spark.archive.mirror>
|
|
</properties>
|
|
</profile>
|
|
|
|
<profile>
|
|
<id>tpcds</id>
|
|
<modules>
|
|
<module>dev/kyuubi-tpcds</module>
|
|
</modules>
|
|
</profile>
|
|
|
|
<profile>
|
|
<id>java-8</id>
|
|
<activation>
|
|
<jdk>1.8</jdk>
|
|
</activation>
|
|
<properties>
|
|
<java.version>1.8</java.version>
|
|
</properties>
|
|
</profile>
|
|
|
|
<profile>
|
|
<id>java-11</id>
|
|
<activation>
|
|
<jdk>11</jdk>
|
|
</activation>
|
|
<properties>
|
|
<java.version>11</java.version>
|
|
</properties>
|
|
</profile>
|
|
|
|
<profile>
|
|
<id>spark-3.0</id>
|
|
<properties>
|
|
<spark.version>3.0.3</spark.version>
|
|
<delta.version>0.8.0</delta.version>
|
|
<maven.plugin.scalatest.exclude.tags>org.apache.kyuubi.tags.ExtendedSQLTest</maven.plugin.scalatest.exclude.tags>
|
|
</properties>
|
|
</profile>
|
|
|
|
<profile>
|
|
<id>spark-3.1</id>
|
|
<properties>
|
|
<spark.version>3.1.2</spark.version>
|
|
<delta.version>1.0.0</delta.version>
|
|
<maven.plugin.scalatest.exclude.tags>org.apache.kyuubi.tags.ExtendedSQLTest</maven.plugin.scalatest.exclude.tags>
|
|
</properties>
|
|
</profile>
|
|
|
|
<profile>
|
|
<id>spark-master</id>
|
|
<properties>
|
|
<spark.version>3.2.0-SNAPSHOT</spark.version>
|
|
<maven.plugin.scalatest.exclude.tags>org.apache.kyuubi.tags.ExtendedSQLTest,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.IcebergTest</maven.plugin.scalatest.exclude.tags>
|
|
</properties>
|
|
</profile>
|
|
|
|
<profile>
|
|
<id>spark-hadoop-2.7</id>
|
|
<properties>
|
|
<hadoop.binary.version>2.7</hadoop.binary.version>
|
|
</properties>
|
|
</profile>
|
|
|
|
<profile>
|
|
<id>spark-hadoop-3.2</id>
|
|
<properties>
|
|
<hadoop.binary.version>3.2</hadoop.binary.version>
|
|
</properties>
|
|
</profile>
|
|
|
|
<profile>
|
|
<id>spark-provided</id>
|
|
<properties>
|
|
<spark.archive.download.skip>true</spark.archive.download.skip>
|
|
</properties>
|
|
</profile>
|
|
|
|
<profile>
|
|
<id>kyuubi-extension-spark-3-1</id>
|
|
<modules>
|
|
<module>dev/kyuubi-extension-spark-3-1</module>
|
|
</modules>
|
|
</profile>
|
|
|
|
<profile>
|
|
<id>spark-block-cleaner</id>
|
|
<modules>
|
|
<module>tools/spark-block-cleaner</module>
|
|
</modules>
|
|
</profile>
|
|
|
|
<profile>
|
|
<id>kubernetes</id>
|
|
<modules>
|
|
<module>kubernetes/integration-tests</module>
|
|
</modules>
|
|
</profile>
|
|
|
|
<profile>
|
|
<id>apache-release</id>
|
|
<build>
|
|
<plugins>
|
|
<!-- Prevent the source-release-assembly execution defined in the Apache parent POM
|
|
from running so we can control it ourselves -->
|
|
<plugin>
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
<artifactId>maven-assembly-plugin</artifactId>
|
|
<executions>
|
|
<execution>
|
|
<id>source-release-assembly</id>
|
|
<phase>none</phase>
|
|
</execution>
|
|
</executions>
|
|
</plugin>
|
|
<plugin>
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
<artifactId>maven-deploy-plugin</artifactId>
|
|
<inherited>true</inherited>
|
|
<configuration>
|
|
<updateReleaseInfo>true</updateReleaseInfo>
|
|
</configuration>
|
|
</plugin>
|
|
<plugin>
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
<artifactId>maven-source-plugin</artifactId>
|
|
<executions>
|
|
<execution>
|
|
<id>attach-sources</id>
|
|
<goals>
|
|
<goal>jar-no-fork</goal>
|
|
</goals>
|
|
</execution>
|
|
</executions>
|
|
</plugin>
|
|
<plugin>
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
<artifactId>maven-javadoc-plugin</artifactId>
|
|
<configuration>
|
|
<quiet>true</quiet>
|
|
</configuration>
|
|
<executions>
|
|
<execution>
|
|
<id>attach-javadocs</id>
|
|
<goals>
|
|
<goal>jar</goal>
|
|
</goals>
|
|
</execution>
|
|
</executions>
|
|
</plugin>
|
|
<plugin>
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
<artifactId>maven-gpg-plugin</artifactId>
|
|
<executions>
|
|
<execution>
|
|
<id>sign-release-artifacts</id>
|
|
<phase>verify</phase>
|
|
<goals>
|
|
<goal>sign</goal>
|
|
</goals>
|
|
</execution>
|
|
</executions>
|
|
</plugin>
|
|
</plugins>
|
|
</build>
|
|
</profile>
|
|
</profiles>
|
|
</project>
|