[KYUUBI #488 ] Add kyuubi-shaded module to avoid conflicts with external jars

![turboFei](https://badgen.net/badge/Hello/turboFei/green) [![Closes #488](https://badgen.net/badge/Preview/Closes%20%23488/blue)](https://github.com/yaooqinn/kyuubi/pull/488) ![454](https://badgen.net/badge/%2B/454/red) ![289](https://badgen.net/badge/-/289/green) ![10](https://badgen.net/badge/commits/10/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) ![Feature](https://badgen.net/badge/Label/Feature/) [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
Shade curator dependency in kyuubi-ha module to avoid impaction by external jars.

### _How was this patch tested?_
Existing UT.

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #488 from turboFei/shade_curator.

Closes #488

c88414c [fwang12] Use wildcards when selecting more than 6 elements
a2dfbf6 [fwang12] add doc for install shaded jars
224cc38 [fwang12] remove transactions
c7b9df3 [fwang12] address comments
81fc123 [fwang12] scala style
408c978 [fwang12] create kyuubi-shaded module to shade dependencies
6b15b45 [fwang12] shade hive service rpc
b4b3b1a [fwang12] include curator-client
65086ed [fwang12] address comments
a46f00f [fwang12] Shade curator dependencies to avoid impaction by external jars

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Kent Yao <yao@apache.org>

2021-04-09 11:29:24 +08:00

2.0 KiB

Raw Blame History

Building Kyuubi

Building Kyuubi with Apache Maven

Apache Curator and Hive Service RPC are shaded to avoid conflicts with external jars.

You need install these shaded modules locally at first.

Kyuubi is built based on Apache Maven,

./build/mvn clean install -pl :kyuubi-shaded-curator,:kyuubi-shaded-hive-service-rpc -am -Pkyuubi-shaded

./build/mvn clean package -DskipTests

This results in the creation of all sub-modules of Kyuubi project without running any unit test.

If you want to test it manually, you can start Kyuubi directly from the Kyuubi project root by running

bin/kyuubi start

Building a Submodule Individually

For instance, you can build the Kyuubi Common module using:

build/mvn clean package -pl :kyuubi-common -DskipTests

Building Submodules Individually

For instance, you can build the Kyuubi Common module using:

build/mvn clean package -pl :kyuubi-common,:kyuubi-ha -DskipTests

Skipping Some modules

For instance, you can build the Kyuubi modules without Kyuubi Codecov and Assembly modules using:

 mvn clean install -pl '!:kyuubi-codecov,!:kyuubi-assembly' -DskipTests

Building Kyuubi against Different Apache Spark versions

Since v1.1.0, Kyuubi support building with different Spark profiles,

Profile	Default	Since
-Pspark-3.0	Yes	1.0.0
-Pspark-3.1	No	1.1.0

Defining the Apache Mirror for Spark

By default, we use https://archive.apache.org/dist/spark/ to download the built-in Spark release package, but if you find it hard to reach, or the downloading speed is too slow, you can define the spark.archive.mirror property to a suitable Apache mirror site. For instance,

build/mvn clean package -Dspark.archive.mirror=https://mirrors.bfsu.edu.cn/apache/spark/spark-3.0.1

Visit Apache Mirrors and choose a mirror based on your region.

2.0 KiB Raw Blame History