apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

[KYUUBI #1970] Replace mirror-cn profile with Apache officially suggested dlcdn

Browse Source 
### _Why are the changes needed?_

Replace `mirror-cn` profile with Apache officially suggested dlcdn.

mirror-cn is only suite for CN, while dlcdn is much wiser.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1970 from yaooqinn/cdn.

Closes #1970

a325f828 [Kent Yao] Repalce mirror-cn profile with Apache officially suggested dlcdn
a0756b4f [Kent Yao] Repalce mirror-cn profile with Apache officially suggested dlcdn

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
This commit is contained in:
Kent Yao 2022-02-24 12:07:46 +08:00
parent 19335334db
commit 644b917a30
No known key found for this signature in database
GPG Key ID: F7051850A0AF904D
3 changed files with 22 additions and 28 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

24
docs/develop_tools/building.md
View File

@ -73,22 +73,18 @@ Profile | Default | Since
-Pspark-3.1 | No | 1.1.0
## Defining the Apache Mirror for Spark or Flink
## Building with Apache dlcdn site
By default, we use `https://archive.apache.org/dist/spark/` to download the built-in Spark or
use `https://archive.apache.org/dist/flink/` to download the built-in Flink release package,
but if you find it hard to reach, or the downloading speed is too slow, you can define the `spark.archive.mirror`
or `flink.archive.mirror` property to a suitable Apache mirror site. For instance,
By default, we use `https://archive.apache.org/dist/` to download the built-in release packages of engines,
such as Spark or Flink.
But sometimes, you may find it hard to reach, or the download speed is too slow,
then you can define the `apache.archive.dist` by `-Pmirror-cdn` to accelerate to download speed.
For example,
```bash
build/mvn clean package -Dspark.archive.mirror=https://mirrors.bfsu.edu.cn/apache/spark/spark-3.0.1
build/mvn clean package -Pmirror-cdn
```
Visit [Apache Mirrors](http://www.apache.org/mirrors/) and choose a mirror based on your region.
Specifically for developers in China mainland, you can use the pre-defined profile named `mirror-cn`
which use `mirrors.bfsu.edu.cn` to speed up Spark Binary downloading. For instance,
```bash
build/mvn clean package -Pmirror-cn
```
The profile migrates your download repo to the Apache offically suggested site - https://dlcdn.apache.org.
Note that, this site only holds the latest versions of Apache releases. You may fail if the specific version
defined by `spark.version` or `flink.version` is overdue.

15
externals/kyuubi-download/pom.xml vendored
View File

@ -47,11 +47,6 @@
<configuration>
<skip>${spark.archive.download.skip}</skip>
<url>${spark.archive.mirror}/${spark.archive.name}</url>
<!-- sha512>${spark.archive.sha512}</sha512 -->
<outputDirectory>${project.build.directory}</outputDirectory>
<readTimeOut>60000</readTimeOut>
<retries>3</retries>
<unpack>true</unpack>
</configuration>
</execution>
<execution>
@ -63,13 +58,15 @@
<configuration>
<skip>${flink.archive.download.skip}</skip>
<url>${flink.archive.mirror}/${flink.archive.name}</url>
<outputDirectory>${project.build.directory}</outputDirectory>
<readTimeOut>60000</readTimeOut>
<retries>3</retries>
<unpack>true</unpack>
</configuration>
</execution>
</executions>
<configuration>
<outputDirectory>${project.build.directory}</outputDirectory>
<readTimeOut>60000</readTimeOut>
<retries>3</retries>
<unpack>true</unpack>
</configuration>
</plugin>
</plugins>
</build>

11
pom.xml
View File

@ -93,6 +93,7 @@
<scala.binary.version>2.12</scala.binary.version>
<antlr4.version>4.8</antlr4.version>
<apache.archive.dist>https://archive.apache.org/dist</apache.archive.dist>
<bouncycastle.version>1.67</bouncycastle.version>
<codahale.metrics.version>4.2.8</codahale.metrics.version>
<commons-codec.version>1.15</commons-codec.version>
@ -104,7 +105,7 @@
<delta.version>1.1.0</delta.version>
<flink.version>1.14.3</flink.version>
<flink.archive.name>flink-${flink.version}-bin-scala_${scala.binary.version}.tgz</flink.archive.name>
<flink.archive.mirror>https://archive.apache.org/dist/flink/flink-${flink.version}</flink.archive.mirror>
<flink.archive.mirror>${apache.archive.dist}/flink/flink-${flink.version}</flink.archive.mirror>
<flink.archive.download.skip>false</flink.archive.download.skip>
<google.jsr305.version>3.0.2</google.jsr305.version>
<guava.version>30.1-jre</guava.version>
@ -140,7 +141,7 @@
-->
<spark.version>3.2.1</spark.version>
<spark.archive.name>spark-${spark.version}-bin-hadoop3.2.tgz</spark.archive.name>
<spark.archive.mirror>https://archive.apache.org/dist/spark/spark-${spark.version}</spark.archive.mirror>
<spark.archive.mirror>${apache.archive.dist}/spark/spark-${spark.version}</spark.archive.mirror>
<spark.archive.download.skip>false</spark.archive.download.skip>
<swagger.version>2.1.11</swagger.version>
<swagger-ui.version>4.1.3</swagger-ui.version>
@ -1810,10 +1811,10 @@
<profiles>
<profile>
<id>mirror-cn</id>
<id>mirror-cdn</id>
<!-- this profile works only for latest apache releases -->
<properties>
<spark.archive.mirror>https://mirrors.bfsu.edu.cn/apache/spark/spark-${spark.version}</spark.archive.mirror>
<flink.archive.mirror>https://mirrors.bfsu.edu.cn/apache/flink/flink-${flink.version}</flink.archive.mirror>
<apache.archive.dist>https://dlcdn.apache.org</apache.archive.dist>
</properties>
</profile>