Commit Graph

54 Commits

Author SHA1 Message Date
zwangsheng
520bd2653c
[KYUUBI #1480] Fix bug in docker-image-tool & Provider image way to config SPARK instead of local copy
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
1. fix when using `-t` may cause build fail, because tag invalid reference format
2. add new option `-S`; Declare SPARK_HOME in Docker Image. When you configured -S, you need to provide an image
  with Spark as BASE_IMAGE.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1495 from zwangsheng/KYUUBI/1480.

Closes #1480

0b43985a [zwangsheng] replace -d & add usage
7bc5d946 [zwangsheng] use SPARK_HOME_IN_DOCKER
7b68a8ba [zwangsheng] remove spark_binary
765dc284 [zwangsheng] rename
387590d8 [zwangsheng] use as build
8f2420c2 [zwangsheng] Merge branch 'KYUUBI/1480' of https://github.com/zwangsheng/incubator-kyuubi into KYUUBI/1480
e3407060 [zwangsheng] replace
1486b1ee [zwangsheng] replace
a775006c [zwangsheng] add comments
f98a7c23 [zwangsheng] 1480

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2021-12-08 17:34:57 +08:00
zwangsheng
00b4146e79
[KYUUBI #1478] Docker-image-tool.sh awareness of incorrectly configured SPARK_HOME
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

When you use docker-image-tool.sh to help build docker image, you can use a non-existent path as SPARK_HOME.You won't be aware of the problem until Dockerfile does the COPY SPARK_HOME operation.
See #1478
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1479 from zwangsheng/KYUUBI/1478.

Closes #1478

cac129eb [zwangsheng] 1478

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2021-12-01 17:06:14 +08:00
zwangsheng
c357e3f54f
[KYUUBI #1396][K8S] Add docker image tools and replace Dockerfile
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
At the beginning, you just want to add a script to help the user build the image.In the process, the Dockerfile of the existing master branch was discovered.

Although the multi-layer build method can help to build quickly and easily, it is only limited to source code.

In other words, when the user uses the binary package officially released, the Dockerfile is deactivated.

So, In this PR, we move it(the multi-layer Dockerfile) to dev. And use the previous Dockerfile.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1435 from zwangsheng/add-docker-image-tools.

Closes #1396

83d593af [zwangsheng] fix note
30772c5e [zwangsheng] fix
499b3969 [zwangsheng] fix
e64ffb7d [zwangsheng] fix
27b54113 [zwangsheng] fix
fcdfc55e [zwangsheng] dix
3aea0a0b [zwangsheng] fix SPARK_HOME print
67a97083 [zwangsheng] remove help code
74efb245 [zwangsheng] move and modify workflow
f7649e86 [zwangsheng] move and modify workflow
4672a09e [zwangsheng] move and modify workflow
7447ce28 [zwangsheng] move
567478d3 [zwangsheng] move
a224ac0b [zwangsheng] move
cadeef29 [zwangsheng] workflow use dev/Dockerfile
354bc14c [zwangsheng] fix Dockerfile
83561d8b [zwangsheng] build spark into docker image
7d4323c3 [zwangsheng] add kyuubi verison help build tag
4748d09b [zwangsheng] use offical image
4f107d8a [zwangsheng] use offical iamhge
f8c8ba9d [zwangsheng] for test
cecd0485 [zwangsheng] modify
09917320 [zwangsheng] add docker-image

Authored-by: zwangsheng <2213335496@qq.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2021-11-26 17:48:50 +08:00
wForget
b7e94ff45c
[KYUUBI #1439] Add KYUUBI_CTL_JAVA_OPTS to configure java opts of kyuubi-ctl.
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Add KYUUBI_CTL_JAVA_OPTS to configure java opts of kyuubi-ctl. For details: #1439.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [X] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1440 from wForget/KYUUBI-1439.

Closes #1439

bbbf1f23 [wForget] [KYUUBI-1439] Add KYUUBI_CTL_JAVA_OPTS to configure java opts of kyuubi-ctl. update settings.md
da7779a7 [wForget] [KYUUBI-1439] Add KYUUBI_CTL_JAVA_OPTS to configure java opts of kyuubi-ctl.

Authored-by: wForget <643348094@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-11-24 15:27:42 +08:00
fwang12
e5e49b5b1d
[KYUUBI #1414] Add kyuubi-hive-beeline module to support KyuubiConnection and KyuubiStatement
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Add kyuubi-hive-beeline module to support KyuubiConnection and KyuubiStatement.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate
<img width="1410" alt="Screen Shot 2021-11-19 at 1 56 21 PM" src="https://user-images.githubusercontent.com/6757692/142572735-409afe25-b96a-49da-8191-e9a8ced3947b.png">

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1414 from turboFei/rebase-kyuubi_beeline.

Closes #1414

4694bbca [fwang12] return engine launch handle anyway
20ca1400 [fwang12] address comments
2999c30a [fwang12] fix ut
4d3abf4f [fwang12] jars.target.dir to target
4659f1f6 [fwang12] add readme
7e74c5c8 [fwang12] integrate beeline and server
969af045 [fwang12] fix beeline shell
90395ade [fwang12] BeeLine version By Apache Kyuubi
4e63695e [fwang12] fix ut
026e41ef [fwang12] refactor properties
a49677af [fwang12] refactor build and beeline script
ca68eba1 [fwang12] use beeline mode property instead of static method
4a1932e7 [fwang12] cli -> beeline
624281e3 [fwang12] nit
d7d27a09 [fwang12] refactor kyuubi cli assembly
9d9b25a9 [fwang12] simplify the waitLaunchEngineToComplete
cc025ecf [fwang12] Rebase PR_1407

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2021-11-19 15:05:10 +08:00
sunfangbin
fb8bf49dbc
[KYUUBI #1413] Improve usage of zookeeper cli
### _Why are the changes needed?_
This change enabled the JLine support so that the `kyuubi-zookeeper-shell` command can be easier to use with history, prompt and completion, etc. Note that the `jline` version `0.9.94` is dependent on `zookeeper` version `3.4.14` which is a lower version. BTW, users can consider upgrading the `zookeeper` version for the sake of performance improvements and new features.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1413 from murong00/branch-1413.

Closes #1413

2d1ec084 [sunfangbin] Rename shell script to zk-cli
d09f64dc [sunfangbin] Fixes failed ci
ce88357a [sunfangbin] Improve usage of zookeeper cli

Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-11-18 19:17:11 +08:00
sunfangbin
481903fb31
[KYUUBI #1388] Added support a zookeeper shell
### _Why are the changes needed?_
It is useful to support a zookeeper shell so that we can view some metadata if necessary. For example:
```
$ bin/kyuubi-zookeeper-shell -server localhost:2181
Connecting to localhost:2181
Welcome to ZooKeeper!
...
ls /
[zookeeper, kyuubi]
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1388 from murong00/branch-1387.

Closes #1388

7698e79c [sunfangbin] Added support a zookeeper shell

Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
2021-11-15 18:25:43 +08:00
sunfangbin
ff1f78fd57
[KYUUBI #1365] Refine print environment variables to console
### _Why are the changes needed?_
Currently, the enviroment variables are also printed to console when using `bin/kyuubi status` or `bin/kyuubi stop` like below:
```
$ bin/kyuubi status
Using kyuubi environment file /data01/apache-kyuubi-1.4.0-SNAPSHOT-bin-incubating/conf/kyuubi-env.sh to initialize...
JAVA_HOME: /usr/jdk1.8.0_172
KYUUBI_HOME: /data01/apache-kyuubi-1.4.0-SNAPSHOT-bin-incubating
KYUUBI_CONF_DIR: /data01/apache-kyuubi-1.4.0-SNAPSHOT-bin-incubating/conf
KYUUBI_LOG_DIR: /data01/apache-kyuubi-1.4.0-SNAPSHOT-bin-incubating/logs
KYUUBI_PID_DIR: /data01/apache-kyuubi-1.4.0-SNAPSHOT-bin-incubating/pid
KYUUBI_WORK_DIR_ROOT: /data01/apache-kyuubi-1.4.0-SNAPSHOT-bin-incubating/work
SPARK_HOME: /data01/apache-kyuubi-1.4.0-SNAPSHOT-bin-incubating/externals/spark-3.1.2-bin-hadoop3.2
SPARK_CONF_DIR: /data01/apache-kyuubi-1.4.0-SNAPSHOT-bin-incubating/externals/spark-3.1.2-bin-hadoop3.2/conf
HADOOP_CONF_DIR:
Kyuubi is not running
```
In general, these variables are only needed to display when starting a cluster to keep clean style, something like below:
```
$ bin/kyuubi status
Kyuubi is running (pid: 95472)
$ bin/kyuubi stop
Stopping org.apache.kyuubi.server.KyuubiServer
...
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1365 from murong00/branch-1364.

Closes #1365

3856494f [sunfangbin] add checking run command
92686fe1 [sunfangbin] Refine print environment variables to console

Authored-by: sunfangbin <sunfangbin@cmss.chinamobile.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2021-11-12 17:39:41 +08:00
AnybodyHome
04d4179bbe
[KYUUBI #1336] Remove confusing output when kyuubi stop
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
When kyuubi stops, it will print a confuse message `Starting Kyuubi Server ...`.
When kyuubi starts, KYUUBI_HOME will be included in kyuubi environment output print.
So I think this 'echo' is redundant.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1343 from zhenjiaguo/kyuubi-script-fix.

Closes #1336

3b4cc75c [AnybodyHome] Remove confusing output when kyuubi stop

Authored-by: AnybodyHome <zhenjia_guo@163.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-11-05 23:33:32 +08:00
simon
2ae6acda65
[KYUUBI #1039] Kill yarn job when engine initialize timeout and yarnApplicationState is ACCEPTED
kill yarn job when engine initialize timeout and yarnApplicationState is ACCEPTED
#1039

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1119 from simon824/killyarnjob.

Closes #1039

7face4db [simon] get KYUUBI_HOME
da8c3ec4 [Simon] Merge branch 'apache:master' into killyarnjob
95ae8f59 [simon] add appId
8a6ddcd1 [simon] kill application by script
e0cd2af8 [simon] kill application by script
d15d38c5 [simon] kill application by script
1fd1373b [simon] mv sparklauncher to sparkProcessBuilder
5e806426 [simon] fix codestyle
a9cc4505 [simon] fix option No value Exception
9c7ca2f8 [simon] fix option No value Exception
bbfe8e25 [simon] add set sparkHome
04f23c8a [simon] fix codestyle
75599233 [simon] add spark-launcher dep
ef4b2706 [simon] implement by sparkLauncher
59c25b7c [simon] kill yarn application by restful api
49921a48 [simon] fix ut
a31d8f6a [simon] fix ut
1abc6665 [simon] rename killApplication
43a95c1d [simon] Merge branch 'master' into killyarnjob
3d9c12d3 [simon] Merge remote-tracking branch 'upstream/master'
9eaeb16d [simon] fix ut
64ee1b11 [simon] code style
5b905dbd [simon] Merge remote-tracking branch 'upstream/master'
139f3b79 [Simon] Merge branch 'apache:master' into killyarnjob
1a52401d [simon] #1039

Lead-authored-by: simon <zhangshiming@cvte.com>
Co-authored-by: Simon <3656562@qq.com>
Signed-off-by: ulysses-you <ulyssesyou@apache.org>
2021-10-08 16:01:08 +08:00
Cheng Pan
5edc5066f2
[KYUUBI #1013] Enhance bootstrap shell script
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

To make the error message more clear when `SPARK_HOME` is invalid.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1013 from pan3793/sh.

Closes #1013

c9a573b1 [Cheng Pan] nit
e1628640 [Cheng Pan] indent with 2 spaces
68a767f5 [Cheng Pan] Address comments
59d4f076 [Cheng Pan] Enhance load-kyuubi-env.sh

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-09-02 10:15:02 +08:00
timothy65535
1a76d77b6f [KYUUBI #895] Enhance output uses the default spark conf
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
When start the server, If not specified `SPARK_CONF_DIR`, it should output default.
```
KYUUBI_WORK_DIR_ROOT: /root/tmp/kyuubi/work
SPARK_HOME: /root/package/spark-3.1.2-bin-hadoop2.7
SPARK_CONF_DIR:
HADOOP_CONF_DIR: /root/package/hadoop-3.2.0/etc/hadoop
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #896 from timothy65535/ky-895.

Closes #895

9a936a77 [timothy65535] [KYUUBI #895] Enhance output uses the default spark conf

Authored-by: timothy65535 <timothy65535@163.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-08-05 12:39:06 +08:00
hongdd
164bd03799
[KYUUBI #684]Improve kyuubi ctl verbose output (#685)
* [KYUUBI #684]Improve kyuubi ctl verbose output

* fix
2021-06-28 14:09:29 +08:00
peng.zhao
53ba63cd61 [KYUUBI #517][LICENSE] Add license header on kyuubi ASCII Logo
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

Add license header on kyuubi ASCII Logo.

### _How was this patch tested?_
Existing UT.

- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #532 from alex601049502/KYUUBI#517.

Closes #517

38b0d29 [peng.zhao] KYUUBI#517 remove exclude kyuubi-logo
c8102be [peng.zhao] KYUUBI#517 refactor cat logo
90bf6dd [peng.zhao] KYUUBI#517 add new line
8b02f56 [peng.zhao] KYUUBI#529 refactor function name
516894e [peng.zhao] echo logo

Authored-by: peng.zhao <peng.zhao@idiaoyan.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
2021-04-15 18:38:03 +08:00
fwang12
091a9bec8f [KYUUBI #304] Add kyuubi-ctl shell script
![turboFei](https://badgen.net/badge/Hello/turboFei/green) [![Closes #512](https://badgen.net/badge/Preview/Closes%20%23512/blue)](https://github.com/yaooqinn/kyuubi/pull/512) ![46](https://badgen.net/badge/%2B/46/red) ![0](https://badgen.net/badge/-/0/green) ![1](https://badgen.net/badge/commits/1/yellow) [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Add shell script `kyuubi-ctl` to launch kyuubi service control client.

### _How was this patch tested?_
Run test locally.
![image](https://user-images.githubusercontent.com/6757692/114349132-14bafe00-9b9a-11eb-8e29-53eaa5be4c03.png)

- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #512 from turboFei/kyuubi_ctl_shell.

Closes #304

b2a24c0 [fwang12] Add kyuubi-ctl shell script

Authored-by: fwang12 <fwang12@ebay.com>
Signed-off-by: Cheng Pan <379377944@qq.com>
2021-04-12 15:50:09 +08:00
watermen
fc437a41a4
[KYUUBI #455] Fix Hadoop Version for Dev start stript
![watermen](https://badgen.net/badge/Hello/watermen/green) [![Closes #455](https://badgen.net/badge/Preview/Closes%20%23455/blue)](https://github.com/yaooqinn/kyuubi/pull/455) ![2](https://badgen.net/badge/%2B/2/red) ![2](https://badgen.net/badge/-/2/green) ![1](https://badgen.net/badge/commits/1/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) [<img width="16" alt="Powered by Pull Request Badge" src="https://user-images.githubusercontent.com/1393946/111216524-d2bb8e00-85d4-11eb-821b-ed4c00989c02.png">](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

Closes #455 from watermen/hadoopversion.

9e2f259 [watermen] Fix Hadoop Version for Dev start stript

Authored-by: watermen <qiyadong2010@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-03-26 11:57:36 +08:00
Kent Yao
28db93cc75
[KYUUBI #361] Remove Hive Suffix for built in
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #361](https://badgen.net/badge/Preview/Closes%20%23361/blue)](https://github.com/yaooqinn/kyuubi/pull/361) ![5](https://badgen.net/badge/%2B/5/red) ![11](https://badgen.net/badge/-/11/green) ![1](https://badgen.net/badge/commits/1/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

the builtin hive1.2 is no need for builtin anymore

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #361 from yaooqinn/hivesuffix.

99d49fc [Kent Yao] Remove Hive Suffix for built in

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-24 00:13:20 +08:00
Kent Yao
54c306b967
[KYUUBI #356] Fix missing default versions
![yaooqinn](https://badgen.net/badge/Hello/yaooqinn/green) [![Closes #359](https://badgen.net/badge/Preview/Closes%20%23359/blue)](https://github.com/yaooqinn/kyuubi/pull/359) ![3](https://badgen.net/badge/%2B/3/red) ![2](https://badgen.net/badge/-/2/green) ![1](https://badgen.net/badge/commits/1/yellow) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->

the default versions may be missing if 'MVN' is absent,

```
SPARK_HOME: /Users/kentyao/kyuubi/externals/kyuubi-download/target/spark--bin-hadoop
```

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

```
SPARK_HOME: /Users/kentyao/kyuubi/externals/kyuubi-download/target/spark-3.0.2-bin-hadoop2.7
SPARK_CONF_DIR:
HADOOP_CONF_DIR:

Candidate Kyuubi lib /Users/kentyao/kyuubi/jars doesn't exist, searching development environment...
Starting org.apache.kyuubi.server.KyuubiServer, logging to /Users/kentyao/kyuubi/logs/kyuubi-kentyao-org.apache.kyuubi.server.KyuubiServer-hulk.local.out
Welcome to
  __  __                           __
 /\ \/\ \                         /\ \      __
 \ \ \/'/'  __  __  __  __  __  __\ \ \____/\_\
  \ \ , <  /\ \/\ \/\ \/\ \/\ \/\ \\ \ '__`\/\ \
   \ \ \\`\\ \ \_\ \ \ \_\ \ \ \_\ \\ \ \L\ \ \ \
    \ \_\ \_\/`____ \ \____/\ \____/ \ \_,__/\ \_\
     \/_/\/_/`/___/> \/___/  \/___/   \/___/  \/_/
                /\___/
                \/__/
```

Closes #359 from yaooqinn/356.

2014862 [Kent Yao] [KYUUBI #356] Fix missing default versions

Authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-23 16:15:49 +08:00
ulysses-you
bacff6c59a [KYUUBI #356] Update built-in Spark version to 3.0.2
![ulysses-you](https://badgen.net/badge/Hello/ulysses-you/green) [![Closes #356](https://badgen.net/badge/Preview/Closes%20%23356/blue)](https://github.com/yaooqinn/kyuubi/pull/356) ![11](https://badgen.net/badge/%2B/11/red) ![3](https://badgen.net/badge/-/3/green) ![2](https://badgen.net/badge/commits/2/yellow) ![Target Issue](https://badgen.net/badge/Missing/Target%20Issue/ff0000) ![Test Plan](https://badgen.net/badge/Missing/Test%20Plan/ff0000) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Spark 3.0.2 has been released now.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

Closes #356 from ulysses-you/spark-3.0.2.

c791e19 [ulysses-you] fix
ad6c564 [ulysses-you] 3.0.2

Authored-by: ulysses-you <ulyssesyou18@gmail.com>
Signed-off-by: ulysses-you <ulyssesyou18@gmail.com>
2021-02-20 18:26:26 +08:00
Cheng Pan
600b014070
[KYUUBI #312] Support run kyuubi in foreground
![pan3793](https://badgen.net/badge/Hello/pan3793/green) [![Closes #352](https://badgen.net/badge/Preview/Closes%20%23352/blue)](https://github.com/yaooqinn/kyuubi/pull/352) ![26](https://badgen.net/badge/%2B/26/red) ![3](https://badgen.net/badge/-/3/green) ![3](https://badgen.net/badge/commits/3/yellow) ![Feature](https://badgen.net/badge/Label/Feature/) [&#10088;?&#10089;](https://pullrequestbadge.com/?utm_medium=github&utm_source=yaooqinn&utm_campaign=badge_info)<!-- PR-BADGE: PLEASE DO NOT REMOVE THIS COMMENT -->

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/yaooqinn/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
close #312

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/tools/testing.html#running-tests) locally before make a pull request

![image](https://user-images.githubusercontent.com/26535726/107225226-81f3cb00-6a53-11eb-8c1e-1f0d5f8275b4.png)

Closes #352 from pan3793/KYUUBI-312.

1a48504 [Cheng Pan] update help
08f589d [Cheng Pan] update doc and help
563f39e [Cheng Pan] [KYUUBI-312] support run kyuubi in foreground

Authored-by: Cheng Pan <379377944@qq.com>
Signed-off-by: Kent Yao <yao@apache.org>
2021-02-08 23:53:06 +08:00
Kent Yao
91d0ce302e add HADOOP_CONF_DIR to classpath 2020-11-19 00:30:25 +08:00
Kent Yao
64b83a48c5 Show possible error even timeout 2020-11-16 22:56:25 +08:00
Kent Yao
9bd4139453 Remove Kyuubi useless deps 2020-11-16 20:52:45 +08:00
Kent Yao
91f622f94f Remove Kyuubi 0.8 and before 2020-11-16 20:12:08 +08:00
Kent Yao
e92071a67c Suppport making distribution 2020-09-25 15:40:41 +08:00
Kent Yao
bae3ac071c Fix log4j 2020-09-21 18:08:54 +08:00
Kent Yao
470d7ca90d Make man service work 2020-09-21 14:05:34 +08:00
Kent Yao
01d50d3ec7 add tests for meta operations 2020-09-09 16:15:45 +08:00
Kent Yao
06178e52fe KyuubiServer Part 1 2020-09-08 14:45:40 +08:00
Kent Yao
86e1a7aac3 SparkProcessBuilder 2020-08-18 17:22:08 +08:00
Kent Yao
7fb0b1a32f Enable EmbeddedZkServer for KyuubiServer 2020-06-17 22:42:19 +08:00
Kent Yao
b9a50d62ee new scripts for new framework 2020-05-22 19:16:04 +08:00
Guillem Nieto
a131ce0535 Squashed commit of the following:
commit 065238d830e1bf237cf5b5d74f987827cbe3ba61
Author: Guillem Nieto <gnieto@scopely.com>
Date:   Thu May 9 17:01:37 2019 +0200

    Cast to int wc jar count

    While using Kyuubi on my local environment (OSX) I'm having issues with
    `wc` returning the result with a preffixed tab character.

    Due to this issue, it fails to compare "	1" with "1" and script
    is aborted due to duplicated JARs found.

    To solve that, I'm casting `wc` result to integer and perform integer
    comparisions, which seems to solve the issue.
2019-05-10 12:01:35 +08:00
Kent Yao
13c68b5550
[KYUUBI-167]Handling ApplicationMaster token expiry to support long caching sparkcontext with dynamic executor allocation (#168)
* [KYUUBI-167]Handling ApplicationMaster token expiry to support long caching sparkcontext with dynamic executor allocation - fix #167

* fix ut

* add ut

* fix ut

* add doc

* fix https://github.com/apache/spark/pull/24120

* logger

* typo

* comment/dist jar

* fix ut

* typo
2019-03-19 13:51:08 +08:00
Kent Yao
128928b2fc typo 2019-02-21 17:46:19 +08:00
Kent Yao
07a744d486 add logofor kyuubi 2018-12-29 11:38:30 +08:00
Kent Yao
6676ebfc01 init commit for making kyuubi server a sub-project fix #90 2018-06-22 14:20:04 +08:00
Kent Yao
2d386fff98 typo mistake 2018-05-22 16:26:00 +08:00
Kent Yao
e113a7cf52 bin dir 2018-05-21 19:23:37 +08:00
Kent Yao
57721c0e6a export kyuubi home in kyuubi env 2018-05-21 19:19:28 +08:00
Kent Yao
f1486322ee suitable stop script 2018-05-21 19:12:12 +08:00
Kent Yao
f9d6188f39 add kyuubi submit 2018-05-20 21:42:25 +08:00
Kent Yao
756e9b7f21 .sh 2018-05-20 00:08:04 +08:00
Kent Yao
92b9afebdb disable nice 2018-05-20 00:07:13 +08:00
Kent Yao
f1f96ea82f export kyuubi home 2018-05-20 00:04:35 +08:00
Kent Yao
cfe8f27f71 add permit, export kyuubi home 2018-05-19 23:59:10 +08:00
Kent Yao
4ea8d4d4b2 inline start scripts 2018-05-19 23:52:20 +08:00
Kent Yao
828585e6ea fixes #55 add script for making distribution 2018-04-09 14:59:06 +08:00
Kent Yao
1f7d7993dc mv mvn to build 2018-01-20 00:01:36 +08:00
Kent Yao
58a9a3f998 add stop script 2018-01-19 23:43:24 +08:00