208ab3af62
835 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Cheng Pan
|
208ab3af62
|
[KYUUBI #4852] Switch to Kyuubi Shaded Zookeeper
### _Why are the changes needed?_ This PR aims to migrate the vanilla Zookeeper and Curator to the Kyuubi Shaded Zookeeper. It's the first step to adapting JDK 17. There is a known issue [ZOOKEEPER-3779](https://issues.apache.org/jira/browse/ZOOKEEPER-3779) that Zookeeper 3.4 client can not run on JDK 14 and above, in https://github.com/apache/kyuubi-shaded/pull/5, we fixed this issue by a surgical. With the above fixing, zk-3.4 and zk-3.6 clients both work well on JDK 17, we just randomly pick some cases to make sure zk-3.6 is tested zk-3.4 client supports zk-3.4+ server, but zk-3.6 client only supports zk-3.5+ server; in the meanwhile, zk-3.4 is adopted widely, (CDH 5/6, HDP, EMR created before 2023). We are sticky to zk-3.4 to ensure that Kyuubi can be out-of-box in the most existing Hadoop cluster but also provide zk-3.6 as an alternative(simply replace the kyuubi-shaded-zk-3.4 jar w/ kyuubi-shaded-zk-3.6, or build w/ -Pzookeeper-3.6) for users who concerns that zk-3.4 is EOL. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4852 from pan3793/shaded-zk. Closes #4852 d960cc945 [Cheng Pan] remove staging repo 1b3622080 [Cheng Pan] Switch to Kyuubi Shaded Zookeeper Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
cd47b13127
|
[KYUUBI #4855] Change socket "connection refused" message match pattern
### _Why are the changes needed?_ The error message changed in JDK 17, we should change the match pattern accordingly. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4855 from pan3793/conn-refuse. Closes #4855 78178b140 [Cheng Pan] Change socket "connection refused" message match pattern Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
fwang12
|
b8ee006ca4 |
[KYUUBI #4807][FOLLOWUP] Fix flaky test "basic batch rest client"
### _Why are the changes needed?_ Using `eventually` to fix the flaky test. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4845 from turboFei/flaky_test_log. Closes #4807 3f11bb048 [fwang12] Fix flaky test - basic batch rest client Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
fwang12
|
514a6a961a
|
[KYUUBI #4415][FOLLOWUP] Align the operation handle in server/engine for ExecuteScala, ExecutePython and PlanOnlyStatement
### _Why are the changes needed?_ As title. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4832 from turboFei/scala_python_handle. Closes #4415 a5a44dfa0 [fwang12] ut eaf7f004f [fwang12] ut c8d7a5c82 [fwang12] save Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
liangbowen
|
28216324cd
|
[KYUUBI #4807] [TEST] Retry more times to fix flaky test "spnego batch rest client"
### _Why are the changes needed?_ - retry more times for `spnego batch rest client` ut of `BatchRestApiSuite` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4807 from bowenliang123/fix-spnego-batch. Closes #4807 2b10ba5b3 [liangbowen] update f1563c784 [liangbowen] try to retry in longer time with eventually Authored-by: liangbowen <liangbowen@gf.com.cn> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
fwang12
|
f20f61e686 |
[KYUUBI #4829] Support to expose operation metrics
### _Why are the changes needed?_ We need operation metrics to know the operation details. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4829 from turboFei/op_metrics. Closes #4829 8749405b2 [fwang12] comment 188fbf5ae [fwang12] fix ut 658f1f1c5 [fwang12] expose operation metrics Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
fwang12
|
c38614a372 |
[KYUUBI #4841] [MINOR] Log more info for closing session
### _Why are the changes needed?_ To provide more insights that the session is closed by requests or because of idle more than timeout. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4841 from turboFei/close_log. Closes #4841 959a7ba79 [fwang12] log 0e3f3d609 [fwang12] save Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
Cheng Pan
|
66dfe805d2
|
[KYUUBI #4840] Return cached appInfo iif both op and app are terminated
### _Why are the changes needed?_ When a batch job fails, there is a chance that the op state changed to ERROR, but engine_state has not been updated, w/ the current implementation, currentApplicationInfo always returns the cached outdated appInfo instead of fetching appInfo from the application manager, thus causing the client can not see the correct app status. ``` mysql> select state, engine_state, peer_instance_closed from metadata; +-------+--------------+----------------------+ | state | engine_state | peer_instance_closed | +-------+--------------+----------------------+ | ERROR | RUNNING | 0 | | ERROR | RUNNING | 0 | +-------+--------------+----------------------+ ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4840 from pan3793/appInfo. Closes #4840 f5dddc9db [Cheng Pan] close on error 7c5767e11 [Cheng Pan] Return cached appInfo iif both op and app are terminated Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
wforget
|
272673a41f
|
[KYUUBI #4830][TEST] Fix flaky test "support to interrupt the thrift request if remote engine is broken"
### _Why are the changes needed?_ Move 'support to interrupt the thrift request if remote engine is broken' ut to `KyuubiOperationPerConnectionSuite` and fix it. close #4830 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4833 from wForget/KYUUBI-4830. Closes #4830 838021175 [wforget] test 214d22559 [wforget] fix c7b96f0e2 [wforget] fix fbf837ffd [wforget] fix e6ada6af9 [wforget] [KYUUBI #4830] Fix flaky test: KyuubiOperationPerUserSuite: max result rows Authored-by: wforget <643348094@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
fwang12
|
3fc23970c6 |
[KYUUBI #4792] [MINOR] Enhance hardcode session keywords and remove unused code
### _Why are the changes needed?_ As title. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4792 from turboFei/remove_unused. Closes #4792 fe568af7e [fwang12] server conf 97f510020 [fwang12] save c44e70a58 [fwang12] remove unused code Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
fwang12
|
369c21a611
|
[KYUUBI #4816] [K8S] Correct the implementation of cleanup terminated appInfo
### _Why are the changes needed?_ For `LocalCache`, `put` operation will `remove` the existing element. |
||
|
fwang12
|
299df0d7c2 |
[KYUUBI #4811] Do not update app info after batch or application terminated
### _Why are the changes needed?_ If the application has been terminated, it is not needed to update application. and It can prevent that, the correct application info is overwritten by NOT_FOUND state. If the batch has been terminated, we shall get the batch report from metastore. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4811 from turboFei/k8s_status. Closes #4811 8fc6fd6ab [fwang12] check app id defined 87b0797e0 [fwang12] if batch state is terminal, get state from metadata store 488433e05 [fwang12] save Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
liangbowen
|
d73ec64b37 |
[KYUUBI #4733] Introduce Kafka event logger for server events
### _Why are the changes needed?_ - introduce new event logger type `KAFKA` - send server events to the Kafka topic with initializing and closing Kafka producer properly with server's lifecyle - use Kafka 3.4.0 as the client version, and tested with Kakfa servers of 2.8.x and 3.4.x ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4733 from bowenliang123/kafka-logger. Closes #4733 b5220d234 [liangbowen] introduce kafka server event logger Authored-by: liangbowen <liangbowen@gf.com.cn> Signed-off-by: liangbowen <liangbowen@gf.com.cn> |
||
|
fwang12
|
6ae0c8b141 |
[KYUUBI #4801] Using different engine submit timeout config for kubernetes and yarn
### _Why are the changes needed?_ We shall use different engine submit timeout for different resource manager. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4801 from turboFei/engine_submit_timeout. Closes #4801 e34852a64 [fwang12] nit ad69008e7 [fwang12] 1.7.2 db11330c5 [fwang12] save Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
Cheng Pan
|
ae3b81395c
|
[KYUUBI #4798] Allows BatchJobSubmission to run in sync mode
### _Why are the changes needed?_ Currently, BatchJobSubmission is only allowed to run in async mode, this PR makes the `shouldRunAsync` configurable and allows BatchJobSubmission to run in sync mode. (To minimize the change, in sync mode, the real submission and monitoring still happen on the exec pool, the BatchJobSubmission just blocks until the batch is finished) This PR also refactors the constructor parameters of `KyuubiBatchSessionImpl`, and unwrapped the BatchRequest to make it fit the Batch V2 design. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4798 from pan3793/batch-sync. Closes #4798 38eee2708 [Cheng Pan] Allows BatchJobSubmission run in sync mode Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
d7417ce44f
|
[KYUUBI #4791] Add helper method to simplify REST enabled judgment
### _Why are the changes needed?_ The REST enabled judgment will be used in other places, e.g. the developing batch API v2 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4791 from pan3793/conf. Closes #4791 264566569 [Cheng Pan] Add helper method to simplify REST enabled judgement Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
fwang12
|
07e26a85e0 |
[KYUUBI #4786] Support yarn-client and yarn-cluster for YarnApplicationOperation
### _Why are the changes needed?_ The spark master might be yarn-client and yarn-cluster ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4786 from turboFei/yarn_client_cluster. Closes #4786 accab6b81 [fwang12] Support yarn-client and yarn-cluster Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
fwang12
|
66de0ad8a0
|
[KYUUBI #4780] Get engine application info with interval to prevent frequent call to resource manager
### _Why are the changes needed?_ To prevent frequent call to resource manager. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4780 from turboFei/engine_ref. Closes #4780 09f67c699 [fwang12] re-order 88c1cb33c [fwang12] sleep Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
liangbowen
|
e4fe4814ec
|
[KYUUBI #4783] Use pnpm v8 and update pnpm lockfile to v6 schema
### _Why are the changes needed?_ - v8 is now the stable version of pnpm - lockfile v6 is used by default since pnpm v8.0, which improves the readability of the lockfile by removing hashes from package IDs and more resistant to git merge conflicts (https://github.com/pnpm/pnpm/releases/tag/v8.0.0) - pnpm 8 supports nodejs 16 and 18 (https://pnpm.io/installation#compatibility) ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4783 from bowenliang123/pnpm8. Closes #4783 8b3db3820 [liangbowen] use pnpm v8 and update lock file to v6 Authored-by: liangbowen <liangbowen@gf.com.cn> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
liangbowen
|
430f6d5901 |
[KYUUBI #4777] Deregister event handlers when stopping server with event handler made auto-closeable
### _Why are the changes needed?_ - deregister event handlers when stopping Kyuubiserver, by deregister them from EventBus's handler Registry - change `EventHandler` from `type` to `trait` and make it extending `AutoCloseable` - implement `close` method in `JsonLoggingEventHandler` for closing writers and streams ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4777 from bowenliang123/closable-eventlogger. Closes #4777 db1ad5d73 [liangbowen] make EventBus.deregisterAll method synchronized 648471ba1 [liangbowen] update d28931d3c [liangbowen] re-register event loggers in ut 7121fa33a [liangbowen] make EventHandler closable, and de-register all event handlers when stopping server Authored-by: liangbowen <liangbowen@gf.com.cn> Signed-off-by: liangbowen <liangbowen@gf.com.cn> |
||
|
zwangsheng
|
4762edc622
|
[KYUUBI #4757][UI] Move Statistics to Management Menu
### _Why are the changes needed?_ Close #4757 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4758 from zwangsheng/KYUUBI_4757. Closes #4757 2daa5fc84 [zwangsheng] fix comments b1d5177ce [zwangsheng] [KYUUBI #4757][UI] Move Statistics to Management Menu Authored-by: zwangsheng <2213335496@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
fwang12
|
94c72734ca |
[KYUUBI #4767] Correct the submit time for BatchJobSubmission and check applicationInfo if submitted application
### _Why are the changes needed?_ - if the kyuubi instance is unreachable, we should not transfer the batch metadata create time as batch submit time - we should always wait the kyuubi instance recovery - here using a fake submit time to prevent that the batch be marked as terminated if application state is NOT_FOUND - Inside the BatchJobSubmission, using the first get application info time as batch submit time. In this pr, I also record whether the batch operation submit batch. If it did submit batch application and the app is not started, we need to mark the batch state as ERROR. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4767 from turboFei/submit_time. Closes #4767 9d4df0f91 [fwang12] save 3e56a39cb [fwang12] runtime exception -> kyuubi exception 5cac15ec5 [fwang12] nit 92d5000be [fwang12] nit 3678f8f2c [fwang12] wait the app to monitor d51fb2636 [fwang12] save 708ad20ce [fwang12] refactor 98d49c64e [fwang12] wait 1adbefd59 [fwang12] revert f3e4f2a11 [fwang12] wait app id ready before monitoring a3bfe6f56 [fwang12] check app started 7530b5118 [fwang12] check submit app and final state a41e81d0e [fwang12] refactor e4217da03 [fwang12] _app start time 3d1e8f022 [fwang12] fake submit timeout 06c8f0a22 [fwang12] correct submit time Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
remzi
|
efe81ac0b1
|
[KYUUBI #4738] AdminResource.getEngineSpace should use primary group name on GROUP share level
### _Why are the changes needed?_ Closes #4738 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4740 from HaoYang670/4738_fix_GROUP_REST_get_engine. Closes #4738 d044b2ac2 [remzi] fmt 68630250a [remzi] add tests 47f2dc205 [remzi] Merge remote-tracking branch 'upstream/master' into 4738_fix_GROUP_REST_get_engine 60d40f5b7 [remzi] fix style c077fcd84 [remzi] fix style 1126c1c72 [remzi] Merge remote-tracking branch 'upstream/master' into 4738_fix_GROUP_REST_get_engine 1b03ef559 [remzi] address review comments dbb6b17ec [remzi] restore gitignore df528ff5b [remzi] ignore scala metals things 91789f968 [remzi] use group name Authored-by: remzi <13716567376yh@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
fwang12
|
8d424ef435 |
[KYUUBI #4746] Do not recreate async request executor if has been shutdown
### _Why are the changes needed?_ After #4480, there should be only one asyncRequestExecutor in one KyuubiSyncThriftClient ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4746 from turboFei/engine_alive. Closes #4746 9d2fa5fe9 [fwang12] fix typo d90263b65 [fwang12] check 38aefdf39 [fwang12] close protocol first 6eb61780f [fwang12] do not renew executor 64e66a857 [fwang12] close protocol first Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
fwang12
|
ccacb33c69 |
[KYUUBI #4739] Add operation lock instead of locking state Enumeration
### _Why are the changes needed?_ We meet an issue that cause all the operation stuck when closing operation. Because now all the operations try to lock a Scala Enumeration val. And if one of them stuck, all the others will be keep stuck. In this pr, I add a lock for each operation. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4739 from turboFei/op_lock. Closes #4739 535400a42 [fwang12] revert a93438927 [fwang12] lockInterruptibly 274abc9db [fwang12] utils ceda7314f [fwang12] op lock Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
Cheng Pan
|
6876f82053
|
[KYUUBI #4744] [TEST] Remove Hudi integration tests
### _Why are the changes needed?_ This PR aims to remove Hudi integration tests from the Kyuubi project. Actually, there is no obvious benefit to running Hudi tests w/ Kyuubi, since the real work happens on the compute engine and Hudi integration. Besides, Hudi's horrible dependency management brings significant maintenance efforts to the Kyuubi community. This change only affects tests, does not affect any functionality. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4744 from pan3793/remove-hudi. Closes #4744 ea99f747e [Cheng Pan] Remove Hudi integration tests Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
fwang12
|
a59e27a7ae |
[KYUUBI #4719] Support submission timeout for yarn application manager and get the applicationInfo in-memory
### _Why are the changes needed?_ To prevent the create batch operation stuck. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4719 from turboFei/yarn_app_mgr. Closes #4719 1b0da9315 [fwang12] check timeout Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
zwangsheng
|
f4a56efec2
|
[KYUUBI #3652][UI] Add Kyuubi Server Management page
### _Why are the changes needed?_ Close #3652 Add Kyuubi Server Management page ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request  Closes #4726 from zwangsheng/KYUUBI_3652. Closes #3652 702bb5505 [zwangsheng] remvoe usless code 15f8214f2 [zwangsheng] [KYUUBI #3652][UI] Add Kyuubi Server Manangement page Authored-by: zwangsheng <2213335496@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
fwang12
|
ec8596c9ee
|
[KYUUBI #4731] Support batch session conf advisor
### _Why are the changes needed?_ As title. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4731 from turboFei/batch_session_conf_overlay. Closes #4731 5cf73d1db [fwang12] save Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
zwangsheng
|
f6331a2a0f |
[KYUUBI #3653][REST] AdminResource add list kyuubi server api
### _Why are the changes needed?_ Add List Kyuubi Server Api for `AdminResource` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4670 from zwangsheng/KYUUBI_3653. Closes #3653 b91a6c617 [zwangsheng] fxi 4271d0fd0 [zwangsheng] fix comments e14f8cd55 [zwangsheng] [KYUUBI #3653][REST] AdminResource add list server api Authored-by: zwangsheng <2213335496@qq.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
Paul Lin
|
553b2aafe7
|
[KYUUBI #4367] Support Flink 1.17
### _Why are the changes needed?_ Support Flink 1.17 and Flink SQL gateway. 1. Drop Flink 1.15 2. Migrate API to Flink SQL Gateway 3. Support Flink 1.17 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4368 from link3280/flink-1.17. Closes #4367 8eb4da6c0 [Paul Lin] [KYUUBI #4367] Fix test failure 81a10f6be [Paul Lin] [KYUUBI #4367] Fix test failure 23d87ba1d [Paul Lin] [KYUUBI #4367] Rename delegation package to shim 5c9d0aa84 [Paul Lin] [KYUUBI #4367] Improve code style 56567fcd7 [Paul Lin] [KYUUBI #4367] Improve java.long.Long usage 417d37b27 [Paul Lin] Update externals/kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/FlinkEngineUtils.scala 08f89991a [Paul Lin] [KYUUBI #4367] Fix ambiguous reference ed950600c [Paul Lin] [KYUUBI #4367] Fix spotless 7b28eaf11 [Paul Lin] [KYUUBI #4367] Improve code style for iterations c2a23d571 [Paul Lin] [KYUUBI #4367] Improve code style for error messages and iterations 7e36e70c7 [Paul Lin] [KYUUBI #4367] Improve code style for java.lang.Boolean 4ef8c5209 [Paul Lin] [KYUUBI #4367] Improve code style for java.util.* 8530aec2a [Paul Lin] [KYUUBI #4367] Remove unnecessary classes 1c41e4400 [Paul Lin] [KYUUBI #4367] Remove unnecessary variables 33eeb37ee [Paul Lin] [KYUUBI #4367] Remove unnecessary reflection code e1e5cd2cf [Paul Lin] [KYUUBI #4367] Fix IncompatibleClassChangeError 3520a5153 [Paul Lin] [KYUUBI #4367] Fix IncompatibleClassChangeError 42cce7a54 [Paul Lin] [KYUUBI #4367] Replace vanilla reflection with kyuubi refection tools 20e9913e3 [Paul Lin] [KYUUBI #4367] Fix FlinkProcessBuilder test error a02e01adf [Paul Lin] [KYUUBI #4367] Improve code style 20e1a559e [Paul Lin] [KYUUBI #4367] Use kyuubi refection tools 9b2072e45 [Paul Lin] [KYUUBI #4367] Improve flink version match 7ce1e9a12 [Paul Lin] [KYUUBI #4367] Fix local engine tagged as YARN app fd0c88d15 [Paul Lin] Revert "[KYUUBI #4367] Filter out non kyuubi prefixed conf in flink login engine" f71c6014e [Paul Lin] [KYUUBI #4367] Fix local engine tagged as YARN app b7d46f57d [Paul Lin] [KYUUBI #4367] Filter out non kyuubi prefixed conf in flink login engine 47beb1a78 [Paul Lin] [KYUUBI #4367] Refactor Flink engine tests 7e1a198ca [Paul Lin] [KYUUBI #4367] Fix flink sql gateway jar not included in local mode e851d9732 [Paul Lin] [KYUUBI #4367] Disable query id test for flink 1.16 7291e27fa [Paul Lin] [KYUUBI #4367] Remove profile for flink-1.15 54cfe3bbc [Paul Lin] [KYUUBI #4367] Fix udf not found in local flink engine tests 1a7833bf2 [Paul Lin] [KYUUBI #4367] Fix test failure in PlanOnlyStatementSuit 700ee04db [Paul Lin] [KYUUBI #4367] Fix FLINK_CONF_DIR not set in ut b685ff139 [Paul Lin] [KYUUBI #4367] Improve code style 29728c042 [Paul Lin] [KYUUBI #4367] Fix Flink conf dir not found 799c93876 [Paul Lin] [KYUUBI #4367] Fix NoSuchFieldException 614ecc335 [Paul Lin] [KYUUBI #4367] Fix reflection failures 6a08d0bbe [Paul Lin] [KYUUBI #4367] Fix NPE in dependencies d289495c0 [Paul Lin] [KYUUBI #4367] Flink FlinkSQLEngine capabilities with Flink 1.16 ef6f4d4ff [Paul Lin] [KYUUBI #4367] Remove support for Flink 1.15 e18b3c2ed [Paul Lin] [KYUUBI #4367] Fix Flink SessionManager compatibility issue 49e0a94be [Paul Lin] feat: Support Flink 1.17 Authored-by: Paul Lin <paullin3280@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Fu Chen
|
db46b5b320
|
[KYUUBI #4711] JDBC client should catch task failed exception instead of NPE in the incremental mode
### _Why are the changes needed?_
Since the job was lazily submitted in the incremental mode, the engine should not catch the task failed exception even though the operation is in the terminal state.
Before this PR:
```
0: jdbc:hive2://0.0.0.0:10009/> set kyuubi.operation.incremental.collect=true;
+---------------------------------------+--------+
| key | value |
+---------------------------------------+--------+
| kyuubi.operation.incremental.collect | true |
+---------------------------------------+--------+
0: jdbc:hive2://0.0.0.0:10009/> SELECT raise_error('custom error message');
Error: (state=,code=0)
0: jdbc:hive2://0.0.0.0:10009/>
```
kyuubi server log
```
2023-04-14 18:47:50.185 ERROR org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Error fetching results:
java.lang.NullPointerException: null
at org.apache.kyuubi.server.BackendServiceMetric.$anonfun$fetchResults$1(BackendServiceMetric.scala:191) ~[classes/:?]
at org.apache.kyuubi.metrics.MetricsSystem$.timerTracing(MetricsSystem.scala:111) ~[classes/:?]
at org.apache.kyuubi.server.BackendServiceMetric.fetchResults(BackendServiceMetric.scala:187) ~[classes/:?]
at org.apache.kyuubi.server.BackendServiceMetric.fetchResults$(BackendServiceMetric.scala:182) ~[classes/:?]
at org.apache.kyuubi.server.KyuubiServer$$anon$1.fetchResults(KyuubiServer.scala:147) ~[classes/:?]
at org.apache.kyuubi.service.TFrontendService.FetchResults(TFrontendService.scala:530) [classes/:?]
```
After this PR:
```
0: jdbc:hive2://0.0.0.0:10009/> set kyuubi.operation.incremental.collect=true;
+---------------------------------------+--------+
| key | value |
+---------------------------------------+--------+
| kyuubi.operation.incremental.collect | true |
+---------------------------------------+--------+
0: jdbc:hive2://0.0.0.0:10009/> SELECT raise_error('custom error message');
Error: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 1 times, most recent failure: Lost task 0.0 in stage 3.0 (TID 3) (0.0.0.0 executor driver): java.lang.RuntimeException: custom error message
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.project_doConsume_0$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:760)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:364)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:136)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2672)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2608)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2607)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2607)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1182)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1182)
at scala.Option.foreach(Option.scala:407)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1182)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2860)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2802)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2791)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:952)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2228)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2249)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2268)
at org.apache.spark.rdd.RDD.collectPartition$1(RDD.scala:1036)
at org.apache.spark.rdd.RDD.$anonfun$toLocalIterator$3(RDD.scala:1038)
at org.apache.spark.rdd.RDD.$anonfun$toLocalIterator$3$adapted(RDD.scala:1038)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:491)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.kyuubi.operation.IterableFetchIterator.hasNext(FetchIterator.scala:97)
at scala.collection.Iterator$SliceIterator.hasNext(Iterator.scala:268)
at scala.collection.Iterator.toStream(Iterator.scala:1417)
at scala.collection.Iterator.toStream$(Iterator.scala:1416)
at scala.collection.AbstractIterator.toStream(Iterator.scala:1431)
at scala.collection.TraversableOnce.toSeq(TraversableOnce.scala:354)
at scala.collection.TraversableOnce.toSeq$(TraversableOnce.scala:354)
at scala.collection.AbstractIterator.toSeq(Iterator.scala:1431)
at org.apache.kyuubi.engine.spark.operation.SparkOperation.$anonfun$getNextRowSet$1(SparkOperation.scala:265)
at org.apache.kyuubi.engine.spark.operation.SparkOperation.$anonfun$withLocalProperties$1(SparkOperation.scala:155)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
at org.apache.kyuubi.engine.spark.operation.SparkOperation.withLocalProperties(SparkOperation.scala:139)
at org.apache.kyuubi.engine.spark.operation.SparkOperation.getNextRowSet(SparkOperation.scala:243)
at org.apache.kyuubi.operation.OperationManager.getOperationNextRowSet(OperationManager.scala:141)
at org.apache.kyuubi.session.AbstractSession.fetchResults(AbstractSession.scala:240)
at org.apache.kyuubi.service.AbstractBackendService.fetchResults(AbstractBackendService.scala:214)
at org.apache.kyuubi.service.TFrontendService.FetchResults(TFrontendService.scala:530)
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837)
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: custom error message
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.project_doConsume_0$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:760)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:364)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:136)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
... 3 more (state=,code=0)
0: jdbc:hive2://0.0.0.0:10009/>
```
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes #4711 from cfmcgrady/incremental-show-error-msg.
Closes #4711
66bb527ce [Fu Chen] JDBC client should catch task failed exception in the incremental mode
Authored-by: Fu Chen <cfmcgrady@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
|
||
|
fwang12
|
fa60e4c70b |
[KYUUBI #4691] [REST] Configure FAIL_ON_UNKNOWN_PROPERTIES to false for KyuubiScalaObjectMapper
### _Why are the changes needed?_ Do not failed on unknown properties in server side. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4691 from turboFei/ignore. Closes #4691 a03cb5be0 [fwang12] Ignore c406878e7 [fwang12] Fast return batch info when post batches Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
zwangsheng
|
f5ef4018ee
|
[KYUUBI #3654][UI] Add Engine Manager Page
### _Why are the changes needed?_ Close #3654 Add Engine Manager Page for UI ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request  Closes #4674 from zwangsheng/KYUUBI_3654. Closes #3654 b18c7d2d9 [zwangsheng] fix style 75b61350a [zwangsheng] fix style a064203fc [zwangsheng] I18n da61ea2fe [zwangsheng] [KYUUBI #3654][UI] Engine Manager Page Authored-by: zwangsheng <2213335496@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Paul Lin
|
b315123a6b
|
[KYUUBI #1652] Support Flink yarn application mode
### _Why are the changes needed?_ Flink yarn application mode is crucial for the production usage of Flink engine. To test this PR locally, we should: 1) set `flink.execution.target=yarn-application` in `kyuubi-defaults.conf`. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4604 from link3280/KYUUBI-1652. Closes #1652 49b454f1e [Paul Lin] [KYUUBI #1652] Delay access to thrift services to stablize tests b91b64bf6 [Paul Lin] Revert "[KYUUBI #1652] Avoid hadoop conf injecting into kyuubi conf" c9f710b0f [Paul Lin] [KYUUBI #1652] Avoid hadoop conf injecting into kyuubi conf cde8a5477 [Paul Lin] [KYUUBI #1652] Improve docs edba0ec79 [Paul Lin] [KYUUBI #1652] Improve codestyle e03e055ae [Paul Lin] [KYUUBI #1652] Update docs according to the comments 490559cd8 [Paul Lin] [KYUUBI #1652] Update docs 769d1a8fa [Paul Lin] [KYUUBI #1652] Move zookeeper to test scope bafb3f5a4 [Paul Lin] [KYUUBI #1652] Fix flink-it test dd40c72b8 [Paul Lin] [KYUUBI #1652] Update docs 36c993fc2 [Paul Lin] [KYUUBI #1652] Fix javax.activation not found in flink-it 2a751bdd6 [Paul Lin] [KYUUBI #1652] Introduce EmbeddedZookeeper in Flink yarn tests 0933b7082 [Paul Lin] [KYUUBI #1652] Fix spotless issue b858f7df6 [Paul Lin] [KYUUBI #1652] Fix Flink submit timeout because failing to find hadoop conf 15801b598 [Paul Lin] [KYUUBI #1652] Replace unused jaxb b210615e4 [Paul Lin] Update externals/kyuubi-flink-sql-engine/pom.xml 24b23da2c [Paul Lin] [KYUUBI #1652] Update jaxb scope to test 240efae1a [Paul Lin] [KYUUBI #1652] Update jaxb scope to runtime 0e9a508b6 [Paul Lin] [KYUUBI #1652] Update jaxb scope to runtime b5dbd3346 [Paul Lin] [KYUUBI #1652] Fix jdk11 jaxb ClassNotFoundException 72ba3ee6d [Paul Lin] [KYUUBI #1652] Update tm memory to 1gb 4e10ea21f [Paul Lin] [KYUUBI #1652] Refactor flink engin tests e9cec4a65 [Paul Lin] [KYUUBI #1652] Add flink-it tests 6eb9fd3ad [Paul Lin] [KYUUBI #1652] Fix ProcessBuilder tests 6aca061e6 [Paul Lin] [KYUUBI #1652] Fix ClassNotFoundException 7581a2a0d [Paul Lin] [KYUUBI #1652] Fix missing minicluster 412c34571 [Paul Lin] [KYUUBI #1652] Remove flink-yarn dependencies 0eafbd7b0 [Paul Lin] Update externals/kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/result/ResultSet.scala ee2c64d04 [Paul Lin] [KYUUBI #1652] Add Flink YARN application tests a72627393 [Paul Lin] [KYUUBI #1652] Avoid flink-yarn dependencies a75cb2579 [Paul Lin] [KYUUBI #1652] Fix test issue b7e173f30 [Paul Lin] [KYUUBI #1652] Replace file-based Kyuubi conf with cli args 693ad6529 [Paul Lin] [KYUUBI #1652] Removed unused imports 68e0081e1 [Paul Lin] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/flink/FlinkProcessBuilder.scala ba021de9d [Paul Lin] [KYUUBI #1652] Search flink-sql.* jars and add them to pipeline jars 0846babbd [Paul Lin] [KYUUBI #1652] Avoid Scala bug 56413fe83 [Paul Lin] [KYUUBI #1652] Improve tmp files cleanup 8bdb672c4 [Paul Lin] [KYUUBI #1652] Explicitly load Kyuubi conf on Flink engine start 0b6325000 [Paul Lin] [KYUUBI #1652] Fix test failures 0ba03e439 [Paul Lin] [KYUUBI #1652] Fix wrong Flink args 00f036b04 [Paul Lin] [KYUUBI #1652] Remove unused util methods dfd2777ac [Paul Lin] [KYUUBI ##1652] Support Flink yarn application mode Authored-by: Paul Lin <paullin3280@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
He Zhao
|
015b800156
|
[KYUUBI #3650][UI] Add Operation Statistics Page
### _Why are the changes needed?_ Close #3650 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request  Closes #4663 from zwangsheng/KYUUBI_3650. Closes #3650 277e544e6 [Cheng Pan] Update kyuubi-server/web-ui/src/views/operation/operation-statistics/index.vue 688cfb44e [zwangsheng] i18n ed314d7c8 [zwangsheng] [KYUUBI #3650][UI] Add Operation Statistics Page c1965031a [zwangsheng] [KYUUBI #3650][UI] Add Operation Statistics Page 65779b878 [zwangsheng] [KYUUBI #3650][UI] Add Operation Statistics Page Lead-authored-by: He Zhao <hezhao2@cisco.com> Co-authored-by: zwangsheng <2213335496@qq.com> Co-authored-by: Cheng Pan <pan3793@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
zwangsheng
|
0c6ba94930
|
[KYUUBI #4622][UI] Session api use admin api for security
### _Why are the changes needed?_ Close #4622 Currently, we don't have a login for the UI, so we can't differentiate between users. So, we currently maintain a cautious attitude towards the kyuubi ui open API(especially the ability to delete resources) and choose `AdminResource`, which helps us filter non-Admin requests (although the current strategy is not perfect, it is a safe and quick step). Change api `api/v1/XXX` => `api/v1/admin/XXX` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4631 from zwangsheng/KYUUBI_4622. Closes #4622 3a611896f [zwangsheng] [KYUUBI #4622][UI] Session api use admin resource for security Authored-by: zwangsheng <2213335496@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
zwangsheng
|
d9e14f239d
|
[KYUUBI #4623][K8S] KubernetesApplicationOperation uses Informer instead of list
### _Why are the changes needed?_ Close #4623 To reduce the pressure on the Api Server (which use the kubernetes client polls with label to find the `spark driver pod` when multiple Rest Application are running at the same time), use informer, the kubernetes-recommended method of maintaining the application state. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request - [x] Run CI Closes #4625 from zwangsheng/KYUUBI_4623. Closes #4623 a415bef7f [Cheng Pan] nit 136d0db4d [Cheng Pan] 171 b5d3c237a [Cheng Pan] re-generate conf bf14ad870 [Cheng Pan] nit 9ee7e04f9 [Cheng Pan] nit 301162ea0 [Cheng Pan] nit 1d426922b [Cheng Pan] nit b95d7a650 [Cheng Pan] improve cc8d2c7f4 [zwangsheng] fix comments d017bafdf [zwangsheng] Set resycn 0 28f9a70d9 [zwangsheng] Reorder func & slow get app info 22d9c1662 [zwangsheng] fix setting 8e0940334 [zwangsheng] fix comments 10965d3df [zwangsheng] Rename fileter function => isSparkEnginePod b02677154 [zwangsheng] rename 78c9fdb17 [zwangsheng] fix comments 6d31f70d1 [zwangsheng] Fix IT Test f43bba2b9 [zwangsheng] fix 17e4f55eb [zwangsheng] debug be8da790e [zwangsheng] debug 0db45a513 [zwangsheng] retest a93786abc [zwangsheng] Fix style 652ee837e [zwangsheng] Add Setting & Debug 4add7e4e2 [zwangsheng] improve 1f4341237 [zwangsheng] remove unused import 35acd6106 [zwangsheng] fix compile 05dfc598e [zwangsheng] [KYUUBI #4623][Improvement][K8S] Remove cached app info when out of time 4ab530e99 [zwangsheng] [KYUUBI #4623][Improvement][K8S] kubernetesApplicationOperation Using Informer instead of list Lead-authored-by: zwangsheng <2213335496@qq.com> Co-authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
wangyepeng2
|
4c2f1e6abe
|
[KYUUBI #3872][BATCH] Skip setting proxy user when keytab is provided
### _Why are the changes needed?_ Fix #3872 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [X] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4640 from wangyepeng2/proxy_user_fix. Closes #3872 c7228b916 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala b6f41d1b5 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkBatchProcessBuilder.scala ecf3b3b18 [wangyepeng2] change function name to setupKerberos 63fc5deef [wangyepeng2] fix:spark batch submit should not set proxy user when keytab is used Lead-authored-by: wangyepeng2 <yepeng.wang@convertlab.com> Co-authored-by: Cheng Pan <pan3793@gmail.com> Co-authored-by: wangyepeng2 <44561824+wangyepeng2@users.noreply.github.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
yehere
|
ca93a4e5c2
|
[KYUUBI #4325] Support replace preparedStatement for Trino-jdbc
### _Why are the changes needed?_ close #4325 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4417 from yehere/kyuubi-4325. Closes #4325 7b2864b53 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc,update the dependency file, run './build/dependency.sh --replace' 749b1c15c [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc,Code optimization 33ea9ba2b [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc,Code optimization 568418a21 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc,Add test case for DEALLOCATE PREPARE 358a8e3b8 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc,Support DEALLOCATE PREPARE 7d4a32402 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc 91392add6 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc 63bf8c462 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc f5b7fb786 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc b0476a79d [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc 1a8f147a0 [yehere] [KYUUBI #4325] Support replace preparedStatement for Trino-jdbc Authored-by: yehere <867171931@qq.com> Signed-off-by: ulyssesyou <ulyssesyou@apache.org> |
||
|
fwang12
|
0fdf145e29 |
[KYUUBI #4639] Support to specify confOverlay when executing statement with RESTful API
### _Why are the changes needed?_ As title. With this pr, customer can execute SCALA code with `confOverlay`. ``` kyuubi.operation.language=SCALA ``` execute PYTHON code with ``` kyuubi.operation.language=PYTHON ``` ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4639 from turboFei/init_scala. Closes #4639 cdf828f9a [fwang12] add ut f4f2bc883 [fwang12] doc 2fe8a1659 [fwang12] save f840cb4d9 [fwang12] conf overlay eb49537ea [fwang12] conf overlay Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
Tianlin Liao
|
3815bc4a71 |
[KYUUBI #4611] support all sessions api in SessionRestApi
### _Why are the changes needed?_ Close #4611 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4612 from lightning-L/kyuubi-4602. Closes #4611 9ad95c6ad [Tianlin Liao] fix 0b3792e3e [Tianlin Liao] fix 7db90d570 [Tianlin Liao] [KYUUBI #4611] support all sessions api in SessionRestApi Authored-by: Tianlin Liao <tiliao@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
zwangsheng
|
f3986d2d67
|
[KYUUBI #4618][REST] Admin Resource list operations with sessionHandle filter
### _Why are the changes needed?_ Close #4618 `api/v1/admin/operations` add queryParams sessionHandle ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4621 from zwangsheng/KYUUBI_4618. Closes #4618 1f1e40213 [zwangsheng] [KYUUBI #4618][REST] Admin Resource list operations with sessionhandle filter Authored-by: zwangsheng <2213335496@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Tianlin Liao
|
6fd3353c78 |
[KYUUBI #4609] get engineRefId in KyuubiConnection
### _Why are the changes needed?_ Close #4609 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4610 from lightning-L/kyuubi-4609. Closes #4609 c40fd71ed [Tianlin Liao] fix test case 8f2d96cf0 [Tianlin Liao] [KYUUBI #4609] get engineRefId in KyuubiConnection Authored-by: Tianlin Liao <tiliao@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
Xieming LI
|
cee2f00f91 |
[KYUUBI #4480] Engine alive probe should close thrift connection on engine lost
### _Why are the changes needed?_ As described in https://github.com/apache/kyuubi/issues/4457, when the backend engine becomes unresponsive, the beeline will hang because `client.getOperationStatus(_remoteOpHandle)` in `ExecuteStatement #waitStatementComplete()` would never receive any response. |
||
|
He Zhao
|
0d5eaa2d0b
|
[KYUUBI #3646][UI] Init Session Statistic Page
### _Why are the changes needed?_ Init Session Statistic Page Close #3646 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request  Closes #4564 from zwangsheng/KYUUBI_3646. Closes #3646 cbe0842ba [zwangsheng] [KYUUBI #3646] fix style 930cbb12a [zwangsheng] [KYUUBI #3646] bracket same line d2ab1d5fd [zwangsheng] [KYUUBI #3646] Fix i18n about status 7f059e2df [zwangsheng] [KYUUBI #3646] Fix i18n about status 452c3ee5a [zwangsheng] [KYUUBI #3646] Remove unused style class 8967652bc [zwangsheng] [KYUUBI #3646] Add date fns license 70f2472c2 [zwangsheng] [KYUUBI #3646] install date-fns 8a3e845ff [zwangsheng] [KYUUBI #3646][UI] Init Session Statistic Page Lead-authored-by: He Zhao <hezhao2@cisco.com> Co-authored-by: zwangsheng <2213335496@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
fwang12
|
351bab3676 |
[KYUUBI #4584] Post the session exception to EventBus in time
### _Why are the changes needed?_ When session exception thrown, post it to event bus in time. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4584 from turboFei/handle_post. Closes #4584 920ee29a4 [fwang12] post exception in time Authored-by: fwang12 <fwang12@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
odone
|
f9701b03d7 |
[KYUUBI #4579] Fix the unstable situation of test cases for Trino-Fronted-Service
### _Why are the changes needed?_ In the cancellation implementation of Trino frontend service, closing the operation will also close the session. There is no need to check the status of the operation here. ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4579 from iodone/trino-fronted-test-fix. Closes #4579 75e02dd1 [odone] Fix the unstable situation of test cases. Authored-by: odone <odone.zhang@gmail.com> Signed-off-by: ulyssesyou <ulyssesyou@apache.org> |
||
|
Tianlin Liao
|
141919308e |
[KYUUBI #4523] support close engine session gracefully
### _Why are the changes needed?_ ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4545 from lightning-L/kyuubi-4523. Closes #4523 2ee634cd5 [Tianlin Liao] modify conf description; add test case 12b930456 [Tianlin Liao] refer to HIVE-13415 c8a9db6a2 [Tianlin Liao] minor fix 9942d2515 [Tianlin Liao] move logic to SessionManager.closeSession 62b990b3e [Tianlin Liao] [KYUUBI #4523] support close engine session gracefully Authored-by: Tianlin Liao <tiliao@ebay.com> Signed-off-by: fwang12 <fwang12@ebay.com> |
||
|
zwangsheng
|
824aba6070
|
[KYUUBI #3646][Improvement][UI] Front-end style should bracket same line
### _Why are the changes needed?_ Should bracket same line. According to https://github.com/npetruzzelli/prettier-config-standard, should set true to bracketSameLine and also drop jsxBracketSameLine, which been deprecated. Close #3646 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request - [x] CI Closes #4571 from zwangsheng/KYUUBI_4570. Closes #3646 2b61d1173 [zwangsheng] [KYUUBI #3646] bracket same line Authored-by: zwangsheng <2213335496@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
