84928184fc
82 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
wangzhigang
|
84928184fc
|
[KYUUBI #7121] Improve operation timeout management with configurable executors
### Why are the changes needed? The current mechanism for handling operation timeouts in Kyuubi creates a new `ScheduledExecutorService` with a dedicated thread for each operation. In scenarios with a large number of concurrent operations, this results in excessive thread creation, which consumes substantial system resources and may adversely affect server performance and stability. This PR introduces a shared `ScheduledThreadPool` within the Operation Manager to centrally schedule operation timeouts. This approach avoids the overhead of creating an excessive number of threads, thereby reducing the system load. Additionally, both the pool size and thread keep-alive time are configurable via the `OPERATION_TIMEOUT_POOL_SIZE` and `OPERATION_TIMEOUT_POOL_KEEPALIVE_TIME` parameters. ### How was this patch tested? A new unit test for `newDaemonScheduledThreadPool` was added to `ThreadUtilsSuite.scala`. Furthermore, a dedicated `TimeoutSchedulerSuite` was introduced to verify operation timeout behavior. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #7121 from wangzhigang1999/master. Closes #7121 df7688dbf [wangzhigang] Refactor timeout management configuration and improve documentation 2b03b1e68 [wangzhigang] Remove deprecated `ThreadPoolTimeoutExecutor` class following refactor of operation timeout management. 52a8a516a [wangzhigang] Refactor operation timeout management to use per-OperationManager scheduler 7e46d47f8 [wangzhigang] Refactor timeout management by introducing ThreadPoolTimeoutExecutor f7f10881a [wangzhigang] Add operation timeout management with ThreadPoolTimeoutExecutor d8cd6c7d4 [wangzhigang] Update .gitignore to exclude .bloop and .metals directories Lead-authored-by: wangzhigang <wangzhigang1999@live.cn> Co-authored-by: wangzhigang <wzg443064@alibaba-inc.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
taylor.fan
|
127c736a8f
|
[KYUUBI #6926] Add SERVER_LOCAL engine share level
### Why are the changes needed? As clarified in https://github.com/apache/kyuubi/issues/6926, there are some scenarios user want to launch engine on each kyuubi server. SERVER_LOCAL engine share level implement this function by extracting local host address as subdomain, in which case each kyuubi server's engine is unique. ### How was this patch tested? ### Was this patch authored or co-authored using generative AI tooling? No Closes #7013 from taylor12805/share_level_server_local. Closes #6926 ba201bb72 [taylor.fan] [KYUUBI #6926] update format 42f0a4f7d [taylor.fan] [KYUUBI #6926] move host address to subdomain e06de79ad [taylor.fan] [KYUUBI #6926] Add SERVER_LOCAL engine share level Authored-by: taylor.fan <taylor.fan@vipshop.com> Signed-off-by: Kent Yao <yao@apache.org> |
||
|
Wang, Fei
|
29b6076319 |
[KYUUBI #7043] Support to construct the batch info from metadata directly
### Why are the changes needed? Add an option to allow construct the batch info from metadata directly instead of redirecting the requests to reduce the RPC latency. ### How was this patch tested? Minor change and Existing GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #7043 from turboFei/support_no_redirect. Closes #7043 7f7a2fb80 [Wang, Fei] comments bb0e324a1 [Wang, Fei] save Authored-by: Wang, Fei <fwang12@ebay.com> Signed-off-by: Wang, Fei <fwang12@ebay.com> |
||
|
Wang, Fei
|
4fc201e85d |
[KYUUBI #7027] Support to initialize kubernetes clients on kyuubi server startup
### Why are the changes needed? This ensure the Kyuubi server is promptly informed for any Kubernetes resource changes after startup. It is highly recommend to set it for multiple Kyuubi instances mode. ### How was this patch tested? Existing GA and Integration testing. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #7027 from turboFei/k8s_client_init. Closes #7027 393b9960a [Wang, Fei] server only a640278c4 [Wang, Fei] refresh Authored-by: Wang, Fei <fwang12@ebay.com> Signed-off-by: Wang, Fei <fwang12@ebay.com> |
||
|
Cheng Pan
|
fff1841054
|
[KYUUBI #6876] Support rolling spark.kubernetes.file.upload.path
### Why are the changes needed?
The vanilla Spark neither support rolling nor expiration mechanism for `spark.kubernetes.file.upload.path`, if you use file system that does not support TTL, e.g. HDFS, additional cleanup mechanisms are needed to prevent the files in this directory from growing indefinitely.
This PR proposes to let `spark.kubernetes.file.upload.path` support placeholders `{{YEAR}}`, `{{MONTH}}` and `{{DAY}}` and introduce a switch `kyuubi.kubernetes.spark.autoCreateFileUploadPath.enabled` to let Kyuubi server create the directory with 777 permission automatically before submitting Spark application.
For example, the user can configure the below configurations in `kyuubi-defaults.conf` to enable monthly rolling support for `spark.kubernetes.file.upload.path`
```
kyuubi.kubernetes.spark.autoCreateFileUploadPath.enabled=true
spark.kubernetes.file.upload.path=hdfs://hadoop-cluster/spark-upload-{{YEAR}}{{MONTH}}
```
Note that: spark would create sub dir `s"spark-upload-${UUID.randomUUID()}"` under the `spark.kubernetes.file.upload.path` for each uploading, the administer still needs to clean up the staging directory periodically.
For example:
```
hdfs://hadoop-cluster/spark-upload-202412/spark-upload-f2b71340-dc1d-4940-89e2-c5fc31614eb4
hdfs://hadoop-cluster/spark-upload-202412/spark-upload-173a8653-4d3e-48c0-b8ab-b7f92ae582d6
hdfs://hadoop-cluster/spark-upload-202501/spark-upload-3b22710f-a4a0-40bb-a3a8-16e481038a63
```
Administer can safely delete the `hdfs://hadoop-cluster/spark-upload-202412` after 20250101
### How was this patch tested?
New UTs are added.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #6876 from pan3793/rolling-upload.
Closes #6876
6614bf29c [Cheng Pan] comment
5d5cb3eb3 [Cheng Pan] docs
343adaefb [Cheng Pan] review
3eade8bc4 [Cheng Pan] fix
706989778 [Cheng Pan] docs
38953dc3f [Cheng Pan] Support rolling spark.kubernetes.file.upload.path
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
|
||
|
Wang, Fei
|
aa33521cf7
|
[KYUUBI #6864] Support to return prometheus metrics with instance label
### Why are the changes needed? For my use case, the instances are not human readable, so I prefer to return the FQDN. <img width="1483" alt="image" src="https://github.com/user-attachments/assets/92045517-456f-4087-8a36-9e3e4bea2f1d" /> ### How was this patch tested? Integration testing. ``` (base) ➜ dist git:(prometheus_label_2) cat conf/kyuubi-defaults.conf kyuubi.metrics.prometheus.metrics.instance.enabled=true kyuubi.zookeeper.embedded.client.port.address=localhost kyuubi.frontend.bind.host=localhost ``` <img width="1692" alt="image" src="https://github.com/user-attachments/assets/0b60d504-62ec-418d-880b-f8a2f00d5550" /> ### Was this patch authored or co-authored using generative AI tooling? No. Closes #6864 from turboFei/prometheus_label_2. Closes #6864 d24571ccb [Wang, Fei] match 6a6a5110b [Wang, Fei] comments c3046d4a1 [Wang, Fei] save fb2021a31 [Wang, Fei] revert 42395945e [Wang, Fei] compatible 17b7007f5 [Wang, Fei] add instance label Authored-by: Wang, Fei <fwang12@ebay.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Wang, Fei
|
3167692732
|
[KYUUBI #6829] Add metrics for batch pending max elapse time
### Why are the changes needed?
1. add metrics `kyuubi.operartion.batch_pending_max_elapse` for the batch pending max elapse time, which is helpful for batch health monitoring, and we can send alert if the batch pending elapse time too long
2. For `GET /api/v1/batches` api, limit the max time window for listing batches, which is helpful that, we want to reserve more metadata in kyuubi server end, for example: 90 days, but for list batches, we just want to allow user to search the last 7 days. It is optional. And if `create_time` is specified, order by `create_time` instead of `key_id`.
|
||
|
naive-zhang
|
eb1b5996c9
|
[KYUUBI #6815] JDBC Engine supports Oracle
# Description Currently, Kyuubi supports JDBC engines with limited dialects, and I extend the dialects to support Oracle. * Introduce Oracle support in JDBC Engine * Adding dialects and tests for Oracle ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Add tests of `OperationWithOracleEngineSuite`, `OracleOperationSuite`, `OracleSessionSuite` and `OracleStatementSuite`. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6815 from naive-zhang/jdbc-oracle. Closes #6815 0ffad5b6b [native-zhang] add some brief comments on the caller side for the implementation of Oracle JDBC engine 6f469a135 [naive-zhang] Merge branch 'apache:master' into jdbc-oracle ae70710e6 [Cheng Pan] Update externals/kyuubi-jdbc-engine/src/main/scala/org/apache/kyuubi/engine/jdbc/dialect/OracleSQLDialect.scala 171d06b9e [native-zhang] use another implementation of transform decimal into int, in engine instead of KyuubiBaseResultSet 7cb74d28e [naive-zhang] Merge branch 'apache:master' into jdbc-oracle ccd7cae8b [naive-zhang] remove redundant override methods in OracleSQLDialect.scala a7da4a646 [naive-zhang] remove redundant impl of getTableTypesOperation in OracleSQLDialect.scala 70b49fcba [naive-zhang] Use the single line string if SQL fits in one line, otherwise write it in a pretty style e58348460 [naive-zhang] Update externals/kyuubi-jdbc-engine/src/main/scala/org/apache/kyuubi/engine/jdbc/dialect/OracleSQLDialect.scala b33e97a08 [naive-zhang] remove redundant testcontainers-scala-oracle-xe dependency in pom.xml 4c967b98e [naive-zhang] use gvenzl/oracle-free:23.5-slim with docker-compose for test case 0215e6d49 [naive-zhang] Merge branch 'apache:master' into jdbc-oracle d688b4706 [naive-zhang] change oracle image into gvenzl/oracle-free:23.5-slim abf983727 [naive-zhang] fix code style checking error in KyuubiConf.scala d1e82edb1 [naive-zhang] fix code style checking error in settings.md aa2e2e9ba [naive-zhang] adjust wired space in OracleSQLDialect b43cea421 [naive-zhang] add oracle configuration for kyuubi.engine.jdbc.connection.provider 397c1cfec [naive-zhang] Merge branch 'apache:master' into jdbc-oracle 2f1b5ed0b [naive-zhang] add jdbc support for Oracle Lead-authored-by: naive-zhang <xinsen.zhang.0571@gmail.com> Co-authored-by: native-zhang <xinsen.zhang.0571@gmail.com> Co-authored-by: Cheng Pan <pan3793@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
wforget
|
1e9d68b000 |
[KYUUBI #6368] Flink engine supports user impersonation
# 🔍 Description ## Issue References 🔗 This pull request fixes #6368 ## Describe Your Solution 🔧 Support impersonation mode for flink sql engine. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [X] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 Test in hadoop-testing env. Connection: ``` beeline -u "jdbc:hive2://hadoop-master1.orb.local:10009/default;hive.server2.proxy.user=spark;principal=kyuubi/_HOSTTEST.ORG?kyuubi.engine.type=FLINK_SQL;flink.execution.target=yarn-application;kyuubi.engine.share.level=CONNECTION;kyuubi.engine.flink.doAs.enabled=true;" ``` sql: ``` select 1; ``` result:  launch engine command: ``` 2024-06-12 03:22:10.242 INFO KyuubiSessionManager-exec-pool: Thread-62 org.apache.kyuubi.engine.EngineRef: Launching engine: /opt/flink-1.18.1/bin/flink run-application \ -t yarn-application \ -Dyarn.ship-files=/opt/flink/opt/flink-sql-client-1.18.1.jar;/opt/flink/opt/flink-sql-gateway-1.18.1.jar;/etc/hive/conf/hive-site.xml \ -Dyarn.application.name=kyuubi_CONNECTION_FLINK_SQL_spark_6170b9aa-c690-4b50-938f-d59cca9aa2d6 \ -Dyarn.tags=KYUUBI,6170b9aa-c690-4b50-938f-d59cca9aa2d6 \ -Dcontainerized.master.env.FLINK_CONF_DIR=. \ -Dcontainerized.master.env.HIVE_CONF_DIR=. \ -Dyarn.security.appmaster.delegation.token.services=kyuubi \ -Dsecurity.delegation.token.provider.HiveServer2.enabled=false \ -Dsecurity.delegation.token.provider.hbase.enabled=false \ -Dexecution.target=yarn-application \ -Dsecurity.module.factory.classes=org.apache.flink.runtime.security.modules.JaasModuleFactory;org.apache.flink.runtime.security.modules.ZookeeperModuleFa ctory \ -Dsecurity.delegation.token.provider.hadoopfs.enabled=false \ -c org.apache.kyuubi.engine.flink.FlinkSQLEngine /opt/apache-kyuubi-1.10.0-SNAPSHOT-bin/externals/engines/flink/kyuubi-flink-sql-engine_2.12-1.10.0-SNAPS HOT.jar \ --conf kyuubi.session.user=spark \ --conf kyuubi.client.ipAddress=172.20.0.5 \ --conf kyuubi.engine.credentials=SERUUwACJnRocmlmdDovL2hhZG9vcC1tYXN0ZXIxLm9yYi5sb2NhbDo5MDgzRQAFc3BhcmsEaGl2ZShreXV1YmkvaGFkb29wLW1hc3RlcjEub3JiLmxvY2Fs QFRFU1QuT1JHigGQCneevIoBkC6EIrwWDxSg03pnAB8dA295wh+Dim7Fx4FNxhVISVZFX0RFTEVHQVRJT05fVE9LRU4ADzE3Mi4yMC4wLjU6ODAyMEEABXNwYXJrAChreXV1YmkvaGFkb29wLW1hc3RlcjEub3JiL mxvY2FsQFRFU1QuT1JHigGQCneekIoBkC6EIpBHHBSket0SQnlXT5EIMN0U2fUKFRIVvBVIREZTX0RFTEVHQVRJT05fVE9LRU4PMTcyLjIwLjAuNTo4MDIwAA== \ --conf kyuubi.engine.flink.doAs.enabled=true \ --conf kyuubi.engine.hive.extra.classpath=/opt/hadoop/share/hadoop/client/*:/opt/hadoop/share/hadoop/mapreduce/* \ --conf kyuubi.engine.share.level=CONNECTION \ --conf kyuubi.engine.submit.time=1718162530017 \ --conf kyuubi.engine.type=FLINK_SQL \ --conf kyuubi.frontend.protocols=THRIFT_BINARY,REST \ --conf kyuubi.ha.addresses=hadoop-master1.orb.local:2181 \ --conf kyuubi.ha.engine.ref.id=6170b9aa-c690-4b50-938f-d59cca9aa2d6 \ --conf kyuubi.ha.namespace=/kyuubi_1.10.0-SNAPSHOT_CONNECTION_FLINK_SQL/spark/6170b9aa-c690-4b50-938f-d59cca9aa2d6 \ --conf kyuubi.server.ipAddress=172.20.0.5 \ --conf kyuubi.session.connection.url=hadoop-master1.orb.local:10009 \ --conf kyuubi.session.engine.startup.waitCompletion=false \ --conf kyuubi.session.real.user=spark ``` launch engine log:  jobmanager job: ``` 2024-06-12 03:22:26,400 INFO org.apache.flink.runtime.security.token.DefaultDelegationTokenManager [] - Loading delegation token providers 2024-06-12 03:22:26,992 INFO org.apache.kyuubi.engine.flink.security.token.KyuubiDelegationTokenProvider [] - Renew delegation token with engine credentials: SERUUwACJnRocmlmdDovL2hhZG9vcC1tYXN0ZXIxLm9yYi5sb2NhbDo5MDgzRQAFc3BhcmsEaGl2ZShreXV1YmkvaGFkb29wLW1hc3RlcjEub3JiLmxvY2FsQFRFU1QuT1JHigGQCneevIoBkC6EIrwWDxSg03pnAB8dA295wh+Dim7Fx4FNxhVISVZFX0RFTEVHQVRJT05fVE9LRU4ADzE3Mi4yMC4wLjU6ODAyMEEABXNwYXJrAChreXV1YmkvaGFkb29wLW1hc3RlcjEub3JiLmxvY2FsQFRFU1QuT1JHigGQCneekIoBkC6EIpBHHBSket0SQnlXT5EIMN0U2fUKFRIVvBVIREZTX0RFTEVHQVRJT05fVE9LRU4PMTcyLjIwLjAuNTo4MDIwAA== 2024-06-12 03:22:27,100 INFO org.apache.kyuubi.engine.flink.FlinkEngineUtils [] - Add new unknown token Kind: HIVE_DELEGATION_TOKEN, Service: , Ident: 00 05 73 70 61 72 6b 04 68 69 76 65 28 6b 79 75 75 62 69 2f 68 61 64 6f 6f 70 2d 6d 61 73 74 65 72 31 2e 6f 72 62 2e 6c 6f 63 61 6c 40 54 45 53 54 2e 4f 52 47 8a 01 90 0a 77 9e bc 8a 01 90 2e 84 22 bc 16 0f 2024-06-12 03:22:27,104 WARN org.apache.kyuubi.engine.flink.FlinkEngineUtils [] - Ignore token with earlier issue date: Kind: HDFS_DELEGATION_TOKEN, Service: 172.20.0.5:8020, Ident: (token for spark: HDFS_DELEGATION_TOKEN owner=spark, renewer=, realUser=kyuubi/hadoop-master1.orb.localTEST.ORG, issueDate=1718162529936, maxDate=1718767329936, sequenceNumber=71, masterKeyId=28) 2024-06-12 03:22:27,104 INFO org.apache.kyuubi.engine.flink.FlinkEngineUtils [] - Update delegation tokens. The number of tokens sent by the server is 2. The actual number of updated tokens is 1. ...... 4-06-12 03:22:29,414 INFO org.apache.flink.runtime.security.token.DefaultDelegationTokenManager [] - Starting tokens update task 2024-06-12 03:22:29,415 INFO org.apache.flink.runtime.security.token.DelegationTokenReceiverRepository [] - New delegation tokens arrived, sending them to receivers 2024-06-12 03:22:29,422 INFO org.apache.kyuubi.engine.flink.security.token.KyuubiDelegationTokenReceiver [] - Updating delegation tokens for current user 2024-06-12 03:22:29,422 INFO org.apache.kyuubi.engine.flink.security.token.KyuubiDelegationTokenReceiver [] - Token Service: Identifier:[10, 13, 10, 9, 8, 10, 16, -78, -36, -49, -17, -5, 49, 16, 1, 16, -100, -112, -60, -127, -8, -1, -1, -1, -1, 1] 2024-06-12 03:22:29,422 INFO org.apache.kyuubi.engine.flink.security.token.KyuubiDelegationTokenReceiver [] - Token Service: Identifier:[0, 5, 115, 112, 97, 114, 107, 4, 104, 105, 118, 101, 40, 107, 121, 117, 117, 98, 105, 47, 104, 97, 100, 111, 111, 112, 45, 109, 97, 115, 116, 101, 114, 49, 46, 111, 114, 98, 46, 108, 111, 99, 97, 108, 64, 84, 69, 83, 84, 46, 79, 82, 71, -118, 1, -112, 10, 119, -98, -68, -118, 1, -112, 46, -124, 34, -68, 22, 15] 2024-06-12 03:22:29,422 INFO org.apache.kyuubi.engine.flink.security.token.KyuubiDelegationTokenReceiver [] - Token Service:172.20.0.5:8020 Identifier:[0, 5, 115, 112, 97, 114, 107, 0, 40, 107, 121, 117, 117, 98, 105, 47, 104, 97, 100, 111, 111, 112, 45, 109, 97, 115, 116, 101, 114, 49, 46, 111, 114, 98, 46, 108, 111, 99, 97, 108, 64, 84, 69, 83, 84, 46, 79, 82, 71, -118, 1, -112, 10, 119, -98, -112, -118, 1, -112, 46, -124, 34, -112, 71, 28] 2024-06-12 03:22:29,422 INFO org.apache.kyuubi.engine.flink.security.token.KyuubiDelegationTokenReceiver [] - Updated delegation tokens for current user successfully ``` taskmanager log: ``` 2024-06-12 03:45:06,622 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Receive initial delegation tokens from resource manager 2024-06-12 03:45:06,627 INFO org.apache.flink.runtime.security.token.DelegationTokenReceiverRepository [] - New delegation tokens arrived, sending them to receivers 2024-06-12 03:45:06,628 INFO org.apache.kyuubi.engine.flink.security.token.KyuubiDelegationTokenReceiver [] - Updating delegation tokens for current user 2024-06-12 03:45:06,629 INFO org.apache.kyuubi.engine.flink.security.token.KyuubiDelegationTokenReceiver [] - Token Service: Identifier:[10, 13, 10, 9, 8, 10, 16, -78, -36, -49, -17, -5, 49, 16, 1, 16, -100, -112, -60, -127, -8, -1, -1, -1, -1, 1] 2024-06-12 03:45:06,630 INFO org.apache.kyuubi.engine.flink.security.token.KyuubiDelegationTokenReceiver [] - Token Service: Identifier:[0, 5, 115, 112, 97, 114, 107, 4, 104, 105, 118, 101, 40, 107, 121, 117, 117, 98, 105, 47, 104, 97, 100, 111, 111, 112, 45, 109, 97, 115, 116, 101, 114, 49, 46, 111, 114, 98, 46, 108, 111, 99, 97, 108, 64, 84, 69, 83, 84, 46, 79, 82, 71, -118, 1, -112, 10, 119, -98, -68, -118, 1, -112, 46, -124, 34, -68, 22, 15] 2024-06-12 03:45:06,630 INFO org.apache.kyuubi.engine.flink.security.token.KyuubiDelegationTokenReceiver [] - Token Service:172.20.0.5:8020 Identifier:[0, 5, 115, 112, 97, 114, 107, 0, 40, 107, 121, 117, 117, 98, 105, 47, 104, 97, 100, 111, 111, 112, 45, 109, 97, 115, 116, 101, 114, 49, 46, 111, 114, 98, 46, 108, 111, 99, 97, 108, 64, 84, 69, 83, 84, 46, 79, 82, 71, -118, 1, -112, 10, 119, -98, -112, -118, 1, -112, 46, -124, 34, -112, 71, 28] 2024-06-12 03:45:06,636 INFO org.apache.kyuubi.engine.flink.security.token.KyuubiDelegationTokenReceiver [] - Updated delegation tokens for current user successfully 2024-06-12 03:45:06,636 INFO org.apache.flink.runtime.security.token.DelegationTokenReceiverRepository [] - Delegation tokens sent to receivers ``` #### Related Unit Tests --- # Checklist 📝 - [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6383 from wForget/KYUUBI-6368. Closes #6368 47df43ef0 [wforget] remove doAsEnabled 984b96c74 [wforget] update settings.md c7f8d474e [wforget] make generateTokenFile conf to internal 8632176b1 [wforget] address comments 2ec270e8a [wforget] licenses ed0e22f4e [wforget] separate kyuubi-flink-token-provider module b66b855b6 [wforget] address comment d4fc2bd1d [wforget] fix 1a3dc4643 [wforget] fix style 825e2a7a0 [wforget] address comments a679ba1c2 [wforget] revert remove renewer cdd499b95 [wforget] fix and comment 19caec6c0 [wforget] pass token to submit process b2991d419 [wforget] fix 7c3bdde1b [wforget] remove security.delegation.tokens.enabled check 8987c9176 [wforget] fix 5bd8cfe7c [wforget] fix 08992642d [wforget] Implement KyuubiDelegationToken Provider/Receiver fa16d7def [wforget] enable delegation token manager e50db7497 [wforget] [KYUUBI #6368] Support impersonation mode for flink sql engine Authored-by: wforget <643348094@qq.com> Signed-off-by: Bowen Liang <liangbowen@gf.com.cn> |
||
|
Bowen Liang
|
fb65a12936 |
[KYUUBI #6756] [REST] Check max file size of uploaded resource and extra resources in batch creation
# 🔍 Description ## Issue References 🔗 This pull request fixes # ## Describe Your Solution 🔧 Check the uploaded resource files when creating batch via REST API - add config `kyuubi.batch.resource.file.max.size` for resource file's max size in bytes - add config `kyuubi.batch.extra.resource.file.max.size` for each extra resource file's max size in bytes ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6756 from bowenliang123/resource-maxsize. Closes #6756 5c409c425 [Bowen Liang] nit 4b16bcfc4 [Bowen Liang] nit 743920d25 [Bowen Liang] check resource file size max size Authored-by: Bowen Liang <liangbowen@gf.com.cn> Signed-off-by: Bowen Liang <liangbowen@gf.com.cn> |
||
|
taylor.fan
|
851fb5ae5c
|
[KYUUBI #6704] Disable periodic gc if set interval to 0
# 🔍 Description ## Issue References 🔗 This pull request fixes https://github.com/apache/kyuubi/issues/6704 ## Describe Your Solution 🔧 if periodic gc is set to 0, there is no need to perform an explicit gc. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [x] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6725 from taylor12805/master. Closes #6704 a52ddda62 [Bowen Liang] update doc b84a32f35 [Bowen Liang] make periodic gc thead pool lazy 2d4bd7c05 [Bowen Liang] update doc in spark style 3e04604b0 [taylor.fan] [KYUUBI #6704] disable periodic gc if set interval to 0 bf20b134b [taylor.fan] [KYUUBI #6704] disable periodic gc if set interval to 0 c2b7c3078 [taylor.fan] [KYUUBI #6704] disable periodic gc if set interval to 0 6182075fc [taylor.fan] [KYUUBI #6704] disable periodic gc if set interval to 0 52b1c078b [taylor.fan] [KYUUBI #6704] disable periodic gc if set interval to 0 ccf19cf24 [taylor.fan] [KYUUBI #6704] disable periodic gc if set interval to 0 affd67c88 [taylor.fan] [KYUUBI #6704] disable periodic gc if set interval to 0 d4ee164d1 [taylor.fan] disable periodic gc if set interval to 0 Lead-authored-by: taylor.fan <taylor.fan@vipshop.com> Co-authored-by: Bowen Liang <liangbowen@gf.com.cn> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Wang, Fei
|
f8431da7ac |
[KYUUBI #6686] Ignore Spark pod container state if pod is terminated
# 🔍 Description ## Issue References 🔗 To close #6686  The pod already in failed state, and the driver container is in waiting state. We shall mark the application terminated and ignore the container state. ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6690 from turboFei/pod_state. Closes #6686 0d4c8a255 [Wang, Fei] comments d60b901c1 [Wang, Fei] check pod terminated Authored-by: Wang, Fei <fwang12@ebay.com> Signed-off-by: Wang, Fei <fwang12@ebay.com> |
||
|
Bowen Liang
|
bef3d5590f |
[KYUUBI #6645] Size based eviction for server-side temp files cleanup
# 🔍 Description ## Issue References 🔗 This pull request fixes # ## Describe Your Solution 🔧 - adding `maximumSize` to support size based eviction for server-side temp files cleanup in `TempFileService` - size-based eviction is disabled by default , with `maximumSize` set to optional by default - time-based eviction time is now extended from 14 days to 30 days by default ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6645 from bowenliang123/temp-file-size-evict. Closes #6645 e1f166b6a [liangbowen] docs 0b2d5aa6e [liangbowen] increase default SERVER_TEMP_FILE_EXPIRE_TIME to 30 days ee87da56a [liangbowen] make config optional 0607efcd7 [Bowen Liang] import 9cc777660 [liangbowen] update f9e4de00e [Bowen Liang] docs 55bf238d3 [liangbowen] size Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn> Co-authored-by: liangbowen <liangbowen@gf.com.cn> Signed-off-by: Bowen Liang <liangbowen@gf.com.cn> |
||
|
Bowen Liang
|
db57e9365d |
[KYUUBI #6587] Periodically expire temp files and operation logs on server to avoid memeory leak by Files.deleteOnExit
# 🔍 Description ## Issue References 🔗 - ## Describe Your Solution 🔧 Fix the memory leak on server caused by `Files.deleteOnExit`. For long-running Kyuubi server instances, some operation log files and batch job upload files are marked for deletion at exit using `Files.deleteOnExit`. However, the `files` list within the `DeleteOnExitHook` by `Files.deleteOnExit` method continuously accumulates file paths without being cleaned up, leading to a memory leak issue. This PR fix this issue by: 1. introduce a new util `FileExpirationUtils` for similar use of `Files.deleteOnExit`, with exposed method for evict file path from the list to prevent accumulative path list 2. adding a service `TempFileService ` in server module, periodical clean-up the files for operation logging path, uploaded resources and etc. And it evict the paths in `TempFileCleanupUtils` instance after cleanup. 3. add the new config `kyuubi.server.tempFile.expireTime` with a default value of 7 days, to control How often to trigger a file expiration clean-up for stale files ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6587 from bowenliang123/file-expiration. Closes #6587 e23b72e08 [liangbowen] change to P14D acaf370e7 [liangbowen] change config name to kyuubi.server.tempFile.expireTime 6c7ddd527 [liangbowen] import ed1e4d76f [liangbowen] comment: ConcurrentHashMap.newKeySet fbf73ccb4 [liangbowen] update 34d3fc71c [liangbowen] add guava to common module's dep 49c10e5ef [Bowen Liang] file expiration Lead-authored-by: Bowen Liang <liangbowen@gf.com.cn> Co-authored-by: liangbowen <liangbowen@gf.com.cn> Co-authored-by: Bowen Liang <liangbowen@gf.com.cn> Signed-off-by: liangbowen <liangbowen@gf.com.cn> |
||
|
Cheng Pan
|
11de72f117
|
[KYUUBI #6594] Port HIVE-26633: Make thrift client maxMessageSize configurable
# 🔍 Description Fix #6594. This PR ports HIVE-26633(https://github.com/apache/hive/pull/3674): Make thrift client maxMessageSize configurable to fix a regression after upgrading Thrift 0.16 in 1.9.0. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6631 from pan3793/thrift-max-size. Closes #6594 e4841c88e [Cheng Pan] [KYUUBI #6594] Port HIVE-26633: Make thrift client maxMessageSize configurable Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
George314159
|
a4390a785a |
[KYUUBI #6618] Support http bearer token authentication for REST protocol
# 🔍 Description ## Issue References 🔗 This pull request fixes #6618 ## Describe Your Solution 🔧 It is a subtask of #6590 This PR is to support http bearer token authentication for REST protocol. In addition to BasicAuthenticationHandler, BearerAuthenticationHandler will be added to handle http bear token authentication. They will both support CUSTOM AuthType. In order to distinguish them, two new configurations are added: kyuubi.authentication.custom.basic.class and kyuubi.authentication.custom.bearer.class. For http bear token custom authentication, users could implement the new 'org.apache.kyuubi.service.authentication.TokenAuthenticationProvider', and specify it in the configuration. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6608 from George314159/authentication. Closes #6618 d07a30f83 [Wang, Fei] fix UT 6499c9986 [George314159] Update Test Case da519a9c6 [George314159] Update based on comments f47160148 [Wang, Fei] Refine UT 544422399 [George314159] Add test suite for custom authentication f2bbfbf7e [Wang, Fei] comments & refine a733c0e8f [George314159] Remove unused val 6f669d46c [George314159] Fix 650b88d4e [George314159] Update based on comments 5bc2bac58 [George314159] Update based on comments 1893889db [George314159] Update based on Comments ddee882e9 [George314159] Fix Style 379a563fa [George314159] Support http bearer token authentication Lead-authored-by: George314159 <hua16732@gmail.com> Co-authored-by: Wang, Fei <fwang12@ebay.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
zhang_yao
|
7c20e697ba |
[KYUUBI #6615] Make Jetty sending server version in response configurable
# 🔍 Description ## Issue References 🔗 This pull request fixes #6615 ## Describe Your Solution 🔧 Add a config item that controls whether Jetty should send its version in response. Sending Jetty version could be disabled by calling HttpConfiguration::setSendServerVersion(false) ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Compiled and tested manually. --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6616 from paul8263/KYUUBI-6615. Closes #6615 c1567fdfa [zhang_yao] [KYUUBI #6615] Make Jetty sending server version in response configurable Authored-by: zhang_yao <xzhangyao@126.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
senmiaoliu
|
72396e7f2d |
[KYUUBI #6585] Support Multiple Configuration Files in FileSessionConfAdvisor
# 🔍 Description ## Issue References 🔗 This pull request fixes # ## Describe Your Solution 🔧 The current FileSessionConfAdvisor only supports configuring a single file through kyuubi.session.conf.profiler. However, users may need to use multiple configuration files, and the existing setup does not support this. To address this need, we have updated the FileSessionConfAdvisor to support multiple configuration files. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6585 from lsm1/branch-file-advidor-support-chain. Closes #6585 15b6524e0 [senmiaoliu] fix style fcedc4fe3 [senmiaoliu] FileSessionConfAdvisor support multi files Authored-by: senmiaoliu <senmiaoliu@trip.com> Signed-off-by: senmiaoliu <senmiaoliu@trip.com> |
||
|
xorsum
|
c28d955c0c
|
[KYUUBI #6566] Fix typos in KyuubiConf
# 🔍 Description ## Issue References 🔗 fix typos ## Describe Your Solution 🔧 anderror -> an error deprected -> deprecated ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6566 from XorSum/fix/kyuubi-conf-typo. Closes #6566 5c270ad0b [xorsum] fix typo in KyuubiConf Authored-by: xorsum <xorsum@outlook.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
jiaoqingbo
|
ef943ecb3b
|
[KYUUBI #6524] Trino engine supports insecure configuration
# 🔍 Description ## Issue References 🔗 This pull request fixes #6524 ## Describe Your Solution 🔧 Trino engine supports insecure configuration, just as trino client supports --insecure parameter ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6525 from jiaoqingbo/6524. Closes #6524 b414b2e05 [jiaoqingbo] update settings.md 129d40742 [jiaoqingbo] [KYUUBI #6524] Trino engine supports insecure configuration 24f374b38 [jiaoqingbo] Merge branch 'master' of https://github.com/jiaoqingbo/incubator-kyuubi e89268e4b [jiaoqingbo] [KYUUBI #6508] Add the key-value pairs in optimizedConf to session conf Authored-by: jiaoqingbo <1178404354@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
senmiaoliu
|
02e1aa3063
|
[KYUUBI #6387] [TRINO] Trino engine improve operation log
trino engine increment mode support fetch log # 🔍 Description ## Issue References 🔗 This pull request fixes # ## Describe Your Solution 🔧 1. Trino engine supports outputting final status information, and we can also support outputting running information. 2. In increment mode, the operation status is set to FINISHED quickly, which prevents the client from fetching detailed operation logs. The client is unaware of the exact execution status of the operation. It should be possible to explicitly check if the ResultSet has more data (hasNext), and delay setting the status to output more operation information. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6387 from lsm1/branch-trino-print-info. Closes #6387 b9b733a2f [senmiaoliu] trino engine output running info trino engine increment mode support fetch log Authored-by: senmiaoliu <senmiaoliu@trip.com> Signed-off-by: Shaoyun Chen <csy@apache.org> |
||
|
Wang, Fei
|
3439ea03f2 |
[KYUUBI #6377] Fix isCommand check and set min rows threshold for saveToFile
# 🔍 Description ## Issue References 🔗 This pull request fixes # I found that, with saveToFile enabled with the default min size threshold, even I run a simple `set` command, It also save the result to file. <img width="1718" alt="image" src="https://github.com/apache/kyuubi/assets/6757692/5bcc0da1-201a-453a-8568-d1bfadd7adef"> I think we need to skip this kind of queries. ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6377 from turboFei/check_is_DQL. Closes #6377 da9c2a921 [Wang, Fei] ut 04e20db5f [Wang, Fei] conf 8f20ed84b [Wang, Fei] refine the check f558dcca5 [Wang, Fei] ut c81340333 [Wang, Fei] DQL Authored-by: Wang, Fei <fwang12@ebay.com> Signed-off-by: Wang, Fei <fwang12@ebay.com> |
||
|
wforget
|
ef28a615a6 |
[KYUUBI #6338] Support connecting Kyuubi using Hive JDBC driver 4.0
# 🔍 Description ## Issue References 🔗 This pull request fixes #6338 ## Describe Your Solution 🔧 Support `kyuubi.server.thrift.resultset.default.fetchsize` conf to respect `hive.server2.thrift.resultset.default.fetch.size` hive conf. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [X] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests KyuubiTBinaryFrontendServiceSuite.test("test kyuubi.server.thrift.resultset.default.fetch.size") --- # Checklist 📝 - [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6340 from wForget/KYUUBI-6338. Closes #6338 acd73a16b [wforget] fix 5e64ed3a5 [wforget] comments 48b8ec6cd [wforget] fix style and address comments ca653492c [wforget] [KYUUBI #6338] Support `kyuubi.server.thrift.resultset.default.fetch.size` conf Authored-by: wforget <643348094@qq.com> Signed-off-by: wforget <643348094@qq.com> |
||
|
Wang, Fei
|
90491fc07e |
[KYUUBI #6321] Support to get Spark Kubernetes app URL
# 🔍 Description ## Issue References 🔗 Now for kubernetes application, there is no runtime AppURL return in the application report. In fact, we can get the driver svc and get the spark-ui port, then build the Spark UI URL same with that of spark. |
||
|
senmiaoliu
|
11343cd6ea
|
[KYUUBI #6299] Support disabling Web UI
# 🔍 Description ## Issue References 🔗 This pull request fixes #6299 ## Describe Your Solution 🔧 when disabling web ui, return 404 page ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6311 from lsm1/branch-support-disable-webui. Closes #6299 aa96c2737 [senmiaoliu] remove enable.html 998504710 [senmiaoliu] fix style a2622cbbc [senmiaoliu] disable web ui Authored-by: senmiaoliu <senmiaoliu@trip.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
senmiaoliu
|
9aa5269665
|
[KYUUBI #6253] Support running JDBC engine on YARN AM
# 🔍 Description ## Issue References 🔗 This pull request fixes #6253 ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 <img width="1251" alt="image" src="https://github.com/apache/kyuubi/assets/18713676/b654a300-8c79-4461-9fba-4ad1c913accc"> #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6275 from lsm1/branch-jdbc-engine-on-yarn. Closes #6253 5ed4af041 [senmiaoliu] fix style 86e032688 [senmiaoliu] fix style b3e114445 [senmiaoliu] fix style bf33de300 [senmiaoliu] fix style c38404918 [senmiaoliu] jdbc engine on yarn Authored-by: senmiaoliu <senmiaoliu@trip.com> Signed-off-by: Shaoyun Chen <csy@apache.org> |
||
|
wforget
|
a4c3c0b759 |
[KYUUBI #6216] Support to deny some client ips to make connection
# 🔍 Description ## Issue References 🔗 This pull request fixes #6216 ## Describe Your Solution 🔧 Similar to #4540, sometimes we need to quickly deny requests from some clients, so I added `kyuubi.server.limit.connections.ip.deny.list` to limit client ips. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [X] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests SessionLimiterSuite.test("test session limiter with ip deny list") --- # Checklist 📝 - [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6217 from wForget/KYUUBI-6216. Closes #6216 e0a058c1d [wforget] update ctl f66426eda [wforget] typo 5c5308ad8 [wforget] typo 959242e9b [wforget] Support to deny some client ips to make connection Authored-by: wforget <643348094@qq.com> Signed-off-by: wforget <643348094@qq.com> |
||
|
senmiaoliu
|
a937ae760e
|
[KYUUBI #6243] Add missing Impala alias for JDBC engine
# 🔍 Description ## Issue References 🔗 ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6238 from lsm1/branch-conf-add-Impala. Closes #6243 de3093d6a [senmiaoliu] add Impala conf Authored-by: senmiaoliu <senmiaoliu@trip.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
senmiaoliu
|
03c3b4d69c
|
[KYUUBI #5374] JDBC Engine supports ClickHouse
# 🔍 Description ## Issue References 🔗 This pull request fixes #5374 ## Describe Your Solution 🔧 JDBC Engine supports ClickHouse ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6225 from lsm1/branch-support-clickhouse. Closes #5374 0ce4f6f0b [senmiaoliu] fix style f6ab022b6 [senmiaoliu] use ck jdbc http jar dee6a6bdc [senmiaoliu] add it test aed6b33a9 [senmiaoliu] init clickhouse engine Authored-by: senmiaoliu <senmiaoliu@trip.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
zhouyifan279
|
a196ace284
|
[KYUUBI #6199] Support to run HiveSQLEngine on kerberized YARN
# 🔍 Description ## Issue References 🔗 This pull request implement a feature - Run HiveSQLEngine on kerberized YARN ## Describe Your Solution 🔧 Introduced two configs: - kyuubi.engine.principal - kyuubi.engine.keytab When do submit to a kerberized YARN, submitter uploads `kyuubi.engine.keytab` to application's staging dir. YARN NodeManager downloads keytab to AM's working directory. AM logins to Kerberos using the principal and keytab **Note** I've tried to run HiveSQLEngine with only DelegationTokens but failed. Take SQL `SELECT * FROM a` as an example: Hive handles this simple TableScan SQL by reading directly from table's hdfs file. When Hive invokes `FileInputFormat.getSplits` during reading, `java.io.IOException: Delegation Token can be issued only with kerberos or web authentication` will be thrown. The simplified stacktrace from IDEA is as below: ``` getDelegationToken:734, DFSClient (org.apache.hadoop.hdfs) getDelegationToken:2072, DistributedFileSystem (org.apache.hadoop.hdfs) collectDelegationTokens:108, DelegationTokenIssuer (org.apache.hadoop.security.token) addDelegationTokens:83, DelegationTokenIssuer (org.apache.hadoop.security.token) obtainTokensForNamenodesInternal:143, TokenCache (org.apache.hadoop.mapreduce.security) obtainTokensForNamenodesInternal:102, TokenCache (org.apache.hadoop.mapreduce.security) obtainTokensForNamenodes:81, TokenCache (org.apache.hadoop.mapreduce.security) listStatus:221, FileInputFormat (org.apache.hadoop.mapred) getSplits:332, FileInputFormat (org.apache.hadoop.mapred) getNextSplits:372, FetchOperator (org.apache.hadoop.hive.ql.exec) getRecordReader:304, FetchOperator (org.apache.hadoop.hive.ql.exec) getNextRow:459, FetchOperator (org.apache.hadoop.hive.ql.exec) pushRow:428, FetchOperator (org.apache.hadoop.hive.ql.exec) fetch:147, FetchTask (org.apache.hadoop.hive.ql.exec) getResults:2208, Driver (org.apache.hadoop.hive.ql) getNextRowSet:494, SQLOperation (org.apache.hive.service.cli.operation) getNextRowSetInternal:105, HiveOperation (org.apache.kyuubi.engine.hive.operation) ``` Theoretically, it can be solved by add AM DelegationTokens into `org.apache.hadoop.hive.ql.exec.FetchOperator.job.credentials`. But actually, it is impossible without modifying Hive's source code. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ HiveSQLEngine can not run on a kerberized YARN #### Behavior With This Pull Request 🎉 HiveSQLEngine can run on a kerberized YARN #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6199 from zhouyifan279/kerberized-hive-engine-on-yarn. Closes #6199 383d1cdcb [zhouyifan279] Fix tests 458493a91 [zhouyifan279] Warn if run Hive on YARN without principal and keytab 118afe280 [zhouyifan279] Warn if run Hive on YARN without principal and keytab 41fed0c44 [zhouyifan279] Ignore Principal&Keytab when hadoop security is no enabled. 9e2d86237 [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/hive/HiveProcessBuilder.scala 5ae0a3eac [zhouyifan279] Remove redundant checks 5d3013aaf [zhouyifan279] Use principal & keytab in Local mode 5733dfdcb [zhouyifan279] Use principal & keytab in Local mode 85ce9bb7a [zhouyifan279] Use principal & keytab in Local mode 061223dbe [zhouyifan279] Resolve comments e706936e7 [zhouyifan279] Resolve comments f84c7bccc [zhouyifan279] Support run HiveSQLEngine on kerberized YARN 4d262c847 [zhouyifan279] Support run HiveSQLEngine on kerberized YARN Lead-authored-by: zhouyifan279 <zhouyifan279@gmail.com> Co-authored-by: Cheng Pan <pan3793@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
senmiaoliu
|
67f099a4cd
|
[KYUUBI #6201] Distinguish the collect mode of different engines
# 🔍 Description ## Issue References 🔗 ## Describe Your Solution 🔧 Currently, Spark, JDBC, and TRINO engines support incremental collection, but they share the same configuration option `kyuubi.operation.incremental.collect`. Sometimes, it's necessary to enable incremental collection only for specific engines, which requires distinguishing between them. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6201 from lsm1/branch-distinguish-engine-collect-mode. Closes #6201 3c43d2cac [senmiaoliu] fix conf version aa53231aa [senmiaoliu] distinguish kyuubi.operation.incremental.collect for engine type Authored-by: senmiaoliu <senmiaoliu@trip.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
minyk
|
7413ba584a
|
[KYUUBI #6165] Add some HttpClient options for EngineUIProxyServlet
# 🔍 Description ## Issue References 🔗 This pull request fixes #6165 ## Describe Your Solution 🔧 Add HttpClient options for engine ui proxy servlet. - idleTimeout - maxConnections - maxThreads - requestBufferSize - responseBufferSize - timeout ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ Engine UI proxy requests with large headers( > 4k) are failed with 500 http status. #### Behavior With This Pull Request 🎉 Increase the value of `kyuubi.frontend.rest.proxy.jetty.client.requestBufferSize` properly(ex: 5k), request with large header has succeeded with 200 status. #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6192 from minyk/kyuubi-6165. Closes #6165 6ef26aa43 [minyk] fix kyuubi version of configs 1812a96eb [minyk] apply spotless 3527cada9 [minyk] set initial parameters to the httpclient of proxy servletholder. Authored-by: minyk <minykreva@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
182f4349a7
|
[KYUUBI #6117] Bump Kyuubi Shaded 0.3.0
# 🔍 Description ## Issue References 🔗 This PR upgrades Kyuubi Shaded to 0.3.0, the major changes are: - upgrade Thrift from 0.9.3-1 to 0.16.0 - upgrade JLine used by Zookeeper CLI from 0.9.94 to 2.14.6 - split kyuubi-relocated-thrift from kyuubi-relocated-hive-service-rpc - introduce a light kyuubi-relocated-hive-metastore-client for getting DelegationToken Due to THRIFT-5297 (fixed in 0.14.0) removes requestTimeout and beBackoffSlotLength, the following Kyuubi configurations are removed too. - kyuubi.frontend.login.timeout - kyuubi.frontend.thrift.login.timeout - kyuubi.frontend.backoff.slot.length - kyuubi.frontend.thrift.backoff.slot.length Previously, Zookeeper 3.4 depends on JLine 0.9.94 while Zookeeper 3.6 and Hive Beeline depends on JLine 2.14.6, we pulls different versions of JLine into the binary distribution. Now we align them to a single JLine 2.14.6. ## Describe Your Solution 🔧 Upgrade dependencies and change code to adapt to the breaking changes, also upgrade migration guide to mention the removed configurations. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Pass GA. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6117 from pan3793/shaded-0.3. Closes #6117 5639c9fd8 [Cheng Pan] nit d281cdb4b [Cheng Pan] fix 184e1b95b [Cheng Pan] fix 1628337c8 [Cheng Pan] remove unused conf 24db2d5ad [Cheng Pan] try fix 1e995bb34 [Cheng Pan] nit 1d8e9bce9 [Cheng Pan] dep list 7f33624e8 [Cheng Pan] Bump Kyuubi Shaded 0.3.0 Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Manhua
|
97d02a492d |
[KYUUBI #6119] [DOC] Update doc for HA/Zookeeper Configuration
# 🔍 Description ## Issue References 🔗 This pull request update doc for HA/Zookeeper Configuration ## Describe Your Solution 🔧 Replace conf name with the new ones and mark olds as deprecated ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6119 from kevinjmh/doc-update. Closes #6119 49d0fffa0 [Manhua] [DOC] Update doc for HA/Zookeeper Configuration Authored-by: Manhua <manhua@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
d007d7af10
|
[KYUUBI #6099] Remove support of Derby for Kyuubi metastore
# 🔍 Description ## Issue References 🔗 This pull request removes support of Derby for Kyuubi metastore. ## Describe Your Solution 🔧 Previously, we migrated the embedded DB of Kyuubi metastore from Derby to SQLite, and also marked Derby as deprecated (#4950), now, I propose to remove support of Derby for Kyuubi metastore. Note, that both Derby and SQLite are mainly for testing purposes, and they're not supposed to be used in production. Users should not be surprised by this removal. Also, the Derby we used suffers CVE-2022-46337 > Mitigation: > > Users should upgrade to Java 21 and Derby 10.17.1.0. > > Alternatively, users who wish to remain on older Java versions should build their own Derby distribution from one of the release families to which the fix was backported: 10.16, 10.15, and 10.14. Those are the releases which correspond, respectively, with Java LTS versions 17, 11, and 8. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [x] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Some tests are modified accordingly. Pass GA. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6099 from pan3793/remove-derby. Closes #6099 7e9dfd692 [Cheng Pan] Remove support of Derby for Kyuubi metastore Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
4390cc6fa4
|
[KYUUBI #6058] Make Jetty server stop timeout configurable
# 🔍 Description ## Issue References 🔗 This pull request aims to make the Jetty server stop timeout configurable. The RESTful frontend protocol has been enabled by default since 1.8.0, while a side-effect is observed, the process shutdown is always blocked by the Jetty server until reaches the default 30s timeout. ## Describe Your Solution 🔧 Two configurations are introduced in this PR ``` kyuubi.frontend.trino.jetty.stopTimeout=5s kyuubi.frontend.rest.jetty.stopTimeout=5s ``` while the 5s is chosen as the default value according to https://git.eclipse.org/c/jetty/org.eclipse.jetty.project.git/tree/jetty-server/src/main/config/etc/jetty.xml?h=jetty-9.3.x#n124 ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 1. perform `bin/kyuubi run` 2. press `ctrl + c` to shutdown the Kyuubi process #### Behavior Without This Pull Request ⚰️ takes ~30s to complete the shutdown progress. #### Behavior With This Pull Request 🎉 takes ~5s to complete the shutdown progress. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6058 from pan3793/jetty-timeout. Closes #6058 47d15f9aa [Cheng Pan] fix cbac2b13e [Cheng Pan] Make Jetty server stop timeout configurable Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Fei Wang
|
8c3f471ae3 |
[KYUUBI #5952] Disconnect connections without running operations after engine maxlife time graceful period
# 🔍 Description ## Issue References 🔗 We found that, some kyuubi connections(maybe managed by jdbc connection pool likes hikari) always keep alive, and the engine can not be terminated after exceeds the max life time. So, In this pr, I introduce a graceful period after spark engine max life time, after the graceful period, the connections without running operations will be disconnected forcibly. Close #5952 ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6040 from turboFei/close_on_nooperation_rebase. Closes #5952 0b05ddc0c [Fei Wang] comments 4329a85cf [Fei Wang] async stop b39c9b3a0 [Fei Wang] use short sleep 7c123f612 [Fei Wang] save 45ad3489d [Fei Wang] check no running operation Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: Fei Wang <fwang12@ebay.com> |
||
|
Tigran Manasyan
|
35a910a43e
|
[KYUUBI #5674] Add PostgreSQL as backend database for kyuubi metadata
# 🔍 Description ## Issue References 🔗 This pull request fixes #5674 ## Describe Your Solution 🔧 Currently, Kyuubi supports the use of SQLite and MySQL to store metadata. Although PostgreSQL can be supported through the 'CUSTOM' method, there is a certain usage cost for users, therefore this MR adds support for PostgreSQL as a backend database for kyuubi metadata. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests `org.apache.kyuubi.server.metadata.jdbc.JDBCMetadataStoreSuite` --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6027 from tigrulya-exe/feature/5674-postgresql-metadata-backend. Closes #5674 111658002 [Tigran Manasyan] Fix metastore db type option comment 44a22bfcf [Tigran Manasyan] Fix dependency list and metastore db type option comment b638c8942 [Tigran Manasyan] Add PostgreSQL as backend database for kyuubi metadata Authored-by: Tigran Manasyan <t.manasyan@arenadata.io> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
3f993f405a
|
[KYUUBI #6003] Allow disabling user impersonation on launching engine
# 🔍 Description ## Issue References 🔗 HiveServer2 has a configuration `hive.server2.enable.doAs` to control the execution user between the session user and the server user, Kyuubi's CONNECTION and USER share levels always perform like doAs enabled do. In CDH 5/6, this is disabled by default, users who want to migrate from CDH to Kyuubi may encounter permission issues with the current implementation. ## Describe Your Solution 🔧 This pull request introduces a new configuration `kyuubi.engine.doAs.enabled` to allow enable/disable user impersonation on launching engine. For security purpose, it's not allowed to be overridden by session conf. The change in this PR has certain limitations: - only supports Spark engine - only supports interactive mode, specifically, it does not take effect on Spark batch mode now. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 The first step is passing all existing UTs when `kyuubi.engine.doAs.enabled=true`. Tested on internal Kerberized-environment, when `kyuubi.engine.share.level=CONNECTION` and `kyuubi.engine.doAs.enabled=false`, use user 'spark' to launch engine, and the engine submitted without `--proxy-user spark`, thus engine launched by server user `hive`, then run `select session_user(), current_user()` and returns ``` +-----------------+-----------------+ | session_user() | current_user() | +-----------------+-----------------+ | spark | hive | +-----------------+-----------------+ ``` And I checked the `spark.app.name` and registered path on Zookeeper also expected. ``` +-----------------+--------------------------------------------------------------------------+ | key | value | +-----------------+--------------------------------------------------------------------------+ | spark.app.name | kyuubi_USER_SPARK_SQL_spark_default_51a416e5-6023-4bac-a964-cd9605f17c61 | +-----------------+--------------------------------------------------------------------------+ ``` --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6003 from pan3793/doas. Closes #6003 c4002fef5 [Cheng Pan] grammar add20fd57 [Cheng Pan] nit 8711c2265 [Cheng Pan] address comment 033a32252 [Cheng Pan] 1.9.0 9273b9426 [Cheng Pan] fix a1563e1ca [Cheng Pan] HadoopCredentialsManager e982e2364 [Cheng Pan] Allow disable user impersonation on launching engine Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
d474768d97
|
[KYUUBI #6018] Speed up GetTables operation for Spark session catalog
# 🔍 Description ## Issue References 🔗 This pull request aims to speed up the GetTables operation for the Spark session catalog. As reported in https://github.com/apache/kyuubi/discussions/4956, https://github.com/apache/kyuubi/discussions/5949, the GetTables operation is quite slow in some cases, and in https://github.com/apache/kyuubi/pull/4444, `kyuubi.operation.getTables.ignoreTableProperties` was introduced to speed up the V2 catalog, but not covers session catalog. ## Describe Your Solution 🔧 Extend the scope of `kyuubi.operation.getTables.ignoreTableProperties` to cover the GetTables operation for the Spark session catalog. Currently, the basic step of GetTables in the Spark engine is ``` val catalog: String = getCatalog(spark, catalogName) val databases: Seq[String] = sessionCatalog.listDatabases(schemaPattern) val identifiers: Seq[TableIdentifier] = catalog.listTables(db, tablePattern, includeLocalTempViews = false) val tableObjects: Seq[CatalogTable] = catalog.getTablesByName(identifiers) ``` then filter `tableObjects` with `tableTypes: Set[String]`. The cost of `catalog.getTablesByName(identifiers)` is quite high when the table number is large, e.g. dozen thousand. For some cases, listing tables only for table name display, it is worth speeding up the operation while ignoring some properties(e.g. table comments) and query criteria(specifically in this case, when `kyuubi.operation.getTables.ignoreTableProperties=true`, criteria `tableTypes` will be ignored, and all tables and views will be treated as TABLE to return.) ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Pass GA --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6018 from pan3793/fast-get-table. Closes #6018 058001c6f [Cheng Pan] fix 405b12484 [Cheng Pan] fix 615b7470f [Cheng Pan] Speed up GetTables operation Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
senmiaoliu
|
777b784439
|
[KYUUBI #5968] Support set authentication user for Trino engine
# 🔍 Description ## Issue References 🔗 This pull request fixes #5968 ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #5970 from lsm1/branch-kyuubi-5968. Closes #5968 580eb45d2 [senmiaoliu] fix style a36380cde [senmiaoliu] add auth user conf for trino engine Authored-by: senmiaoliu <senmiaoliu@trip.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
3b2e674364
|
[KYUUBI #5990] Always take the first declared SASL/PLAIN auth type
# 🔍 Description ## Issue References 🔗 https://github.com/apache/kyuubi/pull/5185 changed the type of `kyuubi.authentication` from `Seq`(ordered) to `Set`(unordered), which break the assumption of ``` Note that: for SASL authentication, KERBEROS and PLAIN auth types are supported at the same time, and only the first specified PLAIN auth type is valid. ``` ## Describe Your Solution 🔧 Restore the type to Seq ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 UT is updated --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #5990 from pan3793/auth-plain. Closes #5990 acae25f68 [Cheng Pan] fix doc cef7dba90 [Cheng Pan] fix 87b370ffe [Cheng Pan] Always take the first declared SASL/PLAIN auth type Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
yikaifei
|
518e918fa7
|
[KYUUBI #5867][FOLLOWUP] Remove unused config
# 🔍 Description ## Issue References 🔗 This pull request aims to remove unused config as #5867 followup. ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #5976 from Yikf/remove-INTERVAL. Closes #5867 e4183ffca [yikaifei] [KYUUBI #5867][FOLLOWUP] Remove unused config Authored-by: yikaifei <yikaifei@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Cheng Pan
|
5d59cf1d6a
|
[KYUUBI #5867][FOLLOWUP] Fix generated configuration docs and typo
# 🔍 Description ## Issue References 🔗 This pull request is a follow-up of #5867, to fix generated configuration docs and typo to recover CI. ## Describe Your Solution 🔧 Regenerated docs by performing `dev/gen/gen_all_config_docs.sh` Address comment https://github.com/apache/kyuubi/pull/5868#discussion_r1438080698 ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 Pass GA. --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #5932 from pan3793/fix-docs. Closes #5867 93b8f97b0 [Cheng Pan] docs 0ad0d29f3 [Cheng Pan] typo Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
yikaifei
|
679aca5a6c
|
[KYUUBI #5867] HiveEngine support run on YARN mode
# 🔍 Description ## Issue References 🔗 This PR aims to support hive engine run on yarn mode, close https://github.com/apache/kyuubi/issues/5867 ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [ ] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [x] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5868 from Yikf/hive-on-yarn. Closes #5867 44f7287f5 [yikaifei] fix 3c17d2c4a [yikaifei] fix test 5474ebfba [yikaifei] parse classpath 6b97c4213 [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala 34a67b452 [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala 5e5045e66 [yikaifei] fix app type d1eb5aea7 [yikaifei] fix d89d09cfe [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala 1fa18ba1b [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/ApplicationMaster.scala 1b0b77f4d [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/ApplicationMaster.scala 2ed1d4492 [Cheng Pan] Update kyuubi-common/src/main/scala/org/apache/kyuubi/engine/deploy/yarn/EngineYarnModeSubmitter.scala 98ff19ce6 [yikaifei] HiveEngine support run on YARN mode Lead-authored-by: yikaifei <yikaifei@apache.org> Co-authored-by: Cheng Pan <pan3793@gmail.com> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Kent Yao
|
c18dd61952
|
[KYUUBI #5916] Revert #5916
# 🔍 Description ## Issue References 🔗 This pull request fixes #5916 as a revert ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [x] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5928 from yaooqinn/revert. Closes #5916 5c43c0286 [Kent Yao] [KYUUBI #5916] Revert #5916 Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Bowen Liang
|
b1e726dcf5 |
[KYUUBI #5912] Cleanup unused config kyuubi.frontend.thrift.http.allow.user.substitution
# 🔍 Description ## Issue References 🔗 As decribed. ## Describe Your Solution 🔧 Drop the unused config of KyuubiConf, which are never used and linked in source code and test code: - FRONTEND_THRIFT_HTTP_ALLOW_USER_SUBSTITUTION: `kyuubi.frontend.thrift.http.allow.user.substitution` ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5912 from bowenliang123/unused-configs. Closes #5912 b938058b8 [Bowen Liang] update doc 9fd441dd0 [Bowen Liang] fix 9b0b31ee1 [Bowen Liang] cleanup unused config entries in KyuubiConf Authored-by: Bowen Liang <liangbowen@gf.com.cn> Signed-off-by: Bowen Liang <liangbowen@gf.com.cn> |
||
|
Bowen Liang
|
0f8edb2b2f |
[KYUUBI #5916] Use "(none)" placeholder for default value of un-preset configs on confing doc page
# 🔍 Description ## Issue References 🔗 As descirbed. ## Describe Your Solution 🔧 - Replace the placeholder for default value of un-preset configs on doc page, changed from `<undefined>` to `(none)` - The `(none)` placeholder style is also used in the docs of - Spark : https://spark.apache.org/docs/latest/configuration.html - Flink : https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/ - Trino (as `NONE`): https://trino.io/docs/current/admin/properties.html - Only affects the config doc page, no functional changes. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ No functional changes. Config doc page:  #### Behavior With This Pull Request 🎉 No functional changes. Config doc page:  #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5916 from bowenliang123/none-ph. Closes #5916 3b141dd61 [Bowen Liang] update test 04cf8cd2a [Bowen Liang] Use "(none)" placeholder for default value of unset configs on config page Authored-by: Bowen Liang <liangbowen@gf.com.cn> Signed-off-by: Bowen Liang <liangbowen@gf.com.cn> |
||
|
Bowen Liang
|
5a751027f5 |
[KYUUBI #5905] [DOC] Workaround for long inlined code sample in config page
# 🔍 Description ## Issue References 🔗 ## Describe Your Solution 🔧 Workaround to fix the display issue of long inlined code sample on config page. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️  #### Behavior With This Pull Request 🎉  #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [ ] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5905 from bowenliang123/doc-block. Closes #5905 b4d28ed85 [Bowen Liang] update 7aa995b38 [Bowen Liang] update ef947ea15 [Bowen Liang] update 7b3263bee [Bowen Liang] update aefd206ae [Bowen Liang] update f02ec3acf [Bowen Liang] update d3302d55c [Bowen Liang] fix display by putting url demo in code block Authored-by: Bowen Liang <liangbowen@gf.com.cn> Signed-off-by: Cheng Pan <chengpan@apache.org> |
||
|
Fei Wang
|
fd397d6277 |
[KYUUBI #5900] Support to deregister the engine on open failure
# 🔍 Description ## Issue References 🔗 This pull request fixes #5900 ## Describe Your Solution 🔧 Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [ ] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [ ] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [ ] Pull request title is okay. - [ ] No license issues. - [ ] Milestone correctly set? - [ ] Test coverage is ok - [ ] Assignees are selected. - [ ] Minimum number of approvals - [ ] No changes are requested **Be nice. Be informative.** Closes #5901 from turboFei/delete_engine. Closes #5900 57bcf1f24 [Fei Wang] docs 557714eed [Fei Wang] enum bc479e6e0 [Fei Wang] save 2b2a4952c [Fei Wang] refine f0ace3574 [Fei Wang] reset after close client cf07877fe [Fei Wang] re-version d93dd6add [Fei Wang] refactor Authored-by: Fei Wang <fwang12@ebay.com> Signed-off-by: Fei Wang <fwang12@ebay.com> |