[KYUUBI #6829] Add metrics for batch pending max elapse time
### Why are the changes needed? 1. add metrics `kyuubi.operartion.batch_pending_max_elapse` for the batch pending max elapse time, which is helpful for batch health monitoring, and we can send alert if the batch pending elapse time too long 2. For `GET /api/v1/batches` api, limit the max time window for listing batches, which is helpful that, we want to reserve more metadata in kyuubi server end, for example: 90 days, but for list batches, we just want to allow user to search the last 7 days. It is optional. And if `create_time` is specified, order by `create_time` instead of `key_id`.68a6f48da5/kyuubi-server/src/main/resources/sql/mysql/metadata-store-schema-1.8.0.mysql.sql (L32)### How was this patch tested? GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #6829 from turboFei/batch_pending_time. Closes #6829 ee4f93125 [Wang, Fei] docs bf8169ad4 [Wang, Fei] comments f493a2af8 [Wang, Fei] new config ab7b6db65 [Wang, Fei] ut 168017587 [Wang, Fei] in memory session 510a30b6a [Wang, Fei] batchSearchWindow opt 1e93dd276 [Wang, Fei] save Authored-by: Wang, Fei <fwang12@ebay.com> Signed-off-by: Cheng Pan <chengpan@apache.org> (cherry picked from commit3167692732) Signed-off-by: Cheng Pan <chengpan@apache.org>
This commit is contained in:
Wang, Fei
Cheng Pan
parent
c65f04848d
commit
eb65d3a8ff
@ -377,24 +377,25 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
|
||||
|
||||
### Metadata
|
||||
|
||||
| Key | Default | Meaning | Type | Since |
|
||||
|-------------------------------------------------|----------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
|
||||
| kyuubi.metadata.cleaner.enabled | true | Whether to clean the metadata periodically. If it is enabled, Kyuubi will clean the metadata that is in the terminate state with max age limitation. | boolean | 1.6.0 |
|
||||
| kyuubi.metadata.cleaner.interval | PT30M | The interval to check and clean expired metadata. | duration | 1.6.0 |
|
||||
| kyuubi.metadata.max.age | PT72H | The maximum age of metadata, the metadata exceeding the age will be cleaned. | duration | 1.6.0 |
|
||||
| kyuubi.metadata.recovery.threads | 10 | The number of threads for recovery from the metadata store when the Kyuubi server restarts. | int | 1.6.0 |
|
||||
| kyuubi.metadata.request.async.retry.enabled | true | Whether to retry in async when metadata request failed. When true, return success response immediately even the metadata request failed, and schedule it in background until success, to tolerate long-time metadata store outages w/o blocking the submission request. | boolean | 1.7.0 |
|
||||
| kyuubi.metadata.request.async.retry.queue.size | 65536 | The maximum queue size for buffering metadata requests in memory when the external metadata storage is down. Requests will be dropped if the queue exceeds. Only take affect when kyuubi.metadata.request.async.retry.enabled is `true`. | int | 1.6.0 |
|
||||
| kyuubi.metadata.request.async.retry.threads | 10 | Number of threads in the metadata request async retry manager thread pool. Only take affect when kyuubi.metadata.request.async.retry.enabled is `true`. | int | 1.6.0 |
|
||||
| kyuubi.metadata.request.retry.interval | PT5S | The interval to check and trigger the metadata request retry tasks. | duration | 1.6.0 |
|
||||
| kyuubi.metadata.store.class | org.apache.kyuubi.server.metadata.jdbc.JDBCMetadataStore | Fully qualified class name for server metadata store. | string | 1.6.0 |
|
||||
| kyuubi.metadata.store.jdbc.database.schema.init | true | Whether to init the JDBC metadata store database schema. | boolean | 1.6.0 |
|
||||
| kyuubi.metadata.store.jdbc.database.type | SQLITE | The database type for server jdbc metadata store.<ul> <li>SQLITE: SQLite3, JDBC driver `org.sqlite.JDBC`.</li> <li>MYSQL: MySQL, JDBC driver `com.mysql.cj.jdbc.Driver` (fallback `com.mysql.jdbc.Driver`).</li> <li>POSTGRESQL: PostgreSQL, JDBC driver `org.postgresql.Driver`.</li> <li>CUSTOM: User-defined database type, need to specify corresponding JDBC driver.</li> Note that: The JDBC datasource is powered by HiKariCP, for datasource properties, please specify them with the prefix: kyuubi.metadata.store.jdbc.datasource. For example, kyuubi.metadata.store.jdbc.datasource.connectionTimeout=10000. | string | 1.6.0 |
|
||||
| kyuubi.metadata.store.jdbc.driver | <undefined> | JDBC driver class name for server jdbc metadata store. | string | 1.6.0 |
|
||||
| kyuubi.metadata.store.jdbc.password || The password for server JDBC metadata store. | string | 1.6.0 |
|
||||
| kyuubi.metadata.store.jdbc.priority.enabled | false | Whether to enable the priority scheduling for batch impl v2. When false, ignore kyuubi.batch.priority and use the FIFO ordering strategy for batch job scheduling. Note: this feature may cause significant performance issues when using MySQL 5.7 as the metastore backend due to the lack of support for mixed order index. See more details at KYUUBI #5329. | boolean | 1.8.0 |
|
||||
| kyuubi.metadata.store.jdbc.url | jdbc:sqlite:<KYUUBI_HOME>/kyuubi_state_store.db | The JDBC url for server JDBC metadata store. By default, it is a SQLite database url, and the state information is not shared across Kyuubi instances. To enable high availability for multiple kyuubi instances, please specify a production JDBC url. Note: this value support the variables substitution: `<KYUUBI_HOME>`. | string | 1.6.0 |
|
||||
| kyuubi.metadata.store.jdbc.user || The username for server JDBC metadata store. | string | 1.6.0 |
|
||||
| Key | Default | Meaning | Type | Since |
|
||||
|-------------------------------------------------|----------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|--------|
|
||||
| kyuubi.metadata.cleaner.enabled | true | Whether to clean the metadata periodically. If it is enabled, Kyuubi will clean the metadata that is in the terminate state with max age limitation. | boolean | 1.6.0 |
|
||||
| kyuubi.metadata.cleaner.interval | PT30M | The interval to check and clean expired metadata. | duration | 1.6.0 |
|
||||
| kyuubi.metadata.max.age | PT72H | The maximum age of metadata, the metadata exceeding the age will be cleaned. | duration | 1.6.0 |
|
||||
| kyuubi.metadata.recovery.threads | 10 | The number of threads for recovery from the metadata store when the Kyuubi server restarts. | int | 1.6.0 |
|
||||
| kyuubi.metadata.request.async.retry.enabled | true | Whether to retry in async when metadata request failed. When true, return success response immediately even the metadata request failed, and schedule it in background until success, to tolerate long-time metadata store outages w/o blocking the submission request. | boolean | 1.7.0 |
|
||||
| kyuubi.metadata.request.async.retry.queue.size | 65536 | The maximum queue size for buffering metadata requests in memory when the external metadata storage is down. Requests will be dropped if the queue exceeds. Only take affect when kyuubi.metadata.request.async.retry.enabled is `true`. | int | 1.6.0 |
|
||||
| kyuubi.metadata.request.async.retry.threads | 10 | Number of threads in the metadata request async retry manager thread pool. Only take affect when kyuubi.metadata.request.async.retry.enabled is `true`. | int | 1.6.0 |
|
||||
| kyuubi.metadata.request.retry.interval | PT5S | The interval to check and trigger the metadata request retry tasks. | duration | 1.6.0 |
|
||||
| kyuubi.metadata.search.window | <undefined> | The time window to restrict user queries to metadata within a specific period, starting from the current time to the past. It only affects `GET /api/v1/batches` API. You may want to set this to short period to improve query performance and reduce load on the metadata store when administer want to reserve the metadata for long time. The side-effects is that, the metadata created outside the window will not be invisible to users. If it is undefined, all metadata will be visible for users. | duration | 1.10.1 |
|
||||
| kyuubi.metadata.store.class | org.apache.kyuubi.server.metadata.jdbc.JDBCMetadataStore | Fully qualified class name for server metadata store. | string | 1.6.0 |
|
||||
| kyuubi.metadata.store.jdbc.database.schema.init | true | Whether to init the JDBC metadata store database schema. | boolean | 1.6.0 |
|
||||
| kyuubi.metadata.store.jdbc.database.type | SQLITE | The database type for server jdbc metadata store.<ul> <li>SQLITE: SQLite3, JDBC driver `org.sqlite.JDBC`.</li> <li>MYSQL: MySQL, JDBC driver `com.mysql.cj.jdbc.Driver` (fallback `com.mysql.jdbc.Driver`).</li> <li>POSTGRESQL: PostgreSQL, JDBC driver `org.postgresql.Driver`.</li> <li>CUSTOM: User-defined database type, need to specify corresponding JDBC driver.</li> Note that: The JDBC datasource is powered by HiKariCP, for datasource properties, please specify them with the prefix: kyuubi.metadata.store.jdbc.datasource. For example, kyuubi.metadata.store.jdbc.datasource.connectionTimeout=10000. | string | 1.6.0 |
|
||||
| kyuubi.metadata.store.jdbc.driver | <undefined> | JDBC driver class name for server jdbc metadata store. | string | 1.6.0 |
|
||||
| kyuubi.metadata.store.jdbc.password || The password for server JDBC metadata store. | string | 1.6.0 |
|
||||
| kyuubi.metadata.store.jdbc.priority.enabled | false | Whether to enable the priority scheduling for batch impl v2. When false, ignore kyuubi.batch.priority and use the FIFO ordering strategy for batch job scheduling. Note: this feature may cause significant performance issues when using MySQL 5.7 as the metastore backend due to the lack of support for mixed order index. See more details at KYUUBI #5329. | boolean | 1.8.0 |
|
||||
| kyuubi.metadata.store.jdbc.url | jdbc:sqlite:<KYUUBI_HOME>/kyuubi_state_store.db | The JDBC url for server JDBC metadata store. By default, it is a SQLite database url, and the state information is not shared across Kyuubi instances. To enable high availability for multiple kyuubi instances, please specify a production JDBC url. Note: this value support the variables substitution: `<KYUUBI_HOME>`. | string | 1.6.0 |
|
||||
| kyuubi.metadata.store.jdbc.user || The username for server JDBC metadata store. | string | 1.6.0 |
|
||||
|
||||
### Metrics
|
||||
|
||||
|
||||
@ -40,56 +40,57 @@ The metrics system is configured via `$KYUUBI_HOME/conf/kyuubi-defaults.conf`.
|
||||
|
||||
These metrics include:
|
||||
|
||||
| Metrics Prefix | Metrics Suffix | Type | Since | Description |
|
||||
|--------------------------------------------------|----------------------------------------|-----------|-------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| `kyuubi.exec.pool.threads.alive` | | gauge | 1.2.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> threads keepAlive in the backend executive thread pool</div> |
|
||||
| `kyuubi.exec.pool.threads.active` | | gauge | 1.2.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> threads active in the backend executive thread pool</div> |
|
||||
| `kyuubi.exec.pool.work_queue.size` | | gauge | 1.7.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> work queue size in the backend executive thread pool</div> |
|
||||
| `kyuubi.connection.total` | | counter | 1.2.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> cumulative connection count</div> |
|
||||
| `kyuubi.connection.total` | `${sessionType}` | counter | 1.7.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> cumulative connection count with session type `${sessionType}`</div> |
|
||||
| `kyuubi.connection.opened` | | gauge | 1.2.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> current active connection count</div> |
|
||||
| `kyuubi.connection.opened` | `${user}` | counter | 1.2.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> current active connections count requested by a `${user}`</div> |
|
||||
| `kyuubi.connection.opened` | `${user}`</br>`${sessionType}` | counter | 1.7.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> current active connections count requested by a `${user}` with session type `${sessionType}`</div> |
|
||||
| `kyuubi.connection.opened` | `${sessionType}` | counter | 1.7.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> current active connections count with session type `${sessionType}`</div> |
|
||||
| `kyuubi.connection.failed` | | counter | 1.2.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> cumulative failed connection count</div> |
|
||||
| `kyuubi.connection.failed` | `${user}` | counter | 1.2.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> cumulative failed connections for a `${user}`</div> |
|
||||
| `kyuubi.connection.failed` | `${sessionType}` | counter | 1.7.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> cumulative failed connection count with session type `${sessionType}`</div> |
|
||||
| `kyuubi.operation.total` | | counter | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> cumulative opened operation count</div> |
|
||||
| `kyuubi.operation.total` | `${operationType}` | counter | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> cumulative opened count for the operation `${operationType}`</div> |
|
||||
| `kyuubi.operation.opened` | | gauge | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> current opened operation count</div> |
|
||||
| `kyuubi.operation.opened` | `${operationType}` | counter | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> current opened count for the operation `${operationType}`</div> |
|
||||
| `kyuubi.operation.failed` | `${operationType}`<br/>`.${errorType}` | counter | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> cumulative failed count for the operation `${operationType}` with a particular `${errorType}`, e.g. `execute_statement.AnalysisException`</div> |
|
||||
| `kyuubi.operation.state` | `${operationState}` | meter | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi operation state rate </div> |
|
||||
| `kyuubi.operation.exec_time` | `${operationType}` | histogram | 1.7.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> execution time histogram for the operation `${operationType}`, now only `ExecuteStatement` is enabled. </div> |
|
||||
| `kyuubi.engine.total` | | counter | 1.2.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> cumulative created engines</div> |
|
||||
| `kyuubi.engine.timeout` | | counter | 1.2.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> cumulative timeout engines</div> |
|
||||
| `kyuubi.engine.failed` | `${user}` | counter | 1.2.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> cumulative explicitly failed engine count for a `${user}`</div> |
|
||||
| `kyuubi.engine.failed` | `${errorType}` | counter | 1.2.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> cumulative explicitly failed engine count for a particular `${errorType}`, e.g. `ClassNotFoundException`</div> |
|
||||
| `kyuubi.backend_service.open_session` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `openSession` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.close_session` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `closeSession` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.get_info` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `getInfo` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.execute_statement` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `executeStatement` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.get_type_info` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `getTypeInfo` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.get_catalogs` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `getCatalogs` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.get_schemas` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `getSchemas` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.get_tables` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `getTables` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.get_table_types` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `getTableTypes` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.get_columns` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `getColumns` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.get_functions` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `getFunctions` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.get_operation_status` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `getOperationStatus` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.cancel_operation` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `cancelOperation` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.close_operation` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `closeOperation` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.get_result_set_metadata` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `getResultSetMetadata` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.fetch_results` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `fetchResults` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.fetch_log_rows_rate` | | meter | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `fetchResults` method that fetch log rows rate </div> |
|
||||
| `kyuubi.backend_service.fetch_result_rows_rate` | | meter | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `fetchResults` method that fetch result rows rate </div> |
|
||||
| `kyuubi.backend_service.get_primary_keys` | | meter | 1.6.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `get_primary_keys` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.get_cross_reference` | | meter | 1.6.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `get_cross_reference` method execution time and rate </div> |
|
||||
| `kyuubi.operation.state` | `${operationType}`<br/>`.${state}` | meter | 1.6.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> The `${operationType}` with a particular `${state}` rate, e.g. `BatchJobSubmission.pending`, `BatchJobSubmission.finished`. Note that, the terminal states are cumulative, but the intermediate ones are not. </div> |
|
||||
| `kyuubi.metadata.request.opened` | | counter | 1.6.1 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> current opened count for the metadata requests </div> |
|
||||
| `kyuubi.metadata.request.total` | | meter | 1.6.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> metadata requests time and rate </div> |
|
||||
| `kyuubi.metadata.request.failed` | | meter | 1.6.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> metadata requests failure time and rate </div> |
|
||||
| `kyuubi.metadata.request.retrying` | | meter | 1.6.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> retrying metadata requests time and rate, it is not cumulative </div> |
|
||||
| Metrics Prefix | Metrics Suffix | Type | Since | Description |
|
||||
|--------------------------------------------------|----------------------------------------|-----------|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| `kyuubi.exec.pool.threads.alive` | | gauge | 1.2.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> threads keepAlive in the backend executive thread pool</div> |
|
||||
| `kyuubi.exec.pool.threads.active` | | gauge | 1.2.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> threads active in the backend executive thread pool</div> |
|
||||
| `kyuubi.exec.pool.work_queue.size` | | gauge | 1.7.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> work queue size in the backend executive thread pool</div> |
|
||||
| `kyuubi.connection.total` | | counter | 1.2.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> cumulative connection count</div> |
|
||||
| `kyuubi.connection.total` | `${sessionType}` | counter | 1.7.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> cumulative connection count with session type `${sessionType}`</div> |
|
||||
| `kyuubi.connection.opened` | | gauge | 1.2.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> current active connection count</div> |
|
||||
| `kyuubi.connection.opened` | `${user}` | counter | 1.2.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> current active connections count requested by a `${user}`</div> |
|
||||
| `kyuubi.connection.opened` | `${user}`</br>`${sessionType}` | counter | 1.7.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> current active connections count requested by a `${user}` with session type `${sessionType}`</div> |
|
||||
| `kyuubi.connection.opened` | `${sessionType}` | counter | 1.7.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> current active connections count with session type `${sessionType}`</div> |
|
||||
| `kyuubi.connection.failed` | | counter | 1.2.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> cumulative failed connection count</div> |
|
||||
| `kyuubi.connection.failed` | `${user}` | counter | 1.2.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> cumulative failed connections for a `${user}`</div> |
|
||||
| `kyuubi.connection.failed` | `${sessionType}` | counter | 1.7.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> cumulative failed connection count with session type `${sessionType}`</div> |
|
||||
| `kyuubi.operation.total` | | counter | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> cumulative opened operation count</div> |
|
||||
| `kyuubi.operation.total` | `${operationType}` | counter | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> cumulative opened count for the operation `${operationType}`</div> |
|
||||
| `kyuubi.operation.opened` | | gauge | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> current opened operation count</div> |
|
||||
| `kyuubi.operation.opened` | `${operationType}` | counter | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> current opened count for the operation `${operationType}`</div> |
|
||||
| `kyuubi.operation.failed` | `${operationType}`<br/>`.${errorType}` | counter | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> cumulative failed count for the operation `${operationType}` with a particular `${errorType}`, e.g. `execute_statement.AnalysisException`</div> |
|
||||
| `kyuubi.operation.state` | `${operationState}` | meter | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi operation state rate </div> |
|
||||
| `kyuubi.operation.exec_time` | `${operationType}` | histogram | 1.7.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> execution time histogram for the operation `${operationType}`, now only `ExecuteStatement` is enabled. </div> |
|
||||
| `kyuubi.engine.total` | | counter | 1.2.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> cumulative created engines</div> |
|
||||
| `kyuubi.engine.timeout` | | counter | 1.2.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> cumulative timeout engines</div> |
|
||||
| `kyuubi.engine.failed` | `${user}` | counter | 1.2.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> cumulative explicitly failed engine count for a `${user}`</div> |
|
||||
| `kyuubi.engine.failed` | `${errorType}` | counter | 1.2.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> cumulative explicitly failed engine count for a particular `${errorType}`, e.g. `ClassNotFoundException`</div> |
|
||||
| `kyuubi.backend_service.open_session` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `openSession` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.close_session` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `closeSession` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.get_info` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `getInfo` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.execute_statement` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `executeStatement` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.get_type_info` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `getTypeInfo` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.get_catalogs` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `getCatalogs` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.get_schemas` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `getSchemas` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.get_tables` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `getTables` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.get_table_types` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `getTableTypes` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.get_columns` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `getColumns` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.get_functions` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `getFunctions` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.get_operation_status` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `getOperationStatus` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.cancel_operation` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `cancelOperation` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.close_operation` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `closeOperation` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.get_result_set_metadata` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `getResultSetMetadata` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.fetch_results` | | timer | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `fetchResults` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.fetch_log_rows_rate` | | meter | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `fetchResults` method that fetch log rows rate </div> |
|
||||
| `kyuubi.backend_service.fetch_result_rows_rate` | | meter | 1.5.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `fetchResults` method that fetch result rows rate </div> |
|
||||
| `kyuubi.backend_service.get_primary_keys` | | meter | 1.6.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `get_primary_keys` method execution time and rate </div> |
|
||||
| `kyuubi.backend_service.get_cross_reference` | | meter | 1.6.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> kyuubi backend service `get_cross_reference` method execution time and rate </div> |
|
||||
| `kyuubi.operation.state` | `${operationType}`<br/>`.${state}` | meter | 1.6.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> The `${operationType}` with a particular `${state}` rate, e.g. `BatchJobSubmission.pending`, `BatchJobSubmission.finished`. Note that, the terminal states are cumulative, but the intermediate ones are not. </div> |
|
||||
| `kyuubi.metadata.request.opened` | | counter | 1.6.1 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> current opened count for the metadata requests </div> |
|
||||
| `kyuubi.metadata.request.total` | | meter | 1.6.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> metadata requests time and rate </div> |
|
||||
| `kyuubi.metadata.request.failed` | | meter | 1.6.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> metadata requests failure time and rate </div> |
|
||||
| `kyuubi.metadata.request.retrying` | | meter | 1.6.0 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> retrying metadata requests time and rate, it is not cumulative </div> |
|
||||
| `kyuubi.operartion.batch_pending_max_elapse` | | gauge | 1.10.1 | <div style='width: 150pt;word-wrap: break-word;white-space: normal'> the batch pending max elapsed time on current kyuubi instance </div> |
|
||||
|
||||
Before v1.5.0, if you use these metrics:
|
||||
- `kyuubi.statement.total`
|
||||
|
||||
@ -2023,6 +2023,19 @@ object KyuubiConf {
|
||||
.intConf
|
||||
.createWithDefault(65536)
|
||||
|
||||
val METADATA_SEARCH_WINDOW: OptionalConfigEntry[Long] =
|
||||
buildConf("kyuubi.metadata.search.window")
|
||||
.doc("The time window to restrict user queries to metadata within a specific period, " +
|
||||
"starting from the current time to the past. It only affects `GET /api/v1/batches` API. " +
|
||||
"You may want to set this to short period to improve query performance and reduce load " +
|
||||
"on the metadata store when administer want to reserve the metadata for long time. " +
|
||||
"The side-effects is that, the metadata created outside the window will not be " +
|
||||
"invisible to users. If it is undefined, all metadata will be visible for users.")
|
||||
.version("1.10.1")
|
||||
.timeConf
|
||||
.checkValue(_ > 0, "must be positive number")
|
||||
.createOptional
|
||||
|
||||
val ENGINE_EXEC_WAIT_QUEUE_SIZE: ConfigEntry[Int] =
|
||||
buildConf("kyuubi.backend.engine.exec.pool.wait.queue.size")
|
||||
.doc("Size of the wait queue for the operation execution thread pool in SQL engine" +
|
||||
|
||||
@ -64,6 +64,7 @@ object MetricsConstants {
|
||||
final val OPERATION_TOTAL: String = OPERATION + "total"
|
||||
final val OPERATION_STATE: String = OPERATION + "state"
|
||||
final val OPERATION_EXEC_TIME: String = OPERATION + "exec_time"
|
||||
final val OPERATION_BATCH_PENDING_MAX_ELAPSE: String = OPERATION + "batch_pending_max_elapse"
|
||||
|
||||
final private val BACKEND_SERVICE = KYUUBI + "backend_service."
|
||||
final val BS_FETCH_LOG_ROWS_RATE = BACKEND_SERVICE + "fetch_log_rows_rate"
|
||||
|
||||
@ -424,6 +424,14 @@ class BatchJobSubmission(
|
||||
Utils.deleteDirectoryRecursively(session.resourceUploadFolderPath.toFile)
|
||||
}
|
||||
}
|
||||
|
||||
def getPendingElapsedTime: Long = {
|
||||
if (state == OperationState.PENDING) {
|
||||
System.currentTimeMillis() - createTime
|
||||
} else {
|
||||
0L
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
object BatchJobSubmission {
|
||||
|
||||
@ -31,12 +31,14 @@ import org.eclipse.jetty.servlet.{ErrorPageErrorHandler, FilterHolder}
|
||||
import org.apache.kyuubi.{KyuubiException, Utils}
|
||||
import org.apache.kyuubi.config.KyuubiConf
|
||||
import org.apache.kyuubi.config.KyuubiConf._
|
||||
import org.apache.kyuubi.metrics.MetricsConstants.OPERATION_BATCH_PENDING_MAX_ELAPSE
|
||||
import org.apache.kyuubi.metrics.MetricsSystem
|
||||
import org.apache.kyuubi.server.api.v1.ApiRootResource
|
||||
import org.apache.kyuubi.server.http.authentication.{AuthenticationFilter, KyuubiHttpAuthenticationFactory}
|
||||
import org.apache.kyuubi.server.ui.{JettyServer, JettyUtils}
|
||||
import org.apache.kyuubi.service.{AbstractFrontendService, Serverable, Service, ServiceUtils}
|
||||
import org.apache.kyuubi.service.authentication.{AuthTypes, AuthUtils}
|
||||
import org.apache.kyuubi.session.{KyuubiSessionManager, SessionHandle}
|
||||
import org.apache.kyuubi.session.{KyuubiBatchSession, KyuubiSessionManager, SessionHandle}
|
||||
import org.apache.kyuubi.util.{JavaUtils, ThreadUtils}
|
||||
import org.apache.kyuubi.util.ThreadUtils.scheduleTolerableRunnableWithFixedDelay
|
||||
|
||||
@ -202,6 +204,14 @@ class KyuubiRestFrontendService(override val serverable: Serverable)
|
||||
}
|
||||
}
|
||||
|
||||
private def getBatchPendingMaxElapse(): Long = {
|
||||
val batchPendingElapseTimes = sessionManager.allSessions().map {
|
||||
case session: KyuubiBatchSession => session.batchJobSubmissionOp.getPendingElapsedTime
|
||||
case _ => 0L
|
||||
}
|
||||
if (batchPendingElapseTimes.isEmpty) 0L else batchPendingElapseTimes.max
|
||||
}
|
||||
|
||||
def waitForServerStarted(): Unit = {
|
||||
// block until the HTTP server is started, otherwise, we may get
|
||||
// the wrong HTTP server port -1
|
||||
@ -220,6 +230,9 @@ class KyuubiRestFrontendService(override val serverable: Serverable)
|
||||
isStarted.set(true)
|
||||
startBatchChecker()
|
||||
recoverBatchSessions()
|
||||
MetricsSystem.tracing { ms =>
|
||||
ms.registerGauge(OPERATION_BATCH_PENDING_MAX_ELAPSE, getBatchPendingMaxElapse, 0)
|
||||
}
|
||||
} catch {
|
||||
case e: Exception => throw new KyuubiException(s"Cannot start $getName", e)
|
||||
}
|
||||
|
||||
@ -67,6 +67,7 @@ private[v1] class BatchesResource extends ApiRequestContext with Logging {
|
||||
fe.getConf.get(ENGINE_SECURITY_ENABLED)
|
||||
private lazy val resourceFileMaxSize = fe.getConf.get(BATCH_RESOURCE_FILE_MAX_SIZE)
|
||||
private lazy val extraResourceFileMaxSize = fe.getConf.get(BATCH_EXTRA_RESOURCE_FILE_MAX_SIZE)
|
||||
private lazy val metadataSearchWindow = fe.getConf.get(METADATA_SEARCH_WINDOW)
|
||||
|
||||
private def batchV2Enabled(reqConf: Map[String, String]): Boolean = {
|
||||
fe.getConf.get(BATCH_SUBMITTER_ENABLED) &&
|
||||
@ -420,13 +421,15 @@ private[v1] class BatchesResource extends ApiRequestContext with Logging {
|
||||
s"The valid batch state can be one of the following: ${VALID_BATCH_STATES.mkString(",")}")
|
||||
}
|
||||
|
||||
val createTimeFilter =
|
||||
math.max(createTime, metadataSearchWindow.map(System.currentTimeMillis() - _).getOrElse(0L))
|
||||
val filter = MetadataFilter(
|
||||
sessionType = SessionType.BATCH,
|
||||
engineType = batchType,
|
||||
username = batchUser,
|
||||
state = batchState,
|
||||
requestName = batchName,
|
||||
createTime = createTime,
|
||||
createTime = createTimeFilter,
|
||||
endTime = endTime)
|
||||
val batches = sessionManager.getBatchesFromMetadataStore(filter, from, size, desc)
|
||||
new GetBatchesResponse(from, batches.size, batches.asJava)
|
||||
|
||||
@ -139,8 +139,13 @@ class MetadataManager extends AbstractService("MetadataManager") {
|
||||
from: Int,
|
||||
size: Int,
|
||||
desc: Boolean = false): Seq[Batch] = {
|
||||
withMetadataRequestMetrics(_metadataStore.getMetadataList(filter, from, size, desc)).map(
|
||||
buildBatch)
|
||||
withMetadataRequestMetrics(_metadataStore.getMetadataList(
|
||||
filter,
|
||||
from,
|
||||
size,
|
||||
// if create_file field is set, order by create_time, which is faster, otherwise by key_id
|
||||
orderBy = if (filter.createTime > 0) Some("create_time") else Some("key_id"),
|
||||
direction = if (desc) "DESC" else "ASC")).map(buildBatch)
|
||||
}
|
||||
|
||||
def countBatch(
|
||||
|
||||
@ -58,13 +58,16 @@ trait MetadataStore extends Closeable {
|
||||
* @param from the metadata offset.
|
||||
* @param size the size to get.
|
||||
* @param desc the order of metadata list.
|
||||
* @param orderBy the order by column, default is the auto increment primary key, `key_id`.
|
||||
* @param direction the order direction, default is `ASC`.
|
||||
* @return selected metadata list.
|
||||
*/
|
||||
def getMetadataList(
|
||||
filter: MetadataFilter,
|
||||
from: Int,
|
||||
size: Int,
|
||||
desc: Boolean = false): Seq[Metadata]
|
||||
orderBy: Option[String] = Some("key_id"),
|
||||
direction: String = "ASC"): Seq[Metadata]
|
||||
|
||||
/**
|
||||
* Count the metadata list with filter conditions.
|
||||
|
||||
@ -257,15 +257,15 @@ class JDBCMetadataStore(conf: KyuubiConf) extends MetadataStore with Logging {
|
||||
filter: MetadataFilter,
|
||||
from: Int,
|
||||
size: Int,
|
||||
desc: Boolean = false): Seq[Metadata] = {
|
||||
orderBy: Option[String] = Some("key_id"),
|
||||
direction: String = "ASC"): Seq[Metadata] = {
|
||||
val queryBuilder = new StringBuilder
|
||||
val params = ListBuffer[Any]()
|
||||
queryBuilder.append("SELECT ")
|
||||
queryBuilder.append(METADATA_COLUMNS)
|
||||
queryBuilder.append(s" FROM $METADATA_TABLE")
|
||||
queryBuilder.append(s" ${assembleWhereClause(filter, params)}")
|
||||
queryBuilder.append(" ORDER BY key_id ")
|
||||
queryBuilder.append(if (desc) "DESC " else "ASC ")
|
||||
orderBy.foreach(o => queryBuilder.append(s" ORDER BY $o $direction "))
|
||||
queryBuilder.append(dialect.limitClause(size, from))
|
||||
val query = queryBuilder.toString
|
||||
JdbcUtils.withConnection { connection =>
|
||||
|
||||
@ -486,6 +486,7 @@ abstract class BatchesResourceSuiteBase extends KyuubiFunSuite
|
||||
.queryParam("from", "0")
|
||||
.queryParam("size", "1")
|
||||
.queryParam("desc", "true")
|
||||
.queryParam("createTime", "1")
|
||||
.request(MediaType.APPLICATION_JSON_TYPE)
|
||||
.header(AUTHORIZATION_HEADER, basicAuthorizationHeader("anonymous"))
|
||||
.get()
|
||||
|
||||
Loading…
Reference in New Issue
Block a user