apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

[KYUUBI #5185] [Improvement] Use Set collection for order-insensitive configs

Browse Source 
### _Why are the changes needed?_

- Use `Set` collection for order-insensitive configs instead of `Seq`
  - kyuubi.frontend.thrift.binary.ssl.disallowed.protocols
  - kyuubi.authentication
  - kyuubi.authentication.ldap.groupFilter
  - kyuubi.authentication.ldap.userFilter
  - kyuubi.kubernetes.context.allow.list
  - kyuubi.kubernetes.namespace.allow.list
  - kyuubi.session.conf.ignore.list
  - kyuubi.session.conf.restrict.list
  - kyuubi.session.local.dir.allow.list
  - kyuubi.batch.conf.ignore.list
  - kyuubi.engine.deregister.exception.classes
  - kyuubi.engine.deregister.exception.messages
  - kyuubi.operation.plan.only.excludes
  - kyuubi.server.limit.connections.user.unlimited.list
  - kyuubi.server.administrators
  - kyuubi.metrics.reporters
- Support skipping blank elements

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No.

Closes #5185 from bowenliang123/conf-toset.

Closes #5185

c656af78a [liangbowen] conf to set

Authored-by: liangbowen <liangbowen@gf.com.cn>
Signed-off-by: liangbowen <liangbowen@gf.com.cn>
This commit is contained in:
liangbowen 2023-08-25 10:28:15 +08:00
parent f42a7d6cca
commit beea6d5fce
30 changed files with 125 additions and 108 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

32
docs/configuration/settings.md
View File

@ -33,7 +33,7 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
| Key | Default | Meaning | Type | Since |
|-----------------------------------------------|-------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------|-------|
| kyuubi.authentication | NONE | A comma-separated list of client authentication types.<ul> <li>NOSASL: raw transport.</li> <li>NONE: no authentication check.</li> <li>KERBEROS: Kerberos/GSSAPI authentication.</li> <li>CUSTOM: User-defined authentication.</li> <li>JDBC: JDBC query authentication.</li> <li>LDAP: Lightweight Directory Access Protocol authentication.</li></ul>The following tree describes the catalog of each option.<ul> <li><code>NOSASL</code></li> <li>SASL <ul> <li>SASL/PLAIN</li> <ul> <li><code>NONE</code></li> <li><code>LDAP</code></li> <li><code>JDBC</code></li> <li><code>CUSTOM</code></li> </ul> <li>SASL/GSSAPI <ul> <li><code>KERBEROS</code></li> </ul> </li> </ul> </li></ul> Note that: for SASL authentication, KERBEROS and PLAIN auth types are supported at the same time, and only the first specified PLAIN auth type is valid. | seq | 1.0.0 |
| kyuubi.authentication | NONE | A comma-separated list of client authentication types.<ul> <li>NOSASL: raw transport.</li> <li>NONE: no authentication check.</li> <li>KERBEROS: Kerberos/GSSAPI authentication.</li> <li>CUSTOM: User-defined authentication.</li> <li>JDBC: JDBC query authentication.</li> <li>LDAP: Lightweight Directory Access Protocol authentication.</li></ul>The following tree describes the catalog of each option.<ul> <li><code>NOSASL</code></li> <li>SASL <ul> <li>SASL/PLAIN</li> <ul> <li><code>NONE</code></li> <li><code>LDAP</code></li> <li><code>JDBC</code></li> <li><code>CUSTOM</code></li> </ul> <li>SASL/GSSAPI <ul> <li><code>KERBEROS</code></li> </ul> </li> </ul> </li></ul> Note that: for SASL authentication, KERBEROS and PLAIN auth types are supported at the same time, and only the first specified PLAIN auth type is valid. | set | 1.0.0 |
| kyuubi.authentication.custom.class | &lt;undefined&gt; | User-defined authentication implementation of org.apache.kyuubi.service.authentication.PasswdAuthenticationProvider | string | 1.3.0 |
| kyuubi.authentication.jdbc.driver.class | &lt;undefined&gt; | Driver class name for JDBC Authentication Provider. | string | 1.6.0 |
| kyuubi.authentication.jdbc.password | &lt;undefined&gt; | Database password for JDBC Authentication Provider. | string | 1.6.0 |
@ -47,12 +47,12 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
| kyuubi.authentication.ldap.domain | &lt;undefined&gt; | LDAP domain. | string | 1.0.0 |
| kyuubi.authentication.ldap.groupClassKey | groupOfNames | LDAP attribute name on the group entry that is to be used in LDAP group searches. For example: group, groupOfNames or groupOfUniqueNames. | string | 1.7.0 |
| kyuubi.authentication.ldap.groupDNPattern | &lt;undefined&gt; | COLON-separated list of patterns to use to find DNs for group entities in this directory. Use %s where the actual group name is to be substituted for. For example: CN=%s,CN=Groups,DC=subdomain,DC=domain,DC=com. | string | 1.7.0 |
| kyuubi.authentication.ldap.groupFilter || COMMA-separated list of LDAP Group names (short name not full DNs). For example: HiveAdmins,HadoopAdmins,Administrators | seq | 1.7.0 |
| kyuubi.authentication.ldap.groupFilter || COMMA-separated list of LDAP Group names (short name not full DNs). For example: HiveAdmins,HadoopAdmins,Administrators | set | 1.7.0 |
| kyuubi.authentication.ldap.groupMembershipKey | member | LDAP attribute name on the group object that contains the list of distinguished names for the user, group, and contact objects that are members of the group. For example: member, uniqueMember or memberUid | string | 1.7.0 |
| kyuubi.authentication.ldap.guidKey | uid | LDAP attribute name whose values are unique in this LDAP server. For example: uid or CN. | string | 1.2.0 |
| kyuubi.authentication.ldap.url | &lt;undefined&gt; | SPACE character separated LDAP connection URL(s). | string | 1.0.0 |
| kyuubi.authentication.ldap.userDNPattern | &lt;undefined&gt; | COLON-separated list of patterns to use to find DNs for users in this directory. Use %s where the actual group name is to be substituted for. For example: CN=%s,CN=Users,DC=subdomain,DC=domain,DC=com. | string | 1.7.0 |
| kyuubi.authentication.ldap.userFilter || COMMA-separated list of LDAP usernames (just short names, not full DNs). For example: hiveuser,impalauser,hiveadmin,hadoopadmin | seq | 1.7.0 |
| kyuubi.authentication.ldap.userFilter || COMMA-separated list of LDAP usernames (just short names, not full DNs). For example: hiveuser,impalauser,hiveadmin,hadoopadmin | set | 1.7.0 |
| kyuubi.authentication.ldap.userMembershipKey | &lt;undefined&gt; | LDAP attribute name on the user object that contains groups of which the user is a direct member, except for the primary group, which is represented by the primaryGroupId. For example: memberOf | string | 1.7.0 |
| kyuubi.authentication.sasl.qop | auth | Sasl QOP enable higher levels of protection for Kyuubi communication with clients.<ul> <li>auth - authentication only (default)</li> <li>auth-int - authentication plus integrity protection</li> <li>auth-conf - authentication plus integrity and confidentiality protection. This is applicable only if Kyuubi is configured to use Kerberos authentication.</li> </ul> | string | 1.0.0 |
@ -79,7 +79,7 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
|---------------------------------------------|---------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
| kyuubi.batch.application.check.interval | PT5S | The interval to check batch job application information. | duration | 1.6.0 |
| kyuubi.batch.application.starvation.timeout | PT3M | Threshold above which to warn batch application may be starved. | duration | 1.7.0 |
| kyuubi.batch.conf.ignore.list || A comma-separated list of ignored keys for batch conf. If the batch conf contains any of them, the key and the corresponding value will be removed silently during batch job submission. Note that this rule is for server-side protection defined via administrators to prevent some essential configs from tampering. You can also pre-define some config for batch job submission with the prefix: kyuubi.batchConf.[batchType]. For example, you can pre-define `spark.master` for the Spark batch job with key `kyuubi.batchConf.spark.spark.master`. | seq | 1.6.0 |
| kyuubi.batch.conf.ignore.list || A comma-separated list of ignored keys for batch conf. If the batch conf contains any of them, the key and the corresponding value will be removed silently during batch job submission. Note that this rule is for server-side protection defined via administrators to prevent some essential configs from tampering. You can also pre-define some config for batch job submission with the prefix: kyuubi.batchConf.[batchType]. For example, you can pre-define `spark.master` for the Spark batch job with key `kyuubi.batchConf.spark.spark.master`. | set | 1.6.0 |
| kyuubi.batch.session.idle.timeout | PT6H | Batch session idle timeout, it will be closed when it's not accessed for this duration | duration | 1.6.2 |
### Credentials
@ -132,8 +132,8 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
| kyuubi.engine.chat.memory | 1g | The heap memory for the Chat engine | string | 1.8.0 |
| kyuubi.engine.chat.provider | ECHO | The provider for the Chat engine. Candidates: <ul> <li>ECHO: simply replies a welcome message.</li> <li>GPT: a.k.a ChatGPT, powered by OpenAI.</li></ul> | string | 1.8.0 |
| kyuubi.engine.connection.url.use.hostname | true | (deprecated) When true, the engine registers with hostname to zookeeper. When Spark runs on K8s with cluster mode, set to false to ensure that server can connect to engine | boolean | 1.3.0 |
| kyuubi.engine.deregister.exception.classes || A comma-separated list of exception classes. If there is any exception thrown, whose class matches the specified classes, the engine would deregister itself. | seq | 1.2.0 |
| kyuubi.engine.deregister.exception.messages || A comma-separated list of exception messages. If there is any exception thrown, whose message or stacktrace matches the specified message list, the engine would deregister itself. | seq | 1.2.0 |
| kyuubi.engine.deregister.exception.classes || A comma-separated list of exception classes. If there is any exception thrown, whose class matches the specified classes, the engine would deregister itself. | set | 1.2.0 |
| kyuubi.engine.deregister.exception.messages || A comma-separated list of exception messages. If there is any exception thrown, whose message or stacktrace matches the specified message list, the engine would deregister itself. | set | 1.2.0 |
| kyuubi.engine.deregister.exception.ttl | PT30M | Time to live(TTL) for exceptions pattern specified in kyuubi.engine.deregister.exception.classes and kyuubi.engine.deregister.exception.messages to deregister engines. Once the total error count hits the kyuubi.engine.deregister.job.max.failures within the TTL, an engine will deregister itself and wait for self-terminated. Otherwise, we suppose that the engine has recovered from temporary failures. | duration | 1.2.0 |
| kyuubi.engine.deregister.job.max.failures | 4 | Number of failures of job before deregistering the engine. | int | 1.2.0 |
| kyuubi.engine.event.json.log.path | file:///tmp/kyuubi/events | The location where all the engine events go for the built-in JSON logger.<ul><li>Local Path: start with 'file://'</li><li>HDFS Path: start with 'hdfs://'</li></ul> | string | 1.3.0 |
@ -236,7 +236,7 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
| kyuubi.frontend.thrift.backoff.slot.length | PT0.1S | Time to back off during login to the thrift frontend service. | duration | 1.4.0 |
| kyuubi.frontend.thrift.binary.bind.host | &lt;undefined&gt; | Hostname or IP of the machine on which to run the thrift frontend service via the binary protocol. | string | 1.4.0 |
| kyuubi.frontend.thrift.binary.bind.port | 10009 | Port of the machine on which to run the thrift frontend service via the binary protocol. | int | 1.4.0 |
| kyuubi.frontend.thrift.binary.ssl.disallowed.protocols | SSLv2,SSLv3 | SSL versions to disallow for Kyuubi thrift binary frontend. | seq | 1.7.0 |
| kyuubi.frontend.thrift.binary.ssl.disallowed.protocols | SSLv2,SSLv3 | SSL versions to disallow for Kyuubi thrift binary frontend. | set | 1.7.0 |
| kyuubi.frontend.thrift.binary.ssl.enabled | false | Set this to true for using SSL encryption in thrift binary frontend server. | boolean | 1.7.0 |
| kyuubi.frontend.thrift.binary.ssl.include.ciphersuites || A comma-separated list of include SSL cipher suite names for thrift binary frontend. | seq | 1.7.0 |
| kyuubi.frontend.thrift.http.allow.user.substitution | true | Allow alternate user to be specified as part of open connection request when using HTTP transport mode. | boolean | 1.6.0 |
@ -317,10 +317,10 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
| kyuubi.kubernetes.authenticate.oauthToken | &lt;undefined&gt; | The OAuth token to use when authenticating against the Kubernetes API server. Note that unlike, the other authentication options, this must be the exact string value of the token to use for the authentication. | string | 1.7.0 |
| kyuubi.kubernetes.authenticate.oauthTokenFile | &lt;undefined&gt; | Path to the file containing the OAuth token to use when authenticating against the Kubernetes API server. Specify this as a path as opposed to a URI (i.e. do not provide a scheme) | string | 1.7.0 |
| kyuubi.kubernetes.context | &lt;undefined&gt; | The desired context from your kubernetes config file used to configure the K8s client for interacting with the cluster. | string | 1.6.0 |
| kyuubi.kubernetes.context.allow.list || The allowed kubernetes context list, if it is empty, there is no kubernetes context limitation. | seq | 1.8.0 |
| kyuubi.kubernetes.context.allow.list || The allowed kubernetes context list, if it is empty, there is no kubernetes context limitation. | set | 1.8.0 |
| kyuubi.kubernetes.master.address | &lt;undefined&gt; | The internal Kubernetes master (API server) address to be used for kyuubi. | string | 1.7.0 |
| kyuubi.kubernetes.namespace | default | The namespace that will be used for running the kyuubi pods and find engines. | string | 1.7.0 |
| kyuubi.kubernetes.namespace.allow.list || The allowed kubernetes namespace list, if it is empty, there is no kubernetes namespace limitation. | seq | 1.8.0 |
| kyuubi.kubernetes.namespace.allow.list || The allowed kubernetes namespace list, if it is empty, there is no kubernetes namespace limitation. | set | 1.8.0 |
| kyuubi.kubernetes.terminatedApplicationRetainPeriod | PT5M | The period for which the Kyuubi server retains application information after the application terminates. | duration | 1.7.1 |
| kyuubi.kubernetes.trust.certificates | false | If set to true then client can submit to kubernetes cluster only with token | boolean | 1.7.0 |
@ -354,7 +354,7 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
| kyuubi.metrics.json.location | metrics | Where the JSON metrics file located | string | 1.2.0 |
| kyuubi.metrics.prometheus.path | /metrics | URI context path of prometheus metrics HTTP server | string | 1.2.0 |
| kyuubi.metrics.prometheus.port | 10019 | Prometheus metrics HTTP server port | int | 1.2.0 |
| kyuubi.metrics.reporters | JSON | A comma-separated list for all metrics reporters<ul> <li>CONSOLE - ConsoleReporter which outputs measurements to CONSOLE periodically.</li> <li>JMX - JmxReporter which listens for new metrics and exposes them as MBeans.</li> <li>JSON - JsonReporter which outputs measurements to json file periodically.</li> <li>PROMETHEUS - PrometheusReporter which exposes metrics in Prometheus format.</li> <li>SLF4J - Slf4jReporter which outputs measurements to system log periodically.</li></ul> | seq | 1.2.0 |
| kyuubi.metrics.reporters | JSON | A comma-separated list for all metrics reporters<ul> <li>CONSOLE - ConsoleReporter which outputs measurements to CONSOLE periodically.</li> <li>JMX - JmxReporter which listens for new metrics and exposes them as MBeans.</li> <li>JSON - JsonReporter which outputs measurements to json file periodically.</li> <li>PROMETHEUS - PrometheusReporter which exposes metrics in Prometheus format.</li> <li>SLF4J - Slf4jReporter which outputs measurements to system log periodically.</li></ul> | set | 1.2.0 |
| kyuubi.metrics.slf4j.interval | PT5S | How often should report metrics to SLF4J logger | duration | 1.2.0 |
### Operation
@ -366,7 +366,7 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
| kyuubi.operation.interrupt.on.cancel | true | When true, all running tasks will be interrupted if one cancels a query. When false, all running tasks will remain until finished. | boolean | 1.2.0 |
| kyuubi.operation.language | SQL | Choose a programing language for the following inputs<ul><li>SQL: (Default) Run all following statements as SQL queries.</li><li>SCALA: Run all following input as scala codes</li><li>PYTHON: (Experimental) Run all following input as Python codes with Spark engine</li></ul> | string | 1.5.0 |
| kyuubi.operation.log.dir.root | server_operation_logs | Root directory for query operation log at server-side. | string | 1.4.0 |
| kyuubi.operation.plan.only.excludes | ResetCommand,SetCommand,SetNamespaceCommand,UseStatement,SetCatalogAndNamespace | Comma-separated list of query plan names, in the form of simple class names, i.e, for `SET abc=xyz`, the value will be `SetCommand`. For those auxiliary plans, such as `switch databases`, `set properties`, or `create temporary view` etc., which are used for setup evaluating environments for analyzing actual queries, we can use this config to exclude them and let them take effect. See also kyuubi.operation.plan.only.mode. | seq | 1.5.0 |
| kyuubi.operation.plan.only.excludes | SetCatalogAndNamespace,UseStatement,SetNamespaceCommand,SetCommand,ResetCommand | Comma-separated list of query plan names, in the form of simple class names, i.e, for `SET abc=xyz`, the value will be `SetCommand`. For those auxiliary plans, such as `switch databases`, `set properties`, or `create temporary view` etc., which are used for setup evaluating environments for analyzing actual queries, we can use this config to exclude them and let them take effect. See also kyuubi.operation.plan.only.mode. | set | 1.5.0 |
| kyuubi.operation.plan.only.mode | none | Configures the statement performed mode, The value can be 'parse', 'analyze', 'optimize', 'optimize_with_stats', 'physical', 'execution', or 'none', when it is 'none', indicate to the statement will be fully executed, otherwise only way without executing the query. different engines currently support different modes, the Spark engine supports all modes, and the Flink engine supports 'parse', 'physical', and 'execution', other engines do not support planOnly currently. | string | 1.4.0 |
| kyuubi.operation.plan.only.output.style | plain | Configures the planOnly output style. The value can be 'plain' or 'json', and the default value is 'plain'. This configuration supports only the output styles of the Spark engine | string | 1.7.0 |
| kyuubi.operation.progress.enabled | false | Whether to enable the operation progress. When true, the operation progress will be returned in `GetOperationStatus`. | boolean | 1.6.0 |
@ -382,7 +382,7 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
| Key | Default | Meaning | Type | Since |
|----------------------------------------------------------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
| kyuubi.server.administrators || Comma-separated list of Kyuubi service administrators. We use this config to grant admin permission to any service accounts. | seq | 1.8.0 |
| kyuubi.server.administrators || Comma-separated list of Kyuubi service administrators. We use this config to grant admin permission to any service accounts. | set | 1.8.0 |
| kyuubi.server.info.provider | ENGINE | The server information provider name, some clients may rely on this information to check the server compatibilities and functionalities. <li>SERVER: Return Kyuubi server information.</li> <li>ENGINE: Return Kyuubi engine information.</li> | string | 1.6.1 |
| kyuubi.server.limit.batch.connections.per.ipaddress | &lt;undefined&gt; | Maximum kyuubi server batch connections per ipaddress. Any user exceeding this limit will not be allowed to connect. | int | 1.7.0 |
| kyuubi.server.limit.batch.connections.per.user | &lt;undefined&gt; | Maximum kyuubi server batch connections per user. Any user exceeding this limit will not be allowed to connect. | int | 1.7.0 |
@ -391,7 +391,7 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
| kyuubi.server.limit.connections.per.ipaddress | &lt;undefined&gt; | Maximum kyuubi server connections per ipaddress. Any user exceeding this limit will not be allowed to connect. | int | 1.6.0 |
| kyuubi.server.limit.connections.per.user | &lt;undefined&gt; | Maximum kyuubi server connections per user. Any user exceeding this limit will not be allowed to connect. | int | 1.6.0 |
| kyuubi.server.limit.connections.per.user.ipaddress | &lt;undefined&gt; | Maximum kyuubi server connections per user:ipaddress combination. Any user-ipaddress exceeding this limit will not be allowed to connect. | int | 1.6.0 |
| kyuubi.server.limit.connections.user.unlimited.list || The maximum connections of the user in the white list will not be limited. | seq | 1.7.0 |
| kyuubi.server.limit.connections.user.unlimited.list || The maximum connections of the user in the white list will not be limited. | set | 1.7.0 |
| kyuubi.server.name | &lt;undefined&gt; | The name of Kyuubi Server. | string | 1.5.0 |
| kyuubi.server.periodicGC.interval | PT30M | How often to trigger a garbage collection. | duration | 1.7.0 |
| kyuubi.server.redaction.regex | &lt;undefined&gt; | Regex to decide which Kyuubi contain sensitive information. When this regex matches a property key or value, the value is redacted from the various logs. || 1.6.0 |
@ -404,9 +404,9 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
| kyuubi.session.close.on.disconnect | true | Session will be closed when client disconnects from kyuubi gateway. Set this to false to have session outlive its parent connection. | boolean | 1.8.0 |
| kyuubi.session.conf.advisor | &lt;undefined&gt; | A config advisor plugin for Kyuubi Server. This plugin can provide some custom configs for different users or session configs and overwrite the session configs before opening a new session. This config value should be a subclass of `org.apache.kyuubi.plugin.SessionConfAdvisor` which has a zero-arg constructor. | string | 1.5.0 |
| kyuubi.session.conf.file.reload.interval | PT10M | When `FileSessionConfAdvisor` is used, this configuration defines the expired time of `$KYUUBI_CONF_DIR/kyuubi-session-<profile>.conf` in the cache. After exceeding this value, the file will be reloaded. | duration | 1.7.0 |
| kyuubi.session.conf.ignore.list || A comma-separated list of ignored keys. If the client connection contains any of them, the key and the corresponding value will be removed silently during engine bootstrap and connection setup. Note that this rule is for server-side protection defined via administrators to prevent some essential configs from tampering but will not forbid users to set dynamic configurations via SET syntax. | seq | 1.2.0 |
| kyuubi.session.conf.ignore.list || A comma-separated list of ignored keys. If the client connection contains any of them, the key and the corresponding value will be removed silently during engine bootstrap and connection setup. Note that this rule is for server-side protection defined via administrators to prevent some essential configs from tampering but will not forbid users to set dynamic configurations via SET syntax. | set | 1.2.0 |
| kyuubi.session.conf.profile | &lt;undefined&gt; | Specify a profile to load session-level configurations from `$KYUUBI_CONF_DIR/kyuubi-session-<profile>.conf`. This configuration will be ignored if the file does not exist. This configuration only takes effect when `kyuubi.session.conf.advisor` is set as `org.apache.kyuubi.session.FileSessionConfAdvisor`. | string | 1.7.0 |
| kyuubi.session.conf.restrict.list || A comma-separated list of restricted keys. If the client connection contains any of them, the connection will be rejected explicitly during engine bootstrap and connection setup. Note that this rule is for server-side protection defined via administrators to prevent some essential configs from tampering but will not forbid users to set dynamic configurations via SET syntax. | seq | 1.2.0 |
| kyuubi.session.conf.restrict.list || A comma-separated list of restricted keys. If the client connection contains any of them, the connection will be rejected explicitly during engine bootstrap and connection setup. Note that this rule is for server-side protection defined via administrators to prevent some essential configs from tampering but will not forbid users to set dynamic configurations via SET syntax. | set | 1.2.0 |
| kyuubi.session.engine.alive.probe.enabled | false | Whether to enable the engine alive probe, it true, we will create a companion thrift client that keeps sending simple requests to check whether the engine is alive. | boolean | 1.6.0 |
| kyuubi.session.engine.alive.probe.interval | PT10S | The interval for engine alive probe. | duration | 1.6.0 |
| kyuubi.session.engine.alive.timeout | PT2M | The timeout for engine alive. If there is no alive probe success in the last timeout window, the engine will be marked as no-alive. | duration | 1.6.0 |
@ -440,7 +440,7 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
| kyuubi.session.engine.trino.showProgress.debug | false | When true, show the progress debug info in the Trino engine log. | boolean | 1.6.0 |
| kyuubi.session.group.provider | hadoop | A group provider plugin for Kyuubi Server. This plugin can provide primary group and groups information for different users or session configs. This config value should be a subclass of `org.apache.kyuubi.plugin.GroupProvider` which has a zero-arg constructor. Kyuubi provides the following built-in implementations: <li>hadoop: delegate the user group mapping to hadoop UserGroupInformation.</li> | string | 1.7.0 |
| kyuubi.session.idle.timeout | PT6H | session idle timeout, it will be closed when it's not accessed for this duration | duration | 1.2.0 |
| kyuubi.session.local.dir.allow.list || The local dir list that are allowed to access by the kyuubi session application. End-users might set some parameters such as `spark.files` and it will upload some local files when launching the kyuubi engine, if the local dir allow list is defined, kyuubi will check whether the path to upload is in the allow list. Note that, if it is empty, there is no limitation for that. And please use absolute paths. | seq | 1.6.0 |
| kyuubi.session.local.dir.allow.list || The local dir list that are allowed to access by the kyuubi session application. End-users might set some parameters such as `spark.files` and it will upload some local files when launching the kyuubi engine, if the local dir allow list is defined, kyuubi will check whether the path to upload is in the allow list. Note that, if it is empty, there is no limitation for that. And please use absolute paths. | set | 1.6.0 |
| kyuubi.session.name | &lt;undefined&gt; | A human readable name of the session and we use empty string by default. This name will be recorded in the event. Note that, we only apply this value from session conf. | string | 1.4.0 |
| kyuubi.session.timeout | PT6H | (deprecated)session timeout, it will be closed when it's not accessed for this duration | duration | 1.0.0 |
| kyuubi.session.user.sign.enabled | false | Whether to verify the integrity of session user name on the engine side, e.g. Authz plugin in Spark. | boolean | 1.7.0 |

4
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/PlanOnlyStatement.scala vendored
View File

@ -41,8 +41,8 @@ class PlanOnlyStatement(
extends SparkOperation(session) {
private val operationLog: OperationLog = OperationLog.createOperationLog(session, getHandle)
private val planExcludes: Seq[String] = {
spark.conf.getOption(OPERATION_PLAN_ONLY_EXCLUDES.key).map(_.split(",").map(_.trim).toSeq)
private val planExcludes: Set[String] = {
spark.conf.getOption(OPERATION_PLAN_ONLY_EXCLUDES.key).map(_.split(",").map(_.trim).toSet)
.getOrElse(session.sessionManager.getConf.get(OPERATION_PLAN_ONLY_EXCLUDES))
}

6
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SparkSQLEngineListener.scala vendored
View File

@ -40,9 +40,9 @@ import org.apache.kyuubi.service.{Serverable, ServiceState}
class SparkSQLEngineListener(server: Serverable) extends SparkListener with Logging {
// the conf of server is null before initialized, use lazy val here
private lazy val deregisterExceptions: Seq[String] =
private lazy val deregisterExceptions: Set[String] =
server.getConf.get(ENGINE_DEREGISTER_EXCEPTION_CLASSES)
private lazy val deregisterMessages: Seq[String] =
private lazy val deregisterMessages: Set[String] =
server.getConf.get(ENGINE_DEREGISTER_EXCEPTION_MESSAGES)
private lazy val deregisterExceptionTTL: Long =
server.getConf.get(ENGINE_DEREGISTER_EXCEPTION_TTL)
@ -74,7 +74,7 @@ class SparkSQLEngineListener(server: Serverable) extends SparkListener with Logg
case JobFailed(e) if e != null =>
val cause = findCause(e)
var deregisterInfo: Option[String] = None
if (deregisterExceptions.exists(_.equals(cause.getClass.getCanonicalName))) {
if (deregisterExceptions.contains(cause.getClass.getCanonicalName)) {
deregisterInfo = Some("Job failed exception class is in the set of " +
s"${ENGINE_DEREGISTER_EXCEPTION_CLASSES.key}, deregistering the engine.")
} else if (deregisterMessages.exists(stringifyException(cause).contains)) {

7
kyuubi-common/src/main/scala/org/apache/kyuubi/config/ConfigBuilder.scala
View File

@ -206,7 +206,12 @@ private[kyuubi] case class TypedConfigBuilder[T](
/** Turns the config entry into a sequence of values of the underlying type. */
def toSequence(sp: String = ","): TypedConfigBuilder[Seq[T]] = {
parent._type = "seq"
TypedConfigBuilder(parent, strToSeq(_, fromStr, sp), seqToStr(_, toStr))
TypedConfigBuilder(parent, strToSeq(_, fromStr, sp), iterableToStr(_, toStr))
}
def toSet(sp: String = ",", skipBlank: Boolean = true): TypedConfigBuilder[Set[T]] = {
parent._type = "set"
TypedConfigBuilder(parent, strToSet(_, fromStr, sp, skipBlank), iterableToStr(_, toStr))
}
def createOptional: OptionalConfigEntry[T] = {

10
kyuubi-common/src/main/scala/org/apache/kyuubi/config/ConfigHelpers.scala
View File

@ -17,6 +17,8 @@
package org.apache.kyuubi.config
import org.apache.commons.lang3.StringUtils
import org.apache.kyuubi.Utils
object ConfigHelpers {
@ -25,7 +27,11 @@ object ConfigHelpers {
Utils.strToSeq(str, sp).map(converter)
}
def seqToStr[T](v: Seq[T], stringConverter: T => String): String = {
v.map(stringConverter).mkString(",")
def strToSet[T](str: String, converter: String => T, sp: String, skipBlank: Boolean): Set[T] = {
Utils.strToSeq(str, sp).filter(!skipBlank || StringUtils.isNotBlank(_)).map(converter).toSet
}
def iterableToStr[T](v: Iterable[T], stringConverter: T => String, sp: String = ","): String = {
v.map(stringConverter).mkString(sp)
}
}

90
kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
View File

@ -483,13 +483,13 @@ object KyuubiConf {
.stringConf
.createOptional
val FRONTEND_THRIFT_BINARY_SSL_DISALLOWED_PROTOCOLS: ConfigEntry[Seq[String]] =
val FRONTEND_THRIFT_BINARY_SSL_DISALLOWED_PROTOCOLS: ConfigEntry[Set[String]] =
buildConf("kyuubi.frontend.thrift.binary.ssl.disallowed.protocols")
.doc("SSL versions to disallow for Kyuubi thrift binary frontend.")
.version("1.7.0")
.stringConf
.toSequence()
.createWithDefault(Seq("SSLv2", "SSLv3"))
.toSet()
.createWithDefault(Set("SSLv2", "SSLv3"))
val FRONTEND_THRIFT_BINARY_SSL_INCLUDE_CIPHER_SUITES: ConfigEntry[Seq[String]] =
buildConf("kyuubi.frontend.thrift.binary.ssl.include.ciphersuites")
@ -765,7 +765,7 @@ object KyuubiConf {
.stringConf
.createWithDefault("X-Real-IP")
val AUTHENTICATION_METHOD: ConfigEntry[Seq[String]] = buildConf("kyuubi.authentication")
val AUTHENTICATION_METHOD: ConfigEntry[Set[String]] = buildConf("kyuubi.authentication")
.doc("A comma-separated list of client authentication types." +
"<ul>" +
" <li>NOSASL: raw transport.</li>" +
@ -801,11 +801,11 @@ object KyuubiConf {
.serverOnly
.stringConf
.transformToUpperCase
.toSequence()
.toSet()
.checkValue(
_.forall(AuthTypes.values.map(_.toString).contains),
s"the authentication type should be one or more of ${AuthTypes.values.mkString(",")}")
.createWithDefault(Seq(AuthTypes.NONE.toString))
.createWithDefault(Set(AuthTypes.NONE.toString))
val AUTHENTICATION_CUSTOM_CLASS: OptionalConfigEntry[String] =
buildConf("kyuubi.authentication.custom.class")
@ -861,25 +861,25 @@ object KyuubiConf {
.stringConf
.createOptional
val AUTHENTICATION_LDAP_GROUP_FILTER: ConfigEntry[Seq[String]] =
val AUTHENTICATION_LDAP_GROUP_FILTER: ConfigEntry[Set[String]] =
buildConf("kyuubi.authentication.ldap.groupFilter")
.doc("COMMA-separated list of LDAP Group names (short name not full DNs). " +
"For example: HiveAdmins,HadoopAdmins,Administrators")
.version("1.7.0")
.serverOnly
.stringConf
.toSequence()
.createWithDefault(Nil)
.toSet()
.createWithDefault(Set.empty)
val AUTHENTICATION_LDAP_USER_FILTER: ConfigEntry[Seq[String]] =
val AUTHENTICATION_LDAP_USER_FILTER: ConfigEntry[Set[String]] =
buildConf("kyuubi.authentication.ldap.userFilter")
.doc("COMMA-separated list of LDAP usernames (just short names, not full DNs). " +
"For example: hiveuser,impalauser,hiveadmin,hadoopadmin")
.version("1.7.0")
.serverOnly
.stringConf
.toSequence()
.createWithDefault(Nil)
.toSet()
.createWithDefault(Set.empty)
val AUTHENTICATION_LDAP_GUID_KEY: ConfigEntry[String] =
buildConf("kyuubi.authentication.ldap.guidKey")
@ -1142,14 +1142,14 @@ object KyuubiConf {
.stringConf
.createOptional
val KUBERNETES_CONTEXT_ALLOW_LIST: ConfigEntry[Seq[String]] =
val KUBERNETES_CONTEXT_ALLOW_LIST: ConfigEntry[Set[String]] =
buildConf("kyuubi.kubernetes.context.allow.list")
.doc("The allowed kubernetes context list, if it is empty," +
" there is no kubernetes context limitation.")
.version("1.8.0")
.stringConf
.toSequence()
.createWithDefault(Nil)
.toSet()
.createWithDefault(Set.empty)
val KUBERNETES_NAMESPACE: ConfigEntry[String] =
buildConf("kyuubi.kubernetes.namespace")
@ -1158,14 +1158,14 @@ object KyuubiConf {
.stringConf
.createWithDefault("default")
val KUBERNETES_NAMESPACE_ALLOW_LIST: ConfigEntry[Seq[String]] =
val KUBERNETES_NAMESPACE_ALLOW_LIST: ConfigEntry[Set[String]] =
buildConf("kyuubi.kubernetes.namespace.allow.list")
.doc("The allowed kubernetes namespace list, if it is empty," +
" there is no kubernetes namespace limitation.")
.version("1.8.0")
.stringConf
.toSequence()
.createWithDefault(Nil)
.toSet()
.createWithDefault(Set.empty)
val KUBERNETES_MASTER: OptionalConfigEntry[String] =
buildConf("kyuubi.kubernetes.master.address")
@ -1517,7 +1517,7 @@ object KyuubiConf {
.timeConf
.createWithDefault(Duration.ofMinutes(30L).toMillis)
val SESSION_CONF_IGNORE_LIST: ConfigEntry[Seq[String]] =
val SESSION_CONF_IGNORE_LIST: ConfigEntry[Set[String]] =
buildConf("kyuubi.session.conf.ignore.list")
.doc("A comma-separated list of ignored keys. If the client connection contains any of" +
" them, the key and the corresponding value will be removed silently during engine" +
@ -1527,10 +1527,10 @@ object KyuubiConf {
" configurations via SET syntax.")
.version("1.2.0")
.stringConf
.toSequence()
.createWithDefault(Nil)
.toSet()
.createWithDefault(Set.empty)
val SESSION_CONF_RESTRICT_LIST: ConfigEntry[Seq[String]] =
val SESSION_CONF_RESTRICT_LIST: ConfigEntry[Set[String]] =
buildConf("kyuubi.session.conf.restrict.list")
.doc("A comma-separated list of restricted keys. If the client connection contains any of" +
" them, the connection will be rejected explicitly during engine bootstrap and connection" +
@ -1540,8 +1540,8 @@ object KyuubiConf {
" configurations via SET syntax.")
.version("1.2.0")
.stringConf
.toSequence()
.createWithDefault(Nil)
.toSet()
.createWithDefault(Set.empty)
val SESSION_USER_SIGN_ENABLED: ConfigEntry[Boolean] =
buildConf("kyuubi.session.user.sign.enabled")
@ -1589,7 +1589,7 @@ object KyuubiConf {
.booleanConf
.createWithDefault(true)
val SESSION_LOCAL_DIR_ALLOW_LIST: ConfigEntry[Seq[String]] =
val SESSION_LOCAL_DIR_ALLOW_LIST: ConfigEntry[Set[String]] =
buildConf("kyuubi.session.local.dir.allow.list")
.doc("The local dir list that are allowed to access by the kyuubi session application. " +
" End-users might set some parameters such as `spark.files` and it will " +
@ -1602,8 +1602,8 @@ object KyuubiConf {
.stringConf
.checkValue(dir => dir.startsWith(File.separator), "the dir should be absolute path")
.transform(dir => dir.stripSuffix(File.separator) + File.separator)
.toSequence()
.createWithDefault(Nil)
.toSet()
.createWithDefault(Set.empty)
val BATCH_APPLICATION_CHECK_INTERVAL: ConfigEntry[Long] =
buildConf("kyuubi.batch.application.check.interval")
@ -1619,7 +1619,7 @@ object KyuubiConf {
.timeConf
.createWithDefault(Duration.ofMinutes(3).toMillis)
val BATCH_CONF_IGNORE_LIST: ConfigEntry[Seq[String]] =
val BATCH_CONF_IGNORE_LIST: ConfigEntry[Set[String]] =
buildConf("kyuubi.batch.conf.ignore.list")
.doc("A comma-separated list of ignored keys for batch conf. If the batch conf contains" +
" any of them, the key and the corresponding value will be removed silently during batch" +
@ -1631,8 +1631,8 @@ object KyuubiConf {
" for the Spark batch job with key `kyuubi.batchConf.spark.spark.master`.")
.version("1.6.0")
.stringConf
.toSequence()
.createWithDefault(Nil)
.toSet()
.createWithDefault(Set.empty)
val BATCH_INTERNAL_REST_CLIENT_SOCKET_TIMEOUT: ConfigEntry[Long] =
buildConf("kyuubi.batch.internal.rest.client.socket.timeout")
@ -2076,24 +2076,24 @@ object KyuubiConf {
.toSequence(";")
.createWithDefault(Nil)
val ENGINE_DEREGISTER_EXCEPTION_CLASSES: ConfigEntry[Seq[String]] =
val ENGINE_DEREGISTER_EXCEPTION_CLASSES: ConfigEntry[Set[String]] =
buildConf("kyuubi.engine.deregister.exception.classes")
.doc("A comma-separated list of exception classes. If there is any exception thrown," +
" whose class matches the specified classes, the engine would deregister itself.")
.version("1.2.0")
.stringConf
.toSequence()
.createWithDefault(Nil)
.toSet()
.createWithDefault(Set.empty)
val ENGINE_DEREGISTER_EXCEPTION_MESSAGES: ConfigEntry[Seq[String]] =
val ENGINE_DEREGISTER_EXCEPTION_MESSAGES: ConfigEntry[Set[String]] =
buildConf("kyuubi.engine.deregister.exception.messages")
.doc("A comma-separated list of exception messages. If there is any exception thrown," +
" whose message or stacktrace matches the specified message list, the engine would" +
" deregister itself.")
.version("1.2.0")
.stringConf
.toSequence()
.createWithDefault(Nil)
.toSet()
.createWithDefault(Set.empty)
val ENGINE_DEREGISTER_JOB_MAX_FAILURES: ConfigEntry[Int] =
buildConf("kyuubi.engine.deregister.job.max.failures")
@ -2400,7 +2400,7 @@ object KyuubiConf {
"'plain', 'json'.")
.createWithDefault(PlainStyle.name)
val OPERATION_PLAN_ONLY_EXCLUDES: ConfigEntry[Seq[String]] =
val OPERATION_PLAN_ONLY_EXCLUDES: ConfigEntry[Set[String]] =
buildConf("kyuubi.operation.plan.only.excludes")
.doc("Comma-separated list of query plan names, in the form of simple class names, i.e, " +
"for `SET abc=xyz`, the value will be `SetCommand`. For those auxiliary plans, such as " +
@ -2410,8 +2410,8 @@ object KyuubiConf {
s"See also ${OPERATION_PLAN_ONLY_MODE.key}.")
.version("1.5.0")
.stringConf
.toSequence()
.createWithDefault(Seq(
.toSet()
.createWithDefault(Set(
"ResetCommand",
"SetCommand",
"SetNamespaceCommand",
@ -2614,14 +2614,14 @@ object KyuubiConf {
.intConf
.createOptional
val SERVER_LIMIT_CONNECTIONS_USER_UNLIMITED_LIST: ConfigEntry[Seq[String]] =
val SERVER_LIMIT_CONNECTIONS_USER_UNLIMITED_LIST: ConfigEntry[Set[String]] =
buildConf("kyuubi.server.limit.connections.user.unlimited.list")
.doc("The maximum connections of the user in the white list will not be limited.")
.version("1.7.0")
.serverOnly
.stringConf
.toSequence()
.createWithDefault(Nil)
.toSet()
.createWithDefault(Set.empty)
val SERVER_LIMIT_BATCH_CONNECTIONS_PER_USER: OptionalConfigEntry[Int] =
buildConf("kyuubi.server.limit.batch.connections.per.user")
@ -2683,15 +2683,15 @@ object KyuubiConf {
.timeConf
.createWithDefaultString("PT30M")
val SERVER_ADMINISTRATORS: ConfigEntry[Seq[String]] =
val SERVER_ADMINISTRATORS: ConfigEntry[Set[String]] =
buildConf("kyuubi.server.administrators")
.doc("Comma-separated list of Kyuubi service administrators. " +
"We use this config to grant admin permission to any service accounts.")
.version("1.8.0")
.serverOnly
.stringConf
.toSequence()
.createWithDefault(Nil)
.toSet()
.createWithDefault(Set.empty)
val OPERATION_SPARK_LISTENER_ENABLED: ConfigEntry[Boolean] =
buildConf("kyuubi.operation.spark.listener.enabled")

2
kyuubi-common/src/main/scala/org/apache/kyuubi/service/TBinaryFrontendService.scala
View File

@ -134,7 +134,7 @@ abstract class TBinaryFrontendService(name: String)
keyStorePassword: String,
keyStoreType: Option[String],
keyStoreAlgorithm: Option[String],
disallowedSslProtocols: Seq[String],
disallowedSslProtocols: Set[String],
includeCipherSuites: Seq[String]): TServerSocket = {
val params =
if (includeCipherSuites.nonEmpty) {

2
kyuubi-common/src/main/scala/org/apache/kyuubi/service/authentication/KyuubiAuthenticationFactory.scala
View File

@ -39,7 +39,7 @@ class KyuubiAuthenticationFactory(conf: KyuubiConf, isServer: Boolean = true) ex
private val authTypes = conf.get(AUTHENTICATION_METHOD).map(AuthTypes.withName)
private val none = authTypes.contains(NONE)
private val noSasl = authTypes == Seq(NOSASL)
private val noSasl = authTypes == Set(NOSASL)
private val kerberosEnabled = authTypes.contains(KERBEROS)
private val plainAuthTypeOpt = authTypes.filterNot(_.equals(KERBEROS))
.filterNot(_.equals(NOSASL)).headOption

4
kyuubi-common/src/main/scala/org/apache/kyuubi/service/authentication/ldap/GroupFilterFactory.scala
View File

@ -38,7 +38,7 @@ object GroupFilterFactory extends FilterFactory {
}
}
class GroupMembershipKeyFilter(groupFilter: Seq[String]) extends Filter with Logging {
class GroupMembershipKeyFilter(groupFilter: Set[String]) extends Filter with Logging {
@throws[AuthenticationException]
override def apply(ldap: DirSearch, user: String): Unit = {
@ -70,7 +70,7 @@ class GroupMembershipKeyFilter(groupFilter: Seq[String]) extends Filter with Log
}
}
class UserMembershipKeyFilter(groupFilter: Seq[String]) extends Filter with Logging {
class UserMembershipKeyFilter(groupFilter: Set[String]) extends Filter with Logging {
@throws[AuthenticationException]
override def apply(ldap: DirSearch, user: String): Unit = {
info(s"Authenticating user '$user' using $classOf[UserMembershipKeyFilter].getSimpleName")

4
kyuubi-common/src/main/scala/org/apache/kyuubi/service/authentication/ldap/UserFilterFactory.scala
View File

@ -29,9 +29,9 @@ object UserFilterFactory extends FilterFactory with Logging {
}
}
class UserFilter(_userFilter: Seq[String]) extends Filter with Logging {
class UserFilter(_userFilter: Set[String]) extends Filter with Logging {
lazy val userFilter: Seq[String] = _userFilter.map(_.toLowerCase)
lazy val userFilter: Set[String] = _userFilter.map(_.toLowerCase)
@throws[AuthenticationException]
override def apply(ldap: DirSearch, user: String): Unit = {

6
kyuubi-common/src/main/scala/org/apache/kyuubi/session/SessionManager.scala
View File

@ -265,10 +265,10 @@ abstract class SessionManager(name: String) extends CompositeService(name) {
conf.get(ENGINE_EXEC_KEEPALIVE_TIME)
}
_confRestrictList = conf.get(SESSION_CONF_RESTRICT_LIST).toSet
_confIgnoreList = conf.get(SESSION_CONF_IGNORE_LIST).toSet +
_confRestrictList = conf.get(SESSION_CONF_RESTRICT_LIST)
_confIgnoreList = conf.get(SESSION_CONF_IGNORE_LIST) +
s"${SESSION_USER_SIGN_ENABLED.key}"
_batchConfIgnoreList = conf.get(BATCH_CONF_IGNORE_LIST).toSet
_batchConfIgnoreList = conf.get(BATCH_CONF_IGNORE_LIST)
execPool = ThreadUtils.newDaemonQueuedThreadPool(
poolSize,

6
kyuubi-common/src/test/scala/org/apache/kyuubi/config/ConfigBuilderSuite.scala
View File

@ -87,6 +87,12 @@ class ConfigBuilderSuite extends KyuubiFunSuite {
.createWithDefault(Seq("hehe"))
assert(stringConfUpperSeq.defaultVal.get === Seq("HEHE"))
val stringConfSet = ConfigBuilder("kyuubi.string.conf.set")
.stringConf
.toSet()
.createWithDefault(Set("hehe", "haha"))
assert(stringConfSet.defaultVal.get === Set("hehe", "haha"))
val stringConfLower = ConfigBuilder("kyuubi.string.conf.lower")
.stringConf
.transformToLowerCase

4
kyuubi-common/src/test/scala/org/apache/kyuubi/service/TFrontendServiceSuite.scala
View File

@ -40,8 +40,8 @@ class TFrontendServiceSuite extends KyuubiFunSuite {
.set(KyuubiConf.SESSION_CHECK_INTERVAL, Duration.ofSeconds(5).toMillis)
.set(KyuubiConf.SESSION_IDLE_TIMEOUT, Duration.ofSeconds(5).toMillis)
.set(KyuubiConf.OPERATION_IDLE_TIMEOUT, Duration.ofSeconds(20).toMillis)
.set(KyuubiConf.SESSION_CONF_RESTRICT_LIST, Seq("spark.*"))
.set(KyuubiConf.SESSION_CONF_IGNORE_LIST, Seq("session.engine.*"))
.set(KyuubiConf.SESSION_CONF_RESTRICT_LIST, Set("spark.*"))
.set(KyuubiConf.SESSION_CONF_IGNORE_LIST, Set("session.engine.*"))
private def withSessionHandle(f: (TCLIService.Iface, TSessionHandle) => Unit): Unit = {
TClientTestUtils.withSessionHandle(server.frontendServices.head.connectionUrl, Map.empty)(f)

10
kyuubi-common/src/test/scala/org/apache/kyuubi/service/authentication/KyuubiAuthenticationFactorySuite.scala
View File

@ -56,21 +56,21 @@ class KyuubiAuthenticationFactorySuite extends KyuubiFunSuite {
}
test("AuthType Other") {
val conf = KyuubiConf().set(KyuubiConf.AUTHENTICATION_METHOD, Seq("INVALID"))
val conf = KyuubiConf().set(KyuubiConf.AUTHENTICATION_METHOD, Set("INVALID"))
val e = intercept[IllegalArgumentException](new KyuubiAuthenticationFactory(conf))
assert(e.getMessage contains "the authentication type should be one or more of" +
" NOSASL,NONE,LDAP,JDBC,KERBEROS,CUSTOM")
}
test("AuthType LDAP") {
val conf = KyuubiConf().set(KyuubiConf.AUTHENTICATION_METHOD, Seq("LDAP"))
val conf = KyuubiConf().set(KyuubiConf.AUTHENTICATION_METHOD, Set("LDAP"))
val authFactory = new KyuubiAuthenticationFactory(conf)
authFactory.getTTransportFactory
assert(Security.getProviders.exists(_.isInstanceOf[SaslPlainProvider]))
}
test("AuthType KERBEROS w/o keytab/principal") {
val conf = KyuubiConf().set(KyuubiConf.AUTHENTICATION_METHOD, Seq("KERBEROS"))
val conf = KyuubiConf().set(KyuubiConf.AUTHENTICATION_METHOD, Set("KERBEROS"))
val factory = new KyuubiAuthenticationFactory(conf)
val e = intercept[LoginException](factory.getTTransportFactory)
@ -78,11 +78,11 @@ class KyuubiAuthenticationFactorySuite extends KyuubiFunSuite {
}
test("AuthType is NOSASL if only NOSASL is specified") {
val conf = KyuubiConf().set(KyuubiConf.AUTHENTICATION_METHOD, Seq("NOSASL"))
val conf = KyuubiConf().set(KyuubiConf.AUTHENTICATION_METHOD, Set("NOSASL"))
var factory = new KyuubiAuthenticationFactory(conf)
!factory.getTTransportFactory.isInstanceOf[TSaslServerTransport.Factory]
conf.set(KyuubiConf.AUTHENTICATION_METHOD, Seq("NOSASL", "NONE"))
conf.set(KyuubiConf.AUTHENTICATION_METHOD, Set("NOSASL", "NONE"))
factory = new KyuubiAuthenticationFactory(conf)
factory.getTTransportFactory.isInstanceOf[TSaslServerTransport.Factory]
}

6
kyuubi-metrics/src/main/scala/org/apache/kyuubi/metrics/MetricsConf.scala
View File

@ -32,7 +32,7 @@ object MetricsConf {
.booleanConf
.createWithDefault(true)
val METRICS_REPORTERS: ConfigEntry[Seq[String]] = buildConf("kyuubi.metrics.reporters")
val METRICS_REPORTERS: ConfigEntry[Set[String]] = buildConf("kyuubi.metrics.reporters")
.doc("A comma-separated list for all metrics reporters" +
"<ul>" +
" <li>CONSOLE - ConsoleReporter which outputs measurements to CONSOLE periodically.</li>" +
@ -44,11 +44,11 @@ object MetricsConf {
.version("1.2.0")
.stringConf
.transformToUpperCase
.toSequence()
.toSet()
.checkValue(
_.forall(ReporterType.values.map(_.toString).contains),
s"the reporter type should be one or more of ${ReporterType.values.mkString(",")}")
.createWithDefault(Seq(JSON.toString))
.createWithDefault(Set(JSON.toString))
val METRICS_CONSOLE_INTERVAL: ConfigEntry[Long] = buildConf("kyuubi.metrics.console.interval")
.doc("How often should report metrics to console")

4
kyuubi-metrics/src/test/scala/org/apache/kyuubi/metrics/MetricsSystemSuite.scala
View File

@ -49,7 +49,7 @@ class MetricsSystemSuite extends KyuubiFunSuite {
val conf = KyuubiConf()
.set(MetricsConf.METRICS_ENABLED, true)
.set(MetricsConf.METRICS_REPORTERS, Seq(ReporterType.PROMETHEUS.toString))
.set(MetricsConf.METRICS_REPORTERS, Set(ReporterType.PROMETHEUS.toString))
.set(MetricsConf.METRICS_PROMETHEUS_PORT, 0) // random port
.set(MetricsConf.METRICS_PROMETHEUS_PATH, testContextPath)
val metricsSystem = new MetricsSystem()
@ -77,7 +77,7 @@ class MetricsSystemSuite extends KyuubiFunSuite {
.set(MetricsConf.METRICS_ENABLED, true)
.set(
MetricsConf.METRICS_REPORTERS,
ReporterType.values.filterNot(_ == ReporterType.PROMETHEUS).map(_.toString).toSeq)
ReporterType.values.filterNot(_ == ReporterType.PROMETHEUS).map(_.toString))
.set(MetricsConf.METRICS_JSON_INTERVAL, Duration.ofSeconds(1).toMillis)
.set(MetricsConf.METRICS_JSON_LOCATION, reportPath.toString)
val metricsSystem = new MetricsSystem()

4
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KubernetesApplicationOperation.scala
View File

@ -43,9 +43,9 @@ class KubernetesApplicationOperation extends ApplicationOperation with Logging {
private var submitTimeout: Long = _
private var kyuubiConf: KyuubiConf = _
private def allowedContexts: Seq[String] =
private def allowedContexts: Set[String] =
kyuubiConf.get(KyuubiConf.KUBERNETES_CONTEXT_ALLOW_LIST)
private def allowedNamespaces: Seq[String] =
private def allowedNamespaces: Set[String] =
kyuubiConf.get(KyuubiConf.KUBERNETES_NAMESPACE_ALLOW_LIST)
// key is kyuubi_unique_key

2
kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/AdminResource.scala
View File

@ -45,7 +45,7 @@ import org.apache.kyuubi.shaded.zookeeper.KeeperException.NoNodeException
@Tag(name = "Admin")
@Produces(Array(MediaType.APPLICATION_JSON))
private[v1] class AdminResource extends ApiRequestContext with Logging {
private lazy val administrators = fe.getConf.get(KyuubiConf.SERVER_ADMINISTRATORS).toSet +
private lazy val administrators = fe.getConf.get(KyuubiConf.SERVER_ADMINISTRATORS) +
Utils.currentUser
@ApiResponse(

2
kyuubi-server/src/main/scala/org/apache/kyuubi/server/http/authentication/AuthenticationFilter.scala
View File

@ -57,7 +57,7 @@ class AuthenticationFilter(conf: KyuubiConf) extends Filter with Logging {
val authTypes = conf.get(AUTHENTICATION_METHOD).map(AuthTypes.withName)
val spnegoKerberosEnabled = authTypes.contains(KERBEROS)
val basicAuthTypeOpt = {
if (authTypes == Seq(NOSASL)) {
if (authTypes == Set(NOSASL)) {
authTypes.headOption
} else {
authTypes.filterNot(_.equals(KERBEROS)).filterNot(_.equals(NOSASL)).headOption

6
kyuubi-server/src/main/scala/org/apache/kyuubi/session/KyuubiSessionManager.scala
View File

@ -356,7 +356,7 @@ class KyuubiSessionManager private (name: String) extends SessionManager(name) {
}
private[kyuubi] def refreshUnlimitedUsers(conf: KyuubiConf): Unit = {
val unlimitedUsers = conf.get(SERVER_LIMIT_CONNECTIONS_USER_UNLIMITED_LIST).toSet
val unlimitedUsers = conf.get(SERVER_LIMIT_CONNECTIONS_USER_UNLIMITED_LIST)
limiter.foreach(SessionLimiter.resetUnlimitedUsers(_, unlimitedUsers))
batchLimiter.foreach(SessionLimiter.resetUnlimitedUsers(_, unlimitedUsers))
}
@ -365,9 +365,9 @@ class KyuubiSessionManager private (name: String) extends SessionManager(name) {
userLimit: Int,
ipAddressLimit: Int,
userIpAddressLimit: Int,
userUnlimitedList: Seq[String]): Option[SessionLimiter] = {
userUnlimitedList: Set[String]): Option[SessionLimiter] = {
Seq(userLimit, ipAddressLimit, userIpAddressLimit).find(_ > 0).map(_ =>
SessionLimiter(userLimit, ipAddressLimit, userIpAddressLimit, userUnlimitedList.toSet))
SessionLimiter(userLimit, ipAddressLimit, userIpAddressLimit, userUnlimitedList))
}
private def startEngineAliveChecker(): Unit = {

2
kyuubi-server/src/test/scala/org/apache/kyuubi/RestClientTestHelper.scala
View File

@ -48,7 +48,7 @@ trait RestClientTestHelper extends RestFrontendTestHelper with KerberizedTestHel
UserGroupInformation.setConfiguration(config)
assert(UserGroupInformation.isSecurityEnabled)
val conf = KyuubiConf().set(KyuubiConf.AUTHENTICATION_METHOD, Seq("KERBEROS", "LDAP", "CUSTOM"))
val conf = KyuubiConf().set(KyuubiConf.AUTHENTICATION_METHOD, Set("KERBEROS", "LDAP", "CUSTOM"))
.set(KyuubiConf.SERVER_KEYTAB.key, testKeytab)
.set(KyuubiConf.SERVER_PRINCIPAL, testPrincipal)
.set(KyuubiConf.SERVER_SPNEGO_KEYTAB, testKeytab)

2
kyuubi-server/src/test/scala/org/apache/kyuubi/WithKyuubiServerOnYarn.scala
View File

@ -80,7 +80,7 @@ class KyuubiOperationYarnClusterSuite extends WithKyuubiServerOnYarn with HiveJD
override protected val conf: KyuubiConf = {
new KyuubiConf()
.set(s"$KYUUBI_BATCH_CONF_PREFIX.spark.spark.master", "yarn")
.set(BATCH_CONF_IGNORE_LIST, Seq("spark.master"))
.set(BATCH_CONF_IGNORE_LIST, Set("spark.master"))
.set(BATCH_APPLICATION_CHECK_INTERVAL, 3000L)
}

2
kyuubi-server/src/test/scala/org/apache/kyuubi/engine/KyuubiApplicationManagerSuite.scala
View File

@ -24,7 +24,7 @@ import org.apache.kyuubi.engine.KubernetesApplicationOperation.LABEL_KYUUBI_UNIQ
class KyuubiApplicationManagerSuite extends KyuubiFunSuite {
test("application access path") {
val localDirLimitConf = KyuubiConf()
.set(KyuubiConf.SESSION_LOCAL_DIR_ALLOW_LIST, Seq("/apache/kyuubi"))
.set(KyuubiConf.SESSION_LOCAL_DIR_ALLOW_LIST, Set("/apache/kyuubi"))
val noLocalDirLimitConf = KyuubiConf()
var path = "/apache/kyuubi/a.jar"

2
kyuubi-server/src/test/scala/org/apache/kyuubi/operation/KyuubiOperationKerberosAndPlainAuthSuite.scala
View File

@ -64,7 +64,7 @@ class KyuubiOperationKerberosAndPlainAuthSuite extends WithKyuubiServer with Ker
assert(UserGroupInformation.isSecurityEnabled)
KyuubiConf()
.set(KyuubiConf.AUTHENTICATION_METHOD, Seq("KERBEROS", "LDAP", "CUSTOM"))
.set(KyuubiConf.AUTHENTICATION_METHOD, Set("KERBEROS", "LDAP", "CUSTOM"))
.set(KyuubiConf.SERVER_KEYTAB, testKeytab)
.set(KyuubiConf.SERVER_PRINCIPAL, testPrincipal)
.set(KyuubiConf.AUTHENTICATION_LDAP_URL, ldapUrl)

2
kyuubi-server/src/test/scala/org/apache/kyuubi/operation/thrift/http/KyuubiOperationThriftHttpKerberosAndPlainAuthSuite.scala
View File

@ -49,7 +49,7 @@ class KyuubiOperationThriftHttpKerberosAndPlainAuthSuite
UserGroupInformation.setConfiguration(config)
assert(UserGroupInformation.isSecurityEnabled)
KyuubiConf().set(KyuubiConf.AUTHENTICATION_METHOD, Seq("KERBEROS", "LDAP", "CUSTOM"))
KyuubiConf().set(KyuubiConf.AUTHENTICATION_METHOD, Set("KERBEROS", "LDAP", "CUSTOM"))
.set(KyuubiConf.SERVER_KEYTAB, testKeytab)
.set(KyuubiConf.SERVER_PRINCIPAL, testPrincipal)
.set(KyuubiConf.AUTHENTICATION_LDAP_URL, ldapUrl)

4
kyuubi-server/src/test/scala/org/apache/kyuubi/server/BackendServiceMetricSuite.scala
View File

@ -25,7 +25,7 @@ import org.scalatest.time.SpanSugar.convertIntToGrainOfTime
import org.apache.kyuubi.{Utils, WithKyuubiServer}
import org.apache.kyuubi.config.KyuubiConf
import org.apache.kyuubi.metrics.MetricsConf
import org.apache.kyuubi.metrics.{MetricsConf, ReporterType}
import org.apache.kyuubi.metrics.MetricsConstants._
import org.apache.kyuubi.operation.HiveJDBCTestHelper
@ -36,7 +36,7 @@ class BackendServiceMetricSuite extends WithKyuubiServer with HiveJDBCTestHelper
val reportPath: Path = Utils.createTempDir()
override protected val conf: KyuubiConf = {
KyuubiConf()
.set(MetricsConf.METRICS_REPORTERS, Seq("JSON"))
.set(MetricsConf.METRICS_REPORTERS, Set(ReporterType.JSON.toString))
.set(MetricsConf.METRICS_JSON_LOCATION, reportPath.toString)
.set(MetricsConf.METRICS_JSON_INTERVAL, Duration.ofMillis(100).toMillis)
}

2
kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/AdminResourceSuite.scala
View File

@ -48,7 +48,7 @@ class AdminResourceSuite extends KyuubiFunSuite with RestFrontendTestHelper {
private val engineMgr = new KyuubiApplicationManager()
override protected lazy val conf: KyuubiConf = KyuubiConf()
.set(KyuubiConf.SERVER_ADMINISTRATORS, Seq("admin001"))
.set(KyuubiConf.SERVER_ADMINISTRATORS, Set("admin001"))
private val encodeAuthorization: String = {
new String(

2
kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/BatchesResourceSuite.scala
View File

@ -78,7 +78,7 @@ abstract class BatchesResourceSuiteBase extends KyuubiFunSuite
.set(KyuubiConf.BATCH_IMPL_VERSION, batchVersion)
.set(
KyuubiConf.SESSION_LOCAL_DIR_ALLOW_LIST,
Seq(Paths.get(sparkBatchTestResource.get).getParent.toString))
Set(Paths.get(sparkBatchTestResource.get).getParent.toString))
customConf.foreach { case (k, v) => kyuubiConf.set(k, v) }
kyuubiConf
}

2
kyuubi-server/src/test/scala/org/apache/kyuubi/server/rest/client/AdminCtlSuite.scala
View File

@ -56,7 +56,7 @@ class AdminCtlSuite extends RestClientTestHelper with TestPrematureExit {
val id = UUID.randomUUID().toString
conf.set(HighAvailabilityConf.HA_NAMESPACE, "kyuubi_test")
conf.set(KyuubiConf.ENGINE_IDLE_TIMEOUT, 180000L)
conf.set(KyuubiConf.AUTHENTICATION_METHOD, Seq("LDAP", "CUSTOM"))
conf.set(KyuubiConf.AUTHENTICATION_METHOD, Set("LDAP", "CUSTOM"))
conf.set(KyuubiConf.GROUP_PROVIDER, "hadoop")
val user = ldapUser

2
kyuubi-server/src/test/scala/org/apache/kyuubi/server/rest/client/AdminRestApiSuite.scala
View File

@ -50,7 +50,7 @@ class AdminRestApiSuite extends RestClientTestHelper {
val id = UUID.randomUUID().toString
conf.set(HighAvailabilityConf.HA_NAMESPACE, "kyuubi_test")
conf.set(KyuubiConf.ENGINE_IDLE_TIMEOUT, 180000L)
conf.set(KyuubiConf.AUTHENTICATION_METHOD, Seq("LDAP", "CUSTOM"))
conf.set(KyuubiConf.AUTHENTICATION_METHOD, Set("LDAP", "CUSTOM"))
conf.set(KyuubiConf.GROUP_PROVIDER, "hadoop")
val user = ldapUser
val engine = new EngineRef(conf.clone, user, PluginLoader.loadGroupProvider(conf), id, null)