[KYUUBI #4483] Enable REST frontend protocol by default

### _Why are the changes needed?_

REST frontend protocol was introduced in Kyuubi 1.4.0 https://github.com/apache/kyuubi/pull/1349, more and more users are using REST in recent days, and to simplify the planned Web UI developments, I think we should enable the REST by default for master(1.8.0-SNAPSHOT).

I do not remove the "experiment" because we don't have strong confident to ensure the API stability in this time, I would like to defer it until the community has confident to mark it as stable.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4483 from pan3793/enable-rest.

Closes #4483

66fef0224 [Cheng Pan] http
91a29b453 [Cheng Pan] nit
29a4826da [Cheng Pan] Enable REST frontend protocol in default

Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
This commit is contained in:
Cheng Pan 2023-03-09 02:15:47 +08:00
parent 3595d5f65e
commit a47bf6e633
No known key found for this signature in database
GPG Key ID: 8001952629BCC75D
5 changed files with 66 additions and 63 deletions

View File

@ -80,7 +80,7 @@ server:
# REST API protocol (experimental)
rest:
enabled: false
enabled: true
port: 10099
service:
type: ClusterIP

View File

@ -21,8 +21,9 @@
# kyuubi.authentication NONE
#
# kyuubi.frontend.bind.host 10.0.0.1
# kyuubi.frontend.protocols THRIFT_BINARY
# kyuubi.frontend.protocols THRIFT_BINARY,REST
# kyuubi.frontend.thrift.binary.bind.port 10009
# kyuubi.frontend.rest.bind.port 10099
#
# kyuubi.engine.type SPARK_SQL
# kyuubi.engine.share.level USER

View File

@ -182,65 +182,65 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
### Frontend
| Key | Default | Meaning | Type | Since |
|--------------------------------------------------------|-------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
| kyuubi.frontend.backoff.slot.length | PT0.1S | (deprecated) Time to back off during login to the thrift frontend service. | duration | 1.0.0 |
| kyuubi.frontend.bind.host | &lt;undefined&gt; | Hostname or IP of the machine on which to run the frontend services. | string | 1.0.0 |
| kyuubi.frontend.bind.port | 10009 | (deprecated) Port of the machine on which to run the thrift frontend service via the binary protocol. | int | 1.0.0 |
| kyuubi.frontend.connection.url.use.hostname | true | When true, frontend services prefer hostname, otherwise, ip address. Note that, the default value is set to `false` when engine running on Kubernetes to prevent potential network issues. | boolean | 1.5.0 |
| kyuubi.frontend.login.timeout | PT20S | (deprecated) Timeout for Thrift clients during login to the thrift frontend service. | duration | 1.0.0 |
| kyuubi.frontend.max.message.size | 104857600 | (deprecated) Maximum message size in bytes a Kyuubi server will accept. | int | 1.0.0 |
| kyuubi.frontend.max.worker.threads | 999 | (deprecated) Maximum number of threads in the frontend worker thread pool for the thrift frontend service | int | 1.0.0 |
| kyuubi.frontend.min.worker.threads | 9 | (deprecated) Minimum number of threads in the frontend worker thread pool for the thrift frontend service | int | 1.0.0 |
| kyuubi.frontend.mysql.bind.host | &lt;undefined&gt; | Hostname or IP of the machine on which to run the MySQL frontend service. | string | 1.4.0 |
| kyuubi.frontend.mysql.bind.port | 3309 | Port of the machine on which to run the MySQL frontend service. | int | 1.4.0 |
| kyuubi.frontend.mysql.max.worker.threads | 999 | Maximum number of threads in the command execution thread pool for the MySQL frontend service | int | 1.4.0 |
| kyuubi.frontend.mysql.min.worker.threads | 9 | Minimum number of threads in the command execution thread pool for the MySQL frontend service | int | 1.4.0 |
| kyuubi.frontend.mysql.netty.worker.threads | &lt;undefined&gt; | Number of thread in the netty worker event loop of MySQL frontend service. Use min(cpu_cores, 8) in default. | int | 1.4.0 |
| kyuubi.frontend.mysql.worker.keepalive.time | PT1M | Time(ms) that an idle async thread of the command execution thread pool will wait for a new task to arrive before terminating in MySQL frontend service | duration | 1.4.0 |
| kyuubi.frontend.protocols | THRIFT_BINARY | A comma-separated list for all frontend protocols <ul> <li>THRIFT_BINARY - HiveServer2 compatible thrift binary protocol.</li> <li>THRIFT_HTTP - HiveServer2 compatible thrift http protocol.</li> <li>REST - Kyuubi defined REST API(experimental).</li> <li>MYSQL - MySQL compatible text protocol(experimental).</li> <li>TRINO - Trino compatible http protocol(experimental).</li> </ul> | seq | 1.4.0 |
| kyuubi.frontend.proxy.http.client.ip.header | X-Real-IP | The HTTP header to record the real client IP address. If your server is behind a load balancer or other proxy, the server will see this load balancer or proxy IP address as the client IP address, to get around this common issue, most load balancers or proxies offer the ability to record the real remote IP address in an HTTP header that will be added to the request for other devices to use. Note that, because the header value can be specified to any IP address, so it will not be used for authentication. | string | 1.6.0 |
| kyuubi.frontend.rest.bind.host | &lt;undefined&gt; | Hostname or IP of the machine on which to run the REST frontend service. | string | 1.4.0 |
| kyuubi.frontend.rest.bind.port | 10099 | Port of the machine on which to run the REST frontend service. | int | 1.4.0 |
| kyuubi.frontend.rest.max.worker.threads | 999 | Maximum number of threads in the frontend worker thread pool for the rest frontend service | int | 1.6.2 |
| kyuubi.frontend.ssl.keystore.algorithm | &lt;undefined&gt; | SSL certificate keystore algorithm. | string | 1.7.0 |
| kyuubi.frontend.ssl.keystore.password | &lt;undefined&gt; | SSL certificate keystore password. | string | 1.7.0 |
| kyuubi.frontend.ssl.keystore.path | &lt;undefined&gt; | SSL certificate keystore location. | string | 1.7.0 |
| kyuubi.frontend.ssl.keystore.type | &lt;undefined&gt; | SSL certificate keystore type. | string | 1.7.0 |
| kyuubi.frontend.thrift.backoff.slot.length | PT0.1S | Time to back off during login to the thrift frontend service. | duration | 1.4.0 |
| kyuubi.frontend.thrift.binary.bind.host | &lt;undefined&gt; | Hostname or IP of the machine on which to run the thrift frontend service via the binary protocol. | string | 1.4.0 |
| kyuubi.frontend.thrift.binary.bind.port | 10009 | Port of the machine on which to run the thrift frontend service via the binary protocol. | int | 1.4.0 |
| kyuubi.frontend.thrift.binary.ssl.disallowed.protocols | SSLv2,SSLv3 | SSL versions to disallow for Kyuubi thrift binary frontend. | seq | 1.7.0 |
| kyuubi.frontend.thrift.binary.ssl.enabled | false | Set this to true for using SSL encryption in thrift binary frontend server. | boolean | 1.7.0 |
| kyuubi.frontend.thrift.binary.ssl.include.ciphersuites || A comma-separated list of include SSL cipher suite names for thrift binary frontend. | seq | 1.7.0 |
| kyuubi.frontend.thrift.http.allow.user.substitution | true | Allow alternate user to be specified as part of open connection request when using HTTP transport mode. | boolean | 1.6.0 |
| kyuubi.frontend.thrift.http.bind.host | &lt;undefined&gt; | Hostname or IP of the machine on which to run the thrift frontend service via http protocol. | string | 1.6.0 |
| kyuubi.frontend.thrift.http.bind.port | 10010 | Port of the machine on which to run the thrift frontend service via http protocol. | int | 1.6.0 |
| kyuubi.frontend.thrift.http.compression.enabled | true | Enable thrift http compression via Jetty compression support | boolean | 1.6.0 |
| kyuubi.frontend.thrift.http.cookie.auth.enabled | true | When true, Kyuubi in HTTP transport mode, will use cookie-based authentication mechanism | boolean | 1.6.0 |
| kyuubi.frontend.thrift.http.cookie.domain | &lt;undefined&gt; | Domain for the Kyuubi generated cookies | string | 1.6.0 |
| kyuubi.frontend.thrift.http.cookie.is.httponly | true | HttpOnly attribute of the Kyuubi generated cookie. | boolean | 1.6.0 |
| kyuubi.frontend.thrift.http.cookie.max.age | 86400 | Maximum age in seconds for server side cookie used by Kyuubi in HTTP mode. | int | 1.6.0 |
| kyuubi.frontend.thrift.http.cookie.path | &lt;undefined&gt; | Path for the Kyuubi generated cookies | string | 1.6.0 |
| kyuubi.frontend.thrift.http.max.idle.time | PT30M | Maximum idle time for a connection on the server when in HTTP mode. | duration | 1.6.0 |
| kyuubi.frontend.thrift.http.path | cliservice | Path component of URL endpoint when in HTTP mode. | string | 1.6.0 |
| kyuubi.frontend.thrift.http.request.header.size | 6144 | Request header size in bytes, when using HTTP transport mode. Jetty defaults used. | int | 1.6.0 |
| kyuubi.frontend.thrift.http.response.header.size | 6144 | Response header size in bytes, when using HTTP transport mode. Jetty defaults used. | int | 1.6.0 |
| kyuubi.frontend.thrift.http.ssl.exclude.ciphersuites || A comma-separated list of exclude SSL cipher suite names for thrift http frontend. | seq | 1.7.0 |
| kyuubi.frontend.thrift.http.ssl.keystore.password | &lt;undefined&gt; | SSL certificate keystore password. | string | 1.6.0 |
| kyuubi.frontend.thrift.http.ssl.keystore.path | &lt;undefined&gt; | SSL certificate keystore location. | string | 1.6.0 |
| kyuubi.frontend.thrift.http.ssl.protocol.blacklist | SSLv2,SSLv3 | SSL Versions to disable when using HTTP transport mode. | seq | 1.6.0 |
| kyuubi.frontend.thrift.http.use.SSL | false | Set this to true for using SSL encryption in http mode. | boolean | 1.6.0 |
| kyuubi.frontend.thrift.http.xsrf.filter.enabled | false | If enabled, Kyuubi will block any requests made to it over HTTP if an X-XSRF-HEADER header is not present | boolean | 1.6.0 |
| kyuubi.frontend.thrift.login.timeout | PT20S | Timeout for Thrift clients during login to the thrift frontend service. | duration | 1.4.0 |
| kyuubi.frontend.thrift.max.message.size | 104857600 | Maximum message size in bytes a Kyuubi server will accept. | int | 1.4.0 |
| kyuubi.frontend.thrift.max.worker.threads | 999 | Maximum number of threads in the frontend worker thread pool for the thrift frontend service | int | 1.4.0 |
| kyuubi.frontend.thrift.min.worker.threads | 9 | Minimum number of threads in the frontend worker thread pool for the thrift frontend service | int | 1.4.0 |
| kyuubi.frontend.thrift.worker.keepalive.time | PT1M | Keep-alive time (in milliseconds) for an idle worker thread | duration | 1.4.0 |
| kyuubi.frontend.trino.bind.host | &lt;undefined&gt; | Hostname or IP of the machine on which to run the TRINO frontend service. | string | 1.7.0 |
| kyuubi.frontend.trino.bind.port | 10999 | Port of the machine on which to run the TRINO frontend service. | int | 1.7.0 |
| kyuubi.frontend.trino.max.worker.threads | 999 | Maximum number of threads in the frontend worker thread pool for the Trino frontend service | int | 1.7.0 |
| kyuubi.frontend.worker.keepalive.time | PT1M | (deprecated) Keep-alive time (in milliseconds) for an idle worker thread | duration | 1.0.0 |
| Key | Default | Meaning | Type | Since |
|--------------------------------------------------------|--------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
| kyuubi.frontend.backoff.slot.length | PT0.1S | (deprecated) Time to back off during login to the thrift frontend service. | duration | 1.0.0 |
| kyuubi.frontend.bind.host | &lt;undefined&gt; | Hostname or IP of the machine on which to run the frontend services. | string | 1.0.0 |
| kyuubi.frontend.bind.port | 10009 | (deprecated) Port of the machine on which to run the thrift frontend service via the binary protocol. | int | 1.0.0 |
| kyuubi.frontend.connection.url.use.hostname | true | When true, frontend services prefer hostname, otherwise, ip address. Note that, the default value is set to `false` when engine running on Kubernetes to prevent potential network issues. | boolean | 1.5.0 |
| kyuubi.frontend.login.timeout | PT20S | (deprecated) Timeout for Thrift clients during login to the thrift frontend service. | duration | 1.0.0 |
| kyuubi.frontend.max.message.size | 104857600 | (deprecated) Maximum message size in bytes a Kyuubi server will accept. | int | 1.0.0 |
| kyuubi.frontend.max.worker.threads | 999 | (deprecated) Maximum number of threads in the frontend worker thread pool for the thrift frontend service | int | 1.0.0 |
| kyuubi.frontend.min.worker.threads | 9 | (deprecated) Minimum number of threads in the frontend worker thread pool for the thrift frontend service | int | 1.0.0 |
| kyuubi.frontend.mysql.bind.host | &lt;undefined&gt; | Hostname or IP of the machine on which to run the MySQL frontend service. | string | 1.4.0 |
| kyuubi.frontend.mysql.bind.port | 3309 | Port of the machine on which to run the MySQL frontend service. | int | 1.4.0 |
| kyuubi.frontend.mysql.max.worker.threads | 999 | Maximum number of threads in the command execution thread pool for the MySQL frontend service | int | 1.4.0 |
| kyuubi.frontend.mysql.min.worker.threads | 9 | Minimum number of threads in the command execution thread pool for the MySQL frontend service | int | 1.4.0 |
| kyuubi.frontend.mysql.netty.worker.threads | &lt;undefined&gt; | Number of thread in the netty worker event loop of MySQL frontend service. Use min(cpu_cores, 8) in default. | int | 1.4.0 |
| kyuubi.frontend.mysql.worker.keepalive.time | PT1M | Time(ms) that an idle async thread of the command execution thread pool will wait for a new task to arrive before terminating in MySQL frontend service | duration | 1.4.0 |
| kyuubi.frontend.protocols | THRIFT_BINARY,REST | A comma-separated list for all frontend protocols <ul> <li>THRIFT_BINARY - HiveServer2 compatible thrift binary protocol.</li> <li>THRIFT_HTTP - HiveServer2 compatible thrift http protocol.</li> <li>REST - Kyuubi defined REST API(experimental).</li> <li>MYSQL - MySQL compatible text protocol(experimental).</li> <li>TRINO - Trino compatible http protocol(experimental).</li> </ul> | seq | 1.4.0 |
| kyuubi.frontend.proxy.http.client.ip.header | X-Real-IP | The HTTP header to record the real client IP address. If your server is behind a load balancer or other proxy, the server will see this load balancer or proxy IP address as the client IP address, to get around this common issue, most load balancers or proxies offer the ability to record the real remote IP address in an HTTP header that will be added to the request for other devices to use. Note that, because the header value can be specified to any IP address, so it will not be used for authentication. | string | 1.6.0 |
| kyuubi.frontend.rest.bind.host | &lt;undefined&gt; | Hostname or IP of the machine on which to run the REST frontend service. | string | 1.4.0 |
| kyuubi.frontend.rest.bind.port | 10099 | Port of the machine on which to run the REST frontend service. | int | 1.4.0 |
| kyuubi.frontend.rest.max.worker.threads | 999 | Maximum number of threads in the frontend worker thread pool for the rest frontend service | int | 1.6.2 |
| kyuubi.frontend.ssl.keystore.algorithm | &lt;undefined&gt; | SSL certificate keystore algorithm. | string | 1.7.0 |
| kyuubi.frontend.ssl.keystore.password | &lt;undefined&gt; | SSL certificate keystore password. | string | 1.7.0 |
| kyuubi.frontend.ssl.keystore.path | &lt;undefined&gt; | SSL certificate keystore location. | string | 1.7.0 |
| kyuubi.frontend.ssl.keystore.type | &lt;undefined&gt; | SSL certificate keystore type. | string | 1.7.0 |
| kyuubi.frontend.thrift.backoff.slot.length | PT0.1S | Time to back off during login to the thrift frontend service. | duration | 1.4.0 |
| kyuubi.frontend.thrift.binary.bind.host | &lt;undefined&gt; | Hostname or IP of the machine on which to run the thrift frontend service via the binary protocol. | string | 1.4.0 |
| kyuubi.frontend.thrift.binary.bind.port | 10009 | Port of the machine on which to run the thrift frontend service via the binary protocol. | int | 1.4.0 |
| kyuubi.frontend.thrift.binary.ssl.disallowed.protocols | SSLv2,SSLv3 | SSL versions to disallow for Kyuubi thrift binary frontend. | seq | 1.7.0 |
| kyuubi.frontend.thrift.binary.ssl.enabled | false | Set this to true for using SSL encryption in thrift binary frontend server. | boolean | 1.7.0 |
| kyuubi.frontend.thrift.binary.ssl.include.ciphersuites || A comma-separated list of include SSL cipher suite names for thrift binary frontend. | seq | 1.7.0 |
| kyuubi.frontend.thrift.http.allow.user.substitution | true | Allow alternate user to be specified as part of open connection request when using HTTP transport mode. | boolean | 1.6.0 |
| kyuubi.frontend.thrift.http.bind.host | &lt;undefined&gt; | Hostname or IP of the machine on which to run the thrift frontend service via http protocol. | string | 1.6.0 |
| kyuubi.frontend.thrift.http.bind.port | 10010 | Port of the machine on which to run the thrift frontend service via http protocol. | int | 1.6.0 |
| kyuubi.frontend.thrift.http.compression.enabled | true | Enable thrift http compression via Jetty compression support | boolean | 1.6.0 |
| kyuubi.frontend.thrift.http.cookie.auth.enabled | true | When true, Kyuubi in HTTP transport mode, will use cookie-based authentication mechanism | boolean | 1.6.0 |
| kyuubi.frontend.thrift.http.cookie.domain | &lt;undefined&gt; | Domain for the Kyuubi generated cookies | string | 1.6.0 |
| kyuubi.frontend.thrift.http.cookie.is.httponly | true | HttpOnly attribute of the Kyuubi generated cookie. | boolean | 1.6.0 |
| kyuubi.frontend.thrift.http.cookie.max.age | 86400 | Maximum age in seconds for server side cookie used by Kyuubi in HTTP mode. | int | 1.6.0 |
| kyuubi.frontend.thrift.http.cookie.path | &lt;undefined&gt; | Path for the Kyuubi generated cookies | string | 1.6.0 |
| kyuubi.frontend.thrift.http.max.idle.time | PT30M | Maximum idle time for a connection on the server when in HTTP mode. | duration | 1.6.0 |
| kyuubi.frontend.thrift.http.path | cliservice | Path component of URL endpoint when in HTTP mode. | string | 1.6.0 |
| kyuubi.frontend.thrift.http.request.header.size | 6144 | Request header size in bytes, when using HTTP transport mode. Jetty defaults used. | int | 1.6.0 |
| kyuubi.frontend.thrift.http.response.header.size | 6144 | Response header size in bytes, when using HTTP transport mode. Jetty defaults used. | int | 1.6.0 |
| kyuubi.frontend.thrift.http.ssl.exclude.ciphersuites || A comma-separated list of exclude SSL cipher suite names for thrift http frontend. | seq | 1.7.0 |
| kyuubi.frontend.thrift.http.ssl.keystore.password | &lt;undefined&gt; | SSL certificate keystore password. | string | 1.6.0 |
| kyuubi.frontend.thrift.http.ssl.keystore.path | &lt;undefined&gt; | SSL certificate keystore location. | string | 1.6.0 |
| kyuubi.frontend.thrift.http.ssl.protocol.blacklist | SSLv2,SSLv3 | SSL Versions to disable when using HTTP transport mode. | seq | 1.6.0 |
| kyuubi.frontend.thrift.http.use.SSL | false | Set this to true for using SSL encryption in http mode. | boolean | 1.6.0 |
| kyuubi.frontend.thrift.http.xsrf.filter.enabled | false | If enabled, Kyuubi will block any requests made to it over HTTP if an X-XSRF-HEADER header is not present | boolean | 1.6.0 |
| kyuubi.frontend.thrift.login.timeout | PT20S | Timeout for Thrift clients during login to the thrift frontend service. | duration | 1.4.0 |
| kyuubi.frontend.thrift.max.message.size | 104857600 | Maximum message size in bytes a Kyuubi server will accept. | int | 1.4.0 |
| kyuubi.frontend.thrift.max.worker.threads | 999 | Maximum number of threads in the frontend worker thread pool for the thrift frontend service | int | 1.4.0 |
| kyuubi.frontend.thrift.min.worker.threads | 9 | Minimum number of threads in the frontend worker thread pool for the thrift frontend service | int | 1.4.0 |
| kyuubi.frontend.thrift.worker.keepalive.time | PT1M | Keep-alive time (in milliseconds) for an idle worker thread | duration | 1.4.0 |
| kyuubi.frontend.trino.bind.host | &lt;undefined&gt; | Hostname or IP of the machine on which to run the TRINO frontend service. | string | 1.7.0 |
| kyuubi.frontend.trino.bind.port | 10999 | Port of the machine on which to run the TRINO frontend service. | int | 1.7.0 |
| kyuubi.frontend.trino.max.worker.threads | 999 | Maximum number of threads in the frontend worker thread pool for the Trino frontend service | int | 1.7.0 |
| kyuubi.frontend.worker.keepalive.time | PT1M | (deprecated) Keep-alive time (in milliseconds) for an idle worker thread | duration | 1.0.0 |
### Ha

View File

@ -391,7 +391,9 @@ object KyuubiConf {
.checkValue(
_.forall(FrontendProtocols.values.map(_.toString).contains),
s"the frontend protocol should be one or more of ${FrontendProtocols.values.mkString(",")}")
.createWithDefault(Seq(FrontendProtocols.THRIFT_BINARY.toString))
.createWithDefault(Seq(
FrontendProtocols.THRIFT_BINARY.toString,
FrontendProtocols.REST.toString))
val FRONTEND_BIND_HOST: OptionalConfigEntry[String] = buildConf("kyuubi.frontend.bind.host")
.doc("Hostname or IP of the machine on which to run the frontend services.")

View File

@ -174,7 +174,6 @@ class KyuubiRestFrontendService(override val serverable: Serverable)
server.start()
recoverBatchSessions()
isStarted.set(true)
info(s"$getName has started at ${server.getServerUri}")
startBatchChecker()
startInternal()
} catch {
@ -182,6 +181,7 @@ class KyuubiRestFrontendService(override val serverable: Serverable)
}
}
super.start()
info(s"Exposing REST endpoint at: http://${server.getServerUri}")
}
override def stop(): Unit = synchronized {