apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

[KYUUBI #3545][KYUUBI #3563] Support restrict spark configurations

Browse Source 
### _Why are the changes needed?_

ban end-user from security settings

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #3564 from cfmcgrady/kyuubi-3563.

Closes #3545

Closes #3563

9d912b11 [Fu Chen] rename
ee44f7df [Fu Chen] fix ci
158f1552 [Fu Chen] address comment
3cbf4794 [Fu Chen] doc
0125a862 [Fu Chen] address comment
e4e554f5 [Kent Yao] Update extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AuthzConfigurationCheckExtension.scala
a8d35efa [Fu Chen] ban end-user from security settings

Lead-authored-by: Fu Chen <cfmcgrady@gmail.com>
Co-authored-by: Kent Yao <yao@apache.org>
Signed-off-by: Kent Yao <yao@apache.org>
This commit is contained in:
Fu Chen 2022-09-30 15:43:58 +08:00 committed by Kent Yao
parent 8207b10620
commit bb50c52c2f
No known key found for this signature in database
GPG Key ID: F7051850A0AF904D
4 changed files with 153 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

53
docs/security/authorization/spark/overview.rst
View File

@ -28,7 +28,7 @@ Authorization in Kyuubi
-----------------------
Storage-based Authorization
***************************
^^^^^^^^^^^^^^^^^^^^^^^^^^^
As Kyuubi supports multi tenancy, a tenant can only visit authorized resources,
including computing resources, data, etc.
@ -41,7 +41,7 @@ as well as their permissions.
Storage-based authorization offers users with database, table and partition-level coarse-gained access control.
SQL-standard authorization with Ranger
**************************************
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
A SQL-standard authorization usually offers a row/colum-level fine-grained access control to meet the real-world data security need.
@ -59,4 +59,51 @@ Kyuubi Spark Authz Plugin itself provides general purpose for ACL management for
It is not necessary to deploy it with the Kyuubi server and engine, and can be used as an extension for any Spark SQL jobs.
However, the authorization always requires a robust authentication layer and multi tenancy support, so Kyuubi is a perfect match.
.. _Apache Ranger: https://ranger.apache.org/
Restrict security configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
End-users can disable the AuthZ plugin by modifying Spark's configuration. For example:
.. code-block:: sql
select * from parquet.`/path/to/table`
.. code-block:: sql
set spark.sql.optimizer.excludedRules=org.apache.kyuubi.plugin.spark.authz.ranger.RuleAuthorization
Kyuubi provides a mechanism to ban security configurations to enhance the security of production environments
.. note:: How do we modify the Spark engine configurations please refer to the documentation `Spark Configurations`_
Restrict session level config
*****************************
You can specify config `kyuubi.session.conf.ignore.list` values and config `kyuubi.session.conf.restrict.list` values to disable changing session+ level configuration on the server side. For example:
.. code-block::
kyuubi.session.conf.ignore.list spark.driver.memory,spark.sql.optimizer.excludedRules
.. code-block::
kyuubi.session.conf.restrict.list spark.driver.memory,spark.sql.optimizer.excludedRules
Restrict operation level config
*******************************
You can specify config `spark.kyuubi.conf.restricted.list` values to disable changing operation level configuration on the engine side, this means that the config key in the restricted list cannot set dynamic configuration via SET syntax. For examples:
.. code-block::
spark.kyuubi.conf.restricted.list spark.sql.adaptive.enabled,spark.sql.adaptive.skewJoin.enabled
.. note::
1. Note that config `spark.sql.runSQLOnFiles` values and config `spark.sql.extensions` values are by default in the engine restriction configuration list
2. A set statement with key equal to `spark.sql.optimizer.excludedRules` and value containing `org.apache.kyuubi.plugin.spark.authz.ranger.*` also does not allow modification.
.. _Apache Ranger: https://ranger.apache.org/
.. _Spark Configurations: ../../../deployment/settings.html#spark-configurations

46
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AuthzConfigurationChecker.scala Normal file
View File

@ -0,0 +1,46 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.kyuubi.plugin.spark.authz.ranger
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
import org.apache.spark.sql.execution.command.SetCommand
import org.apache.kyuubi.plugin.spark.authz.AccessControlException
/**
* For banning end-users from set restricted spark configurations
*/
case class AuthzConfigurationChecker(spark: SparkSession) extends (LogicalPlan => Unit) {
final val RESTRICT_LIST_KEY = "spark.kyuubi.conf.restricted.list"
private val restrictedConfList: Set[String] =
Set(RESTRICT_LIST_KEY, "spark.sql.runSQLOnFiles", "spark.sql.extensions") ++
spark.conf.getOption(RESTRICT_LIST_KEY).map(_.split(',').toSet).getOrElse(Set.empty)
override def apply(plan: LogicalPlan): Unit = plan match {
case SetCommand(Some((
"spark.sql.optimizer.excludedRules",
Some(v)))) if v.contains("org.apache.kyuubi.plugin.spark.authz.ranger") =>
throw new AccessControlException("Excluding Authz security rules is not allowed")
case SetCommand(Some((k, Some(_)))) if restrictedConfList.contains(k) =>
throw new AccessControlException(s"Modifying config $k is not allowed")
case _ =>
}
}

1
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/RangerSparkExtension.scala
View File

@ -39,6 +39,7 @@ class RangerSparkExtension extends (SparkSessionExtensions => Unit) {
SparkRangerAdminPlugin.init()
override def apply(v1: SparkSessionExtensions): Unit = {
v1.injectCheckRule(AuthzConfigurationChecker)
v1.injectResolutionRule(_ => new RuleReplaceShowObjectCommands())
v1.injectResolutionRule(_ => new RuleApplyPermanentViewMarker())
v1.injectResolutionRule(new RuleApplyRowFilterAndDataMasking(_))

56
extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/AuthzConfigurationCheckerSuite.scala Normal file
View File

@ -0,0 +1,56 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.kyuubi.plugin.spark.authz.ranger
import org.scalatest.BeforeAndAfterAll
// scalastyle:off
import org.scalatest.funsuite.AnyFunSuite
import org.apache.kyuubi.plugin.spark.authz.{AccessControlException, SparkSessionProvider}
class AuthzConfigurationCheckerSuite extends AnyFunSuite with SparkSessionProvider
with BeforeAndAfterAll {
override protected val catalogImpl: String = "in-memory"
override def afterAll(): Unit = {
spark.stop()
super.afterAll()
}
test("apply spark configuration restriction rules") {
sql("set spark.kyuubi.conf.restricted.list=spark.sql.abc,spark.sql.xyz")
val extension = AuthzConfigurationChecker(spark)
val p1 = sql("set spark.sql.runSQLOnFiles=true").queryExecution.analyzed
intercept[AccessControlException](extension.apply(p1))
val p2 = sql("set spark.sql.runSQLOnFiles=false").queryExecution.analyzed
intercept[AccessControlException](extension.apply(p2))
val p3 = sql("set spark.sql.runSQLOnFiles").queryExecution.analyzed
extension.apply(p3)
val p4 = sql("set spark.sql.abc=xyz").queryExecution.analyzed
intercept[AccessControlException](extension.apply(p4))
val p5 = sql("set spark.sql.xyz=abc").queryExecution.analyzed
intercept[AccessControlException](extension.apply(p5))
val p6 = sql("set spark.kyuubi.conf.restricted.list=123").queryExecution.analyzed
intercept[AccessControlException](extension.apply(p6))
val p7 = sql("set spark.sql.efg=hijk").queryExecution.analyzed
extension.apply(p7)
val p8 = sql(
s"set spark.sql.optimizer.excludedRules=${classOf[RuleAuthorization].getName}").queryExecution.analyzed
intercept[AccessControlException](extension.apply(p8))
}
}