apache/kyuubi
1
0
Fork 0
Code Issues Pull Requests Actions 5 Packages Projects Releases Wiki Activity

[KYUUBI #1675][FEATURE] Redact secret information from SparkSQLEngine log

Browse Source 
<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Secret informations on log is dangerous on the production environment. Secret info should be redacted in some way.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate
capture from Spark Driver's STDERR
![Screen Shot 2022-01-04 at 11 56 43 AM](https://user-images.githubusercontent.com/1802676/148022789-f005748e-4efe-47da-8d2d-87ecdd233910.png)

- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #1677 from minyk/kyuubi-1675.

Closes #1675

6a390575 [minyk] redact statement in the log

Authored-by: minyk <minykreva@gmail.com>
Signed-off-by: Kent Yao <yao@apache.org>
This commit is contained in:
minyk 2022-01-04 19:30:42 +08:00 committed by Kent Yao
parent 47533100f6
commit 0f8ebb6006
No known key found for this signature in database
GPG Key ID: F7051850A0AF904D
2 changed files with 5 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/SparkOperation.scala vendored
View File

@ -58,7 +58,8 @@ abstract class SparkOperation(opType: OperationType, session: Session)
protected def resultSchema: StructType
def redactedStatement: String = redact(spark.sessionState.conf.stringRedactionPattern, statement)
override def redactedStatement: String =
redact(spark.sessionState.conf.stringRedactionPattern, statement)
protected def cleanup(targetState: OperationState): Unit = state.synchronized {
if (!isTerminalState(state)) {

4
kyuubi-common/src/main/scala/org/apache/kyuubi/operation/AbstractOperation.scala
View File

@ -60,6 +60,8 @@ abstract class AbstractOperation(opType: OperationType, session: Session)
def statement: String = opType.toString
def redactedStatement: String = statement
protected def setHasResultSet(hasResultSet: Boolean): Unit = {
this.hasResultSet = hasResultSet
handle.setHasResultSet(hasResultSet)
@ -80,7 +82,7 @@ abstract class AbstractOperation(opType: OperationType, session: Session)
case _ =>
}
info(s"Processing ${session.user}'s query[$statementId]: ${state.name} -> ${newState.name}," +
s" statement: $statement$timeCost")
s" statement: $redactedStatement$timeCost")
state = newState
lastAccessTime = System.currentTimeMillis()
}