[SPARK-20070][SQL] Redact DataSourceScanExec treeString

[hvanhovell]

What changes were proposed in this pull request?

The explain output of DataSourceScanExec can contain sensitive information (like Amazon keys). Such information should not end up in logs, or be exposed to non privileged users.

This PR addresses this by adding a redaction facility for the DataSourceScanExec.treeString. A user can enable this by setting a regex in the spark.redaction.string.regex configuration.

How was this patch tested?

Added a unit test to check the output of DataSourceScanExec.

[hvanhovell]

cc @liufengdb

[rxin]

LGTM

[SparkQA]

Test build #75093 has finished for PR 17397 at commit 2648ce5.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[liufengdb]

lgtm

[gatorsmile]

Nit: * */ -> */

[gatorsmile]

Maybe we can verify the user input or capture the exception? If users input incorrect value, it could get a confusing error message later.

Unclosed group near index 19
spark-~!@#$%^^&*(12
                   ^
java.util.regex.PatternSyntaxException: Unclosed group near index 19
spark-~!@#$%^^&*(12
                   ^
	at java.util.regex.Pattern.error(Pattern.java:1955)
	at java.util.regex.Pattern.accept(Pattern.java:1813)
	at java.util.regex.Pattern.group0(Pattern.java:2908)
	at java.util.regex.Pattern.sequence(Pattern.java:2051)
	at java.util.regex.Pattern.expr(Pattern.java:1996)
	at java.util.regex.Pattern.compile(Pattern.java:1696)
	at java.util.regex.Pattern.<init>(Pattern.java:1351)
	at java.util.regex.Pattern.compile(Pattern.java:1028)
	at scala.util.matching.Regex.<init>(Regex.scala:191)
	at scala.collection.immutable.StringLike$class.r(StringLike.scala:255)
	at scala.collection.immutable.StringOps.r(StringOps.scala:29)
	at scala.collection.immutable.StringLike$class.r(StringLike.scala:244)
	at scala.collection.immutable.StringOps.r(StringOps.scala:29)
	at org.apache.spark.util.Utils$.redact(Utils.scala:2605)
	at org.apache.spark.sql.execution.DataSourceScanExec$class.org$apache$spark$sql$execution$DataSourceScanExec$$redact(DataSourceScanExec.scala:69)
	at org.apache.spark.sql.execution.DataSourceScanExec$$anonfun$3.apply(DataSourceScanExec.scala:53)
	at org.apache.spark.sql.execution.DataSourceScanExec$$anonfun$3.apply(DataSourceScanExec.scala:51)

[SparkQA]

Test build #75100 has finished for PR 17397 at commit 2c160ea.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gatorsmile]

Thanks! @hvanhovell Could we also output the conf name?

spark-[{. is not a valid regex
java.lang.IllegalArgumentException: spark-[{. is not a valid regex
	at org.apache.spark.internal.config.ConfigHelpers$.regexFromString(ConfigBuilder.scala:74)
	at org.apache.spark.internal.config.ConfigBuilder$$anonfun$regexConf$1.apply(ConfigBuilder.scala:228)
	at org.apache.spark.internal.config.ConfigBuilder$$anonfun$regexConf$1.apply(ConfigBuilder.scala:228)
	at scala.Option.map(Option.scala:146)
	at org.apache.spark.internal.config.ConfigEntryWithDefault.readFrom(ConfigEntry.scala:79)
	at org.apache.spark.SparkConf.get(SparkConf.scala:259)
	at org.apache.spark.util.Utils$.redact(Utils.scala:2604)
	at org.apache.spark.sql.execution.DataSourceScanExec$class.org$apache$spark$sql$execution$DataSourceScanExec$$redact(DataSourceScanExec.scala:69)

[SparkQA]

Test build #75166 has finished for PR 17397 at commit cff62f6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gatorsmile]

LGTM pending Jenkins

[SparkQA]

Test build #75174 has finished for PR 17397 at commit 73cd8c4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gatorsmile]

Thanks! Merging to master.

[ueshin]

@hvanhovell This seems to have broken the 2.10 build:

[error] /home/jenkins/workspace/spark-master-compile-maven-scala-2.10/core/src/main/scala/org/apache/spark/internal/config/ConfigBuilder.scala:228: value regex is not a member of scala.util.matching.Regex
[error]     new TypedConfigBuilder(this, regexFromString(_, this.key), _.regex)
[error]                                                                  ^

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/job/spark-master-compile-maven-scala-2.10/4004/console

[hvanhovell]

@ueshin here you go: #17420

[ueshin]

I see, thanks!

[ueshin]

@hvanhovell DataSourceScanExecRedactionSuite introduced by this pr seems to cause another issue when the basePath is too long to redact:

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-master-test-maven-hadoop-2.6/2895/testReport/org.apache.spark.sql.execution/DataSourceScanExecRedactionSuite/treeString_is_redacted/

[gatorsmile]

Sorry, I did not catch it. Let me submit a follow-up to fix this test case issue.

[gatorsmile]

Submitted the PR #17448. Please check whether the fix is appropriate.