Skip to content

[SPARK-34454][SQL] Mark legacy SQL configs as internal #31577

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 3 commits into from
Closed

[SPARK-34454][SQL] Mark legacy SQL configs as internal #31577

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
871dc96
Add a test
MaxGekk Feb 17, 2021
111635d
Mark legacy configs as internal
MaxGekk Feb 17, 2021
84aed2f
.legacy. -> spark.sql.legacy.
MaxGekk Feb 18, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2850,6 +2850,7 @@ object SQLConf {
.createWithDefault(100)

val LEGACY_ALLOW_HASH_ON_MAPTYPE = buildConf("spark.sql.legacy.allowHashOnMapType")
.internal()
.doc("When set to true, hash expressions can be applied on elements of MapType. Otherwise, " +
"an analysis exception will be thrown.")
.version("3.0.0")
Expand Down
1 change: 1 addition & 0 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,6 +195,7 @@ object StaticSQLConf {

val SQL_LEGACY_SESSION_INIT_WITH_DEFAULTS =
buildStaticConf("spark.sql.legacy.sessionInitWithConfigDefaults")
.internal()
.doc("Flag to revert to legacy behavior where a cloned SparkSession receives SparkConf " +
"defaults, dropping any overrides in its parent SparkSession.")
.version("3.0.0")
Expand Down
10 changes: 10 additions & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/internal/SQLConfSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -451,4 +451,14 @@ class SQLConfSuite extends QueryTest with SharedSparkSession {
val e2 = intercept[ParseException](sql("set time zone interval 19 hours"))
assert(e2.getMessage contains "The interval value must be in the range of [-18, +18] hours")
}

test("SPARK-34454: configs from the legacy namespace should be internal") {
val nonInternalLegacyConfigs = spark.sessionState.conf.getAllDefinedConfs
.filter { case (key, _, _, _) => key.contains("spark.sql.legacy.") }
assert(nonInternalLegacyConfigs.isEmpty,
s"""
|Non internal legacy SQL configs:
|${nonInternalLegacyConfigs.map(_._1).mkString("\n")}
|""".stripMargin)
}
}