Skip to content

[SPARK-31522][SQL] Hive metastore client initialization related configurations should be static #28302

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yaooqinn wants to merge 6 commits into from
Closed

[SPARK-31522][SQL] Hive metastore client initialization related configurations should be static #28302

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
d0cf0ec
[SPARK-31522][SQL] Hive metastore client initialization related confi…
yaooqinn Apr 22, 2020
344f558
nit
yaooqinn Apr 22, 2020
5c15a98
fix test
yaooqinn Apr 23, 2020
09b87ff
spark.sql.hive.version should be runtime for backward compatibility
yaooqinn Apr 23, 2020
c72ba70
fix test
yaooqinn Apr 23, 2020
7a26872
fix tests
yaooqinn Apr 23, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 5 additions & 6 deletions sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ private[spark] object HiveUtils extends Logging {
/** The version of hive used internally by Spark SQL. */
val builtinHiveVersion: String = if (isHive23) hiveVersion else "1.2.1"

val HIVE_METASTORE_VERSION = buildConf("spark.sql.hive.metastore.version")
val HIVE_METASTORE_VERSION = buildStaticConf("spark.sql.hive.metastore.version")
.doc("Version of the Hive metastore. Available options are " +
"<code>0.12.0</code> through <code>2.3.7</code> and " +
"<code>3.0.0</code> through <code>3.1.2</code>.")
Expand All @@ -75,10 +75,9 @@ private[spark] object HiveUtils extends Logging {
val FAKE_HIVE_VERSION = buildConf("spark.sql.hive.version")
.doc(s"deprecated, please use ${HIVE_METASTORE_VERSION.key} to get the Hive version in Spark.")
.version("1.1.1")
.stringConf
.createWithDefault(builtinHiveVersion)
.fallbackConf(HIVE_METASTORE_VERSION)

val HIVE_METASTORE_JARS = buildConf("spark.sql.hive.metastore.jars")
val HIVE_METASTORE_JARS = buildStaticConf("spark.sql.hive.metastore.jars")
.doc(s"""
| Location of the jars that should be used to instantiate the HiveMetastoreClient.
| This property can be one of three options: "
Expand Down Expand Up @@ -137,7 +136,7 @@ private[spark] object HiveUtils extends Logging {
.booleanConf
.createWithDefault(true)

val HIVE_METASTORE_SHARED_PREFIXES = buildConf("spark.sql.hive.metastore.sharedPrefixes")
val HIVE_METASTORE_SHARED_PREFIXES = buildStaticConf("spark.sql.hive.metastore.sharedPrefixes")
.doc("A comma separated list of class prefixes that should be loaded using the classloader " +
"that is shared between Spark SQL and a specific version of Hive. An example of classes " +
"that should be shared is JDBC drivers that are needed to talk to the metastore. Other " +
Expand All @@ -151,7 +150,7 @@ private[spark] object HiveUtils extends Logging {
private def jdbcPrefixes = Seq(
"com.mysql.jdbc", "org.postgresql", "com.microsoft.sqlserver", "oracle.jdbc")

val HIVE_METASTORE_BARRIER_PREFIXES = buildConf("spark.sql.hive.metastore.barrierPrefixes")
val HIVE_METASTORE_BARRIER_PREFIXES = buildStaticConf("spark.sql.hive.metastore.barrierPrefixes")
.doc("A comma separated list of class prefixes that should explicitly be reloaded for each " +
"version of Hive that Spark SQL is communicating with. For example, Hive UDFs that are " +
"declared in a prefix that typically would be shared (i.e. <code>org.apache.spark.*</code>).")
Expand Down
10 changes: 10 additions & 0 deletions sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2493,6 +2493,16 @@ abstract class SQLQuerySuiteBase extends QueryTest with SQLTestUtils with TestHi
}
}
}

test("SPARK-31522: hive metastore related configurations should be static") {
Seq("spark.sql.hive.metastore.version",
"spark.sql.hive.metastore.jars",
"spark.sql.hive.metastore.sharedPrefixes",
"spark.sql.hive.metastore.barrierPrefixes").foreach { key =>
val e = intercept[AnalysisException](sql(s"set $key=abc"))
assert(e.getMessage.contains("Cannot modify the value of a static config"))
}
}
}

class SQLQuerySuite extends SQLQuerySuiteBase with DisableAdaptiveExecutionSuite
Expand Down