Skip to content

[SPARK-19664][SQL]put hive.metastore.warehouse.dir in hadoopconf to overwrite its original value #16996

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
windpiger wants to merge 7 commits into from
Closed

[SPARK-19664][SQL]put hive.metastore.warehouse.dir in hadoopconf to overwrite its original value #16996

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
92c1452
[SPARK-19664][SQL]put hive.metastore.warehouse.dir in hadoopconf to o…
windpiger Feb 20, 2017
ac0a1c6
add some loginfo, and fix a bug
windpiger Feb 20, 2017
91b9fd2
remove a log info
windpiger Feb 21, 2017
7429cd8
always set in hadoopconf
windpiger Feb 23, 2017
26ba212
modify a comment
windpiger Feb 23, 2017
1d18636
modify a comment
windpiger Feb 23, 2017
86deb62
remove hive.metastore.warehouse.dir in sparkconf
windpiger Feb 24, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,9 +42,12 @@ private[sql] class SharedState(val sparkContext: SparkContext) extends Logging {
val warehousePath = {
val configFile = Utils.getContextOrSparkClassLoader.getResource("hive-site.xml")
if (configFile != null) {
logInfo(s"loading hive config file: $configFile")
sparkContext.hadoopConfiguration.addResource(configFile)
}

// hive.metastore.warehouse.dir only stay in hadoopConf
sparkContext.conf.remove("hive.metastore.warehouse.dir")
// Set the Hive metastore warehouse path to the one we use
val hiveWarehouseDir = sparkContext.hadoopConfiguration.get("hive.metastore.warehouse.dir")
if (hiveWarehouseDir != null && !sparkContext.conf.contains(WAREHOUSE_PATH.key)) {
Expand All @@ -61,10 +64,11 @@ private[sql] class SharedState(val sparkContext: SparkContext) extends Logging {
// When neither spark.sql.warehouse.dir nor hive.metastore.warehouse.dir is set,
// we will set hive.metastore.warehouse.dir to the default value of spark.sql.warehouse.dir.
val sparkWarehouseDir = sparkContext.conf.get(WAREHOUSE_PATH)
sparkContext.conf.set("hive.metastore.warehouse.dir", sparkWarehouseDir)
logInfo(s"Setting hive.metastore.warehouse.dir ('$hiveWarehouseDir') to the value of " +
s"${WAREHOUSE_PATH.key} ('$sparkWarehouseDir').")
sparkContext.hadoopConfiguration.set("hive.metastore.warehouse.dir", sparkWarehouseDir)
sparkWarehouseDir
}

}
logInfo(s"Warehouse path is '$warehousePath'.")

Expand Down