Skip to content

[SPARK-28054][SQL] Fix error when insert Hive partitioned table dynamically where partition name is upper case #24886

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
viirya wants to merge 1 commit into from
Closed

[SPARK-28054][SQL] Fix error when insert Hive partitioned table dynamically where partition name is upper case #24886

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/SaveAsHiveFile.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,16 @@ private[hive] trait SaveAsHiveFile extends DataWritingCommand {
jobId = java.util.UUID.randomUUID().toString,
outputPath = outputLocation)

// SPARK-28054: Hive metastore is not case preserving and keeps partition columns
// with lower cased names, Hive will validate the column names in partition spec and
// the partition paths. Besides lowercasing the column names in the partition spec,
// we also need to lowercase the column names in written partition paths.
// scalastyle:off caselocale
val hiveCompatiblePartitionColumns = partitionAttributes.map { attr =>
attr.withName(attr.name.toLowerCase)
Copy link
Member

@gatorsmile gatorsmile Jun 26, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indent.

Copy link
Member Author

@viirya viirya Jun 26, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops..will fix in a follow-up. Thanks.

}
// scalastyle:on caselocale

FileFormatWriter.write(
sparkSession = sparkSession,
plan = plan,
Expand All @@ -91,7 +101,7 @@ private[hive] trait SaveAsHiveFile extends DataWritingCommand {
outputSpec =
FileFormatWriter.OutputSpec(outputLocation, customPartitionLocations, outputColumns),
hadoopConf = hadoopConf,
partitionColumns = partitionAttributes,
partitionColumns = hiveCompatiblePartitionColumns,
bucketSpec = None,
statsTrackers = Seq(basicWriteJobStatsTracker(hadoopConf)),
options = Map.empty)
Expand Down
18 changes: 18 additions & 0 deletions sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1188,6 +1188,24 @@ class HiveQuerySuite extends HiveComparisonTest with SQLTestUtils with BeforeAnd
}
}
}

test("SPARK-28054: Unable to insert partitioned table when partition name is upper case") {
withTable("spark_28054_test") {
sql("set hive.exec.dynamic.partition.mode=nonstrict")
Copy link
Member

@gatorsmile gatorsmile Jun 26, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we set it back?

Use withSQLConf ?

Also do we need to set the case sensitivity conf?

Copy link
Member Author

@viirya viirya Jun 26, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This set follows other tests in same suite. Using withSQLConf is good, yes.

The case sensitivity conf has no effect on this, I think it is fine.

sql("CREATE TABLE spark_28054_test (KEY STRING, VALUE STRING) PARTITIONED BY (DS STRING)")

sql("INSERT INTO TABLE spark_28054_test PARTITION(DS) SELECT 'k' KEY, 'v' VALUE, '1' DS")

assertResult(Array(Row("k", "v", "1"))) {
sql("SELECT * from spark_28054_test").collect()
}

sql("INSERT INTO TABLE spark_28054_test PARTITION(ds) SELECT 'k' key, 'v' value, '2' ds")
assertResult(Array(Row("k", "v", "1"), Row("k", "v", "2"))) {
sql("SELECT * from spark_28054_test").collect()
}
}
}
}

// for SPARK-2180 test
Expand Down