Skip to content

[SPARK-29869][SQL] improve error message in HiveMetastoreCatalog#convertToLogicalRelation #26499

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
LantaoJin wants to merge 10 commits into from
Closed

[SPARK-29869][SQL] improve error message in HiveMetastoreCatalog#convertToLogicalRelation #26499

Changes from 2 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
dd1efc8
[SPARK-29869][SQL] Make error message more meaningful when HiveMetast…
LantaoJin Nov 13, 2019
1071b50
add suggestion
LantaoJin Nov 13, 2019
1c98c38
add ut
LantaoJin Nov 14, 2019
363afb6
fix
LantaoJin Nov 14, 2019
46c16c0
fix
LantaoJin Nov 14, 2019
1df7b8d
fix ut
LantaoJin Nov 15, 2019
9f29abc
address comments
LantaoJin Nov 15, 2019
ad71656
remove useless imports
LantaoJin Nov 15, 2019
2efaafc
address comments
LantaoJin Nov 19, 2019
3288ffd
revert sameType
LantaoJin Nov 19, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -257,8 +257,16 @@ private[hive] class HiveMetastoreCatalog(sparkSession: SparkSession) extends Log
}
// The inferred schema may have different field names as the table schema, we should respect
// it, but also respect the exprId in table relation output.
assert(result.output.length == relation.output.length &&
result.output.zip(relation.output).forall { case (a1, a2) => a1.dataType == a2.dataType })
assert(result.output.length == relation.output.length,
s"Target table has ${result.output.length} columns, " +
s"but source table has ${relation.output.length} columns. " +
s"It may need to recreate the table ${relation.tableMeta.identifier}")
result.output.zip(relation.output).foreach { case (a1, a2) =>
assert(a1.dataType.sameType(a2.dataType),
s"Data type of column ${a1.name} in target table is ${a1.dataType.typeName}, " +
s"but column ${a2.name} in source table is ${a2.dataType.typeName}. " +
s"It may need to recreate the table ${relation.tableMeta.identifier}")
}
val newOutput = result.output.zip(relation.output).map {
case (a1, a2) => a1.withExprId(a2.exprId)
}
Expand Down