[HUDI-3136] Fix merge/insert/show partitions error on Spark3.2 #4490
Conversation
|
@hudi-bot run azure |
| classOf[ValidateDuplicateKeyPayload].getCanonicalName | ||
| } else { | ||
| classOf[DefaultHoodieRecordPayload].getCanonicalName | ||
| classOf[OverwriteWithLatestAvroPayload].getCanonicalName |
| super.run(sparkSession) | ||
| try { | ||
| // Delete all data in the table directory | ||
| super.run(sparkSession) |
There was a problem hiding this comment.
it will throw exception and cause failure?
There was a problem hiding this comment.
According to the process, call the TruncateTableCommand in Spark first and reconstruct the hoodie's table.
Within the TruncateTableCommand in Spark, it will delete the related directories first, and refresh the table which will fail because .hoodie dir has gone and it can't resolve hudi's relation. So I catch this and ignore this.
The related error as following:
Cause: org.apache.hudi.exception.TableNotFoundException: Hoodie table not found in path Unable to find a hudi table for the user provided paths.
at org.apache.hudi.DataSourceUtils.getTablePath(DataSourceUtils.java:85)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:103)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:348)
at org.apache.spark.sql.execution.datasources.FindDataSourceTable.$anonfun$readDataSourceTable$1(DataSourceStrategy.scala:252)
There was a problem hiding this comment.
@YannByron looks like it deserves a log statement or comment in the catch block to help explain the try catch scenario?
| Seq("year=2021/month=02/day=01") | ||
| ) | ||
| checkAnswer(s"show partitions $tableName partition(day=01)")( | ||
| checkAnswer(s"show partitions $tableName partition(day='01')")( |
There was a problem hiding this comment.
here partition must be in string format?
There was a problem hiding this comment.
it's related to the origin type of the field. In this case, day is string type.
|
@hudi-bot run azure |
| super.run(sparkSession) | ||
| try { | ||
| // Delete all data in the table directory | ||
| super.run(sparkSession) |
There was a problem hiding this comment.
@YannByron looks like it deserves a log statement or comment in the catch block to help explain the try catch scenario?
|
@hudi-bot run azure |
4 participants
Tips
What is the purpose of the pull request
(For example: This pull request adds quick-start document.)
Brief change log
(for example:)
Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)
Committer checklist
Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.