[HUDI-5317] Fix insert overwrite table for partitioned table#7793
[HUDI-5317] Fix insert overwrite table for partitioned table#7793leesf merged 4 commits intoapache:masterfrom
Conversation
|
@stream2000 @leesf Could you please help to review |
| } | ||
| }) | ||
| } | ||
| spark.sql(s"insert overwrite table $tblNonPartition select 3, 'a3', 10, 1000") |
There was a problem hiding this comment.
is there test case for insert overwrite partitioned table?
There was a problem hiding this comment.
@leesf Test Insert Overwrite Table for V2 Table is unnecessary, it is completely consistent with Test Insert Overwrite, and their results must be the same in spark2 and spark3.
So I just removed it, and fix spark2 by repalce if (overwrite && catalogTable.partitionFields.isEmpty) with if (overwrite && partitionSpec.isEmpty)
danny0405
added
area:sql
|
cc @jonvex and @lokeshj1703 to take a look :) |
| } | ||
|
|
||
| test("Test Insert Overwrite Table for V2 Table") { | ||
| withSQLConf("hoodie.schema.on.read.enable" -> "true") { |
There was a problem hiding this comment.
V2 Table is only enabled when hoodie.schema.on.read.enable is true, otherwise v2Table.v1TableWrapper will be used(see org.apache.spark.sql.hudi.catalog.HoodieCatalog#loadTable). In V2 table, we can distinguish between insert overwrite partition and insert overwrite table while we can't do this in v1 table, so I add a v2 table test here to test the different behaviors between v1 and v2 table.
There was a problem hiding this comment.
Year, I noticed that you added this config to force the use of the V2 table, but I think in the future, hudi spark3 may use v2 by default instead of being controlled by this config.
Beside, v1 table can also distinguish insert overwrite partition and insert overwrite table by checking partitionSpec is empty or not, so I think the test should be uniform.
|
LGTM |
yihua
added
priority:blocker
6 participants
Change Logs
fix #7365
Test Insert Overwrite Table for V2 Tableis unnecessary, it is completely consistent withTest Insert Overwrite Table, and their results must be the same in spark2 and spark3.So I just removed it, and fix spark2 by repalce
if (overwrite && catalogTable.partitionFields.isEmpty)withif (overwrite && partitionSpec.isEmpty)Impact
low
Risk level (write none, low medium or high below)
low
Documentation Update
None
Contributor's checklist