[HUDI-5317] Fix insert overwrite table for partitioned table

[stream2000]

Change Logs

For sql like insert overwrite table $table select xxx, we expect to drop all data in the table first and then insert the selected data into it. But we found that the 'insert overwrite table' semantic works only for non-partitioned table. For partitioned table, current implementation will drop only partitions involved in the select sub-query, other partitions won't be dropped( which should be dropped as expected).

This pr to solve the problem that insert overwrite table can drop all partitions at first then insert new data.

Impact

Insert overwrite table will drop all partitions at first then insert new data.

Risk level (write none, low medium or high below)

None

Documentation Update

None

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[stream2000]

@leesf Could you please help to review this PR?

[leesf]

@stream2000 would you please check CI failure?

[leesf]

@hudi-bot run azure

[stream2000]

@stream2000 would you please check CI failure?

Seems like some uts were failed. Will fix it

[nsivabalan]

we have two operations relating to insert_overwrite.
1: insert_overwrite_table
2: insert_overwrite.

spark-ds writes supports both operations.
insert_overwrite_table will override entire table. while insert_overwrite will overwrite only matching partitions.

guess in spark-sql, we supported only insert_overwrite. not sure if we can revert the behavior. May be we should consider adding a new write operation in spark-sql for this.

[leesf]

we have two operations relating to insert_overwrite. 1: insert_overwrite_table 2: insert_overwrite.

spark-ds writes supports both operations. insert_overwrite_table will override entire table. while insert_overwrite will overwrite only matching partitions.

guess in spark-sql, we supported only insert_overwrite. not sure if we can revert the behavior. May be we should consider adding a new write operation in spark-sql for this.

@nsivabalan hi, here are my two cents: insert overwrite xxx values(xx,xxx) has very clear semantics, it means overwrite the entire table, insert overwrite xx partition(xx) values(xx,xxx) means insert overwrite partitions, but hudi handles overwrite partitions for overwrite table, which is a definite bug and i do not think we need to introduce a new operation for it.

[stream2000]

@hudi-bot run azure

[leesf]

LGTM

[stream2000]

@hudi-bot run azure

[leesf]

@hudi-bot run azure

[hudi-bot]

CI report:

• c963634 UNKNOWN
• 9e1f64f Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build