Disallow dropping partition column in Iceberg table explicitly#21294
Conversation
|
Isn't it true that this issue only affects prior (historical) specs, and not the latest? So for example, it doesn't affect brand new tables, and it doesn't affect newly added columns? |
That's right, because dropping columns used in current partition spec would explicitly fail in Iceberg implementation. So when we try to drop a partition column, we must firstly drop it from current partition spec. That means, the column exists in a historical spec now. Then we will meet various problems subsequently after dropping it from schema. In my experiment, simultaneously drop the column from partition spec and schema in a single transaction would meet problems in the issue too. Furthermore, this PR can also prevent Iceberg table falling into this situation in case that, for example, if we support dropping column from partition spec only. |
There was a problem hiding this comment.
nit: Personally, I would make the error message more specific
| throw new PrestoException(NOT_SUPPORTED, "This connector does not support drop partition columns"); | |
| throw new PrestoException(NOT_SUPPORTED, "This connector does not support dropping columns which exist in any of the table's partition specs"); |
ab6c619 to
69b37e3
Compare
69b37e3 to
035f136
Compare
Description
When we try to drop a partition column in Iceberg table, we should firstly remove it from the partition spec, then drop the column in table schema. But currently Iceberg have some problems in this scenario, see issue apache/iceberg#4563. This PR explicitly disallow drop partition column until Iceberg fixed it.
Contributor checklist
Release Notes