Refactor Delta's DROP TABLE support for corrupted tables#16651
Refactor Delta's DROP TABLE support for corrupted tables#16651
Conversation
findepi
requested review from
alexjo2144,
ebyhr,
findinpath,
krvikash and
losipiuk
There was a problem hiding this comment.
i linked to the PR that adds syntax.
Once we have syntax supported we should have an uncommented test line here, even before the functionality is supported in Delta
There was a problem hiding this comment.
Note: NullPointerException here (before the refactor)
There was a problem hiding this comment.
Note: before the change we allowed rename. I think it was incidental
There was a problem hiding this comment.
Note: testGetInsertLayoutTableNotFound test was removed, instead of
being updated, since ConnectorMetadata.getInsertLayout cannot be
reached for a corrupted table, as getting column list will fail earlier.
findepi
force-pushed
the
findepi/drop-delta-corrupted
branch
from
bb00968 to
9afbe9f
Compare
|
CI #16658 |
|
The BigQuery CI failures are handled in #16603 |
plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java
Outdated
Show resolved
Hide resolved
This makes it consistent with other methods like that, in particular with `DeltaLakeTableHandle.forOptimize`.
|
Conflict with #16466, rebasing |
findepi
force-pushed
the
findepi/drop-delta-corrupted
branch
from
ab8404e to
6045f41
Compare
There was a problem hiding this comment.
nit: you can extract throwIfCourrupted(ConnectorTableHandle handle) method
There was a problem hiding this comment.
i wanted to have throw in the calling code so that code flow analysis works, but i don't think there is much gain from that.
will change
Previously, a corrupted table handle was represented with a `DeltaLakeTableHandle` missing a `MetadataEntry`. When drop support for corrupted tables was implemented, a connector could have only one table handle class. This commit improves the distinction by introducing a dedicated table handle class to represent corrupted tables. As a necessity, this class is explicitly handled in multiple `DeltaLakeMetadata` methods. This sets viable example to follow for implementing drop support for corrupted Iceberg tables as a follow-up. Note: `testGetInsertLayoutTableNotFound` test was removed, instead of being updated, since `ConnectorMetadata.getInsertLayout` cannot be reached for a corrupted table, as getting column list will fail earlier.
findepi
force-pushed
the
findepi/drop-delta-corrupted
branch
from
6045f41 to
c23ec5e
Compare
|
|
||
| // DROP TABLE should succeed so that users can remove their corrupted table | ||
| getQueryRunner().execute("DROP TABLE " + tableName); | ||
| assertFalse(getQueryRunner().tableExists(getSession(), tableName)); |
There was a problem hiding this comment.
Can we also verify that the table location does not have any files after DROP TABLE execute?
There was a problem hiding this comment.
as discussed offline, in this test we already deleted all the files
// break the table by deleting all its files including transaction log
deleteRecursively(tableLocation, ALLOW_INSECURE);
i like the idea to improve the test. let's delete only transaction log and then add assertion that data files got removed.
since the PR is otherwise ready to go let's handle this as a follow-up
4 participants
Previously, a corrupted table handle was represented with a
DeltaLakeTableHandlemissing aMetadataEntry. When drop support forcorrupted tables was implemented, a connector could have only one table
handle class. This commit improves the distinction by introducing a
dedicated table handle class to represent corrupted tables. As a
necessity, this class is explicitly handled in multiple
DeltaLakeMetadatamethods. This sets viable example to follow forimplementing drop support for corrupted Iceberg tables as a follow-up.
Note:
testGetInsertLayoutTableNotFoundtest was removed, instead ofbeing updated, since
ConnectorMetadata.getInsertLayoutcannot bereached for a corrupted table, as getting column list will fail earlier.