API: Add function for removing Specs from Metadata.json which are no … #3462
Conversation
| MetadataTableUtils.createMetadataTableInstance(table, MetadataTableType.ALL_ENTRIES) | ||
| .newScan() | ||
| .planFiles(), | ||
| fileScanTask -> ((ManifestEntriesTable.ManifestReadTask) (fileScanTask)).partitionSpecId() |
There was a problem hiding this comment.
So the reason I use this api, and not actually accessing the underlying rows of ALL_MANIFESTS table is that for tables which have corrupt specs (from previous bugs) will be unable to actually generate the common "partition" schema which is required to actually manifest rows. This table instead just gets us ManifestFiles which we can check the specID of directly.
|
@rdblue Please let me know what you think about this for the removing unused specs / removing broken and unused specs from table utility |
…longer in use Previously there was no way to remove partition specs from a table once they were added. To fix this we add an api which searches through all reachable mainfest files and records their specsIds. Any specIds which do not find are marked for removal which is done through a serializable commit.
RussellSpitzer
force-pushed
the
RemoveUnusedSpec
branch
from
b23ac11 to
2677026
Compare
|
@aokolnychyi do you think this is still worth doing? I can update the PR if so |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
1 participant
…longer in use
Previously there was no way to remove partition specs from a table once they were
added. To fix this we add an api which searches through all reachable mainfest
files and records their specsIds. Any specIds which do not find are marked for
removal which is done through a serializable commit.