-
Notifications
You must be signed in to change notification settings - Fork 9.2k
HADOOP-17244. S3A directory delete tombstones dir markers prematurely. #2280
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HADOOP-17244. S3A directory delete tombstones dir markers prematurely. #2280
Conversation
This changes directory tree deletion so that only files are incrementally deleted from S3Guard after the objects are deleted; the directories are left alone until metadataStore.deleteSubtree(path) is invoked This avoids directory tombstones being added above files/child directories, which stop that subtree operation from actually finishing. Also: * callback to delete objects splits files and dirs so that any problems deleting the dirs doesn't trigger s3guard updates * new statistic to measure #of objects deleted, alongside request count. * callback listFilesAndEmptyDirectories renamed listFilesAndDirectoryMarkers to clarify behavior. Test enhancements in ITestDirectoryOperationCost, but I didn't manage to replicate it there (didn't quite understand the cause); the FileContext test works for regression testing. Change-Id: I8f96536bc6ffaf548b3da1068b9bfc897d34a993
|
Tests in progress |
|
broke the local fs tests |
* Fix S3A test failures caused by DeleteOperation bugs added * Extend S3A tests * Fix FileContext base test so that raised IOEs are always rethrown regarding change apache#3, I think those IOEs should be thrown first, but as the existing code does the asserts in finally(), I'm leaving it alone Change-Id: Ia3159db7760424264d84fa2e38d2cbb452a9854a
|
Note: I ran the hdfs contract tests in my IDE to verify it was happy too; all good. |
bgaborg
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, +1 pending on integration test results, and which endpoint you tested against.
|
repeatedly tested against london with options showing the error and the -Dkeep option and unguarded. Some transient failures related to local network issues, addressed in HADOOP-17181. |
#2280) This changes directory tree deletion so that only files are incrementally deleted from S3Guard after the objects are deleted; the directories are left alone until metadataStore.deleteSubtree(path) is invoke. This avoids directory tombstones being added above files/child directories, which stop the treewalk and delete phase from working. Also: * Callback to delete objects splits files and dirs so that any problems deleting the dirs doesn't trigger s3guard updates * New statistic to measure #of objects deleted, alongside request count. * Callback listFilesAndEmptyDirectories renamed listFilesAndDirectoryMarkers to clarify behavior. * Test enhancements to replicate the failure and verify the fix Contributed by Steve Loughran Change-Id: I0e6ea2c35e487267033b1664228c8837279a35c7
|
This patch is incomplete. Reverted and will create a new one. Issues
I think I may want to push this delete/no-delete logic into s3guard -it can do the check rather than relying on the layer above. But: we will need to pass down for each path whether a path is a dir or a file, so that s3guard doesn't do a DDB query on every single path entry to work that out for itself |
status
This PR was merged into trunk and branch-3.3, but reverted after it became clear that it was incomplete.
See #2310 for the successor PR
description
This changes directory tree deletion so that only files are incrementally deleted
from S3Guard after the objects are deleted; the directories are left alone
until metadataStore.deleteSubtree(path) is invoked
This avoids directory tombstones being added above files/child directories,
which stop that subtree operation from actually finishing.
Also:
any problems deleting the dirs doesn't trigger s3guard updates
to clarify behavior.
Test enhancements in ITestDirectoryOperationCost, but I didn't manage to
replicate it there (didn't quite understand the cause); the FileContext
test works for regression testing.
Change-Id: I8f96536bc6ffaf548b3da1068b9bfc897d34a993