HADOOP-15183 S3Guard store becomes inconsistent after partial failure of rename#654

Closed

steveloughran wants to merge 22 commits intoapache:trunkfrom

steveloughran:s3/HADOOP-15183-s3guard-rename-failures

Commits on May 17, 2019

HADOOP-15183: previous patch coalesced into a new PR
steveloughran
committed
HADOOP-15183 getting ITestPartialRenamesDeletes tests to work by putting check for multidelete into the correct test case
steveloughran
committed
HADOOP-15183 evolving the tests to simulate the real failure scenario
steveloughran
committed
HADOOP-15183: parallel submission of delete operations to DDB.
steveloughran
committed
HADOOP-15183 peformance boost of prune on on-demand DDB tables:
steveloughran
committed
HADOOP-15183: improve parallel execution of metastore deletes
steveloughran
committed
HADOOP-15183. Minor code cleanup before going more aggressively into failures-during-copy
steveloughran
committed
HADOOP-15183: fix up findbugs warnings (and failure to create directory allocator) by providing a specific allocator callback.
steveloughran
committed
HADOOP-15183: ITestMagiCommitMRJob is failing with __magic still being there in parallel auth test runs. Change the failure to incude a listing so as to work out what is happening.
steveloughran
committed
HADOOP-15183 exploring making the copy parallelized; need to be clearer about copy and update logic first, and want to merge in the versioning stuff from Cerner first
steveloughran
committed
HADOOP-15183 S3Guard and rename failures.
steveloughran
committed
HADOOP-15183: copies are parallelised
steveloughran
committed
HADOOP-15183: rename operation failure handling works in the test cases
steveloughran
committed
HADOOP-15183: progressive rename tracker
steveloughran
committed
HADOOP-15183 rename all happy.
steveloughran
committed
HADOOP-15183: tracking ancestors across move operations.
steveloughran
committed
HADOOP-15183: bulk metastore operation performance/scale
steveloughran
committed
HADOOP-15183: address github review and yetus comments
steveloughran
committed
HADOOP-15183: full end-to-end test to verify that >1 commit will only update parent directories once.
steveloughran
committed

Commits on May 18, 2019

Commits on May 19, 2019

HADOOP-15183 remove diff between this branch and trunk; checkstyle
steveloughran
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-15183 S3Guard store becomes inconsistent after partial failure of rename#654

HADOOP-15183 S3Guard store becomes inconsistent after partial failure of rename#654
steveloughran wants to merge 22 commits intoapache:trunkfrom
steveloughran:s3/HADOOP-15183-s3guard-rename-failures

Commits on May 17, 2019

HADOOP-15183: previous patch coalesced into a new PR

HADOOP-15183 getting ITestPartialRenamesDeletes tests to work by putting check for multidelete into the correct test case

HADOOP-15183 evolving the tests to simulate the real failure scenario

HADOOP-15183: parallel submission of delete operations to DDB.

HADOOP-15183 peformance boost of prune on on-demand DDB tables:

HADOOP-15183: improve parallel execution of metastore deletes

HADOOP-15183. Minor code cleanup before going more aggressively into failures-during-copy

HADOOP-15183: fix up findbugs warnings (and failure to create directory allocator) by providing a specific allocator callback.

HADOOP-15183: ITestMagiCommitMRJob is failing with __magic still being there in parallel auth test runs. Change the failure to incude a listing so as to work out what is happening.

HADOOP-15183 exploring making the copy parallelized; need to be clearer about copy and update logic first, and want to merge in the versioning stuff from Cerner first

HADOOP-15183 S3Guard and rename failures.

HADOOP-15183: copies are parallelised

HADOOP-15183: rename operation failure handling works in the test cases

HADOOP-15183: progressive rename tracker

HADOOP-15183 rename all happy.

HADOOP-15183: tracking ancestors across move operations.

HADOOP-15183: bulk metastore operation performance/scale

HADOOP-15183: address github review and yetus comments

HADOOP-15183: full end-to-end test to verify that >1 commit will only update parent directories once.

Commits on May 18, 2019

HADOOP-15183: teardown improvements in ITestPartialRenamesDeletes

HADOOP-15183 checkstyle

Commits on May 19, 2019

HADOOP-15183 remove diff between this branch and trunk; checkstyle