HADOOP-17215: Support for conditional overwrite.#4
Open
arjun4084346 wants to merge 10 commits intobranch-2.10from
Open
HADOOP-17215: Support for conditional overwrite.#4arjun4084346 wants to merge 10 commits intobranch-2.10from
arjun4084346 wants to merge 10 commits intobranch-2.10from
Conversation
Contributed by Mukund Thakur. (cherry picked from commit 8031c66)
Contributed by Mehakmeet Singh.There (cherry picked from commit 459eb2a)
Contributed by: Mehakmeet Singh. Change-Id: Ib84e7a42f28e064df4c6204fcce33e573360bf42 (cherry picked from commit 7f486f0)
Contributed by Mehakmeet Singh. (cherry picked from commit 3472c3e)
arjun4084346
force-pushed
the
HADOOP-17215_branch-2.10
branch
from
dcd6627 to
a9cf955
Compare
4 tasks
arjun4084346
changed the base branch from
HADOOP-17065_branch-2.10
to
HADOOP-17229_branch-2.10
…nse failure occurs in ABFS (apache#2264) Contributed by Mehakmeet Singh (cherry picked from commit 0d85515)
Contributed by Sneha Vijayarajan
DETAILS:
This change adds config key "fs.azure.enable.conditional.create.overwrite" with
a default of true. When enabled, if create(path, overwrite: true) is invoked
and the file exists, the ABFS driver will first obtain its etag and then attempt
to overwrite the file on the condition that the etag matches. The purpose of this
is to mitigate the non-idempotency of this method. Specifically, in the event of
a network error or similar, the client will retry and this can result in the file
being created more than once which may result in data loss. In essense this is
like a poor man's file handle, and will be addressed more thoroughly in the future
when support for lease is added to ABFS.
TEST RESULTS:
namespace.enabled=true
auth.type=SharedKey
-------------------
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
Tests run: 457, Failures: 0, Errors: 0, Skipped: 42
Tests run: 207, Failures: 0, Errors: 0, Skipped: 24
namespace.enabled=true
auth.type=OAuth
-------------------
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
Tests run: 457, Failures: 0, Errors: 0, Skipped: 74
Tests run: 207, Failures: 0, Errors: 0, Skipped: 140
(cherry picked from commit e31a636)
…out requiring to cherry pick another commit
arjun4084346
force-pushed
the
HADOOP-17215_branch-2.10
branch
from
a9cf955 to
ba8e37d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
It is almost a clean cherry pick of commit e31a636
The reason for this cherry pick is to backport fix for https://issues.apache.org/jira/browse/HADOOP-17215 (apache#2246)
I need to modify TestAbfsClient.java a bit to avoid cherry picking another commit.
How was this patch tested?
Ran mvn test -pl hadoop-tools/hadoop-azure
Ran all integration abfs tests using mvn -T 1C -Dparallel-tests=abfs clean verify with my storage account arjundev.dfs.core.windows.net
Storage account's Primary location: East US, Secondary location: West US
No new unit tests fail. No tests failed. There are some errors; these were negative test cases where error was expected.
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?