Rename Resume optimization: resume only if srcDir etag matches to the one with which rename was started. by saxenapranav · Pull Request #85 · ABFSDriver/AbfsHadoop

saxenapranav · 2023-07-24T12:21:50Z

Resume the rename only if srcDirectory is present and it's etag matches to the one that was there at the time of rename. Else, just delete the pendingJson.
PendingJson will now have Etag information of the srcDir at the time of rename.
There is listStatus done after its found that renamePendingJson is there. But since there can be scenario where rename resume shouldnt be done (like srcDir not there, or original srcDir is changed(etag change)), the second listStatus shall be redundant. In case of not requirement, it will not be called.

:::: AGGREGATED TEST RESULT ::::

HNS-OAuth

[INFO] Results:
[INFO]
[WARNING] Tests run: 140, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:383 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 782, Failures: 0, Errors: 1, Skipped: 189
[INFO] Results:
[INFO]
[WARNING] Tests run: 279, Failures: 0, Errors: 0, Skipped: 45

HNS-SharedKey

[INFO] Results:
[INFO]
[WARNING] Tests run: 140, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:383 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 782, Failures: 0, Errors: 1, Skipped: 189
[INFO] Results:
[INFO]
[WARNING] Tests run: 279, Failures: 0, Errors: 0, Skipped: 45

NonHNS-SharedKey

[INFO] Results:
[INFO]
[WARNING] Tests run: 140, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAzureBlobFileSystemRandomRead.testValidateSeekBounds:276->Assert.assertTrue:42->Assert.fail:89 There should not be any network I/O (elapsedTimeMs=155).
[INFO]
[ERROR] Tests run: 782, Failures: 1, Errors: 0, Skipped: 280
[INFO] Results:
[INFO]
[WARNING] Tests run: 279, Failures: 0, Errors: 0, Skipped: 45

NonHNS-OAuth

[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:181->fuzzyValidate:64 The actual value 10 is not within the expected range: [5.60, 8.40].
[INFO]
[ERROR] Tests run: 140, Failures: 1, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[WARNING] Tests run: 782, Failures: 0, Errors: 0, Skipped: 280
[INFO] Results:
[INFO]
[WARNING] Tests run: 279, Failures: 0, Errors: 0, Skipped: 45

AppendBlob-HNS-OAuth

[INFO] Results:
[INFO]
[WARNING] Tests run: 140, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:383 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 782, Failures: 0, Errors: 1, Skipped: 189
[INFO] Results:
[INFO]
[WARNING] Tests run: 279, Failures: 0, Errors: 0, Skipped: 45

Time taken: 47 mins 6 secs.

azureuser@Hadoop-VM-EAST2:~/hadoop/hadoop-tools/hadoop-azure$ git log
commit bab6499eb99e289d75b98a71ad2c8afac1a248e7 (HEAD -> ABFS_3.3.2_dev_redo_optimization, origin/ABFS_3.3.2_dev_redo_optimization_less, origin/ABFS_3.3.2_dev_redo_optimization, ABFS_3.3.2_dev_redo_optimization_less)
Author: Pranav Saxena <>
Date:   Mon Jul 24 06:41:27 2023 -0700

    asssertion if the listStatus API of FS will give correct result when we are not making redundant listStatus backend call

…we are not making redundant listStatus backend call

snvijaya · 2023-07-25T10:05:45Z

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java

@@ -931,12 +931,28 @@ public FileStatus[] listStatus(final Path f) throws IOException {
          == PrefixMode.BLOB) {


Resume actions to be made only if incoming path is a parent to one of the configured atomic rename paths to stop parsing of all the List calls.

snvijaya · 2023-07-25T10:07:11Z

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java

@@ -931,12 +931,28 @@ public FileStatus[] listStatus(final Path f) throws IOException {
          == PrefixMode.BLOB) {
        FileStatus renamePendingFileStatus
            = getAbfsStore().getRenamePendingFileStatus(result);


Rename variable to renamePendingJSONStatus

refactored.

snvijaya · 2023-07-25T10:13:12Z

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java

@@ -1083,15 +1099,20 @@ private FileStatus getFileStatus(final Path path,
          && getAbfsStore().isAtomicRenameKey(fileStatus.getPath().toUri().getPath())


Same as prev comment, resume only if this qualifies as parent of atomic rename path.

Current code already has the check.

…ctory

… listStatus flow

…eview

saxenapranav · 2023-07-25T13:24:56Z

:::: AGGREGATED TEST RESULT ::::

HNS-OAuth

[INFO] Results:
[INFO]
[WARNING] Tests run: 140, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:383 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 785, Failures: 0, Errors: 1, Skipped: 192
[INFO] Results:
[INFO]
[WARNING] Tests run: 282, Failures: 0, Errors: 0, Skipped: 48

HNS-SharedKey

[INFO] Results:
[INFO]
[WARNING] Tests run: 140, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:366 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 785, Failures: 0, Errors: 1, Skipped: 192
[INFO] Results:
[INFO]
[WARNING] Tests run: 282, Failures: 0, Errors: 0, Skipped: 48

NonHNS-SharedKey

[INFO] Results:
[INFO]
[WARNING] Tests run: 140, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemRename.testRenameResumeThroughListStatusWithSrcDirDeletedJustBeforeResume:2350->assertTracingContextOnRenameResumeProcess:2232 » UnfinishedStubbing
[INFO]
[ERROR] Tests run: 785, Failures: 0, Errors: 1, Skipped: 280
[INFO] Results:
[INFO]
[WARNING] Tests run: 282, Failures: 0, Errors: 0, Skipped: 45

NonHNS-OAuth

[INFO] Results:
[INFO]
[WARNING] Tests run: 140, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAzureBlobFileSystemRandomRead.testValidateSeekBounds:276->Assert.assertTrue:42->Assert.fail:89 There should not be any network I/O (elapsedTimeMs=105).
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemExplictImplicitRename.testImplicitDirectoryIntoSameNameImplicitDestination:601->explicitImplicitDirectoryRenameTestWithDestPathNames:796->createSourcePaths:837->AbstractAbfsIntegrationTest.createAzCopyFile:537 » IO
[INFO]
[ERROR] Tests run: 785, Failures: 1, Errors: 1, Skipped: 280
[INFO] Results:
[INFO]
[WARNING] Tests run: 282, Failures: 0, Errors: 0, Skipped: 45

AppendBlob-HNS-OAuth

[INFO] Results:
[INFO]
[WARNING] Tests run: 140, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAzureBlobFileSystemRandomRead.testSkipBounds:225->Assert.assertTrue:42->Assert.fail:89 There should not be any network I/O (elapsedTimeMs=88).
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:366 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 785, Failures: 1, Errors: 1, Skipped: 192
[INFO] Results:
[INFO]
[WARNING] Tests run: 282, Failures: 0, Errors: 0, Skipped: 48

azureuser@Hadoop-VM-EAST2:~/hadoop/hadoop-tools/hadoop-azure$ git log
commit fbf0a244e882e0a9edf210d87069fb03afcc3d1d (HEAD -> ABFS_3.3.2_dev_redo_optimization, origin/ABFS_3.3.2_dev_redo_optimization_review, origin/ABFS_3.3.2_dev_redo_optimization, ABFS_3.3.2_dev_redo_optimization_review)
Merge: d9a38b893e8 e858497e875
Author: Pranav Saxena <>
Date:   Tue Jul 25 04:59:39 2023 -0700

    Merge branch 'ABFS_3.3.2_dev' into ABFS_3.3.2_dev_redo_optimization_review

… via OpenFileParameters (apache#2975) commit: dcddc6a

…edo;

…he cases of rename resume

… test refactor

…ilestatus

saxenapranav · 2023-07-27T05:35:37Z

:::: AGGREGATED TEST RESULT ::::

HNS-OAuth

[INFO] Results:
[INFO]
[WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:397->lambda$testAcquireRetry$6:403 » TestTimedOut
[INFO]
[ERROR] Tests run: 787, Failures: 0, Errors: 1, Skipped: 194
[INFO] Results:
[INFO]
[WARNING] Tests run: 283, Failures: 0, Errors: 0, Skipped: 49

HNS-SharedKey

[INFO] Results:
[INFO]
[WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAzureBlobFileSystemRandomRead.testValidateSeekBounds:276->Assert.assertTrue:42->Assert.fail:89 There should not be any network I/O (elapsedTimeMs=74).
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:383 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 787, Failures: 1, Errors: 1, Skipped: 194
[INFO] Results:
[INFO]
[WARNING] Tests run: 283, Failures: 0, Errors: 0, Skipped: 49

NonHNS-SharedKey

[INFO] Results:
[INFO]
[WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[WARNING] Tests run: 787, Failures: 0, Errors: 0, Skipped: 279
[INFO] Results:
[INFO]
[WARNING] Tests run: 283, Failures: 0, Errors: 0, Skipped: 45

NonHNS-OAuth

[INFO] Results:
[INFO]
[WARNING] Tests run: 141, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[WARNING] Tests run: 787, Failures: 0, Errors: 0, Skipped: 279
[INFO] Results:
[INFO]
[WARNING] Tests run: 283, Failures: 0, Errors: 0, Skipped: 45

AppendBlob-HNS-OAuth

[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:181->fuzzyValidate:64 The actual value 10 is not within the expected range: [5.60, 8.40].
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAzureBlobFileSystemRandomRead.testValidateSeekBounds:276->Assert.assertTrue:42->Assert.fail:89 There should not be any network I/O (elapsedTimeMs=21).
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:383 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 787, Failures: 1, Errors: 1, Skipped: 194
[INFO] Results:
[INFO]
[WARNING] Tests run: 283, Failures: 0, Errors: 0, Skipped: 49

Time taken: 48 mins 6 secs.

azureuser@Hadoop-VM-EAST2:~/hadoop/hadoop-tools/hadoop-azure$ git log
commit b76972e76fdd921167433c971c74bbb7021e8fc9 (HEAD -> ABFS_3.3.2_dev_redo_optimization, origin/ABFS_3.3.2_dev_redo_optimization_filestatus, origin/ABFS_3.3.2_dev_redo_optimization, ABFS_3.3.2_dev_redo_optimization_filestatus)
Author: Pranav Saxena <>
Date:   Wed Jul 26 20:58:52 2023 -0700

    fix tests in ITestAzureBlobFileSystemFileStatus

saxenapranav · 2023-07-27T05:45:26Z

Thanks @snvijaya for the approval.
The approval was on commit 14104c1

The new changes include:

backport HADOOP-17682. ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters apache/hadoop#2975 at commit: dcddc6a
For the resume, fileStatus is already there with listStatus or getFileStatus for renamePendingJson file. On ABFS_3,3.2_dev, for reading the file, we have to open which needs getFileStatus. This is a redundant call and needs optimization.
This new change on top of approved commit is take the optimization..

anmolanmol1234 · 2023-07-27T05:56:00Z

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java

          fileSystemId, FSOperationType.LISTSTATUS, true, tracingHeaderFormat,
          listener);
-      FileStatus[] result = getAbfsStore().listStatus(qualifiedPath, tracingContext);
+      FileStatus[] result = getAbfsStore().listStatus(qualifiedPath,


Used multiple times, can be stored into a variable

….getPath; qualifiedPath has correct path with the workingDir.

saxenapranav added 10 commits July 20, 2023 00:44

changed signature to bring it to redo method

109a0ba

flowing etag in renameAtomicUtils to write to JSON; read from JSON

e585777

etag check before redo

1ad1c02

redo method doesn't need etag; tests wip

3156efd

added tests

5563827

quote or unquote the etag

08606b4

javadocs changes

38b6dfa

try saving listStatus call if nothing changed

77ec128

asssertion if the listStatus API of FS will give correct result when …

bab6499

…we are not making redundant listStatus backend call

added a comment on leaseAcquire catch in redo method

1dd8204

saxenapranav changed the title ~~Abfs 3.3.2 dev redo optimization~~ Rename Resume optimization: resume only if srcDir etag matches to the one with which rename was started. Jul 25, 2023

saxenapranav marked this pull request as ready for review July 25, 2023 03:55

saxenapranav requested review from anmolanmol1234, anujmodi2021, snvijaya and sreeb-msft and removed request for anmolanmol1234, anujmodi2021 and snvijaya July 25, 2023 03:56

snvijaya requested changes Jul 25, 2023

View reviewed changes

saxenapranav added 3 commits July 25, 2023 03:45

renamePendingJSON in listStatus to be searched only if its atomicDire…

3d04da3

…ctory

added tests; to take qualified path uri for checking if its atomic in…

d9a38b8

… listStatus flow

Merge branch 'ABFS_3.3.2_dev' into ABFS_3.3.2_dev_redo_optimization_r…

fbf0a24

…eview

saxenapranav requested a review from snvijaya July 25, 2023 13:25

saxenapranav and others added 2 commits July 25, 2023 06:35

line spacing refactor

14104c1

HADOOP-17682. ABFS: Support FileStatus input to OpenFileWithOptions()…

a660122

… via OpenFileParameters (apache#2975) commit: dcddc6a

saxenapranav added 7 commits July 26, 2023 04:24

listStatus and getFileStatus will open the inputStream and send for r…

08b8c79

…edo;

modified time to check the number of invocation of getFileStatus in t…

aa2d21d

…he cases of rename resume

directly calling store.openWithOptions from listStatus/getFileStatus;…

e69321e

… test refactor

testRefator

9be288b

Merge branch 'ABFS_3.3.2_dev' into ABFS_3.3.2_dev_redo_optimization_f…

f326e52

…ilestatus

for fileStatus of renamePendingJson, we will only call getPathProperty.

dd12da4

invocation of getPathProperty instead of getFileStatus

549285d

snvijaya approved these changes Jul 27, 2023

View reviewed changes

fix tests in ITestAzureBlobFileSystemFileStatus

b76972e

anmolanmol1234 reviewed Jul 27, 2023

View reviewed changes

saxenapranav added 2 commits July 26, 2023 23:52

fileStatus.getpath().toUri().getPathh() extract variable for reuse

1f77da6

filePathStr = qualifiedPath.toUri().getPath() instead of path.toUri()…

680f18c

….getPath; qualifiedPath has correct path with the workingDir.

anmolanmol1234 approved these changes Jul 27, 2023

View reviewed changes

saxenapranav merged commit 6fc5975 into ABFSDriver:ABFS_3.3.2_dev Jul 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rename Resume optimization: resume only if srcDir etag matches to the one with which rename was started.#85

Rename Resume optimization: resume only if srcDir etag matches to the one with which rename was started.#85
saxenapranav merged 25 commits intoABFSDriver:ABFS_3.3.2_devfrom
saxenapranav:ABFS_3.3.2_dev_redo_optimization

saxenapranav commented Jul 24, 2023 •

edited

Loading

Uh oh!

snvijaya Jul 25, 2023

Uh oh!

saxenapranav Jul 25, 2023

Uh oh!

snvijaya Jul 25, 2023

Uh oh!

saxenapranav Jul 25, 2023

Uh oh!

snvijaya Jul 25, 2023

Uh oh!

saxenapranav Jul 25, 2023

Uh oh!

saxenapranav commented Jul 25, 2023

Uh oh!

saxenapranav commented Jul 27, 2023 •

edited

Loading

Uh oh!

saxenapranav commented Jul 27, 2023

Uh oh!

anmolanmol1234 Jul 27, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		@@ -931,12 +931,28 @@ public FileStatus[] listStatus(final Path f) throws IOException {
		== PrefixMode.BLOB) {

		@@ -1083,15 +1099,20 @@ private FileStatus getFileStatus(final Path path,
		&& getAbfsStore().isAtomicRenameKey(fileStatus.getPath().toUri().getPath())

Conversation

saxenapranav commented Jul 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

HNS-OAuth

HNS-SharedKey

NonHNS-SharedKey

NonHNS-OAuth

AppendBlob-HNS-OAuth

Uh oh!

snvijaya Jul 25, 2023

Choose a reason for hiding this comment

Uh oh!

saxenapranav Jul 25, 2023

Choose a reason for hiding this comment

Uh oh!

snvijaya Jul 25, 2023

Choose a reason for hiding this comment

Uh oh!

saxenapranav Jul 25, 2023

Choose a reason for hiding this comment

Uh oh!

snvijaya Jul 25, 2023

Choose a reason for hiding this comment

Uh oh!

saxenapranav Jul 25, 2023

Choose a reason for hiding this comment

Uh oh!

saxenapranav commented Jul 25, 2023

HNS-OAuth

HNS-SharedKey

NonHNS-SharedKey

NonHNS-OAuth

AppendBlob-HNS-OAuth

Uh oh!

saxenapranav commented Jul 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

HNS-OAuth

HNS-SharedKey

NonHNS-SharedKey

NonHNS-OAuth

AppendBlob-HNS-OAuth

Uh oh!

saxenapranav commented Jul 27, 2023

Uh oh!

anmolanmol1234 Jul 27, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

saxenapranav commented Jul 24, 2023 •

edited

Loading

saxenapranav commented Jul 27, 2023 •

edited

Loading