Producer, consumer design for having listing and renaming in parallel. by saxenapranav · Pull Request #52 · ABFSDriver/AbfsHadoop

saxenapranav · 2023-05-15T07:36:53Z

Details:

To have a way where in list and rename happens in parallel. List API can give max 5000 results which need the client to call server again with a continuation token. This will need number of sequential call to server and then rename starts. Its a not an optimized use of resources. Will increase latency of rename for dev code, chances of OOM happening(keeping info of all blobs in memory before renaming). In this PR, the rename will start as soon as the client has got some blobs from the server. The producer will call the list API and populate a queue which will be dequeued by the consumer. The producer will pause producing until the consumer-lag comes in control. This design of producer-consumer will be used in the deleteDir API changes as well.
Lease control on files being renamed in atomic directory. The source dir will be leased until the whole rename is completed. The blobs in directory are leased when their copy has to start. The reasoning behing this;

/*
               * Conditionally get a lease on the source blob to prevent other writers
               * from changing it. This is used for correctness in HBase when log files
               * are renamed. It generally should do no harm other than take a little
               * more time for other rename scenarios. When the HBase master renames a
               * log file folder, the lease locks out other writers.  This
               * prevents a region server that the master thinks is dead, but is still
               * alive, from committing additional updates.  This is different than
               * when HBase runs on HDFS, where the region server recovers the lease
               * on a log file, to gain exclusive access to it, before it splits it.
               */

This helps in preventing parallel process in appending a blob which is copy in-progress. Or parallel rename on the same sourceDir.
3. RenamePendingJSON will not have the fileList now. When we need to resume, we will list again(it will return only the ones which are present in the srcDir) and rename.

:::: AGGREGATED TEST RESULT ::::

HNS-OAuth

[INFO] Results:
[INFO]
[WARNING] Tests run: 113, Failures: 0, Errors: 0, Skipped: 2
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:356 » TestTimedOut test timed o...
[ERROR] ITestAzureBlobFileSystemOauth.testBlobDataContributor:80 » AccessDenied Operat...
[ERROR] ITestAzureBlobFileSystemOauth.testBlobDataReader:137 » AccessDenied Operation ...
[INFO]
[ERROR] Tests run: 726, Failures: 0, Errors: 3, Skipped: 224
[INFO] Results:
[INFO]
[WARNING] Tests run: 273, Failures: 0, Errors: 0, Skipped: 53

HNS-SharedKey

[INFO] Results:
[INFO]
[WARNING] Tests run: 113, Failures: 0, Errors: 0, Skipped: 3
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsListStatusRemoteIterator.testWithAbfsIteratorDisabled:168 [After removing every iterm found from the iterator, there should be no more elements in the fileNames] expected:<[0]> but was:<[1]>[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:356 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 726, Failures: 1, Errors: 1, Skipped: 180
[INFO] Results:
[INFO]
[WARNING] Tests run: 273, Failures: 0, Errors: 0, Skipped: 41

NonHNS-SharedKey

[INFO] Results:
[INFO]
[WARNING] Tests run: 113, Failures: 0, Errors: 0, Skipped: 3
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsStatistics.testCreateStatistics:108->AbstractAbfsIntegrationTest.assertAbfsStatistics:505->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 Mismatch in directories_created expected:<2> but was:<1>
[ERROR] ITestAzureBlobFileSystemCheckAccess.testCheckAccessForAccountWithoutNS:191 Expecting org.apache.hadoop.security.AccessControlException with text "This request is not authorized to perform this operation using this permission.", 403 but got : "void"
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:356 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 726, Failures: 2, Errors: 1, Skipped: 278
[INFO] Results:
[INFO]
[WARNING] Tests run: 273, Failures: 0, Errors: 0, Skipped: 45

NonHNS-OAuth

[INFO] Results:
[INFO]
[WARNING] Tests run: 113, Failures: 0, Errors: 0, Skipped: 3
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsStatistics.testCreateStatistics:108->AbstractAbfsIntegrationTest.assertAbfsStatistics:505->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 Mismatch in directories_created expected:<2> but was:<1>
[ERROR] ITestAzureBlobFileSystemCheckAccess.testCheckAccessForAccountWithoutNS:191 Expecting org.apache.hadoop.security.AccessControlException with text "This request is not authorized to perform this operation using this permission.", 403 but got : "void"
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemFlush.testTracingHeaderForAppendBlob:319 » IO AppendBl...
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:356 » TestTimedOut test timed o...
[ERROR] ITestAzureBlobFileSystemOauth.testBlobDataContributor:80 » AbfsRestOperation O...
[ERROR] ITestAzureBlobFileSystemOauth.testBlobDataReader:137 » AccessDenied Operation ...
[INFO]
[ERROR] Tests run: 726, Failures: 2, Errors: 4, Skipped: 284
[INFO] Results:
[INFO]
[WARNING] Tests run: 273, Failures: 0, Errors: 0, Skipped: 57

AppendBlob-HNS-OAuth

[INFO] Results:
[INFO]
[WARNING] Tests run: 113, Failures: 0, Errors: 0, Skipped: 2
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsListStatusRemoteIterator.testAbfsIteratorWithHasNext:88 [After removing every iterm found from the iterator, there should be no more elements in the fileNames] expected:<[0]> but was:<[1]>
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:356 » TestTimedOut test timed o...
[ERROR] ITestAzureBlobFileSystemOauth.testBlobDataContributor:80 » AccessDenied Operat...
[ERROR] ITestAzureBlobFileSystemOauth.testBlobDataReader:137 » AccessDenied Operation ...
[INFO]
[ERROR] Tests run: 726, Failures: 1, Errors: 3, Skipped: 224
[INFO] Results:
[INFO]
[WARNING] Tests run: 273, Failures: 0, Errors: 0, Skipped: 53

Time taken: 46 mins 23 secs.

…d after all child are renamed -> tests needs small change

…olean filed in in the method renameBlobDir

…ileSystem, need to check if current file given is not dir or present. mkdirs to be ignored if coming from createNonRecursive

… of copy) when rename on an atomicDir going on.

…ment_change_renameDir renameDir not to have sourceBlobProperty since it already has sourceD…

…Rename

…change Abfs 3.3.2 dev rename major change

saxenapranav · 2023-06-22T14:36:45Z

:::: AGGREGATED TEST RESULT ::::

HNS-OAuth

[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:181->fuzzyValidate:64 The actual value 13 is not within the expected range: [5.60, 8.40].
[INFO]
[ERROR] Tests run: 113, Failures: 1, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:371 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 759, Failures: 0, Errors: 1, Skipped: 170
[INFO] Results:
[INFO]
[WARNING] Tests run: 278, Failures: 0, Errors: 0, Skipped: 44

HNS-SharedKey

[INFO] Results:
[INFO]
[WARNING] Tests run: 113, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAzureBlobFileSystemRandomRead.testValidateSeekBounds:275->Assert.assertTrue:42->Assert.fail:89 There should not be any network I/O (elapsedTimeMs=106).
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:383 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 759, Failures: 1, Errors: 1, Skipped: 170
[INFO] Results:
[INFO]
[WARNING] Tests run: 278, Failures: 0, Errors: 0, Skipped: 44

NonHNS-SharedKey

[INFO] Results:
[INFO]
[WARNING] Tests run: 113, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsStatistics.testCreateStatistics:108->AbstractAbfsIntegrationTest.assertAbfsStatistics:510->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 Mismatch in directories_created expected:<2> but was:<1>
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemExplictImplicitRename.testRenameImplicitDirectoryToNonExistentDstWithParentIsFile:345->explicitImplicitDirectoryRenameTest:758->createSourcePaths:835->AbstractAbfsIntegrationTest.createAzCopyDirectory:524 » IO
[INFO]
[ERROR] Tests run: 759, Failures: 1, Errors: 1, Skipped: 275
[INFO] Results:
[INFO]
[WARNING] Tests run: 278, Failures: 0, Errors: 0, Skipped: 45

NonHNS-OAuth

[INFO] Results:
[INFO]
[WARNING] Tests run: 113, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsStatistics.testCreateStatistics:108->AbstractAbfsIntegrationTest.assertAbfsStatistics:510->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 Mismatch in directories_created expected:<2> but was:<1>
[INFO]
[ERROR] Tests run: 759, Failures: 1, Errors: 0, Skipped: 275
[INFO] Results:
[INFO]
[WARNING] Tests run: 278, Failures: 0, Errors: 0, Skipped: 45

AppendBlob-HNS-OAuth

[INFO] Results:
[INFO]
[WARNING] Tests run: 113, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemDelegationSASForBlobEndpoint.setup:71->AbstractAbfsIntegrationTest.setup:178->AbstractAbfsIntegrationTest.createFileSystem:309 » TokenAccessProvider
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:371 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 759, Failures: 0, Errors: 2, Skipped: 169
[INFO] Results:
[INFO]
[WARNING] Tests run: 278, Failures: 0, Errors: 0, Skipped: 44

Time taken: 49 mins 0 secs.

saxenapranav · 2023-06-22T14:46:25Z

What has changed in the PR from previous review:

All directory related code has been clubbed together.
In case we dont get any heirarchy, we just start the rename process now. We are not checking if it exists / isDirectory / isFile. For copy/delete, doesn't matter if its marker/non-marker blob. if path doesn't exist, server will return exception.
In directory rename, if any blob rename fails which lead to dir rename failure, we will release lease on the srcDirectory marker blob.

snvijaya

Post the planned review of test plan with team, proceed with checkin.

saxenapranav · 2023-06-23T08:25:19Z

:::: AGGREGATED TEST RESULT ::::

HNS-OAuth

[INFO] Results:
[INFO]
[WARNING] Tests run: 140, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:397->lambda$testAcquireRetry$6:403 » TestTimedOut
[INFO]
[ERROR] Tests run: 760, Failures: 0, Errors: 1, Skipped: 171
[INFO] Results:
[INFO]
[WARNING] Tests run: 278, Failures: 0, Errors: 0, Skipped: 44

HNS-SharedKey

[INFO] Results:
[INFO]
[WARNING] Tests run: 140, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:366 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 760, Failures: 0, Errors: 1, Skipped: 171
[INFO] Results:
[INFO]
[WARNING] Tests run: 278, Failures: 0, Errors: 0, Skipped: 44

NonHNS-SharedKey

[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:181->fuzzyValidate:64 The actual value 14 is not within the expected range: [5.60, 8.40].
[INFO]
[ERROR] Tests run: 140, Failures: 1, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsStatistics.testCreateStatistics:108->AbstractAbfsIntegrationTest.assertAbfsStatistics:510->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 Mismatch in directories_created expected:<2> but was:<1>
[INFO]
[ERROR] Tests run: 760, Failures: 1, Errors: 0, Skipped: 275
[INFO] Results:
[INFO]
[WARNING] Tests run: 278, Failures: 0, Errors: 0, Skipped: 45

NonHNS-OAuth

[INFO] Results:
[INFO]
[WARNING] Tests run: 140, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsStatistics.testCreateStatistics:108->AbstractAbfsIntegrationTest.assertAbfsStatistics:510->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 Mismatch in directories_created expected:<2> but was:<1>
[ERROR] ITestAzureBlobFileSystemRandomRead.testValidateSeekBounds:276->Assert.assertTrue:42->Assert.fail:89 There should not be any network I/O (elapsedTimeMs=85).
[INFO]
[ERROR] Tests run: 760, Failures: 2, Errors: 0, Skipped: 275
[INFO] Results:
[INFO]
[WARNING] Tests run: 278, Failures: 0, Errors: 0, Skipped: 45

AppendBlob-HNS-OAuth

[INFO] Results:
[INFO]
[WARNING] Tests run: 140, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAzureBlobFileSystemRandomRead.testValidateSeekBounds:276->Assert.assertTrue:42->Assert.fail:89 There should not be any network I/O (elapsedTimeMs=97).
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:366 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 760, Failures: 1, Errors: 1, Skipped: 171
[INFO] Results:
[INFO]
[WARNING] Tests run: 278, Failures: 0, Errors: 0, Skipped: 44

saxenapranav added 22 commits May 8, 2023 22:36

API calls for blob lease operations added

f5b8fdd

ListBlobProducer/consumer/queue

0acee48

use of prod and consumer for rename

c02bc3e

WIP, made method for renameBlobDir, need to fix redo method

2e4372d

small changes for knowing if queue is completed.

b63dd44

we need to complete queue when no nextmarker; now src would be rename…

9d1a271

…d after all child are renamed -> tests needs small change

abfsBlobLease

295be5a

calling the APIs of AbfsBlobLease

b66bee8

copyBlob and deleteBlob to have leaseId header

8ab8cf0

test fix; http verb for releaseLease

cea438e

add javadocs; lease of blobs in dir on the basis of isAtomicRename bo…

05505c5

…olean filed in in the method renameBlobDir

lease in createNonRecursive

e7625c8

blobConfigTest:mockito.nullable on leaseId; in create() of azureBlobF…

f222b80

…ileSystem, need to check if current file given is not dir or present. mkdirs to be ignored if coming from createNonRecursive

test for parallel rename, append(when file in atomicDir is in process…

4c7b31e

… of copy) when rename on an atomicDir going on.

testParallelCreateNonRecursiveToFilePartOfAtomicDirectoryInRename

d624e87

added tests in ITestAzureBlobFileSystemCreate: for createNonRecursive

de9d5cb

consumer lag config

0b49c60

Added test for listBlobProducer

9b59f66

Added test for exception catch

ba226ed

refactors

2d38311

Merge branch 'ABFS_3.3.2_dev' into ABFS_3.3.2_dev_rename_improvements

8235c0e

minor changes

2360293

saxenapranav marked this pull request as ready for review May 16, 2023 08:47

saxenapranav requested review from anmolanmol1234, snvijaya and sreeb-msft and removed request for anmolanmol1234 and sreeb-msft May 16, 2023 08:47

saxenapranav and others added 10 commits June 19, 2023 21:01

Merge pull request #12 from saxenapranav/ABFS_3.3.2_dev_rename_impove…

cf08e12

…ment_change_renameDir renameDir not to have sourceBlobProperty since it already has sourceD…

testBlobRenameOfDirectoryHavingNeighborWithSamePrefix

89b0ace

removing throw of runtimeException if renewLEase fail

2ab43e3

lease timer cancel

a5420f4

queue spawned only if requeird

c9f9c29

testBlobRenameCancelRenewTimerForLeaseTakenInAtomicRename

0eb5a94

visibleForTesting on getBlobLease

85f393a

assume nonhns for testBlobRenameCancelRenewTimerForLeaseTakenInAtomic…

0903356

…Rename

testBlobRenameServerReturnsOneBlobPerList

c5ae899

renameBlobExecutorService shutdown if any future task fail

137eca9

saxenapranav requested review from anmolanmol1234 and snvijaya June 21, 2023 06:45

saxenapranav and others added 8 commits June 22, 2023 00:12

when renameDir fails, release srcDir lease.

3cc8c69

refactors

a47a508

SOURCE_PATH_NOT_FOUND in case src is not present

8de26ce

refactors + test adjustments

51e7e5a

no need of blobListProperties in RenameAtomicityUtil; general refactors

14050c1

testParallelBlobLeaseOnChildBlobInRenameSrcDir only on blob

7430b1e

refactor

aafc099

Merge pull request #13 from saxenapranav/ABFS_3.3.2_dev_rename_major_…

21e4e26

…change Abfs 3.3.2 dev rename major change

snvijaya approved these changes Jun 22, 2023

View reviewed changes

saxenapranav added 3 commits June 23, 2023 00:23

testBlobAtomicRenameSrcAndDstAreNotLeftLeased

b8b1ef0

Merge branch 'ABFS_3.3.2_dev' into ABFS_3.3.2_dev_rename_improvements

b5ec70d

remove extra line in import

3e8207e

sreeb-msft approved these changes Jun 23, 2023

View reviewed changes

anmolanmol1234 approved these changes Jun 23, 2023

View reviewed changes

saxenapranav merged commit fbfe195 into ABFSDriver:ABFS_3.3.2_dev Jun 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Producer, consumer design for having listing and renaming in parallel.#52

Producer, consumer design for having listing and renaming in parallel.#52
saxenapranav merged 75 commits intoABFSDriver:ABFS_3.3.2_devfrom
saxenapranav:ABFS_3.3.2_dev_rename_improvements

saxenapranav commented May 15, 2023 •

edited

Loading

Uh oh!

saxenapranav commented Jun 22, 2023

Uh oh!

saxenapranav commented Jun 22, 2023

Uh oh!

snvijaya left a comment

Uh oh!

saxenapranav commented Jun 23, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

saxenapranav commented May 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

HNS-OAuth

HNS-SharedKey

NonHNS-SharedKey

NonHNS-OAuth

AppendBlob-HNS-OAuth

Uh oh!

saxenapranav commented Jun 22, 2023

HNS-OAuth

HNS-SharedKey

NonHNS-SharedKey

NonHNS-OAuth

AppendBlob-HNS-OAuth

Uh oh!

saxenapranav commented Jun 22, 2023

Uh oh!

snvijaya left a comment

Choose a reason for hiding this comment

Uh oh!

saxenapranav commented Jun 23, 2023

HNS-OAuth

HNS-SharedKey

NonHNS-SharedKey

NonHNS-OAuth

AppendBlob-HNS-OAuth

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

saxenapranav commented May 15, 2023 •

edited

Loading