RenameDir and DeleteDir on blob to push number of blobs operated on last API call. by saxenapranav · Pull Request #76 · ABFSDriver/AbfsHadoop

saxenapranav · 2023-06-28T06:32:38Z

PR Details:

After a successful rename, last copy and delete will have the number of blobs operated.
When rename resumed from listStatus, all operation for copy / delete will have same primaryId of the ListStatus and have optpye LS.. Similarly in getFileStatus, it will have GF.

Example of final CID in listStatus:

,cid=:aba4021b-81fe-4442-bb11-98222d6bd2e1:53441220-a477-48e6-9f84-b1c9d9b5dc12:dae069a2-06d8-4697-a78b-902f6f9e726c::LS:0:B:2

Example in getFileStatus:

cid=:3b244e98-aaf9-456d-86c2-480824509694:3b9df2a2-bcde-44d7-b5f2-7d65803f30f4:e76becea-4dd5-4d0c-8362-d7a78a996267::GF:0:B:2

:::: AGGREGATED TEST RESULT ::::

HNS-OAuth

[INFO] Results:
[INFO]
[WARNING] Tests run: 140, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:383 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 772, Failures: 0, Errors: 1, Skipped: 182
[INFO] Results:
[INFO]
[WARNING] Tests run: 278, Failures: 0, Errors: 0, Skipped: 44

HNS-SharedKey

[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:181->fuzzyValidate:64 The actual value 9 is not within the expected range: [5.60, 8.40].
[INFO]
[ERROR] Tests run: 140, Failures: 1, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAzureBlobFileSystemRandomRead.testValidateSeekBounds:276->Assert.assertTrue:42->Assert.fail:89 There should not be any network I/O (elapsedTimeMs=28).
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:383 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 772, Failures: 1, Errors: 1, Skipped: 182
[INFO] Results:
[INFO]
[WARNING] Tests run: 278, Failures: 0, Errors: 0, Skipped: 44

NonHNS-SharedKey

[INFO] Results:
[INFO]
[WARNING] Tests run: 140, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsStatistics.testCreateStatistics:108->AbstractAbfsIntegrationTest.assertAbfsStatistics:510->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 Mismatch in directories_created expected:<2> but was:<1>
[ERROR] ITestAzureBlobFileSystemRandomRead.testValidateSeekBounds:276->Assert.assertTrue:42->Assert.fail:89 There should not be any network I/O (elapsedTimeMs=140).
[INFO]
[ERROR] Tests run: 772, Failures: 2, Errors: 0, Skipped: 276
[INFO] Results:
[INFO]
[WARNING] Tests run: 278, Failures: 0, Errors: 0, Skipped: 45

NonHNS-OAuth

[INFO] Results:
[INFO]
[WARNING] Tests run: 140, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsStatistics.testCreateStatistics:108->AbstractAbfsIntegrationTest.assertAbfsStatistics:510->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 Mismatch in directories_created expected:<2> but was:<1>
[INFO]
[ERROR] Tests run: 772, Failures: 1, Errors: 0, Skipped: 276
[INFO] Results:
[INFO]
[WARNING] Tests run: 278, Failures: 0, Errors: 0, Skipped: 45

AppendBlob-HNS-OAuth

[INFO] Results:
[INFO]
[WARNING] Tests run: 140, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:371 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 772, Failures: 0, Errors: 1, Skipped: 182
[INFO] Results:
[INFO]
[WARNING] Tests run: 278, Failures: 0, Errors: 0, Skipped: 44

azureuser@Hadoop-VM-EAST2:~/hadoop/hadoop-tools/hadoop-azure$ git log
commit 460fd867e1ab93693445f56ba2c331bcc4c9f953 (HEAD -> ABFS_3.3.2_dev_rename_tc, origin/ABFS_3.3.2_dev_rename_tc)
Author: Pranav Saxena <>
Date:   Tue Jun 27 22:26:47 2023 -0700

    concurrency in lease release

…_rename_tc

…marker copy/delete

…_rename_tc

…rimaryIdIsSameInAllTheStepsOfBlobRename

saxenapranav · 2023-06-28T07:28:45Z

Example where in rename failed and resumed from getFileStatus
HttpRequest: CopyBlob: 202,,,cid=:059ce918-c917-4f27-8a62-9f7ca1f6b4d6:3b9df2a2-bcde-44d7-b5f2-7d65803f30f4:4ffe3621-21d7-49b3-b9ab-a9300e9cf2da::RN:0:B,rid=c2e0fea1-101e-0046-1590-a91f580000

HttpRequest: DeleteBlob: 202,,,cid=:eb667826-a694-4562-bd99-a1ff0534b861:3b9df2a2-bcde-44d7-b5f2-7d65803f30f4:4ffe3621-21d7-49b3-b9ab-a9300e9cf2da::RN:0:B,rid=c2e0ffa0-101e-0046-0490-a91f5800000

failure happened, now via getFileStatus

2023-06-28 00:17:12,020 DEBUG [JUnit-testBlobRenameResumeWithGetFileStatus]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(383)) - HttpRequest: GetBlobProperties: 200,,,cid=:9e9fa237-4bb9-4f2c-a1f1-c2c1cfa42f59:3b9df2a2-bcde-44d7-b5f2-7d65803f30f4:e76becea-4dd5-4d0c-8362-d7a78a996267::GF:0:B,rid=c2e110e6-101e-0046-3390-a91f58000000,sent=0,recv=0,HEAD,https://pranavsaxenanonhns.blob.core.windows.net/abfs-testcontainer-8589e309-e8c5-4317-9835-4ce19886f932/hbase/testDir?timeout=90
2023-06-28 00:17:12,022 DEBUG [JUnit-testBlobRenameResumeWithGetFileStatus]: services.AbfsClient (AbfsRestOperation.java:completeExecute(230)) - First execution of REST operation - GetBlobProperties
2023-06-28 00:17:12,023 DEBUG [JUnit-testBlobRenameResumeWithGetFileStatus]: services.AbfsClient (AbfsRestOperation.java:signRequest(290)) - Signing request with shared key
2023-06-28 00:17:12,300 DEBUG [JUnit-testBlobRenameResumeWithGetFileStatus]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(383)) - HttpRequest: GetBlobProperties: 200,,,cid=:c09d9db4-51df-419d-ba41-8e5f98453e57:3b9df2a2-bcde-44d7-b5f2-7d65803f30f4:e76becea-4dd5-4d0c-8362-d7a78a996267::GF:0:B,rid=c2e111b4-101e-0046-6d90-a91f58000000,sent=0,recv=0,HEAD,https://pranavsaxenanonhns.blob.core.windows.net/abfs-testcontainer-8589e309-e8c5-4317-9835-4ce19886f932/hbase/testDir-RenamePending.json?timeout=90

2023-06-28 00:17:12,949 DEBUG [JUnit-testBlobRenameResumeWithGetFileStatus]: services.AbfsClient (AbfsRestOperation.java:completeExecute(230)) - First execution of REST operation - LeaseBlob
2023-06-28 00:17:12,950 DEBUG [Thread-27]: services.AbfsClient (AbfsRestOperation.java:completeExecute(230)) - First execution of REST operation - GetListBlobProperties
2023-06-28 00:17:12,950 DEBUG [JUnit-testBlobRenameResumeWithGetFileStatus]: services.AbfsClient (AbfsRestOperation.java:signRequest(290)) - Signing request with shared key
2023-06-28 00:17:12,950 DEBUG [Thread-27]: services.AbfsClient (AbfsRestOperation.java:signRequest(290)) - Signing request with shared key
2023-06-28 00:17:13,232 DEBUG [JUnit-testBlobRenameResumeWithGetFileStatus]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(383)) - HttpRequest: LeaseBlob: 201,,,cid=:f589d056-d3a8-4c69-9c1e-3b8784e5783e:3b9df2a2-bcde-44d7-b5f2-7d65803f30f4:e76becea-4dd5-4d0c-8362-d7a78a996267::GF:0:B,rid=c2e11480-101e-0046-0e90-a91f58000000,sent=0,recv=0,PUT,https://pranavsaxenanonhns.blob.core.windows.net/abfs-testcontainer-8589e309-e8c5-4317-9835-4ce19886f932/hbase/testDir?comp=lease&timeout=90
2023-06-28 00:17:13,234 DEBUG [Thread-27]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(383)) - HttpRequest: GetListBlobProperties: 200,,,cid=:7761c268-4a50-4274-bbab-4248c8b50b62:3b9df2a2-bcde-44d7-b5f2-7d65803f30f4:e76becea-4dd5-4d0c-8362-d7a78a996267::GF:0:B,rid=a9afc46e-f01e-0003-4b90-a9cabb000000,sent=0,recv=0,GET,https://pranavsaxenanonhns.blob.core.windows.net/abfs-testcontainer-8589e309-e8c5-4317-9835-4ce19886f932?restype=container&comp=list&include=metadata&prefix=hbase/testDir/&maxresults=10000&timeout=90
2023-06-28 00:17:13,236 DEBUG [pool-5-thread-1]: services.AbfsClient (AbfsRestOperation.java:completeExecute(230)) - First execution of REST operation - LeaseBlob
2023-06-28 00:17:13,236 DEBUG [pool-5-thread-1]: services.AbfsClient (AbfsRestOperation.java:signRequest(290)) - Signing request with shared key
2023-06-28 00:17:13,517 DEBUG [pool-5-thread-1]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(383)) - HttpRequest: LeaseBlob: 201,,,cid=:5f244d88-b0c5-4ebb-819e-b603dd0a613c:3b9df2a2-bcde-44d7-b5f2-7d65803f30f4:e76becea-4dd5-4d0c-8362-d7a78a996267::GF:0:B,rid=a9afc561-f01e-0003-2490-a9cabb000000,sent=0,recv=0,PUT,https://pranavsaxenanonhns.blob.core.windows.net/abfs-testcontainer-8589e309-e8c5-4317-9835-4ce19886f932/hbase/testDir/file5?comp=lease&timeout=90
2023-06-28 00:17:13,561 DEBUG [pool-5-thread-1]: services.AbfsClient (AbfsRestOperation.java:completeExecute(230)) - First execution of REST operation - CopyBlob
2023-06-28 00:17:13,562 DEBUG [pool-5-thread-1]: services.AbfsClient (AbfsRestOperation.java:signRequest(290)) - Signing request with shared key
2023-06-28 00:17:13,851 DEBUG [pool-5-thread-1]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(383)) - HttpRequest: CopyBlob: 202,,,cid=:067b1971-d610-4e66-a4ef-40afe01954e4:3b9df2a2-bcde-44d7-b5f2-7d65803f30f4:e76becea-4dd5-4d0c-8362-d7a78a996267::GF:0:B,rid=a9afc680-f01e-0003-3390-a9cabb000000,sent=0,recv=0,PUT,https://pranavsaxenanonhns.blob.core.windows.net/abfs-testcontainer-8589e309-e8c5-4317-9835-4ce19886f932/hbase/testDir2/file5?timeout=90
2023-06-28 00:17:13,852 DEBUG [pool-5-thread-1]: services.AbfsClient (AbfsRestOperation.java:completeExecute(230)) - First execution of REST operation - DeleteBlob
2023-06-28 00:17:13,852 DEBUG [pool-5-thread-1]: services.AbfsClient (AbfsRestOperation.java:signRequest(290)) - Signing request with shared key
2023-06-28 00:17:14,128 DEBUG [pool-5-thread-1]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(383)) - HttpRequest: DeleteBlob: 202,,,cid=:324df76a-5c91-46fb-b32b-31959bc2fb81:3b9df2a2-bcde-44d7-b5f2-7d65803f30f4:e76becea-4dd5-4d0c-8362-d7a78a996267::GF:0:B,rid=a9afc779-f01e-0003-1c90-a9cabb000000,sent=0,recv=0,DELETE,https://pranavsaxenanonhns.blob.core.windows.net/abfs-testcontainer-8589e309-e8c5-4317-9835-4ce19886f932/hbase/testDir/file5?timeout=90
2023-06-28 00:17:14,131 DEBUG [JUnit-testBlobRenameResumeWithGetFileStatus]: services.AbfsClient (AbfsRestOperation.java:completeExecute(230)) - First execution of REST operation - CopyBlob
2023-06-28 00:17:14,132 DEBUG [JUnit-testBlobRenameResumeWithGetFileStatus]: services.AbfsClient (AbfsRestOperation.java:signRequest(290)) - Signing request with shared key
2023-06-28 00:17:14,445 DEBUG [JUnit-testBlobRenameResumeWithGetFileStatus]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(383)) - HttpRequest: CopyBlob: 202,,,cid=:eeb9a393-0908-42d3-a5b4-240456f900e3:3b9df2a2-bcde-44d7-b5f2-7d65803f30f4:e76becea-4dd5-4d0c-8362-d7a78a996267::GF:0:B:2,rid=a9afc8ba-f01e-0003-4590-a9cabb000000,sent=0,recv=0,PUT,https://pranavsaxenanonhns.blob.core.windows.net/abfs-testcontainer-8589e309-e8c5-4317-9835-4ce19886f932/hbase/testDir2?timeout=90
2023-06-28 00:17:14,446 DEBUG [JUnit-testBlobRenameResumeWithGetFileStatus]: services.AbfsClient (AbfsRestOperation.java:completeExecute(230)) - First execution of REST operation - DeleteBlob
2023-06-28 00:17:14,446 DEBUG [JUnit-testBlobRenameResumeWithGetFileStatus]: services.AbfsClient (AbfsRestOperation.java:signRequest(290)) - Signing request with shared key
2023-06-28 00:17:14,725 DEBUG [JUnit-testBlobRenameResumeWithGetFileStatus]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(383)) - HttpRequest: DeleteBlob: 202,,,cid=:3b244e98-aaf9-456d-86c2-480824509694:3b9df2a2-bcde-44d7-b5f2-7d65803f30f4:e76becea-4dd5-4d0c-8362-d7a78a996267::GF:0**:B:2**,rid=a9afc9d9-f01e-0003-4690-a9cabb000000,sent=0,recv=0,DELETE,https://pranavsaxenanonhns.blob.core.windows.net/abfs-testcontainer-8589e309-e8c5-4317-9835-4ce19886f932/hbase/testDir?timeout=90

saxenapranav · 2023-06-28T07:37:29Z

Example where rename failed is resumed from listStatus

HttpRequest: CopyBlob: 202,,,cid=:f668e2e5-4ef4-472f-a0cf-f0f7b9b14c70:53441220-a477-48e6-9f84-b1c9d9b5dc12:ad767c15-1b65-4de0-b2c9-0791fa3b4152::RN:0:B,rid=7bd53d60-701e-006f-1292-a9212c000000,sent=0,recv=0,PUT

HttpRequest: DeleteBlob: 202,,,cid=:2a8facea-96b7-472d-89e0-62bdb93b3308:53441220-a477-48e6-9f84-b1c9d9b5dc12:ad767c15-1b65-4de0-b2c9-0791fa3b4152::RN:0:B,rid=ddf48981-f01e-0061-4e92-a9089c000000,sent=0,recv=0,DELETE

Failure happens, now will be taken from listStatus:

UG [JUnit-testBlobRenameResumeWithListStatus]: services.AbfsClient (AbfsRestOperation.java:signRequest(290)) - Signing request with shared key
2023-06-28 00:32:20,007 DEBUG [JUnit-testBlobRenameResumeWithListStatus]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(383)) - HttpRequest: GetBlobProperties: 200,,,cid=:c62ce0a2-eb9a-42df-8619-b7e16c7f57ee:53441220-a477-48e6-9f84-b1c9d9b5dc12:dae069a2-06d8-4697-a78b-902f6f9e726c::LS:0:B,rid=6269ee56-401e-0006-7092-a91860000000,sent=0,recv=0,HEAD,https://pranavsaxenanonhns.blob.core.windows.net/abfs-testcontainer-1d47d77e-d7d8-4821-9c25-42b65ae55df5/hbase?timeout=90
2023-06-28 00:32:20,009 DEBUG [JUnit-testBlobRenameResumeWithListStatus]: services.AbfsClient (AbfsRestOperation.java:completeExecute(230)) - First execution of REST operation - GetListBlobProperties
2023-06-28 00:32:20,009 DEBUG [JUnit-testBlobRenameResumeWithListStatus]: services.AbfsClient (AbfsRestOperation.java:signRequest(290)) - Signing request with shared key
2023-06-28 00:32:20,292 DEBUG [JUnit-testBlobRenameResumeWithListStatus]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(383)) - HttpRequest: GetListBlobProperties: 200,,,cid=:5ca74812-053d-4138-93ef-7796c9d68b65:53441220-a477-48e6-9f84-b1c9d9b5dc12:dae069a2-06d8-4697-a78b-902f6f9e726c::LS:0:B,rid=6269ef74-401e-0006-7092-a91860000000,sent=0,recv=0,GET,https://pranavsaxenanonhns.blob.core.windows.net/abfs-testcontainer-1d47d77e-d7d8-4821-9c25-42b65ae55df5?restype=container&comp=list&include=metadata&delimiter=/&prefix=hbase/&maxresults=5000&timeout=90
2023-06-28 00:32:20,967 DEBUG [JUnit-testBlobRenameResumeWithListStatus]: services.AbfsClient (AbfsRestOperation.java:completeExecute(230)) - First execution of REST operation - LeaseBlob
2023-06-28 00:32:20,968 DEBUG [JUnit-testBlobRenameResumeWithListStatus]: services.AbfsClient (AbfsRestOperation.java:signRequest(290)) - Signing request with shared key
2023-06-28 00:32:20,969 DEBUG [Thread-27]: services.AbfsClient (AbfsRestOperation.java:completeExecute(230)) - First execution of REST operation - GetListBlobProperties
2023-06-28 00:32:20,969 DEBUG [Thread-27]: services.AbfsClient (AbfsRestOperation.java:signRequest(290)) - Signing request with shared key
2023-06-28 00:32:21,243 DEBUG [Thread-27]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(383)) - HttpRequest: GetListBlobProperties: 200,,,cid=:e9cfc582-fb6c-4bdd-ae68-af0698c2e518:53441220-a477-48e6-9f84-b1c9d9b5dc12:dae069a2-06d8-4697-a78b-902f6f9e726c::LS:0:B,rid=7c7ddbf5-801e-006b-1492-a9ac2b000000,sent=0,recv=0,GET,https://pranavsaxenanonhns.blob.core.windows.net/abfs-testcontainer-1d47d77e-d7d8-4821-9c25-42b65ae55df5?restype=container&comp=list&include=metadata&prefix=hbase/testDir/&maxresults=10000&timeout=90
2023-06-28 00:32:21,251 DEBUG [JUnit-testBlobRenameResumeWithListStatus]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(383)) - HttpRequest: LeaseBlob: 201,,,cid=:40c78cb7-2b5a-41aa-bf09-39e71d1b3b1e:53441220-a477-48e6-9f84-b1c9d9b5dc12:dae069a2-06d8-4697-a78b-902f6f9e726c::LS:0:B,rid=6269f33c-401e-0006-6592-a91860000000,sent=0,recv=0,PUT,https://pranavsaxenanonhns.blob.core.windows.net/abfs-testcontainer-1d47d77e-d7d8-4821-9c25-42b65ae55df5/hbase/testDir?comp=lease&timeout=90
2023-06-28 00:32:21,254 DEBUG [pool-5-thread-1]: services.AbfsClient (AbfsRestOperation.java:completeExecute(230)) - First execution of REST operation - LeaseBlob
2023-06-28 00:32:21,254 DEBUG [pool-5-thread-1]: services.AbfsClient (AbfsRestOperation.java:signRequest(290)) - Signing request with shared key
2023-06-28 00:32:21,539 DEBUG [pool-5-thread-1]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(383)) - HttpRequest: LeaseBlob: 201,,,cid=:954e8796-85f7-4e14-b1c5-9c98eb30867e:53441220-a477-48e6-9f84-b1c9d9b5dc12:dae069a2-06d8-4697-a78b-902f6f9e726c::LS:0:B,rid=6269f430-401e-0006-3d92-a91860000000,sent=0,recv=0,PUT,https://pranavsaxenanonhns.blob.core.windows.net/abfs-testcontainer-1d47d77e-d7d8-4821-9c25-42b65ae55df5/hbase/testDir/file5?comp=lease&timeout=90
2023-06-28 00:32:21,572 DEBUG [pool-5-thread-1]: services.AbfsClient (AbfsRestOperation.java:completeExecute(230)) - First execution of REST operation - CopyBlob
2023-06-28 00:32:21,573 DEBUG [pool-5-thread-1]: services.AbfsClient (AbfsRestOperation.java:signRequest(290)) - Signing request with shared key
2023-06-28 00:32:21,861 DEBUG [pool-5-thread-1]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(383)) - HttpRequest: CopyBlob: 202,,,cid=:9c414229-0668-4ec4-810f-84fca3e54bbd:53441220-a477-48e6-9f84-b1c9d9b5dc12:dae069a2-06d8-4697-a78b-902f6f9e726c::LS:0:B,rid=6269f56d-401e-0006-5d92-a91860000000,sent=0,recv=0,PUT,https://pranavsaxenanonhns.blob.core.windows.net/abfs-testcontainer-1d47d77e-d7d8-4821-9c25-42b65ae55df5/hbase/testDir2/file5?timeout=90
2023-06-28 00:32:21,862 DEBUG [pool-5-thread-1]: services.AbfsClient (AbfsRestOperation.java:completeExecute(230)) - First execution of REST operation - DeleteBlob
2023-06-28 00:32:21,862 DEBUG [pool-5-thread-1]: services.AbfsClient (AbfsRestOperation.java:signRequest(290)) - Signing request with shared key
2023-06-28 00:32:22,144 DEBUG [pool-5-thread-1]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(383)) - HttpRequest: DeleteBlob: 202,,,cid=:b37ae6e8-085c-4f73-9d9a-2caf27d3b701:53441220-a477-48e6-9f84-b1c9d9b5dc12:dae069a2-06d8-4697-a78b-902f6f9e726c::LS:0:B,rid=6269f690-401e-0006-6d92-a91860000000,sent=0,recv=0,DELETE,https://pranavsaxenanonhns.blob.core.windows.net/abfs-testcontainer-1d47d77e-d7d8-4821-9c25-42b65ae55df5/hbase/testDir/file5?timeout=90
2023-06-28 00:32:22,148 DEBUG [JUnit-testBlobRenameResumeWithListStatus]: services.AbfsClient (AbfsRestOperation.java:completeExecute(230)) - First execution of REST operation - CopyBlob
2023-06-28 00:32:22,149 DEBUG [JUnit-testBlobRenameResumeWithListStatus]: services.AbfsClient (AbfsRestOperation.java:signRequest(290)) - Signing request with shared key
2023-06-28 00:32:22,440 DEBUG [JUnit-testBlobRenameResumeWithListStatus]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(383)) - HttpRequest: CopyBlob: 202,,,cid=:aba4021b-81fe-4442-bb11-98222d6bd2e1:53441220-a477-48e6-9f84-b1c9d9b5dc12:dae069a2-06d8-4697-a78b-902f6f9e726c::LS:0:B:2,rid=6269f75f-401e-0006-2592-a91860000000,sent=0,recv=0,PUT,https://pranavsaxenanonhns.blob.core.windows.net/abfs-testcontainer-1d47d77e-d7d8-4821-9c25-42b65ae55df5/hbase/testDir2?timeout=90
2023-06-28 00:32:22,441 DEBUG [JUnit-testBlobRenameResumeWithListStatus]: services.AbfsClient (AbfsRestOperation.java:completeExecute(230)) - First execution of REST operation - DeleteBlob
2023-06-28 00:32:22,441 DEBUG [JUnit-testBlobRenameResumeWithListStatus]: services.AbfsClient (AbfsRestOperation.java:signRequest(290)) - Signing request with shared key
2023-06-28 00:32:22,724 DEBUG [JUnit-testBlobRenameResumeWithListStatus]: services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(383)) - HttpRequest: DeleteBlob: 202,,,cid=:eb93d39a-f8d2-4f1e-b34e-4408ca7e8403:53441220-a477-48e6-9f84-b1c9d9b5dc12:dae069a2-06d8-4697-a78b-902f6f9e726c::LS:0:B:2,rid=6269f82f-401e-0006-6492-a91860000000,sent=0,recv=0,DELETE,https://pranavsaxenanonhns.blob.core.windows.net/abfs-testcontainer-1d47d77e-d7d8-4821-9c25-42b65ae55df5/hbase/testDir?timeout=90

anmolanmol1234 · 2023-06-28T09:34:40Z

    }
    renameBlobExecutorService.shutdown();

+    tracingContext.setOperatedBlobCount(renamedBlob.get() + 1);


Since we are already doing an increment and get, why +1 here ?

Increment is being done only for blobs in renamed in parallel

This +1 is for blob which would contain the count value in CID.

anmolanmol1234 · 2023-06-28T09:35:05Z

      deleteBlobExecutorService.shutdown();
    }
+
+    tracingContext.setOperatedBlobCount(deletedBlobCount.get() + 1);


Same here ?

Increment is being done only for blobs in renamed in parallel

This +1 is for blob which would contain the count value in CID.

anmolanmol1234 · 2023-06-28T09:40:37Z

    Assertions.assertThat(lmt).isEqualTo(newLmt);
  }
+
+  @Test


Can you please add a small description for each test as in what the test is trying to achieve ?

Added as javadoc in all new tests.

…milar tests

* tracingContextValidator impl addition * testBlobRenameResumeWithListStatus * Done for getFileStatus as well * general refactor * javadoc; updateRequestId

saxenapranav · 2023-06-29T11:57:52Z

:::: AGGREGATED TEST RESULT ::::

HNS-OAuth

[INFO] Results:
[INFO]
[WARNING] Tests run: 140, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:383 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 770, Failures: 0, Errors: 1, Skipped: 180
[INFO] Results:
[INFO]
[WARNING] Tests run: 278, Failures: 0, Errors: 0, Skipped: 44

HNS-SharedKey

[INFO] Results:
[INFO]
[WARNING] Tests run: 140, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAzureBlobFileSystemRandomRead.testValidateSeekBounds:276->Assert.assertTrue:42->Assert.fail:89 There should not be any network I/O (elapsedTimeMs=61).
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:397->lambda$testAcquireRetry$6:403 » TestTimedOut
[INFO]
[ERROR] Tests run: 770, Failures: 1, Errors: 1, Skipped: 180
[INFO] Results:
[INFO]
[WARNING] Tests run: 278, Failures: 0, Errors: 0, Skipped: 44

NonHNS-SharedKey

[INFO] Results:
[INFO]
[WARNING] Tests run: 140, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsStatistics.testCreateStatistics:108->AbstractAbfsIntegrationTest.assertAbfsStatistics:510->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 Mismatch in directories_created expected:<2> but was:<1>
[INFO]
[ERROR] Tests run: 770, Failures: 1, Errors: 0, Skipped: 276
[INFO] Results:
[INFO]
[WARNING] Tests run: 278, Failures: 0, Errors: 0, Skipped: 45

NonHNS-OAuth

[INFO] Results:
[INFO]
[WARNING] Tests run: 140, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsStatistics.testCreateStatistics:108->AbstractAbfsIntegrationTest.assertAbfsStatistics:510->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 Mismatch in directories_created expected:<2> but was:<1>
[ERROR] ITestAzureBlobFileSystemRandomRead.testSkipBounds:225->Assert.assertTrue:42->Assert.fail:89 There should not be any network I/O (elapsedTimeMs=51).
[INFO]
[ERROR] Tests run: 770, Failures: 2, Errors: 0, Skipped: 276
[INFO] Results:
[INFO]
[WARNING] Tests run: 278, Failures: 0, Errors: 0, Skipped: 45

AppendBlob-HNS-OAuth

[INFO] Results:
[INFO]
[WARNING] Tests run: 140, Failures: 0, Errors: 0, Skipped: 4
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAzureBlobFileSystemRandomRead.testValidateSeekBounds:276->Assert.assertTrue:42->Assert.fail:89 There should not be any network I/O (elapsedTimeMs=45).
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:397->lambda$testAcquireRetry$6:403 » TestTimedOut
[INFO]
[ERROR] Tests run: 770, Failures: 1, Errors: 1, Skipped: 180
[INFO] Results:
[INFO]
[WARNING] Tests run: 278, Failures: 0, Errors: 0, Skipped: 44

Time taken: 43 mins 39 secs.

azureuser@Hadoop-VM-EAST2:~/hadoop/hadoop-tools/hadoop-azure$ git log
commit a943c31 (HEAD -> ABFS_3.3.2_dev_rename_tc, origin/ABFS_3.3.2_dev_rename_tc)
Author: Pranav Saxena 108325433+saxenapranav@users.noreply.github.com
Date: Thu Jun 29 03:51:05 2023 -0700

Abfs 3.3.2 dev tc review (#16)

* tracingContextValidator impl addition

* testBlobRenameResumeWithListStatus

* Done for getFileStatus as well

* general refactor

* javadoc; updateRequestId

saxenapranav · 2023-06-29T12:08:14Z

What has changed in this revision:

TracingHeaderValidator checking if the header's primaryId and operatedBlobCount is correct.
In TracingHeaderValidator, there is a new field added called disableValidation, which if true, the header will not be validated. Why its required:
1. In RenameAtomicUtil, it calls FileSystem's API of open and delete on the RenamePendingJson file.
2. The fileSystem API would create a new TracingContext with different primaryRequestId for the open and delete to work.
3. Once file is read, all the operation that happen for rename resume uses the tracingHeader which has been used by ListStatus API.
4. Till RenameAtomicUtil class read the RenamePendingJson, validations will be disabled. And will be enabled once the read is complete and the process is ready to start rename-resume.

snvijaya

+1

saxenapranav added 23 commits June 16, 2023 03:44

testIfTracingContextPrimaryIdIsSameInAllTheStepsOfBlobRename

221b21f

nit refactor

1ce1479

Merge branch 'ABFS_3.3.2_dev_rename_improvements' into ABFS_3.3.2_dev…

f581bec

…_rename_tc

tracingContext to have new field for total renamed blobs in src-blob …

e16000f

…marker copy/delete

Merge branch 'ABFS_3.3.2_dev_rename_improvements' into ABFS_3.3.2_dev…

816c34b

…_rename_tc

Merge branch 'ABFS_3.3.2_dev' into ABFS_3.3.2_dev_rename_tc

c4533fe

Merge branch 'ABFS_3.3.2_dev_delete' into ABFS_3.3.2_dev_rename_tc

742e357

Merge branch 'ABFS_3.3.2_dev_rename_improvements' into ABFS_3.3.2_dev…

997127c

…_rename_tc

added for deletion

740e312

Merge branch 'ABFS_3.3.2_dev_delete' into ABFS_3.3.2_dev_rename_tc

7df7df9

test wip

673b0f9

Merge branch 'ABFS_3.3.2_dev_delete' into ABFS_3.3.2_dev_rename_tc

a8783f2

testDeleteEmitDeletionCountInClientRequestId

70be95e

testRenameSrcDirDeleteEmitDeletionCountInClientRequestId

641a84b

Merge branch 'ABFS_3.3.2_dev_delete' into ABFS_3.3.2_dev_rename_tc

3439b1b

refactor argument number and added assertion in testIfTracingContextP…

609d65d

…rimaryIdIsSameInAllTheStepsOfBlobRename

testIfTracingContextPrimaryIdIsSameInAllTheStepsOfBlobDelete

4f3e47c

testBlobRenameResumeWithListStatus

643c10f

testBlobRenameResumeWithGetFileStatus

2e9f1ab

require primary key in getFileStatus

1607a11

release all lease after rename failure

1d4b6ba

concurrency in lease release

460fd86

remove non-required imports

c35c714

saxenapranav marked this pull request as ready for review June 28, 2023 06:39

saxenapranav requested review from anmolanmol1234, anujmodi2021, snvijaya and sreeb-msft June 28, 2023 06:39

anmolanmol1234 reviewed Jun 28, 2023

View reviewed changes

added javadocs on the added tests.

8bde5da

anmolanmol1234 approved these changes Jun 28, 2023

View reviewed changes

saxenapranav and others added 2 commits June 28, 2023 22:08

Refactor to bring out common functionality into methods; aggregate si…

024b451

…milar tests

Abfs 3.3.2 dev tc review (#16)

a943c31

* tracingContextValidator impl addition * testBlobRenameResumeWithListStatus * Done for getFileStatus as well * general refactor * javadoc; updateRequestId

snvijaya approved these changes Jun 29, 2023

View reviewed changes

saxenapranav merged commit b29e301 into ABFSDriver:ABFS_3.3.2_dev Jun 29, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RenameDir and DeleteDir on blob to push number of blobs operated on last API call.#76

RenameDir and DeleteDir on blob to push number of blobs operated on last API call.#76
saxenapranav merged 26 commits intoABFSDriver:ABFS_3.3.2_devfrom
saxenapranav:ABFS_3.3.2_dev_rename_tc

saxenapranav commented Jun 28, 2023 •

edited

Loading

Uh oh!

saxenapranav commented Jun 28, 2023 •

edited

Loading

Uh oh!

saxenapranav commented Jun 28, 2023 •

edited

Loading

Uh oh!

anmolanmol1234 Jun 28, 2023

Uh oh!

saxenapranav Jun 28, 2023 •

edited

Loading

Uh oh!

anmolanmol1234 Jun 28, 2023

Uh oh!

saxenapranav Jun 28, 2023

Uh oh!

anmolanmol1234 Jun 28, 2023

Uh oh!

saxenapranav Jun 28, 2023

Uh oh!

saxenapranav commented Jun 29, 2023

Uh oh!

saxenapranav commented Jun 29, 2023

Uh oh!

snvijaya left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

saxenapranav commented Jun 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

HNS-OAuth

HNS-SharedKey

NonHNS-SharedKey

NonHNS-OAuth

AppendBlob-HNS-OAuth

Uh oh!

saxenapranav commented Jun 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

saxenapranav commented Jun 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anmolanmol1234 Jun 28, 2023

Choose a reason for hiding this comment

Uh oh!

saxenapranav Jun 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anmolanmol1234 Jun 28, 2023

Choose a reason for hiding this comment

Uh oh!

saxenapranav Jun 28, 2023

Choose a reason for hiding this comment

Uh oh!

anmolanmol1234 Jun 28, 2023

Choose a reason for hiding this comment

Uh oh!

saxenapranav Jun 28, 2023

Choose a reason for hiding this comment

Uh oh!

saxenapranav commented Jun 29, 2023

HNS-OAuth

HNS-SharedKey

NonHNS-SharedKey

NonHNS-OAuth

AppendBlob-HNS-OAuth

Uh oh!

saxenapranav commented Jun 29, 2023

Uh oh!

snvijaya left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

saxenapranav commented Jun 28, 2023 •

edited

Loading

saxenapranav commented Jun 28, 2023 •

edited

Loading

saxenapranav commented Jun 28, 2023 •

edited

Loading

saxenapranav Jun 28, 2023 •

edited

Loading