Test fixes for Config changes, azcopy, where config is given with dfs endpoint; forceBlobconversion is enabled.

[saxenapranav]

Description of PR

Test fixes for Config changes, azcopy, where config is given with dfs endpoint; forceBlobconversion is enabled.
What was breaking?

1. since, config was having forceBlobConversion, the test around where dfs endpoint was required was not able to get honoured because it was getting converted to blob endpoint.
2. Azcopy needs blob endpoint for the blob api. It was getting dfs endpoint (as far as renameExplicit testClass is concerned). So, the constructor of azcopyHelper will convert .dfs. to .blob..

How was this patch tested?

WITH OLD CONFIGS: BLOB endpoint in config and no force:

NonHNS-SharedKey

[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestAbfsClient.verifyUserAgentPrefix:158 [User-Agent string should contain Partner Service]
Expecting:
<"APN/1.0 Azure Blob FS/3.3.2 (PrivateBuild JavaJRE 1.8.0_362; Linux 5.15.0-1014-azure/amd64; UNKNOWN/UNKNOWN) abfsdriverV2.1">
to contain:
<"Partner Service">
[INFO]
[ERROR] Tests run: 113, Failures: 1, Errors: 0, Skipped: 2
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsDurationTrackers.testAbfsHttpCallsDurations:90->assertDurationTracker:108 [The DurationTracker Named action_http_delete_request Doesn't match the expected value.]
Expecting:
<0.0>
to be greater than:
<0.0>
[ERROR] ITestAbfsNetworkStatistics.testAbfsHttpSendStatistics:180->AbstractAbfsIntegrationTest.assertAbfsStatistics:506->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 Mismatch in connections_made expected:<30> but was:<29>
[ERROR] ITestAbfsStatistics.testCreateStatistics:105->AbstractAbfsIntegrationTest.assertAbfsStatistics:506->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 Mismatch in directories_created expected:<1> but was:<2>
[ERROR] ITestAbfsStatistics.testDeleteStatistics:173->AbstractAbfsIntegrationTest.assertAbfsStatistics:506->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 Mismatch in op_delete expected:<2> but was:<1>
[ERROR] ITestAzureBlobFileSystemCheckAccess.testCheckAccessForAccountWithoutNS:181 Expecting org.apache.hadoop.security.AccessControlException with text "This request is not authorized to perform this operation using this permission.", 403 but got : "void"
[ERROR] ITestAzureBlobFileSystemE2E.testFlushWithFileNotFoundException:216 Expected an exception of type class java.io.FileNotFoundException
[ERROR] ITestAzureBlobFileSystemLease.testSubDir:136->Assert.assertTrue:42->Assert.fail:89 Store leases were not freed[ERROR] ITestAzureBlobFileSystemLease.testTwoWritersCreateAppendWithInfiniteLeaseEnabled:196->twoWriters:168 Expected to find 'The condition specified using HTTP conditional header(s) is not met.' but got unexpected exception: org.apache.hadoop.fs.PathIOException: `abfs://abfs-testcontainer-9c2ae9dc-a007-4b71-9fb9-4bb4608e53ea@pranavsaxenanonhns.blob.core.windows.net/fork-2/test/testTwoWritersCreateAppendWithInfiniteLeaseEnabled/testfile': Input/output error: Unable to acquire lease: Operation failed: "There is already a lease present.", 409, POST, https://pranavsaxenanonhns.dfs.core.windows.net/abfs-testcontainer-9c2ae9dc-a007-4b71-9fb9-4bb4608e53ea/fork-2/test/testTwoWritersCreateAppendWithInfiniteLeaseEnabled/testfile?timeout=90, LeaseAlreadyPresent, "There is already a lease present. RequestId:b3079c4b-101f-0069-3c30-831293000000 Time:2023-05-10T11:16:57.3487767Z"
at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.checkException(AzureBlobFileSystem.java:1812)
at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.append(AzureBlobFileSystem.java:534)
at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1456)
at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemLease.twoWriters(ITestAzureBlobFileSystemLease.java:160)
at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemLease.testTwoWritersCreateAppendWithInfiniteLeaseEnabled(ITestAzureBlobFileSystemLease.java:196)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
Caused by: Unable to acquire lease: Operation failed: "There is already a lease present.", 409, POST, https://pranavsaxenanonhns.dfs.core.windows.net/abfs-testcontainer-9c2ae9dc-a007-4b71-9fb9-4bb4608e53ea/fork-2/test/testTwoWritersCreateAppendWithInfiniteLeaseEnabled/testfile?timeout=90, LeaseAlreadyPresent, "There is already a lease present. RequestId:b3079c4b-101f-0069-3c30-831293000000 Time:2023-05-10T11:16:57.3487767Z"Operation failed: "There is already a lease present.", 409, POST, https://pranavsaxenanonhns.dfs.core.windows.net/abfs-testcontainer-9c2ae9dc-a007-4b71-9fb9-4bb4608e53ea/fork-2/test/testTwoWritersCreateAppendWithInfiniteLeaseEnabled/testfile?timeout=90, LeaseAlreadyPresent, "There is already a lease present. RequestId:b3079c4b-101f-0069-3c30-831293000000 Time:2023-05-10T11:16:57.3487767Z"
at org.apache.hadoop.fs.azurebfs.services.AbfsLease.(AbfsLease.java:117)
at org.apache.hadoop.fs.azurebfs.services.AbfsLease.(AbfsLease.java:85)
at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.maybeCreateLease(AzureBlobFileSystemStore.java:2545)
at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.openFileForWrite(AzureBlobFileSystemStore.java:1202)
at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.append(AzureBlobFileSystem.java:531)
... 15 more
Caused by: Operation failed: "There is already a lease present.", 409, POST, https://pranavsaxenanonhns.dfs.core.windows.net/abfs-testcontainer-9c2ae9dc-a007-4b71-9fb9-4bb4608e53ea/fork-2/test/testTwoWritersCreateAppendWithInfiniteLeaseEnabled/testfile?timeout=90, LeaseAlreadyPresent, "There is already a lease present. RequestId:b3079c4b-101f-0069-3c30-831293000000 Time:2023-05-10T11:16:57.3487767Z"
at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.completeExecute(AbfsRestOperation.java:235)
at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.lambda$execute$0(AbfsRestOperation.java:195)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDurationOfInvocation(IOStatisticsBinding.java:464)
at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:193)
at org.apache.hadoop.fs.azurebfs.services.AbfsClient.acquireLease(AbfsClient.java:509)
at org.apache.hadoop.fs.azurebfs.services.AbfsLease.lambda$acquireLease$0(AbfsLease.java:130)
at org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at org.apache.hadoop.thirdparty.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more

[ERROR] ITestAzureBlobFileSystemMkDir.testDefaultCreateOverwriteDirTest:100->testCreateDirOverwrite:128->AbstractAbfsIntegrationTest.assertAbfsStatistics:506->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 Mismatch in connections_made expected:<3> but was:<4>
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:344 » TestTimedOut test timed o...
[ERROR] ITestAzureBlobFileSystemLease.testTwoWritersCreateAppendNoInfiniteLease:187->twoWriters:175 » IO
[INFO]
[ERROR] Tests run: 719, Failures: 9, Errors: 2, Skipped: 276
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsRestOperationException.testAbfsRestOperationExceptionFormat:79->Assert.assertEquals:633->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 expected:<6> but was:<4>
[INFO]
[ERROR] Tests run: 273, Failures: 1, Errors: 0, Skipped: 45

Time taken: 9 mins 42 secs.

---

With new config: dfs endpoint, force blob:

<property>
    <name>fs.azure.abfs.account.name</name>
    <value>pranavsaxenanonhns.dfs.core.windows.net</value>
</property>
<property>
    <name>fs.azure.account.name</name>
    <value>pranavsaxenanonhns.dfs.core.windows.net</value>
</property>

fs.azure.enable.blob.endpoint true

NonHNS-SharedKey

[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestAbfsClient.verifyUserAgentPrefix:158 [User-Agent string should contain Partner Service]
Expecting:
<"APN/1.0 Azure Blob FS/3.3.2 (PrivateBuild JavaJRE 1.8.0_362; Linux 5.15.0-1014-azure/amd64; UNKNOWN/UNKNOWN) abfsdriverV2.1">
to contain:
<"Partner Service">
[INFO]
[ERROR] Tests run: 113, Failures: 1, Errors: 0, Skipped: 2
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsDurationTrackers.testAbfsHttpCallsDurations:90->assertDurationTracker:108 [The DurationTracker Named action_http_delete_request Doesn't match the expected value.]
Expecting:
<0.0>
to be greater than:
<0.0>
[ERROR] ITestAbfsNetworkStatistics.testAbfsHttpSendStatistics:180->AbstractAbfsIntegrationTest.assertAbfsStatistics:506->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 Mismatch in connections_made expected:<30> but was:<29>
[ERROR] ITestAbfsStatistics.testCreateStatistics:105->AbstractAbfsIntegrationTest.assertAbfsStatistics:506->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 Mismatch in directories_created expected:<1> but was:<2>
[ERROR] ITestAbfsStatistics.testDeleteStatistics:173->AbstractAbfsIntegrationTest.assertAbfsStatistics:506->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 Mismatch in op_delete expected:<2> but was:<1>
[ERROR] ITestAzureBlobFileSystemAppend.testParallelWriteOutputStreamClose:500->Assert.assertEquals:633->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 expected:<200> but was:<400>
[ERROR] ITestAzureBlobFileSystemCheckAccess.testCheckAccessForAccountWithoutNS:181 Expecting org.apache.hadoop.security.AccessControlException with text "This request is not authorized to perform this operation using this permission.", 403 but got : "void"
[ERROR] ITestAzureBlobFileSystemCreate.testCPKOverBlob:762 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidConfigurationValueException to be thrown, but got the result: : AzureBlobFileSystem{uri=abfs://abfs-testcontainer-7d2e1ea0-c7db-4a33-94b7-52a82288afde@pranavsaxenanonhns.blob.core.windows.net, user='azureuser', primaryUserGroup='azureuser'[fs.azure.capability.readahead.safe]}
[ERROR] ITestAzureBlobFileSystemCreate.testCPKOverBlobEmptyKey:773 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidConfigurationValueException to be thrown, but got the result: : AzureBlobFileSystem{uri=abfs://abfs-testcontainer-dc928190-d4b9-4b99-8077-ab89c3d8dcbe@pranavsaxenanonhns.blob.core.windows.net, user='azureuser', primaryUserGroup='azureuser'[fs.azure.capability.readahead.safe]}
[ERROR] ITestAzureBlobFileSystemE2E.testFlushWithFileNotFoundException:216 Expected an exception of type class java.io.FileNotFoundException
[ERROR] ITestAzureBlobFileSystemLease.testSubDir:136->Assert.assertTrue:42->Assert.fail:89 Store leases were not freed
[ERROR] ITestAzureBlobFileSystemLease.testTwoWritersCreateAppendWithInfiniteLeaseEnabled:196->twoWriters:168 Expected to find 'The condition specified using HTTP conditional header(s) is not met.' but got unexpected exception: org.apache.hadoop.fs.PathIOException: `abfs://abfs-testcontainer-db163e6d-536e-4c3b-9134-d9b48b7ba1f5@pranavsaxenanonhns.blob.core.windows.net/fork-8/test/testTwoWritersCreateAppendWithInfiniteLeaseEnabled/testfile': Input/output error: Unable to acquire lease: Operation failed: "There is already a lease present.", 409, POST, https://pranavsaxenanonhns.dfs.core.windows.net/abfs-testcontainer-db163e6d-536e-4c3b-9134-d9b48b7ba1f5/fork-8/test/testTwoWritersCreateAppendWithInfiniteLeaseEnabled/testfile?timeout=90, LeaseAlreadyPresent, "There is already a lease present. RequestId:86b6e920-301f-0023-2132-83b11c000000 Time:2023-05-10T11:31:32.7970523Z"
at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.checkException(AzureBlobFileSystem.java:1812)
at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.append(AzureBlobFileSystem.java:534)
at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1456)
at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemLease.twoWriters(ITestAzureBlobFileSystemLease.java:160)
at org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemLease.testTwoWritersCreateAppendWithInfiniteLeaseEnabled(ITestAzureBlobFileSystemLease.java:196)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
Caused by: Unable to acquire lease: Operation failed: "There is already a lease present.", 409, POST, https://pranavsaxenanonhns.dfs.core.windows.net/abfs-testcontainer-db163e6d-536e-4c3b-9134-d9b48b7ba1f5/fork-8/test/testTwoWritersCreateAppendWithInfiniteLeaseEnabled/testfile?timeout=90, LeaseAlreadyPresent, "There is already a lease present. RequestId:86b6e920-301f-0023-2132-83b11c000000 Time:2023-05-10T11:31:32.7970523Z"Operation failed: "There is already a lease present.", 409, POST, https://pranavsaxenanonhns.dfs.core.windows.net/abfs-testcontainer-db163e6d-536e-4c3b-9134-d9b48b7ba1f5/fork-8/test/testTwoWritersCreateAppendWithInfiniteLeaseEnabled/testfile?timeout=90, LeaseAlreadyPresent, "There is already a lease present. RequestId:86b6e920-301f-0023-2132-83b11c000000 Time:2023-05-10T11:31:32.7970523Z"
at org.apache.hadoop.fs.azurebfs.services.AbfsLease.(AbfsLease.java:117)
at org.apache.hadoop.fs.azurebfs.services.AbfsLease.(AbfsLease.java:85)
at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.maybeCreateLease(AzureBlobFileSystemStore.java:2545)
at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.openFileForWrite(AzureBlobFileSystemStore.java:1202)
at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.append(AzureBlobFileSystem.java:531)
... 15 more
Caused by: Operation failed: "There is already a lease present.", 409, POST, https://pranavsaxenanonhns.dfs.core.windows.net/abfs-testcontainer-db163e6d-536e-4c3b-9134-d9b48b7ba1f5/fork-8/test/testTwoWritersCreateAppendWithInfiniteLeaseEnabled/testfile?timeout=90, LeaseAlreadyPresent, "There is already a lease present. RequestId:86b6e920-301f-0023-2132-83b11c000000 Time:2023-05-10T11:31:32.7970523Z"
at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.completeExecute(AbfsRestOperation.java:235)
at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.lambda$execute$0(AbfsRestOperation.java:195)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDurationOfInvocation(IOStatisticsBinding.java:464)
at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:193)
at org.apache.hadoop.fs.azurebfs.services.AbfsClient.acquireLease(AbfsClient.java:509)
at org.apache.hadoop.fs.azurebfs.services.AbfsLease.lambda$acquireLease$0(AbfsLease.java:130)
at org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at org.apache.hadoop.thirdparty.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at org.apache.hadoop.thirdparty.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more

[ERROR] ITestAzureBlobFileSystemMkDir.testDefaultCreateOverwriteDirTest:100->testCreateDirOverwrite:128->AbstractAbfsIntegrationTest.assertAbfsStatistics:506->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 Mismatch in connections_made expected:<3> but was:<4>
[ERROR] ITestSharedKeyAuth.testWithWrongSharedKey:52 Expecting java.io.IOException with text "Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.", 403 but got : "void"
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:344 » TestTimedOut test timed o...
[ERROR] ITestAzureBlobFileSystemLease.testTwoWritersCreateAppendNoInfiniteLease:187->twoWriters:175 » IO
[INFO]
[ERROR] Tests run: 719, Failures: 13, Errors: 2, Skipped: 276
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsRestOperationException.testAbfsRestOperationExceptionFormat:79->Assert.assertEquals:633->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 expected:<6> but was:<4>
[INFO]
[ERROR] Tests run: 273, Failures: 1, Errors: 0, Skipped: 45

Time taken: 9 mins 36 secs.