Backmerge of Apache/hadoop:trunk into sumangala-patki/hadoop-Hadoop 17912: Conflict resolution

[saxenapranav]

Description of PR

PR raised for HADOOP-17912(https://issues.apache.org/jira/browse/HADOOP-17912) apache#3440 has conflict with Apache/hadoop:trunk and hence can't be merged.

This PR backmerges the trunk commits into sumangala-patki/hadoop:HADOOP-17912 branch and resolves the conflict.

How was this patch tested?

This was tested by running integrated tests with the Azure-storage accounts (US EAST).
Non-HNS Endpoint: pranavsaxenanonhns.blob.core.windows.net
HNS Endpoint: pranavsaxenahns.dfs.core.windows.net

Following is the test-result:

:::: AGGREGATED TEST RESULT ::::

HNS-OAuth
========================
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 107, Failures: 1, Errors: 0, Skipped: 2
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   ITestAbfsStatistics.testOpenAppendRenameExists:244->Assert.assertTrue:53->Assert.assertTrue:42->Assert.fail:87: THIS SUCCEEDED WHEN RAN INDIVIDUALLY
[INFO]
[ERROR] Tests run: 574, Failures: 1, Errors: 0, Skipped: 26
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek:69->testReadWriteAndSeek:110 [Retry was required due to issue on server side] expected:<[0]> but was:<[1]>: THIS SUCCEEDED WHEN RAN INDIVIDUALLY
[INFO]
[ERROR] Tests run: 332, Failures: 1, Errors: 0, Skipped: 41

HNS-SharedKey
========================
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 107, Failures: 1, Errors: 0, Skipped: 2
[INFO] Results:
[INFO]
[WARNING] Tests run: 574, Failures: 0, Errors: 0, Skipped: 26
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek:69->testReadWriteAndSeek:110 [Retry was required due to issue on server side] expected:<[0]> but was:<[1]>: THIS SUCCEEDED WHEN RAN INDIVIDUALLY
[INFO]
[ERROR] Tests run: 332, Failures: 1, Errors: 0, Skipped: 41

NonHNS-SharedKey
========================
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 107, Failures: 1, Errors: 0, Skipped: 2
[INFO] Results:
[INFO]
[WARNING] Tests run: 559, Failures: 0, Errors: 0, Skipped: 268
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek:69->testReadWriteAndSeek:110 [Retry was required due to issue on server side] expected:<[0]> but was:<[1]>: THIS SUCCEEDED WHEN RAN INDIVIDUALLY
[ERROR]   ITestAbfsRenameStageFailure>TestRenameStageFailure.testResilienceAsExpected:126 [resilient commit support] expected:<[tru]e> but was:<[fals]e>
[ERROR]   ITestAbfsTerasort.test_110_teragen:244->executeStage:211->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 teragen(1000, abfs://testcontainer@pranavsaxenanonhns.dfs.core.windows.net/ITestAbfsTerasort/sortin) failed expected:<0> but was:<1>
[ERROR] Errors:
[ERROR]   ITestAbfsJobThroughManifestCommitter.test_0420_validateJob » OutputValidation ...
[ERROR]   ITestAbfsManifestCommitProtocol.testCommitLifecycle » OutputValidation `abfs:/...
[ERROR]   ITestAbfsManifestCommitProtocol.testCommitterWithDuplicatedCommit » OutputValidation
[ERROR]   ITestAbfsManifestCommitProtocol.testConcurrentCommitTaskWithSubDir » OutputValidation
[ERROR]   ITestAbfsManifestCommitProtocol.testMapFileOutputCommitter » OutputValidation ...
[ERROR]   ITestAbfsManifestCommitProtocol.testOutputFormatIntegration » OutputValidation
[ERROR]   ITestAbfsManifestCommitProtocol.testParallelJobsToAdjacentPaths » OutputValidation
[ERROR]   ITestAbfsManifestCommitProtocol.testTwoTaskAttemptsCommit » OutputValidation `...
[INFO]
[ERROR] Tests run: 332, Failures: 3, Errors: 8, Skipped: 46

AppendBlob-HNS-OAuth
========================
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 107, Failures: 1, Errors: 0, Skipped: 2
[INFO] Results:
[INFO]
[WARNING] Tests run: 574, Failures: 0, Errors: 0, Skipped: 26
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek:69->testReadWriteAndSeek:110 [Retry was required due to issue on server side] expected:<[0]> but was:<[1]>: THIS SUCCEEDED WHEN RAN INDIVIDUALLY
[INFO]
[ERROR] Tests run: 332, Failures: 1, Errors: 0, Skipped: 41

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?