[HADOOP-17912] Call destroy from input/output streams and refactor EncryptionAdapter by mariosmeim-db · Pull Request #10 · sumangala17/hadoop

mariosmeim-db · 2022-09-01T13:34:06Z

Description of PR

This PR introduces the following changes:

Refactor EncryptionAdapter so that computeKeys is a private method and is called internally from the adapter instead of publicly (e.g., from AbfsClient). This also removes the need for storing the encoded key and sha strings (which are not destroyed anyway).
Adds null checks in EncryptionAdapter.destroy()
Ensures that the EncryptionAdapter object that is created in AbfsClient. addEncryptionKeyRequestHeaders is destroyed if it was created for an existing file, right after retrieving the key and sha
Calls destroy on the encryption adapter in AbfsInputStream.close() and AbfsOutputStream.close()

How was this patch tested?

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

saxenapranav · 2022-09-07T04:29:06Z

Yetus run:

============================================================================
Generating Reports . . .

============================================================================

+1 overall

/ | _ _ ___ ___ ___ ___ | |
_ | | | |/ __/ __/ _ / __/ __| |
) | || | (| (| /_ _ _|
|/ _,|___||/_()

| Vote | Subsystem | Runtime | Comment

+---------------------------------------------------------------------------
| | | | Prechecks
+---------------------------------------------------------------------------
+---------------------------------------------------------------------------
| | | | trunk Compile Tests
+---------------------------------------------------------------------------
+---------------------------------------------------------------------------
| | | | Patch Compile Tests
+---------------------------------------------------------------------------
+---------------------------------------------------------------------------
| | | | Other Tests
+---------------------------------------------------------------------------
| | | 0m 42s |

|| Subsystem || Report/Notes ||

============================================================================

                          Finished build.

============================================================================

Ref: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute#HowToContribute-Generatingapatch

saxenapranav · 2022-09-07T08:45:37Z

HNS-OAuth

[�[1;34mINFO�[m] Results:
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31mFailures: �[m
[�[1;31mERROR�[m] �[1;31m TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"�[m
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31mTests run: 107, Failures: 1, Errors: 0, Skipped: 2�[m
[�[1;34mINFO�[m] Results:
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31mErrors: �[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsCustomEncryption.testCustomEncryptionCombinations:162->callOperation:237 » AbfsRestOperation�[m
[�[1;31mERROR�[m] �[1;31morg.apache.hadoop.fs.azurebfs.ITestAbfsCustomEncryption.testCustomEncryptionCombinationsENCRYPTION_CONTEXT mode, GR�[m
[�[1;31mERROR�[m] �[1;31m Run 1: ITestAbfsCustomEncryption.testCustomEncryptionCombinations:162->callOperation:249 » AbfsRestOperation�[m
[�[1;34mINFO�[m] �[1;32m Run 2: PASS�[m
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31morg.apache.hadoop.fs.azurebfs.ITestAbfsCustomEncryption.testCustomEncryptionCombinationsENCRYPTION_CONTEXT mode, RE�[m
[�[1;31mERROR�[m] �[1;31m Run 1: ITestAbfsCustomEncryption.testCustomEncryptionCombinations:162->callOperation:233 » AbfsRestOperation�[m
[�[1;34mINFO�[m] �[1;32m Run 2: PASS�[m
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31morg.apache.hadoop.fs.azurebfs.ITestAbfsCustomEncryption.testCustomEncryptionCombinationsENCRYPTION_CONTEXT mode, SR�[m
[�[1;31mERROR�[m] �[1;31m Run 1: ITestAbfsCustomEncryption.testCustomEncryptionCombinations:162->callOperation:254 » AbfsRestOperation�[m
[�[1;34mINFO�[m] �[1;32m Run 2: PASS�[m
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31morg.apache.hadoop.fs.azurebfs.ITestAbfsCustomEncryption.testCustomEncryptionCombinationsENCRYPTION_CONTEXT mode, WR�[m
[�[1;31mERROR�[m] �[1;31m Run 1: ITestAbfsCustomEncryption.testCustomEncryptionCombinations:162->callOperation:235 » AbfsRestOperation�[m
[�[1;34mINFO�[m] �[1;32m Run 2: PASS�[m
[�[1;34mINFO�[m]
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31mTests run: 570, Failures: 0, Errors: 5, Skipped: 26�[m
[�[1;34mINFO�[m] Results:
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31mErrors: �[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31mTests run: 332, Failures: 0, Errors: 7, Skipped: 41�[m

HNS-SharedKey

[�[1;34mINFO�[m] Results:
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31mFailures: �[m
[�[1;31mERROR�[m] �[1;31m TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"�[m
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31mTests run: 107, Failures: 1, Errors: 0, Skipped: 2�[m
[�[1;34mINFO�[m] Results:
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31mErrors: �[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsCustomEncryption.testCustomEncryptionCombinations:162->callOperation:237 » AbfsRestOperation�[m
[�[1;31mERROR�[m] �[1;31morg.apache.hadoop.fs.azurebfs.ITestAbfsCustomEncryption.testCustomEncryptionCombinationsENCRYPTION_CONTEXT mode, GR�[m
[�[1;31mERROR�[m] �[1;31m Run 1: ITestAbfsCustomEncryption.testCustomEncryptionCombinations:162->callOperation:249 » AbfsRestOperation�[m
[�[1;34mINFO�[m] �[1;32m Run 2: PASS�[m
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31morg.apache.hadoop.fs.azurebfs.ITestAbfsCustomEncryption.testCustomEncryptionCombinationsENCRYPTION_CONTEXT mode, RE�[m
[�[1;31mERROR�[m] �[1;31m Run 1: ITestAbfsCustomEncryption.testCustomEncryptionCombinations:162->callOperation:233 » AbfsRestOperation�[m
[�[1;34mINFO�[m] �[1;32m Run 2: PASS�[m
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31morg.apache.hadoop.fs.azurebfs.ITestAbfsCustomEncryption.testCustomEncryptionCombinationsENCRYPTION_CONTEXT mode, SR�[m
[�[1;31mERROR�[m] �[1;31m Run 1: ITestAbfsCustomEncryption.testCustomEncryptionCombinations:162->callOperation:254 » AbfsRestOperation�[m
[�[1;34mINFO�[m] �[1;32m Run 2: PASS�[m
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31morg.apache.hadoop.fs.azurebfs.ITestAbfsCustomEncryption.testCustomEncryptionCombinationsENCRYPTION_CONTEXT mode, WR�[m
[�[1;31mERROR�[m] �[1;31m Run 1: ITestAbfsCustomEncryption.testCustomEncryptionCombinations:162->callOperation:235 » AbfsRestOperation�[m
[�[1;34mINFO�[m] �[1;32m Run 2: PASS�[m
[�[1;34mINFO�[m]
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31mTests run: 570, Failures: 0, Errors: 5, Skipped: 26�[m
[�[1;34mINFO�[m] Results:
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31mErrors: �[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31mTests run: 332, Failures: 0, Errors: 7, Skipped: 41�[m

NonHNS-SharedKey

[�[1;34mINFO�[m] Results:
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31mFailures: �[m
[�[1;31mERROR�[m] �[1;31m TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"�[m
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31mTests run: 107, Failures: 1, Errors: 0, Skipped: 2�[m
[�[1;34mINFO�[m] Results:
[�[1;34mINFO�[m]
[�[1;33mWARNING�[m] �[1;33mTests run: 559, Failures: 0, Errors: 0, Skipped: 268�[m
[�[1;34mINFO�[m] Results:
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31mFailures: �[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsRenameStageFailure>TestRenameStageFailure.testResilienceAsExpected:126 [resilient commit support] expected:<[tru]e> but was:<[fals]e>�[m
[�[1;31mERROR�[m] �[1;31mErrors: �[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsJobThroughManifestCommitter.test_0420_validateJob » OutputValidation ...�[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsManifestCommitProtocol.testCommitLifecycle » OutputValidation abfs:/...�[m [�[1;31mERROR�[m] �[1;31m ITestAbfsManifestCommitProtocol.testCommitterWithDuplicatedCommit » OutputValidation�[m [�[1;31mERROR�[m] �[1;31m ITestAbfsManifestCommitProtocol.testConcurrentCommitTaskWithSubDir » OutputValidation�[m [�[1;31mERROR�[m] �[1;31m ITestAbfsManifestCommitProtocol.testMapFileOutputCommitter » OutputValidation ...�[m [�[1;31mERROR�[m] �[1;31m ITestAbfsManifestCommitProtocol.testOutputFormatIntegration » OutputValidation�[m [�[1;31mERROR�[m] �[1;31m ITestAbfsManifestCommitProtocol.testParallelJobsToAdjacentPaths » OutputValidation�[m [�[1;31mERROR�[m] �[1;31m ITestAbfsManifestCommitProtocol.testTwoTaskAttemptsCommit » OutputValidation ...�[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31mTests run: 332, Failures: 1, Errors: 15, Skipped: 44�[m

AppendBlob-HNS-OAuth

[�[1;34mINFO�[m] Results:
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31mFailures: �[m
[�[1;31mERROR�[m] �[1;31m TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"�[m
[�[1;31mERROR�[m] �[1;31m TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:171->fuzzyValidate:49 The actual value 9 is not within the expected range: [5.60, 8.40].�[m
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31mTests run: 107, Failures: 2, Errors: 0, Skipped: 2�[m
[�[1;34mINFO�[m] Results:
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31mErrors: �[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsCustomEncryption.testCustomEncryptionCombinations:162->callOperation:237 » AbfsRestOperation�[m
[�[1;31mERROR�[m] �[1;31morg.apache.hadoop.fs.azurebfs.ITestAbfsCustomEncryption.testCustomEncryptionCombinationsENCRYPTION_CONTEXT mode, GR�[m
[�[1;31mERROR�[m] �[1;31m Run 1: ITestAbfsCustomEncryption.testCustomEncryptionCombinations:162->callOperation:249 » AbfsRestOperation�[m
[�[1;34mINFO�[m] �[1;32m Run 2: PASS�[m
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31morg.apache.hadoop.fs.azurebfs.ITestAbfsCustomEncryption.testCustomEncryptionCombinationsENCRYPTION_CONTEXT mode, RE�[m
[�[1;31mERROR�[m] �[1;31m Run 1: ITestAbfsCustomEncryption.testCustomEncryptionCombinations:162->callOperation:233 » AbfsRestOperation�[m
[�[1;34mINFO�[m] �[1;32m Run 2: PASS�[m
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31morg.apache.hadoop.fs.azurebfs.ITestAbfsCustomEncryption.testCustomEncryptionCombinationsENCRYPTION_CONTEXT mode, SR�[m
[�[1;31mERROR�[m] �[1;31m Run 1: ITestAbfsCustomEncryption.testCustomEncryptionCombinations:162->callOperation:254 » AbfsRestOperation�[m
[�[1;34mINFO�[m] �[1;32m Run 2: PASS�[m
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31morg.apache.hadoop.fs.azurebfs.ITestAbfsCustomEncryption.testCustomEncryptionCombinationsENCRYPTION_CONTEXT mode, WR�[m
[�[1;31mERROR�[m] �[1;31m Run 1: ITestAbfsCustomEncryption.testCustomEncryptionCombinations:162->callOperation:235 » AbfsRestOperation�[m
[�[1;34mINFO�[m] �[1;32m Run 2: PASS�[m
[�[1;34mINFO�[m]
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31mTests run: 570, Failures: 0, Errors: 5, Skipped: 26�[m
[�[1;34mINFO�[m] Results:
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31mErrors: �[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;31mERROR�[m] �[1;31m ITestAbfsTerasort.setup:117->AbstractAbfsClusterITest.setup:83->AbstractAbfsClusterITest.demandCreateClusterBinding:215->AbstractAbfsClusterITest.createCluster:172 » YarnRuntime�[m
[�[1;34mINFO�[m]
[�[1;31mERROR�[m] �[1;31mTests run: 332, Failures: 0, Errors: 7, Skipped: 41�[m

saxenapranav · 2022-09-07T08:53:21Z

Assertion failure:

Operation failed: "The given customer specified encryption does not match the encryption used to encrypt the blob.", 409, PUT, https://pranavsaxenahns.dfs.core.windows.net/abfs-testcontainer-a2b337f0-8601-4612-a6ac-994c62d2e8dc/testFilef2a573d32b47?action=append&position=3&timeout=90, BlobCustomerSpecifiedEncryptionMismatch, "The given customer specified encryption does not match the encryption used to encrypt the blob. RequestId:80a12a4e-c01f-0091-2196-c2683e000000 Time:2022-09-07T08:47:32.2490222Z"

	at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.completeExecute(AbfsRestOperation.java:231)
	at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.lambda$execute$0(AbfsRestOperation.java:191)
	at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.measureDurationOfInvocation(IOStatisticsBinding.java:494)
	at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDurationOfInvocation(IOStatisticsBinding.java:465)
	at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:189)
	at org.apache.hadoop.fs.azurebfs.services.AbfsClient.append(AbfsClient.java:749)
	at org.apache.hadoop.fs.azurebfs.ITestAbfsCustomEncryption.callOperation(ITestAbfsCustomEncryption.java:237)
	at org.apache.hadoop.fs.azurebfs.ITestAbfsCustomEncryption.testCustomEncryptionCombinations(ITestAbfsCustomEncryption.java:162)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:750)

saxenapranav · 2022-09-07T11:22:51Z

Fix for breaking tests in saxenapranav@521d5bc, requesting you to please add in your PR. @mariosmeim-db

saxenapranav · 2022-09-07T06:30:32Z

...tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java

-    buffer = null; // de-reference the buffer so it can be GC'ed sooner
-    ReadBufferManager.getBufferManager().purgeBuffersForStream(this);
+    try {
+      if (encryptionAdapter != null) encryptionAdapter.destroy();


Should we purgeBuffersForStream before destroying encryptionAdapter(populating 0s in contextKey)?

Reason:

BufferWorkers run on parallel threads from the main thread.

It can happen that we destroyed encryptionAdapter, and before purging the list, bufferWorker picks up buffer and starts to execute and it would fail as the keys would be wrong. Now, bufferWorker thread reaches https://github.com/mariosmeim-db/hadoop/blob/HADOOP-17912-databricks/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ReadBufferManager.java#L474, and then the thread-switches back to main thread, which purges inProgressQueue and completedQueue for the given stream related buffers. Then when bufferWorkerThread will get back to CPU, it would add buffer object in completedQueue and hence increasing heap-memory with time.

mariosmeim-db · 2022-09-09T12:27:52Z

@pranavsaxena-microsoft I addressed your comments, let me know if they make sense.

saxenapranav · 2022-09-14T06:32:54Z

:::: AGGREGATED TEST RESULT ::::

HNS-OAuth

[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 107, Failures: 1, Errors: 0, Skipped: 1
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsRestOperationException.testCustomTokenFetchRetryCount:94->testWithDifferentCustomTokenFetchRetry:123->Assert.assertTrue:42->Assert.fail:89 Number of token fetch retries (4) done, does not match with fs.azure.custom.token.fetch.retry.count configured (0)
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:329 » TestTimedOut test timed o...
[ERROR] ITestAzureBlobFileSystemOauth.testBlobDataContributor:84 » AccessDenied Operat...
[ERROR] ITestAzureBlobFileSystemOauth.testBlobDataReader:143 » AccessDenied Operation ...
[INFO]
[ERROR] Tests run: 574, Failures: 1, Errors: 3, Skipped: 70
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestReadBufferManager.testPurgeBufferManagerForParallelStreams:85 [After closing all streams free list contents should match with [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]]
Expected size:<16> but was:<15> in:
<[14, 15, 9, 7, 12, 11, 10, 8, 6, 5, 1, 3, 4, 2, 0]>
[ERROR] Errors:
[ERROR] ITestAbfsTerasort.test_120_terasort:262->executeStage:206 » IO The ownership o...
[INFO]
[ERROR] Tests run: 332, Failures: 1, Errors: 1, Skipped: 54

HNS-SharedKey

[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 107, Failures: 1, Errors: 0, Skipped: 2
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsRestOperationException.testCustomTokenFetchRetryCount:95->testWithDifferentCustomTokenFetchRetry:123->Assert.assertTrue:42->Assert.fail:89 Number of token fetch retries (7) done, does not match with fs.azure.custom.token.fetch.retry.count configured (3)
[ERROR] ITestAzureBlobFileSystemRandomRead.testValidateSeekBounds:269->Assert.assertTrue:42->Assert.fail:89 There should not be any network I/O (elapsedTimeMs=54).
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:329 » TestTimedOut test timed o...
[INFO]
[ERROR] Tests run: 574, Failures: 2, Errors: 1, Skipped: 26
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestReadBufferManager.testPurgeBufferManagerForParallelStreams:85 [After closing all streams free list contents should match with [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]]
Expected size:<16> but was:<10> in:
<[15, 7, 6, 5, 9, 10, 12, 13, 14, 11]>
[INFO]
[ERROR] Tests run: 332, Failures: 1, Errors: 0, Skipped: 41

NonHNS-SharedKey

[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[ERROR] TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:171->fuzzyValidate:49 The actual value 17 is not within the expected range: [5.60, 8.40].
[INFO]
[ERROR] Tests run: 107, Failures: 2, Errors: 0, Skipped: 2
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:344->lambda$testAcquireRetry$6:345 » TestTimedOut
[INFO]
[ERROR] Tests run: 559, Failures: 0, Errors: 1, Skipped: 268
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAbfsRenameStageFailure>TestRenameStageFailure.testResilienceAsExpected:126 [resilient commit support] expected:<[tru]e> but was:<[fals]e>
[ERROR] ITestAbfsTerasort.test_110_teragen:244->executeStage:211->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89 teragen(1000, abfs://testcontainer@pranavsaxenanonhns.dfs.core.windows.net/ITestAbfsTerasort/sortin) failed expected:<0> but was:<1>
[ERROR] ITestReadBufferManager.testPurgeBufferManagerForParallelStreams:85 [After closing all streams free list contents should match with [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]]
Expected size:<16> but was:<15> in:
<[15, 5, 3, 6, 4, 1, 2, 0, 8, 7, 9, 10, 13, 12, 11]>
[ERROR] Errors:
[ERROR] ITestAbfsJobThroughManifestCommitter.test_0420_validateJob » OutputValidation ...
[ERROR] ITestAbfsManifestCommitProtocol.testCommitLifecycle » OutputValidation abfs:/... [ERROR] ITestAbfsManifestCommitProtocol.testCommitterWithDuplicatedCommit » OutputValidation [ERROR] ITestAbfsManifestCommitProtocol.testConcurrentCommitTaskWithSubDir » OutputValidation [ERROR] ITestAbfsManifestCommitProtocol.testMapFileOutputCommitter » OutputValidation ... [ERROR] ITestAbfsManifestCommitProtocol.testOutputFormatIntegration » OutputValidation [ERROR] ITestAbfsManifestCommitProtocol.testParallelJobsToAdjacentPaths » OutputValidation [ERROR] ITestAbfsManifestCommitProtocol.testTwoTaskAttemptsCommit » OutputValidation ...
[INFO]
[ERROR] Tests run: 332, Failures: 3, Errors: 8, Skipped: 46

AppendBlob-HNS-OAuth

[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399 Expected a org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException to be thrown, but got the result: : "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 107, Failures: 1, Errors: 0, Skipped: 1
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestAzureBlobFileSystemRandomRead.testValidateSeekBounds:269->Assert.assertTrue:42->Assert.fail:89 There should not be any network I/O (elapsedTimeMs=249).
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:329 » TestTimedOut test timed o...
[ERROR] ITestAzureBlobFileSystemOauth.testBlobDataContributor:84 » AccessDenied Operat...
[ERROR] ITestAzureBlobFileSystemOauth.testBlobDataReader:143 » AccessDenied Operation ...
[INFO]
[ERROR] Tests run: 574, Failures: 1, Errors: 3, Skipped: 70
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITestReadBufferManager.testPurgeBufferManagerForParallelStreams:81->assertListEmpty:93 [After closing all streams InProgressList should be empty]
Expected size:<0> but was:<6> in:
<[org.apache.hadoop.fs.azurebfs.services.ReadBuffer@59c2b2e,
org.apache.hadoop.fs.azurebfs.services.ReadBuffer@45c7fd7a,
org.apache.hadoop.fs.azurebfs.services.ReadBuffer@c4c212d,
org.apache.hadoop.fs.azurebfs.services.ReadBuffer@5c9b8641,
org.apache.hadoop.fs.azurebfs.services.ReadBuffer@38ba03a7,
org.apache.hadoop.fs.azurebfs.services.ReadBuffer@426c05e7]>
[ERROR] Errors:
[ERROR] ITestAbfsTerasort.test_120_terasort:262->executeStage:206 » IO The ownership o...
[INFO]
[ERROR] Tests run: 332, Failures: 1, Errors: 1, Skipped: 54

mariosmeim-db added 5 commits September 1, 2022 15:29

initial commit

9505ed9

revert unrelated change

f3abbb9

remove unused import

a3d31f1

minor

5842b28

clean up deps

1f1d9d1

mariosmeim-db marked this pull request as draft September 2, 2022 09:09

mariosmeim-db marked this pull request as ready for review September 6, 2022 14:02

saxenapranav reviewed Sep 7, 2022

View reviewed changes

mariosmeim-db added 2 commits September 7, 2022 16:45

fix test

4f86224

purge buffers before destroying adapter

873b3fe

mariosmeim-db requested a review from saxenapranav September 7, 2022 15:15

sumangala-patki merged commit 873b3fe into sumangala17:HADOOP-17912 Sep 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

[HADOOP-17912] Call destroy from input/output streams and refactor EncryptionAdapter #10

[HADOOP-17912] Call destroy from input/output streams and refactor EncryptionAdapter #10
sumangala-patki merged 7 commits intosumangala17:HADOOP-17912from
mariosmeim-db:HADOOP-17912-databricks

mariosmeim-db commented Sep 1, 2022 •

edited

Loading

Uh oh!

saxenapranav commented Sep 7, 2022

Uh oh!

saxenapranav commented Sep 7, 2022

Uh oh!

saxenapranav commented Sep 7, 2022

Uh oh!

saxenapranav commented Sep 7, 2022 •

edited

Loading

Uh oh!

saxenapranav Sep 7, 2022

Uh oh!

mariosmeim-db commented Sep 9, 2022

Uh oh!

saxenapranav commented Sep 14, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

Conversation

mariosmeim-db commented Sep 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

Uh oh!

saxenapranav commented Sep 7, 2022

Yetus run:

============================================================================ Generating Reports . . .

| Vote | Subsystem | Runtime | Comment

|| Subsystem || Report/Notes ||

============================================================================

============================================================================

Uh oh!

saxenapranav commented Sep 7, 2022

HNS-OAuth

HNS-SharedKey

NonHNS-SharedKey

AppendBlob-HNS-OAuth

Uh oh!

saxenapranav commented Sep 7, 2022

Uh oh!

saxenapranav commented Sep 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

saxenapranav Sep 7, 2022

Choose a reason for hiding this comment

Uh oh!

mariosmeim-db commented Sep 9, 2022

Uh oh!

saxenapranav commented Sep 14, 2022

HNS-OAuth

HNS-SharedKey

NonHNS-SharedKey

AppendBlob-HNS-OAuth

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mariosmeim-db commented Sep 1, 2022 •

edited

Loading

============================================================================
Generating Reports . . .

saxenapranav commented Sep 7, 2022 •

edited

Loading