HDDS-10615. ETag change detected in S3A contract test

[adoroszlai]

What changes were proposed in this pull request?

Fix test failures related to in S3A contract test. The problem is that some parts of the code use "modification time" instead of content hash as ETag.

Tests run: 11, Failures: 0, Errors: 3, Skipped: 1, Time elapsed: 2.998 s <<< FAILURE! - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename
testRenamePopulatesFileAncestors2(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename)  Time elapsed: 1.04 s  <<< ERROR!
org.apache.hadoop.fs.s3a.RemoteFileChangedException: copy `s3a://obs-bucket/test/testRenamePopulatesFileAncestors2/src/dir1/dir2/dir3/fileA': ETag change detected on client during copy. Expected 2024-03-29T18:34:05.086Z got f41ee8006d26bd7fc73828b0a0b350a8

testRenameWithNonEmptySubDir(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename)  Time elapsed: 0.281 s  <<< ERROR!
org.apache.hadoop.fs.s3a.RemoteFileChangedException: copy `s3a://obs-bucket/test/testRenameWithNonEmptySubDir/src1/source.txt': ETag change detected on client during copy. Expected 2024-03-29T18:34:05.352Z got 046d1988b7a01316bb47905ccf3b584a

testRenamePopulatesFileAncestors(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename)  Time elapsed: 0.219 s  <<< ERROR!
org.apache.hadoop.fs.s3a.RemoteFileChangedException: copy `s3a://obs-bucket/test/testRenamePopulatesFileAncestors/source/dir1/dir2/dir3/file4': ETag change detected on client during copy. Expected 2024-03-29T18:34:05.609Z got 67e95334b751fbfc6570e8228183c074


Tests run: 4, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 1.314 s <<< FAILURE! - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractEtag
testLocatedStatusAlsoHasEtag(org.apache.hadoop.fs.contract.s3a.ITestS3AContractEtag)  Time elapsed: 0.117 s  <<< FAILURE!
org.junit.ComparisonFailure: [etag of listLocatedStatus (S3ALocatedFileStatus{path=s3a://obs-bucket/test/testLocatedStatusAlsoHasEtag/src; isDirectory=false; length=11; replication=1; blocksize=33554432; modification_time=1711737272164; access_time=0; owner=...; group=...; permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false}[eTag='2024-03-29T18:34:32.164Z', versionId='']) compared to HEAD value of S3AFileStatus{path=s3a://obs-bucket/test/testLocatedStatusAlsoHasEtag/src; isDirectory=false; length=11; replication=1; blocksize=33554432; modification_time=1711737272000; access_time=0; owner=...; group=...; permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=FALSE eTag=26e078b87fdaa3206ab8bf63a6096c07 versionId=null] expected:<"2[6e078b87fdaa3206ab8bf63a6096c07]"> but was:<"2[024-03-29T18:34:32.164Z]">

https://issues.apache.org/jira/browse/HDDS-10615

How was this patch tested?

ITestS3AContractEtag is enabled as part of the change.

ITestS3AContractRename is still failing with another error for FSO buckets only (HDDS-10665), but the ones related to ETag change are fixed. The test is passing with OBS bucket:

Tests run: 11, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 5.289 s - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename

https://github.com/adoroszlai/ozone/actions/runs/8661440172

[adoroszlai]

@ivandika3 @vtutrinov @whbing plese review

[ivandika3]

Thank you for catching and fixing this. Mostly LGTM.

Left a comment about wrapping double quotes for the response.

[ivandika3]

Shall we standardize so that the ETag returned needs to be surrounded by wrap in quotes? Similar to CopyObjectResponse?

From https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html

However, it seems that the S3A pass even without the quotes. From what I briefly saw from AbstractContractEtagTest, the test does not assert the quotations of the ETag so it will pass regardless of the quotations.

[adoroszlai]

In the AWS page it is interesting that both Key and ETag are defined as string, but only that latter has quotes as part of the value.

[ivandika3]

From what I understand the ETag is specified by RFC to always be surrounded by double quotes.

Ref: aws/aws-sdk-go-v2#1696

@vtutrinov Could you help confirm this?

[vtutrinov]

@ivandika3 sure thing, the RFC (https://datatracker.ietf.org/doc/html/rfc7232#section-2.3) says that the ETag header value is a quoted string

[adoroszlai]

Thanks for the pointers. Updated to wrap in quotes.

[ivandika3]

Thank you for the update. LGTM +1.

[adoroszlai]

Thanks @ivandika3, @vtutrinov for the review.