HADOOP-18339. fix storage class option doesn't work with heap and byte buffer #4669
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
🎊 +1 overall
This message was automatically generated. |
Contributor
Author
|
@steveloughran @mukund-thakur would you be able to take a look? We also have fixing test pr #4489 related to storage class option and ready for review |
steveloughran
requested changes
Sep 1, 2022
Contributor
steveloughran
left a comment
steveloughran
left a comment
There was a problem hiding this comment.
LGTM. some comments.
was that setCannedAcl missing from dir markers?
| PutObjectRequest putObjectRequest = new PutObjectRequest(getBucket(), key, | ||
| inputStream, metadata); | ||
| setOptionalPutRequestParameters(putObjectRequest); | ||
| putObjectRequest.setCannedAcl(cannedACL); |
Contributor
There was a problem hiding this comment.
were we missing this before?
Contributor
Author
There was a problem hiding this comment.
no, it was in newPutObjectRequest(..) before, but I extracted it here because calling that function will also set storage class to directory marker
| return Arrays.asList(new Object[][]{ | ||
| {FAST_UPLOAD_BUFFER_DISK}, | ||
| {FAST_UPLOAD_BUFFER_ARRAY}, | ||
| {FAST_UPLOAD_BYTEBUFFER} |
Contributor
There was a problem hiding this comment.
let's cut the bytebuffer one as we know the array option gives us the coverage we need, and it will save test time
Contributor
Author
|
thanks Steve, I have updated the pr. and no the setCannedAcl wasn't missing, I just extracted it from another function. |
|
🎊 +1 overall
This message was automatically generated. |
steveloughran
approved these changes
Sep 1, 2022
Contributor
steveloughran
left a comment
steveloughran
left a comment
There was a problem hiding this comment.
+1
thanks for this; especially as I'm cherrypicking things internally this week.
while those parameterized tests are great for exploring the configuration space, they are a dangerously easy way to expand test time...it's good to be as efficient as we can while still getting that coverage up
asfgit
pushed a commit
that referenced
this pull request
Sep 1, 2022
…writes to disk. (#4669) Follow-up to HADOOP-12020 Support configuration of different S3 storage classes; S3 storage class is now set when buffering to heap/bytebuffers, and when creating directory markers Contributed by Monthon Klongklaew
HarshitGupta11
pushed a commit
to HarshitGupta11/hadoop
that referenced
this pull request
Nov 28, 2022
…writes to disk. (apache#4669) Follow-up to HADOOP-12020 Support configuration of different S3 storage classes; S3 storage class is now set when buffering to heap/bytebuffers, and when creating directory markers Contributed by Monthon Klongklaew
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
HADOOP-18339. S3A storage class option only picked up when buffering writes to disk.
The problem is that there are two
createPutObjectRequestclasses, one for source file and another one for input stream. Previously, only the first one is picking up storage class option because I thought the second one is used only by creating directory marker which should not have any storage class.This is fixed it by making both of
createPutObjectRequestclasses pick up the storage class option and updatingnewDirectoryMarkerRequestto create PUT request on its own then add parameterized test to verify it.How was this patch tested?
Tested with a bucket in
eu-west-1withmvn -Dparallel-tests -DtestsThreadCount=16 clean verifyFor code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?