HADOOP-18820. Cut AWS v1 support (#5872) #5973
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This removes the AWS V1 SDK as a hadoop-aws runtime dependency. It is still used at compile time so as to build a wrapper class V1ToV2AwsCredentialProviderAdapter which allows v1 credential provider to be used for authentication. All well known credential providers have their classname remapped from v1 to v2 classes prior to instantiation; this wrapper is not needed for them. There is no support for migrating other SDK plugin points (signing, handlers) Access to the v2 S3Client class used by an S3A FileSystem instance is now via a new interface org.apache.hadoop.fs.s3a.S3AInternals; other low-level operations (getObjectMetadata(Path)) have moved. Contributed by Steve Loughran Change-Id: I231fade61b20a206a018d18f5e867f5d3a1fa879
Contributor
Author
|
tested s3 london; mvit -Dparallel-tests -DtestsThreadCount=8 -Dscale existing multipart issue we should be able fix soon. |
mukund-thakur
approved these changes
Aug 21, 2023
Contributor
mukund-thakur
left a comment
mukund-thakur
left a comment
There was a problem hiding this comment.
LGTM +1, already reviewed here #5872
|
💔 -1 overall
This message was automatically generated. |
Change-Id: I37415cd88a1a46f8537adc4c042855d7e349e8cd
Contributor
Author
|
aah, missed that I'd not pushed up the final changes for the build to work. pushed up and retesting for digiligence. I'm tempted to cherrypick the util.Sets class to branch-3.3, as that was the problem |
|
💔 -1 overall
This message was automatically generated. |
Contributor
Author
|
tests happy; merging this into the feature branch |
steveloughran
merged commit 2253c62
into
apache:feature-HADOOP-18073-s3a-sdk-upgrade-3.3
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This removes the AWS V1 SDK as a hadoop-aws runtime dependency.
It is still used at compile time so as to build a wrapper class V1ToV2AwsCredentialProviderAdapter which allows v1 credential provider to be used for authentication.
All well known credential providers have their classname remapped from v1 to v2 classes prior to instantiation; this wrapper is not needed for them.
There is no support for migrating other SDK plugin points (signing, handlers)
Access to the v2 S3Client class used by an S3A FileSystem instance is now via a new interface org.apache.hadoop.fs.s3a.S3AInternals; other low-level operations (getObjectMetadata(Path)) have moved.
Contributed by Steve Loughran
How was this patch tested?
testing in progress
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?