Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HADOOP-18980. S3A credential provider remapping: make extensible #6406

Merged
merged 7 commits into from
Feb 2, 2024
Merged

HADOOP-18980. S3A credential provider remapping: make extensible #6406

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
4964b5a
HADOOP-18980. S3A credential provider remapping: make extensible
virajjasani Jan 4, 2024
5e2fcb5
addendum - checkstyle
virajjasani Jan 8, 2024
576e80b
addendum - addressing comments
virajjasani Jan 26, 2024
c261fcd
addendum - checkstyle
virajjasani Jan 26, 2024
cff702f
move utility usage from config class to s3a util class
virajjasani Jan 31, 2024
f918d23
test + doc changes
virajjasani Feb 1, 2024
7428c13
minor doc changes
virajjasani Feb 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 9 additions & 16 deletions ...op-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -519,36 +519,31 @@ public void testStringCollectionSplitByEquals() {
Map<String, String> splitMap =
StringUtils.getTrimmedStringCollectionSplitByEquals("");
Assertions
.assertThat(splitMap.size())
.assertThat(splitMap)
.describedAs("Map of key value pairs split by equals(=) and comma(,)")
.isEqualTo(0);
.hasSize(0);

splitMap = StringUtils.getTrimmedStringCollectionSplitByEquals(null);
Assertions
.assertThat(splitMap.size())
.assertThat(splitMap)
.describedAs("Map of key value pairs split by equals(=) and comma(,)")
.isEqualTo(0);
.hasSize(0);

splitMap = StringUtils.getTrimmedStringCollectionSplitByEquals(
"element.first.key1 = element.first.val1");
Assertions
.assertThat(splitMap.size())
.describedAs("Map of key value pairs split by equals(=) and comma(,)")
.isEqualTo(1);
Assertions
.assertThat(splitMap)
.describedAs("Map of key value pairs split by equals(=) and comma(,)")
.hasSize(1)
.containsEntry("element.first.key1", "element.first.val1");

splitMap = StringUtils.getTrimmedStringCollectionSplitByEquals(
"element.xyz.key1 =element.abc.val1 , element.xyz.key2= element.abc.val2");
Assertions
.assertThat(splitMap.size())
.describedAs("Map of key value pairs split by equals(=) and comma(,)")
.isEqualTo(2);

Assertions
.assertThat(splitMap)
.describedAs("Map of key value pairs split by equals(=) and comma(,)")
.hasSize(2)
.containsEntry("element.xyz.key1", "element.abc.val1")
.containsEntry("element.xyz.key2", "element.abc.val2");

Expand All @@ -559,13 +554,11 @@ public void testStringCollectionSplitByEquals() {
+ "element.abc.val5 ,\n \n \n "
+ " element.xyz.key6 = element.abc.val6 \n , \n"
+ "element.xyz.key7=element.abc.val7,\n");
Assertions
.assertThat(splitMap.size())
.describedAs("Map of key value pairs split by equals(=) and comma(,)")
.isEqualTo(7);

Assertions
.assertThat(splitMap)
.describedAs("Map of key value pairs split by equals(=) and comma(,)")
.hasSize(7)
.containsEntry("element.xyz.key1", "element.abc.val1")
.containsEntry("element.xyz.key2", "element.abc.val2")
.containsEntry("element.xyz.key3", "element.abc.val3")
Expand Down
19 changes: 19 additions & 0 deletions hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/aws_sdk_upgrade.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,25 @@ The change in interface will mean that custom credential providers will need to
implement `software.amazon.awssdk.auth.credentials.AwsCredentialsProvider` instead of
`com.amazonaws.auth.AWSCredentialsProvider`.

[HADOOP-18980](https://issues.apache.org/jira/browse/HADOOP-18980) introduces extended version of
the credential provider remapping. `fs.s3a.aws.credentials.provider.mapping` can be used to
list comma-separated key-value pairs of mapped credential providers that are separated by
equal operator (=). The key can be used by `fs.s3a.aws.credentials.provider` config, and it
will be translated into the specified value of credential provider class based on the key-value
pair provided by this config.

For example, if `fs.s3a.aws.credentials.provider.mapping` is set with value:

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: prefer triple backslash over indentation as it allows for us to specify the format.

now: should we actually include this as an XML snippet or not? I'm of mixed feelings here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just tried xml snippet to see how it looks and actually it looks better because now the example is much clear. Please let me know what you think of the latest revision.

com.amazonaws.auth.AnonymousAWSCredentials=org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider,
com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper=org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider,
com.amazonaws.auth.InstanceProfileCredentialsProvider=org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider

With the above key-value pairs, if `fs.s3a.aws.credentials.provider` specifies
`com.amazonaws.auth.AnonymousAWSCredentials`, it will be remapped to
`org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider` by S3A while preparing
AWS credential provider list.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "the AWS credential provider list"



### Original V1 `AWSCredentialsProvider` interface

Note how the interface begins with the capitalized "AWS" acronym.
Expand Down
29 changes: 10 additions & 19 deletions ...ools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AAWSCredentialsProvider.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -732,48 +732,42 @@ public void testStringCollectionSplitByEquals() {
S3AUtils.getTrimmedStringCollectionSplitByEquals(
configuration, "custom_key");
Assertions
.assertThat(splitMap.size())
.assertThat(splitMap)
.describedAs(
"Map of key value pairs derived from config, split by equals(=) and comma(,)")
.isEqualTo(0);
.hasSize(0);

splitMap =
S3AUtils.getTrimmedStringCollectionSplitByEquals(
configuration, "not_present");
Assertions
.assertThat(splitMap.size())
.assertThat(splitMap)
.describedAs(
"Map of key value pairs derived from config, split by equals(=) and comma(,)")
.isEqualTo(0);
.hasSize(0);

configuration.set("custom_key", "element.first.key1 = element.first.val1");
splitMap = S3AUtils.getTrimmedStringCollectionSplitByEquals(
configuration, "custom_key");
Assertions
.assertThat(splitMap.size())
.describedAs(
"Map of key value pairs derived from config, split by equals(=) and comma(,)")
.isEqualTo(1);

Assertions
.assertThat(splitMap)
.describedAs(
"Map of key value pairs derived from config, split by equals(=) and comma(,)")
.hasSize(1)
.containsEntry("element.first.key1", "element.first.val1");

configuration.set("custom_key",
"element.xyz.key1 =element.abc.val1 , element.xyz.key2= element.abc.val2");
splitMap =
S3AUtils.getTrimmedStringCollectionSplitByEquals(
configuration, "custom_key");
Assertions
.assertThat(splitMap.size())
.describedAs(
"Map of key value pairs derived from config, split by equals(=) and comma(,)")
.isEqualTo(2);

Assertions
.assertThat(splitMap)
.describedAs(
"Map of key value pairs derived from config, split by equals(=) and comma(,)")
.hasSize(2)
.containsEntry("element.xyz.key1", "element.abc.val1")
.containsEntry("element.xyz.key2", "element.abc.val2");

Expand All @@ -786,15 +780,12 @@ public void testStringCollectionSplitByEquals() {
+ "element.xyz.key7=element.abc.val7,\n");
splitMap = S3AUtils.getTrimmedStringCollectionSplitByEquals(
configuration, "custom_key");
Assertions
.assertThat(splitMap.size())
.describedAs(
"Map of key value pairs derived from config, split by equals(=) and comma(,)")
.isEqualTo(7);

Assertions
.assertThat(splitMap)
.describedAs(
"Map of key value pairs derived from config, split by equals(=) and comma(,)")
.hasSize(7)
.containsEntry("element.xyz.key1", "element.abc.val1")
.containsEntry("element.xyz.key2", "element.abc.val2")
.containsEntry("element.xyz.key3", "element.abc.val3")
Expand Down
Loading