Skip to content

HADOOP-19044: AWS SDK V2 - Update S3A region logic #6482

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HarshitGupta11 wants to merge 1 commit into from
Closed

HADOOP-19044: AWS SDK V2 - Update S3A region logic #6482

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 17 additions & 7 deletions hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -291,15 +291,25 @@ private <BuilderT extends S3BaseClientBuilder<BuilderT, ClientT>, ClientT> void
if (endpoint != null) {
checkArgument(!fipsEnabled,
"%s : %s", ERROR_ENDPOINT_WITH_FIPS, endpoint);
builder.endpointOverride(endpoint);
// No region was configured, try to determine it from the endpoint.
if (region == null) {
region = getS3RegionFromEndpoint(parameters.getEndpoint());
if (region != null) {
origin = "endpoint";
if(parameters.getEndpoint().equals(CENTRAL_ENDPOINT)){
Copy link
Contributor

@ahmarsuhail ahmarsuhail Jan 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if someone configures fs.s3a.endpoint to s3.amazonaws.com and sets region in fs.s3a.endpoint.region to eu-west-1, this code will start ignoring what we have in fs.s3a.endpoint.region and just enable cross region for everything. I think this could be risky as people could just have s.s3a.endpoint to s3.amazonaws.com in their core-site.xml as it doesn't make a difference if you've set your region right.

// this will cause some issues, override this and ignore the endpoint setting
// this is to make it similar to cross region access.
region = Region.of(AWS_S3_DEFAULT_REGION);
builder.crossRegionAccessEnabled(true);
builder.region(region);
origin = "cross region access fallback because of global endpoint";
}
else {
builder.endpointOverride(endpoint);
// No region was configured, try to determine it from the endpoint.
if (region == null) {
region = getS3RegionFromEndpoint(parameters.getEndpoint());
if (region != null) {
origin = "endpoint";
}
}
LOG.debug("Setting endpoint to {}", endpoint);
}
LOG.debug("Setting endpoint to {}", endpoint);
}

if (region != null) {
Expand Down
4 changes: 2 additions & 2 deletions hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEndpointRegion.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ public void testCentralEndpoint() throws Throwable {
describe("Create a client with the central endpoint");
Configuration conf = getConfiguration();

S3Client client = createS3Client(conf, CENTRAL_ENDPOINT, null, US_EAST_1, false);
S3Client client = createS3Client(conf, CENTRAL_ENDPOINT, null, US_EAST_2, false);

expectInterceptorException(client);
}
Expand Down Expand Up @@ -272,7 +272,7 @@ private final class RegionInterceptor implements ExecutionInterceptor {
public void beforeExecution(Context.BeforeExecution context,
ExecutionAttributes executionAttributes) {

if (endpoint != null) {
if (endpoint != null && !endpoint.equals(CENTRAL_ENDPOINT)) {
Assertions.assertThat(
executionAttributes.getAttribute(AwsExecutionAttribute.ENDPOINT_OVERRIDDEN))
.describedAs("Endpoint not overridden").isTrue();
Expand Down