Skip to content

Allow setting the Glue STS endpoint and region #14412

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
findepi merged 3 commits into from
Oct 4, 2022
Merged

Allow setting the Glue STS endpoint and region #14412

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
817e4a8
Allow setting the Glue STS endpoint and region
Sep 29, 2022
fe46e4f
Empty
alexjo2144 Oct 3, 2022
283fad4
empty
findepi Oct 4, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions docs/src/main/sphinx/connector/hive.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -606,6 +606,13 @@ Property Name Description
``hive.metastore.glue.endpoint-url`` Glue API endpoint URL (optional).
Example: ``https://glue.us-east-1.amazonaws.com``

``hive.metastore.glue.sts.region`` AWS region of the STS service to authenticate with. This is
Copy link
Copy Markdown
Member

@findepi findepi Sep 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need a new setting? why this cannot be based on hive.metastore.glue.region?

Copy link
Copy Markdown
Member Author

@alexjo2144 alexjo2144 Oct 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is necessary if we want to support situations like using Minio's STS with AWS Glue.

There's also this: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html

TLDR: By default STS uses the global endpoint which is always available. You can set a region explicitly to reduce latency but STS is not available in all regions. So if we just use the Glue region but you're deployed in one of those places, for example eu-south-1, you need to use either the global endpoint or a diferent region with STS available.

I believe this means we need the setting.

required when running in a GovCloud region.
Example: ``us-gov-east-1``

``hive.metastore.glue.sts.endpoint`` STS endpoint URL to use when authenticating to Glue (optional).
Example: ``https://sts.us-gov-east-1.amazonaws.com``

``hive.metastore.glue.pin-client-to-current-region`` Pin Glue requests to the same region as the EC2 instance
where Trino is running, defaults to ``false``.

Expand Down
19 changes: 18 additions & 1 deletion ...trino-hive/src/main/java/io/trino/plugin/hive/metastore/glue/GlueCredentialsProvider.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,13 @@
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
import com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider;
import com.amazonaws.client.builder.AwsClientBuilder;
import com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClientBuilder;

import javax.inject.Inject;
import javax.inject.Provider;

import static io.trino.plugin.hive.aws.AwsCurrentRegionHolder.getCurrentRegionFromEC2Metadata;
import static java.lang.String.format;

public class GlueCredentialsProvider
Expand All @@ -45,10 +48,24 @@ public GlueCredentialsProvider(GlueHiveMetastoreConfig config)
provider = DefaultAWSCredentialsProviderChain.getInstance();
}
if (config.getIamRole().isPresent()) {
AWSSecurityTokenServiceClientBuilder stsClientBuilder = AWSSecurityTokenServiceClientBuilder
.standard()
.withCredentials(provider);

if (config.getGlueStsEndpointUrl().isPresent() && config.getGlueStsRegion().isPresent()) {
stsClientBuilder.setEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(config.getGlueStsEndpointUrl().get(), config.getGlueStsRegion().get()));
}
else if (config.getGlueStsRegion().isPresent()) {
stsClientBuilder.setRegion(config.getGlueStsRegion().get());
}
else if (config.getPinGlueClientToCurrentRegion()) {
stsClientBuilder.setRegion(getCurrentRegionFromEC2Metadata().getName());
}

provider = new STSAssumeRoleSessionCredentialsProvider
.Builder(config.getIamRole().get(), "trino-session")
.withExternalId(config.getExternalId().orElse(null))
.withLongLivedCredentialsProvider(provider)
Copy link
Copy Markdown
Member Author

@alexjo2144 alexjo2144 Sep 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note for reviewers: withLongLivedCredentialsProvider is deprecated and cannot be used along with withStsClient, that's why this change is here.

.withStsClient(stsClientBuilder.build())
.build();
}
this.credentialsProvider = provider;
Expand Down
28 changes: 28 additions & 0 deletions ...trino-hive/src/main/java/io/trino/plugin/hive/metastore/glue/GlueHiveMetastoreConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ public class GlueHiveMetastoreConfig
{
private Optional<String> glueRegion = Optional.empty();
private Optional<String> glueEndpointUrl = Optional.empty();
private Optional<String> glueStsRegion = Optional.empty();
private Optional<String> glueStsEndpointUrl = Optional.empty();
private boolean pinGlueClientToCurrentRegion;
private int maxGlueErrorRetries = 10;
private int maxGlueConnections = 30;
Expand Down Expand Up @@ -70,6 +72,32 @@ public GlueHiveMetastoreConfig setGlueEndpointUrl(String glueEndpointUrl)
return this;
}

public Optional<String> getGlueStsRegion()
{
return glueStsRegion;
}

@Config("hive.metastore.glue.sts.region")
@ConfigDescription("AWS STS signing region for Glue authentication")
public GlueHiveMetastoreConfig setGlueStsRegion(String glueStsRegion)
{
this.glueStsRegion = Optional.ofNullable(glueStsRegion);
return this;
}

public Optional<String> getGlueStsEndpointUrl()
{
return glueStsEndpointUrl;
}

@Config("hive.metastore.glue.sts.endpoint")
@ConfigDescription("AWS STS endpoint for Glue authentication")
public GlueHiveMetastoreConfig setGlueStsEndpointUrl(String glueStsEndpointUrl)
{
this.glueStsEndpointUrl = Optional.ofNullable(glueStsEndpointUrl);
return this;
}

public boolean getPinGlueClientToCurrentRegion()
{
return pinGlueClientToCurrentRegion;
Expand Down
6 changes: 6 additions & 0 deletions ...o-hive/src/test/java/io/trino/plugin/hive/metastore/glue/TestGlueHiveMetastoreConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ public void testDefaults()
assertRecordedDefaults(recordDefaults(GlueHiveMetastoreConfig.class)
.setGlueRegion(null)
.setGlueEndpointUrl(null)
.setGlueStsRegion(null)
.setGlueStsEndpointUrl(null)
.setPinGlueClientToCurrentRegion(false)
.setMaxGlueConnections(30)
.setMaxGlueErrorRetries(10)
Expand All @@ -53,6 +55,8 @@ public void testExplicitPropertyMapping()
Map<String, String> properties = ImmutableMap.<String, String>builder()
.put("hive.metastore.glue.region", "us-east-1")
.put("hive.metastore.glue.endpoint-url", "http://foo.bar")
.put("hive.metastore.glue.sts.region", "us-east-3")
.put("hive.metastore.glue.sts.endpoint", "http://sts.foo.bar")
.put("hive.metastore.glue.pin-client-to-current-region", "true")
.put("hive.metastore.glue.max-connections", "10")
.put("hive.metastore.glue.max-error-retries", "20")
Expand All @@ -73,6 +77,8 @@ public void testExplicitPropertyMapping()
GlueHiveMetastoreConfig expected = new GlueHiveMetastoreConfig()
.setGlueRegion("us-east-1")
.setGlueEndpointUrl("http://foo.bar")
.setGlueStsRegion("us-east-3")
.setGlueStsEndpointUrl("http://sts.foo.bar")
.setPinGlueClientToCurrentRegion(true)
.setMaxGlueConnections(10)
.setMaxGlueErrorRetries(20)
Expand Down