Skip to content

Use regional STS endpoints for IAM join method#15337

Merged
nklaassen merged 12 commits intomasterfrom
nklaassen/regional-sts
Aug 29, 2022
Merged

Use regional STS endpoints for IAM join method#15337
nklaassen merged 12 commits intomasterfrom
nklaassen/regional-sts

Conversation

@nklaassen
Copy link
Copy Markdown
Contributor

@nklaassen nklaassen commented Aug 9, 2022

Fixes #10415

Currently the IAM join method always uses the global STS endpoint sts.amazonaws.com as the target for sts:GetCallerIdentity requests. These requests are generated and signed on the Node which is joining the cluster, and the Auth server sends them to the actual endpoint to get the response and verify the identity of the Node.

The problem with always using the "global" endpoint, is that it knows nothing about IAM identities in China or GovCloud AWS partitions. Those will need to use the regional endpoint, such as sts.cn-northwest-1.amazonaws.com.cn. There are regional endpoints in the default partition as well, we just haven't been using those.

With this change, the joining Node will attempt to find its local region from its environment or the EC2 IMDS, and generate the signed sts:GetCallerIdentity request for that regional endpoint.

The Auth server must now send the request to the regional endpoint selected by the client. We need to be very careful that the client cannot trick us into using an attacker-controlled endpoint to attempt to verify the client's identity. To do this, we make sure that the given endpoint is a known valid public STS endpoint we will maintain as a static list.

Backward Compatibility

The Auth server will always accept join request from Nodes on older versions which use the global STS endpoint, it is still supported.

Upgraded Nodes connecting to an older Auth server (if we release this in a minor, likely 10.2) will still work: they will fall back to using the global endpoint if there is any error trying to use the regional endpoint, which they would hit if the Auth server is on a version which does not yet support regional endpoints.

@nklaassen nklaassen force-pushed the nklaassen/regional-sts branch from 26e8c22 to 62db6bb Compare August 9, 2022 02:27
@nklaassen
Copy link
Copy Markdown
Contributor Author

friendly ping if you guys can review when you get a chance @gabrielcorado and @zmb3 it looks like the bot chose you for code review as well as docs review

Comment thread lib/auth/join_iam.go Outdated
Comment thread lib/auth/join_iam.go Outdated
Comment thread lib/auth/join_iam.go
if trace.IsNotFound(err) {
// Unfortunately we could not find the region from the IMDS, go with
// the default global endpoint and hope it works.
log.Info("Unable to find the local AWS region from the environment or IMDSv2. " +
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great message. Informative and actionable. Nicely done.

Comment thread lib/auth/join_iam.go Outdated
@github-actions github-actions Bot removed the request for review from r0mant August 29, 2022 16:39
@nklaassen nklaassen enabled auto-merge (squash) August 29, 2022 17:08
@nklaassen nklaassen merged commit cf30661 into master Aug 29, 2022
@github-actions
Copy link
Copy Markdown
Contributor

@nklaassen See the table below for backport results.

Branch Result
branch/v10 Create PR
branch/v8 Failed
branch/v9 Create PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

IAM node joining in China (with auth cluster in aws us-east-2)

4 participants