Fix AWS Roles Anywhere access when using per-session MFA

[marcoandredinis]

AWS IAM Roles Anywhere integration allows users to access AWS resources by calling the rolesanywhere.CreateSession API.

This API receives a certificate (signed by the awsra Teleport CA) and returns the AWS Session temporary credentials.

We don't want the AWS Session to outlive the Teleport identity expiration date, so we set the durationSeconds parameter accordingly.

The Teleport identity expiration date is calculated based on multiple factors:

• hard coded max value
• cluster level configuration
• role configuration
• current teleport user's identity certificate

However, when using Per-Session MFA, this expiration is only 1 minute, which is rejected by rolesanywhere.CreateSession (durationSeconds must be greater than 15 minutes).

It would also be a bad UX if the AWS Session expired 1 minute right after being created.

This PR changes the app login flow so that when using per-session MFA, the TTL for the AWS Session is the same as the one defined when not using single use certs.

In order to keep the implicit rule of not writing credentials into disk when using Per-Session MFA, we added a new flag into the tsh apps login: --env.
This flag ensures that AWS credentials are not stored to disk and are only sent to output.
The format used is compatible with eval. This allow users to easily gain access to AWS using the combination of both: eval "$(tsh apps login .... --env)"

Before

After (with commit 6f2e6473dbd994366554afb5da52e7e224cc484c)

marcodinis@MarcoM3 ~/s/p/awsclira (master) [1]> tsh login --proxy=proxy.example.com:443 --user marco
Enter password for Teleport user marco:
Tap any security key
Detected security key tap
> Profile URL:        https://proxy.example.com:443
  Logged in as:       marco
  Cluster:            proxy.example.com
  Roles:              access, auditor, editor
  Logins:             marcodinis, root
  Kubernetes:         enabled
  Valid until:        2025-11-07 03:00:11 +0000 WET [valid for 11h59m]
  Extensions:         login-ip, permit-agent-forwarding, permit-port-forwarding, permit-pty, private-key-policy

marcodinis@MarcoM3 ~/s/p/awsclira (master)> tsh apps login marco-testmfa-teleportdev-ra --aws-role arn:aws:iam::123456789012:role/MarcoExampleReadS3ByIAM_RA
ERROR: AWS access is configured to use per-session MFA and credentials are only available to a single session. Pass the --env flag to the previous command and export the credentials using eval.
Example:
        eval "$(tsh apps login marco-testmfa-teleportdev-ra --aws-role arn:aws:iam::123456789012:role/MarcoExampleReadS3ByIAM_RA --env)"

You can now run the AWS CLI or other AWS SDK based tools as usual.
Example:
        aws sts get-caller-identity

marcodinis@MarcoM3 ~/s/p/awsclira (master) [1]> eval "$(tsh apps login marco-testmfa-teleportdev-ra --aws-role arn:aws:iam::123456789012:role/MarcoExampleReadS3ByIAM_RA --env)"
MFA is required to access Application "marco-testmfa-teleportdev-ra"
Tap any security key
Detected security key tap
marcodinis@MarcoM3 ~/s/p/awsclira (master)> aws sts get-caller-identity
{
    "UserId": "123:456",
    "Account": "123456789012",
    "Arn": "arn:aws:sts::123456789012:assumed-role/MarcoExampleReadS3ByIAM_RA/456"
}
marcodinis@MarcoM3 ~/s/p/awsclira (master)> tree ~/.tsh/keys/proxy.example.com/
/Users/marcodinis/.tsh/keys/proxy.example.com/
├── cas
│   └── proxy.example.com.pem
├── certs.pem
├── marco
├── marco-ssh
│   └── proxy.example.com-cert.pub
├── marco.crt
├── marco.key
└── marco.pub

3 directories, 7 files

Changelog: Fixes AWS Roles Anywhere cli access when using per-session MFA

Fixes: #59597

[Tener]

I'm not sure this approach is secure. This comments suggest we should ensure the issued certs are kept in-memory, which isn't the scenario displayed by this PR.

func setUserSingleUseCertsTTL(actx *grpcContext, req *authpb.UserCertsRequest) {
	if isLocalProxyCertReq(req) {
		// don't limit the cert expiry to 1 minute for db local proxy tunnel or kube local proxy,
		// because the certs will be kept in-memory by the client to protect
		// against cert/key exfiltration. When MFA is required, cert expiration
		// time is bounded by the lifetime of the local proxy process or the mfa verification interval.
		return
	}

@Joerger would you mind taking a look?

[marcoandredinis]

I'm not sure this approach is secure. This comments suggest we should ensure the issued certs are kept in-memory, which isn't the scenario displayed by this PR.

I could require users to use the tsh proxy app --tunnel ... option to keep things in memory
Then the tsh command executed by ~/.aws/config#profile/credential_process would do a local tcp connection to the local tunnel and fetch the credentials.

That makes me wonder whether the credentials are safer: in-memory but accessible from a TCP connection or in a file, in the user's home directory.

[Joerger]

I'm not sure this approach is secure. This comments suggest we should ensure the issued certs are kept in-memory, which isn't the scenario displayed by this PR.

Agreed, we should not leave long lived MFA-verified certs, which in this case hold plain text AWS Roles Anywhere credentials, in the filesystem. Usually we get around that with tsh proxy ... commands which use the credentials locally to create the tunnel and allow the user to connect to that, or with wrappers like tsh aws which internally create the local proxy with fresh MFA certs. The credentials never leave memory in either case.

However, as Marco pointed out, I don't see exactly how we can make this work with tsh app login or tsh proxy app, as the credential_process command still needs to return the plain text AWS Roles Anywhere credentials. Making the credentials retrievable via TCP instead of file is not any better.

Looking at the docs, I see a warning that basically points out this exact vulnerability concern:

Warning
This topic discusses sourcing credentials from an external process. This could be a security risk if the command to generate the credentials becomes accessible by non-approved processes or users. We recommend that you use the supported, secure alternatives provided by the AWS CLI and AWS to reduce the risk of compromising your credentials. Ensure that you secure the config file and any supporting files and tools to prevent disclosure.

Ensure that your custom credential tool does not write any secret information to StdErr because the SDKs and AWS CLI can capture and log such information, potentially exposing it to unauthorized users.

I think we would need to wrap this into tsh aws so that the login + credential retrieval can be handled within the process in-memory. This would defeat the purpose of the credential_process profile approach though, which (with limited context) seems to be the main driving point of this approach?

I don't see how this feature can support long-lived MFA certs without compromising on its current security invariants, but I still think it's worth investigating deeper. Otherwise, we could take the current approach in this PR and be clear in documentation through warnings that per-session MFA has diminished security effectiveness for AWS Roles Anywhere. Additionally, we should probably hard code the cert expiration to 15m rather than just leaving it as the user's session TTL like we do for other local proxy certs.

Edit: some more ideas:

• Add some kind of tsh app config --headless command as the credential_process. This command would login with a headless flow so that the user could handle the MFA in another terminal / Teleport Connect. Once complete, tsh app config --headless would print the resulting AWS creds directly to stdout, never touching disk. We already have the client infrastructure for this with Headless Polling.
• Use tsh aws wrapper to provide the plain creds (aws_access_key_id, aws_secret_access_key, aws_session_token) to the aws command rather than trying to use credentials_command. This way, tsh can handle the login, MFA check, and provide the raw creds directly rather than through disk.

[rosstimothy]

@rob-picard-teleport what are your thoughts on this?

[rob-picard-teleport]

It took me some time to get up to speed on the flows here, so here's my attempt at documenting my understanding.

First, this flowchart is what happens when the user runs tsh apps login --aws-role <role-arn> <app-name>

flowchart

onAppLogin -->|new: requester name is credential process if aws-arn provided| appLogin --> ClusterClient.IssueUserCertsWithMFA


ClusterClient.IssueUserCertsWithMFA -->|if mfa not required then ReusableMFAResponse is nil| ClusterClient.generateUserCerts -->|if ReusableMFAResponse not nil, MFA required, purpose is single use| GenerateUserCerts

GenerateUserCerts -->|purpose is single use| generateUserSingleUseCerts

generateUserSingleUseCerts -->|ttl is capped at teleport.UserSingleUseCertTTL| userSingleUseCertsGenerate
userSingleUseCertsGenerate --> generateUserCert
generateUserCert --> generateCert
generateCert --> generateAWSClientSideCredentials
generateAWSClientSideCredentials -->|notAfter for AWS creds is set to session ttl| generateAWSRolesAnywhereCredentials

GenerateUserCerts -->|else| ServerWithRoles.GenerateUserCerts
ServerWithRoles.GenerateUserCerts --> ServerWithRoles.generateUserCerts
ServerWithRoles.generateUserCerts --> generateUserCert

Loading

So the first issue is that we generate a single use cert with a 1 minute TTL, which is then propagated to the AWS Roles Anywhere credential request, and fails validation because it must be at least 15 minutes.

We have an implicit rule that no single-use Teleport certificates are written to disk with an expiry greater than teleport.UserSingleUseCertTTL (i.e. 1 minute).

We enforce this rule here:

func setUserSingleUseCertsTTL(actx *grpcContext, req *authpb.UserCertsRequest) {
	if isLocalProxyCertReq(req) {
		// don't limit the cert expiry to 1 minute for db local proxy tunnel or kube local proxy,
		// because the certs will be kept in-memory by the client to protect
		// against cert/key exfiltration. When MFA is required, cert expiration
		// time is bounded by the lifetime of the local proxy process or the mfa verification interval.
		return
	}

	maxExpiry := actx.authServer.GetClock().Now().Add(teleport.UserSingleUseCertTTL)
	if req.Expires.After(maxExpiry) {
		req.Expires = maxExpiry
	}
}

DB local proxy and kube local proxy are not bound here, because they are not written to disk.

In the proposed change, we essentially say:

If the user passes --role-arn to tsh apps login assume this is being called by the AWS CLI. If we end up issuing a single-use cert then treat this a local proxy case like DB and kube. Do not cap the TTL to teleport.UserSingleUseCertTTL.

This solves for the validation error by flooring the TTL to the minimum valid one, but creates a new problem in that the certificate, and the valid AWS credentials generated by that certificate, will be saved to disk with a 15 minute TTL, violating our implicit rule.

A partial solution to this would be to do all of that, but then stop short of saving the credentials. This would keep our rule intact, and we would no longer see validation errors.

With this, if I have per-session MFA enabled, and I use the credential_process method of accessing AWS via Teleport, then Teleport will think every aws command is a new session. This may or may not align with what our users think of as a session on the terminal.

If we want to enable longer terminal sessions, we need to have some way of saving the credentials in memory, without making them available on disk or via TCP (I agree with earlier comments, that this is not a secure alternative).

Could we persist the AWS credentials in environment variables instead of saving them in the Teleport cert?

1. Set the TTL of the AWS session to 15 minutes
2. Set environment variables, e.g. TELEPORT_AWSRA_CREDENTIALS
3. On future invocations via process_command we can pull them from the environment.

This way, we tie the concept of the session in per-session MFA to the terminal session itself. We'd need the user experience to include some sort of initial command and eval to set the environment variables for that terminal session though. I'm not sure what that would look like exactly.

I'm trying to think if there's any benefit to also having data in the certificate on disk. If we also want to tie this into having data in the certificate, we could encrypt the credentials with a local session key, save the encrypted credentials in the certificate, persist the session key as an environment variable, and then decrypt on future invocations from credential_process.

That way, the certificate alone can't do anything important, so it's okay to persist for the full 15 minutes as we do for local proxy cases.

Two more questions:

1. Is checking for --role-arn the best way to verify that we're being called by aws via credential_process config? I would expect users to do this manually sometimes.
2. If we continue with any sort of exception case baked in to the isLocalProxyCertReq can we rename and update the comments to make it more clear that there are expanded use cases.

[marcoandredinis]

• Add some kind of tsh app config --headless command as the credential_process. This command would login with a headless flow so that the user could handle the MFA in another terminal / Teleport Connect. Once complete, tsh app config --headless would print the resulting AWS creds directly to stdout, never touching disk. We already have the client infrastructure for this with Headless Polling.

Does it mean a new AWS Session would be created for each API call?
Let's say the user is running some terraform script, they would need to always reach out for the 2nd factor.

Other per-session mfa flows have a broad concept of a session: the user only needs to do a tap when they start an action.
Eg, if you start a database tunnel (tsh proxy db --tunnel pg --db-user=postgres --db-name=postgres --port 8080), they only need to do a tap once and can connect to the database as many times as they need.

• Use tsh aws wrapper to provide the plain creds (aws_access_key_id, aws_secret_access_key, aws_session_token) to the aws command rather than trying to use credentials_command. This way, tsh can handle the login, MFA check, and provide the raw creds directly rather than through disk.

This has to be compatible with whatever tool you use.
Depending on the tsh aws to be invoked every time, defeats the main motivator in the first place: improve UX when compared to other access flows.

Could we persist the AWS credentials in environment variables instead of saving them in the Teleport cert?

1. Set the TTL of the AWS session to 15 minutes
2. Set environment variables, e.g. TELEPORT_AWSRA_CREDENTIALS
3. On future invocations via process_command we can pull them from the environment.

Where do we set the env vars?
Users are expected to have multiple shell sessions and even GUI apps, each one with its own set of env vars.
Even if we call os.SetEnv, it will only persist in a single shell session.

Maybe I didn't understood your suggestion.

Is checking for --role-arn the best way to verify that we're being called by aws via credential_process config? I would expect users to do this manually sometimes.

It's a bit more than that.
We only assume credential_process is going to be used for Apps which have the IAM Roles Anywhere integration configured:

teleport/tool/tsh/common/app.go

Lines 109 to 111 in 43516ca

	if app.GetAWSRolesAnywhereProfileARN() != "" {
	appCertParams.RequesterName = proto.UserCertsRequest_TSH_APP_AWS_CREDENTIALPROCESS
	}

If we continue with any sort of exception case baked in to the isLocalProxyCertReq can we rename and update the comments to make it more clear that there are expanded use cases.

Sure 👍

---

My own suggestion after reading this:

• when per-session MFA is enabled, set the aws session to 15 minutes

Keep everything else as is: same flow, credentials (only valid for 15 minutes) stored in disk.

[rob-picard-teleport]

Where do we set the env vars?
Users are expected to have multiple shell sessions and even GUI apps, each one with its own set of env vars.
Even if we call {"ok":false,"error":"No data returned"}, it will only persist in a single shell session.

Exactly, per-session MFA would result in one MFA tap for each of those shell sessions.

I don't know enough about the GUI app requirements, or how they work today to have ideas there yet.

UX wise, I'm open to suggestions, but I'm thinking something like:

$ eval(tsh apps login --aws-role <role-arn> --format=shell-session-init <app-name>)
# ... MFA tap ...
$ aws <command>
# ... no tap ...
$ aws <command>
# ... no tap ...

Where the first command (totally a placeholder in terms of the specific arguments and command) emits the env variable for the session key or creds.

My own suggestion after reading this:
when per-session MFA is enabled, set the aws session to 15 minutes
Keep everything else as is: same flow, credentials (only valid for 15 minutes) stored in disk.

This breaks per-session MFA too much. The goal of per-session MFA is that a stolen certificate (e.g. from malware) can't be used on its own.

The benefit of the proposed solution above with environment variables is that it's actually never written to disk, not even for the 1 minute TTL.

[marcoandredinis]

I don't know enough about the GUI app requirements, or how they work today to have ideas there yet.

I'm not talking about a specific GUI app, but as a general type of applications.
As an example, there's a lot of VSCode extensions that can use AWS credentials to interact with AWS APIs.
Eg, https://docs.aws.amazon.com/toolkit-for-vscode/latest/userguide/welcome.html

My point is: if you change ~/.aws/config, then everything plays well.
If you set env vars in a shell, then only that shell session has access to the credentials.

Might be an ok trade off: if you want Per-Session MFA, then there's less flexibility on how you can use the credentials.

UX wise, I'm open to suggestions, but I'm thinking something like:

$ eval(tsh apps login --aws-role <role-arn> --format=shell-session-init <app-name>)
# ... MFA tap ...
$ aws <command>
# ... no tap ...
$ aws <command>
# ... no tap ...

That looks good 👍

Suggested UX:
Access without per-session mfa

$ tsh apps login <app> --aws-role <role arn>
Logged into AWS app "<app>".

Your IAM role:
  <role arn>

Example AWS CLI commands:
  aws --profile <app> s3 ls
  AWS_PROFILE=<app> aws s3 ls

$ aws --profile <app> s3 ls
... list of buckets ...

$ eval(tsh apps login <app> --aws-role <role arn>)
$ aws s3 ls
... list of buckets ...

Access with per-session mfa

$ tsh apps login <app> --aws-role <role arn>
MFA is required to access Application "<app>"
Re-run the command with eval:
  
   eval(tsh apps login <app> --aws-role <role arn>)

Note: only the current shell session will be granted AWS access.

$ eval(tsh apps login <app> --aws-role <role arn>)
MFA is required to access Application "<app>"
Tap any security key
Detected security key tap
$ aws s3 ls
... list of buckets ...

What about the AWS Session TTL? Do we keep it as is: as long as allowed or decrease it to 15min?

[rob-picard-teleport]

I'd love to have something elegant for other apps like that. In theory we could have some kind of opt-in "unsafe" mode but I'd rather have the user just configure per-session MFA or disable it in certain cases, etc.

What about the AWS Session TTL? Do we keep it as is: as long as allowed or decrease it to 15min?

Maybe default to 15 minutes but allow them to configure with a flag? Or maybe it's configured in the resource?

[marcoandredinis]

I'll use the current TTL implementation, so whatever is allowed by the Teleport Role (AWS Session may be limited further by the IAM Role Max Session Duration or the IAM Roles Anywhere Profile Max Session Duration)

[marcoandredinis]

This is ready for another review
cc @Tener @rob-picard-teleport @Joerger @rosstimothy

[Joerger]

re: clarifying saving to disk logic.

Also, do we need to set the AWS_CREDENTIALPROCESS requester name for non single use certs? It looks like this puts us here, so we are limiting non mfa certs with the mfa_verification_interval. I see that you use this for this new error condition, though is this condition even possible to reach? Could the app have an associated RolesAnywhereProfile without a matching IntegrationSubKindAWSRolesAnywhere?

Suggested change

	var singleUseCerts bool
	if app.GetAWSRolesAnywhereProfileARN() != "" {
	singleUseCerts, err = isMFARequiredForAppAccess(cf.Context, tc, appInfo.RouteToApp)
	if err != nil {
	return trace.Wrap(err)
	}
	// When using single use certs (aka per-session MFA), tsh cannot write the certificate or AWS credentials to disk.
	// Instead, ask user to use the `--env` flag which only outputs the credentials, in an eval friendly format.
	if singleUseCerts && !cf.AppLoginAWSEnvOutput {
	return trace.BadParameter(`AWS access is configured to use per-session MFA and credentials are only available to a single session. Pass the --env flag to the previous command and export the credentials using eval.
	Example:
	eval "$(tsh apps login %s --aws-role %s --env)"
	You can now run the AWS CLI or other AWS SDK based tools as usual.
	Example:
	aws sts get-caller-identity`,
	shsprintf.EscapeDefaultContext(app.GetName()),
	shsprintf.EscapeDefaultContext(appInfo.RouteToApp.AWSRoleARN),
	)
	}
	appCertParams.RequesterName = proto.UserCertsRequest_TSH_APP_AWS_CREDENTIALPROCESS
	}
	// When using `tsh app login`, certs should generally be saved to disk, whether standard certs or
	// single-use MFA-verified 1m TTL certs. However, in cases where we are exceeding the standard
	// 1m TTL for single-use certs, we must ensure the certs are not saved to disk.
	saveToDisk := true
	if app.GetAWSRolesAnywhereProfileARN() != "" {
	singleUseCerts, err = isMFARequiredForAppAccess(cf.Context, tc, appInfo.RouteToApp)
	if err != nil {
	return trace.Wrap(err)
	}
	if singleUseCerts {
	// Use the AWS_CREDENTIALPROCESS requester name to get longer TTL single use certs.
	// In exchange, this client must not save the certs to disk. Instead, ask user to use the `--env`
	// flag which only outputs the credentials, in an eval friendly format.
	appCertParams.RequesterName = proto.UserCertsRequest_TSH_APP_AWS_CREDENTIALPROCESS
	saveToDisk = false
	if !cf.AppLoginAWSEnvOutput {
	return trace.BadParameter(`AWS access is configured to use per-session MFA and credentials are only available to a single session. Pass the --env flag to the previous command and export the credentials using eval.
	Example:
	eval "$(tsh apps login %s --aws-role %s --env)"
	You can now run the AWS CLI or other AWS SDK based tools as usual.
	Example:
	aws sts get-caller-identity`,
	shsprintf.EscapeDefaultContext(app.GetName()),
	shsprintf.EscapeDefaultContext(appInfo.RouteToApp.AWSRoleARN),
	)
	}
	}
	}

[marcoandredinis]

Updated the code as suggested (minus some minor changes).

It's clearer, thank you.

[backport-bot-workflows]

@marcoandredinis See the table below for backport results.

Branch	Result
branch/v18	Failed