ConstructionFailure(ConstructionFailure { source: SigningStageError { kind: MissingCredentials } }) on EKS

[christopher-wong]

Describe the bug

aws-sdk-rust not loading credentials from IAM role attached to EC2 within a EKS cluster.

Expected Behavior

I expect the default provider to load credentials from the IAM role attached to the EC2

Current Behavior

ConstructionFailure(ConstructionFailure { source: SigningStageError { kind: MissingCredentials } })

Reproduction Steps

#[tokio::main]
async fn main() {
    tracing_subscriber::registry()
        .with(tracing_subscriber::fmt::layer())
        .init();

    let region_provider = RegionProviderChain::default_provider().or_else("us-west-2");
    let shared_config = aws_config::from_env().region(region_provider).load().await;
    let client = aws_sdk_sts::Client::new(&shared_config);
    let res = client.get_caller_identity().send().await;
    tracing::debug!("caller_identity = {:#?}", res);
}

Possible Solution

No response

Additional Information/Context

No response

Version

├── aws-config v0.54.1
│   ├── aws-credential-types v0.54.1
│   │   ├── aws-smithy-async v0.54.4
│   │   ├── aws-smithy-types v0.54.4
│   ├── aws-http v0.54.1
│   │   ├── aws-credential-types v0.54.1 (*)
│   │   ├── aws-smithy-http v0.54.4
│   │   │   ├── aws-smithy-types v0.54.4 (*)
│   │   ├── aws-smithy-types v0.54.4 (*)
│   │   ├── aws-types v0.54.1
│   │   │   ├── aws-credential-types v0.54.1 (*)
│   │   │   ├── aws-smithy-async v0.54.4 (*)
│   │   │   ├── aws-smithy-client v0.54.4
│   │   │   │   ├── aws-smithy-async v0.54.4 (*)
│   │   │   │   ├── aws-smithy-http v0.54.4 (*)
│   │   │   │   ├── aws-smithy-http-tower v0.54.4
│   │   │   │   │   ├── aws-smithy-http v0.54.4 (*)
│   │   │   │   │   ├── aws-smithy-types v0.54.4 (*)
│   │   │   │   ├── aws-smithy-types v0.54.4 (*)
│   │   │   ├── aws-smithy-http v0.54.4 (*)
│   │   │   ├── aws-smithy-types v0.54.4 (*)
│   ├── aws-sdk-sso v0.24.0
│   │   ├── aws-credential-types v0.54.1 (*)
│   │   ├── aws-endpoint v0.54.1
│   │   │   ├── aws-smithy-http v0.54.4 (*)
│   │   │   ├── aws-smithy-types v0.54.4 (*)
│   │   │   ├── aws-types v0.54.1 (*)
│   │   ├── aws-http v0.54.1 (*)
│   │   ├── aws-sig-auth v0.54.1
│   │   │   ├── aws-credential-types v0.54.1 (*)
│   │   │   ├── aws-sigv4 v0.54.1
│   │   │   │   ├── aws-smithy-http v0.54.4 (*)
│   │   │   ├── aws-smithy-http v0.54.4 (*)
│   │   │   ├── aws-types v0.54.1 (*)
│   │   ├── aws-smithy-async v0.54.4 (*)
│   │   ├── aws-smithy-client v0.54.4 (*)
│   │   ├── aws-smithy-http v0.54.4 (*)
│   │   ├── aws-smithy-http-tower v0.54.4 (*)
│   │   ├── aws-smithy-json v0.54.4
│   │   │   └── aws-smithy-types v0.54.4 (*)
│   │   ├── aws-smithy-types v0.54.4 (*)
│   │   ├── aws-types v0.54.1 (*)
│   ├── aws-sdk-sts v0.24.0
│   │   ├── aws-credential-types v0.54.1 (*)
│   │   ├── aws-endpoint v0.54.1 (*)
│   │   ├── aws-http v0.54.1 (*)
│   │   ├── aws-sig-auth v0.54.1 (*)
│   │   ├── aws-smithy-async v0.54.4 (*)
│   │   ├── aws-smithy-client v0.54.4 (*)
│   │   ├── aws-smithy-http v0.54.4 (*)
│   │   ├── aws-smithy-http-tower v0.54.4 (*)
│   │   ├── aws-smithy-json v0.54.4 (*)
│   │   ├── aws-smithy-query v0.54.4
│   │   │   ├── aws-smithy-types v0.54.4 (*)
│   │   ├── aws-smithy-types v0.54.4 (*)
│   │   ├── aws-smithy-xml v0.54.4
│   │   ├── aws-types v0.54.1 (*)
│   ├── aws-smithy-async v0.54.4 (*)
│   ├── aws-smithy-client v0.54.4 (*)
│   ├── aws-smithy-http v0.54.4 (*)
│   ├── aws-smithy-http-tower v0.54.4 (*)
│   ├── aws-smithy-json v0.54.4 (*)
│   ├── aws-smithy-types v0.54.4 (*)
│   ├── aws-types v0.54.1 (*)
├── aws-sdk-lambda v0.24.0
│   ├── aws-credential-types v0.54.1 (*)
│   ├── aws-endpoint v0.54.1 (*)
│   ├── aws-http v0.54.1 (*)
│   ├── aws-sig-auth v0.54.1 (*)
│   ├── aws-smithy-async v0.54.4 (*)
│   ├── aws-smithy-client v0.54.4 (*)
│   ├── aws-smithy-http v0.54.4 (*)
│   ├── aws-smithy-http-tower v0.54.4 (*)
│   ├── aws-smithy-json v0.54.4 (*)
│   ├── aws-smithy-types v0.54.4 (*)
│   ├── aws-types v0.54.1 (*)
├── aws-sdk-secretsmanager v0.24.0
│   ├── aws-credential-types v0.54.1 (*)
│   ├── aws-endpoint v0.54.1 (*)
│   ├── aws-http v0.54.1 (*)
│   ├── aws-sig-auth v0.54.1 (*)
│   ├── aws-smithy-async v0.54.4 (*)
│   ├── aws-smithy-client v0.54.4 (*)
│   ├── aws-smithy-http v0.54.4 (*)
│   ├── aws-smithy-http-tower v0.54.4 (*)
│   ├── aws-smithy-json v0.54.4 (*)
│   ├── aws-smithy-types v0.54.4 (*)
│   ├── aws-types v0.54.1 (*)
├── aws-sdk-sts v0.24.0 (*)
├── aws-smithy-types v0.54.4 (*)

Environment details (OS name and version, etc.)

rust:1.68

Logs

2023-03-20T19:45:12.041306Z DEBUG myapp_rs: caller_identity = Err(
 ConstructionFailure(
 ConstructionFailure {
 source: SigningStageError {
 kind: MissingCredentials,
 },
 },
 ),
)
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ConstructionFailure(ConstructionFailure { source: SigningStageError { kind: MissingCredentials } })', src/main.rs:73:10
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[rcoh]

have you verified that the credentials actually exist in the environment? Can you look at logs for aws_config=trace and see if there are any errors?

[christopher-wong]

it looks like the SDK is failing to communicate with IDMS and is timing out

2023-03-20T20:39:09.608443Z DEBUG provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}: aws_config::imds::credentials: loading credentials from IMDS
2023-03-20T20:39:10.608874Z DEBUG provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=Ec2InstanceMetadata context=the credential provider was not enabled: could not communicate with IMDS: dispatch failure: timeout: HTTP read timeout occurred after 1s: timed out (CredentialsNotLoaded(CredentialsNotLoaded { source: ImdsCommunicationError { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: HttpTimeoutError { kind: "HTTP read", duration: 1s } } }) } }))

I also verified that IAM is configured properly with another container using aws-sdk-go-v2 and it's able to grab the correct credentials from the EC2 host.

[rcoh]

interesting—EKS uses the web identity token provider normally, not IMDS. One possibility is the docker hop limit—#540

As a side note, since IMDS is the last provider in the chain, a timeout isn't necessarily meaningful info since it means we didn't get credentials anywhere else either.

Do logs indicate that the Go SDK is using IMDS (instead of the web identity token provider?) I'd also want to verify that Go is using IMDSv2. The Rust SDK, being a new SDK does not have support for a fallback to (insecure) IMDSv1 so it may behave slightly differently.

[christopher-wong]

Thanks @rcoh. It looks like the issue was the one-hop limit. I set the instance metadata settings as suggested in #540 and everything started working as expected.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.