EKS Pod Identities does not work with default credential chain

[JoeNorth]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

Using the SDK with EKS Pod Identities results in the following error:

/home/node/app/node_modules/@smithy/credential-provider-imds/dist-cjs/fromContainerMetadata.js:57
            throw new property_provider_1.CredentialsProviderError(`${parsed.hostname} is not a valid container metadata service hostname`, false);
                  ^

CredentialsProviderError: 169.254.170.23 is not a valid container metadata service hostname
    at getCmdsUri (/home/node/app/node_modules/@smithy/credential-provider-imds/dist-cjs/fromContainerMetadata.js:57:19)
    at /home/node/app/node_modules/@smithy/credential-provider-imds/dist-cjs/fromContainerMetadata.js:16:38
    at retry (/home/node/app/node_modules/@smithy/credential-provider-imds/dist-cjs/remoteProvider/retry.js:5:19)
    at /home/node/app/node_modules/@smithy/credential-provider-imds/dist-cjs/fromContainerMetadata.js:15:36
    at /home/node/app/node_modules/@smithy/property-provider/dist-cjs/chain.js:12:39
    at async coalesceProvider (/home/node/app/node_modules/@smithy/property-provider/dist-cjs/memoize.js:14:24)
    at async SignatureV4S3Express.credentialProvider (/home/node/app/node_modules/@smithy/property-provider/dist-cjs/memoize.js:33:24)
    at async SignatureV4S3Express.signRequest (/home/node/app/node_modules/@smithy/signature-v4/dist-cjs/SignatureV4.js:106:29)
    at async /home/node/app/node_modules/@aws-sdk/middleware-signing/dist-cjs/awsAuthMiddleware.js:28:25
    at async /home/node/app/node_modules/@smithy/middleware-retry/dist-cjs/retryMiddleware.js:31:46 {
  tryNextLink: false,
  '$metadata': { attempts: 1, totalRetryDelay: 0 }
}

TL;DR the issue is because the credential provider @aws-sdk/credential-provider-http was never added to the default credential chain. This provider has the logic to support EKS Pod Identities while @smithy/credential-provider-imds does not.

The defaultProvider calls remoteProvider which then checks for the presence of the environment variable defined in ENV_CMDS_FULL_URI from @smithy/credential-provider-imds. The value of ENV_CMDS_FULL_URI being AWS_CONTAINER_CREDENTIALS_FULL_URI which I confirmed is set in the pod environment variables:

Environment:
  AWS_STS_REGIONAL_ENDPOINTS:              regional
  AWS_DEFAULT_REGION:                      us-east-1
  AWS_REGION:                              us-east-1
  AWS_CONTAINER_CREDENTIALS_FULL_URI:      http://169.254.170.23/v1/credentials
  AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE:  /var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token

Since this environment variable is present, remoteProviders then calls fromContainerMetadata in @smithy/credential-provider-imds. fromContainerMetadata then calls getCmdsUri which parses the value of environment variable ENV_CMDS_FULL_URI (which again is AWS_CONTAINER_CREDENTIALS_FULL_URI) to get the hostname.

The logic here breaks down because it's assuming that if ENV_CMDS_FULL_URI is set then the parsed hostname 1) exists and 2) if the parsed hostname exists then it is also in GREENGRASS_HOSTS:

const GREENGRASS_HOSTS = {
  localhost: true,
  "127.0.0.1": true,
};

In this case the hostname exists but the hostname is not in that list. Instead, the hostname is 169.254.170.23 which is the EKS Pod Identity Agent IP which then leaves us with the error we're presented.

Instead the default credential chain should call fromHttp from the provider @aws-sdk/credential-provider-http which does have the correct logic.

SDK version number

@aws-sdk/[email protected]

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v18.19.0

Reproduction Steps

Attempt to run the following sample using EKS Pod Identity

import { fileURLToPath } from "url";
import { ListBucketsCommand, S3Client } from "@aws-sdk/client-s3";

const client = new S3Client({});

export const helloS3 = async () => {
  const command = new ListBucketsCommand({});

  const { Buckets } = await client.send(command);
  console.log("Buckets: ");
  console.log(Buckets.map((bucket) => bucket.Name).join("\n"));
  return Buckets;
};

if (process.argv[1] === fileURLToPath(import.meta.url)) {
  helloS3();
}

Observed Behavior

/home/node/app/node_modules/@smithy/credential-provider-imds/dist-cjs/fromContainerMetadata.js:57
            throw new property_provider_1.CredentialsProviderError(`${parsed.hostname} is not a valid container metadata service hostname`, false);
                  ^

CredentialsProviderError: 169.254.170.23 is not a valid container metadata service hostname
    at getCmdsUri (/home/node/app/node_modules/@smithy/credential-provider-imds/dist-cjs/fromContainerMetadata.js:57:19)
    at /home/node/app/node_modules/@smithy/credential-provider-imds/dist-cjs/fromContainerMetadata.js:16:38
    at retry (/home/node/app/node_modules/@smithy/credential-provider-imds/dist-cjs/remoteProvider/retry.js:5:19)
    at /home/node/app/node_modules/@smithy/credential-provider-imds/dist-cjs/fromContainerMetadata.js:15:36
    at /home/node/app/node_modules/@smithy/property-provider/dist-cjs/chain.js:12:39
    at async coalesceProvider (/home/node/app/node_modules/@smithy/property-provider/dist-cjs/memoize.js:14:24)
    at async SignatureV4S3Express.credentialProvider (/home/node/app/node_modules/@smithy/property-provider/dist-cjs/memoize.js:33:24)
    at async SignatureV4S3Express.signRequest (/home/node/app/node_modules/@smithy/signature-v4/dist-cjs/SignatureV4.js:106:29)
    at async /home/node/app/node_modules/@aws-sdk/middleware-signing/dist-cjs/awsAuthMiddleware.js:28:25
    at async /home/node/app/node_modules/@smithy/middleware-retry/dist-cjs/retryMiddleware.js:31:46 {
  tryNextLink: false,
  '$metadata': { attempts: 1, totalRetryDelay: 0 }
}

Expected Behavior

Expected default credential chain to use the correct provider.

Possible Solution

Update the default credential chain to use the correct provider.

Additional Information/Context

No response

[RanVaknin]

Hi @JoeNorth ,

Thanks for reaching out. As we have discussed, this is because the fromHttp provider is not included in the credential chain. This is now in our backlog.

The workaround is to instruct the Client to use the credential provider explciitly:

import { fromHttp } from '@aws-sdk/credential-provider-http'

const client = new S3Client({
  credentials: fromHttp({
    // add params if necassary
  })
});

Thanks,
Ran~

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.