‼️ (iam): OpenIdConnectProvider defaults to first thumbprint instead of root CA thumbprint

[bleish]

Please add your +1 👍 to let us know you have encountered this

Status: IN-PROGRESS

Overview:

The iam.OpenIdConnectProvider resource contains logic that dynamically fetches the certificate thumbprint required to create an OpenID Connect provider. However, as of now, it mistakenly fetches the leaf certificate of the provider, instead of the root one.

As long as the leaf certificate is valid, this doesn't have an impact your applications. However, once the certificate is rotated, your application will fail to use the provider to authenticate against AWS services. Since leaf certificates are rotated frequently, you are in danger of disruption. This will probably manifest in Access Denied errors.

Workaround:

If you are using the iam.OpenIdConnectProvider construct in conjunction with an EKS cluster:

import * as iam from `aws-cdk-lib/aws-iam`;
import * as eks from `aws-cdk-lib/aws-eks`;

const cluster = new eks.Cluster(this, "EKSCluster", {
  version: eks.KubernetesVersion.V1_21,
});

new iam.OpenIdConnectProvider(this, "OIDCProvider", {
  url: cluster.clusterOpenIdConnectIssuerUrl,
});

Switch to use the OpenIdConnectProvider construct from the EKS library:

import * as eks from `aws-cdk-lib/aws-eks`;

const cluster = new eks.Cluster(this, "EKSCluster", {
  version: eks.KubernetesVersion.V1_21,
});

new eks.OpenIdConnectProvider(this, "OIDCProvider", {
  url: cluster.clusterOpenIdConnectIssuerUrl,
});

The reason this works is because the eks.OpenIdConnectProvider hardcodes the correct thumbprint for EKS.

If you are using the iam.OpenIdConnectProvider in conjunction with other services, make sure you pass the thumbprint explicitly to the construct, instead of relying on its dynamic fetching capabilities.

import * as iam from `aws-cdk-lib/aws-iam`;

new iam.OpenIdConnectProvider(this, "OIDCProvider", {
  url: cluster.clusterOpenIdConnectIssuerUrl,
    clientIds: ["sts.amazonaws.com"],
    thumbprints: ['<pass-thumbprint-here>'],
});

To obtain the correct thumbprint for your provider, follow these instructions.

Solution:

We are working on a fix to the iam.OpenIdConnectProvider construct so that if correctly fetches the root certificate thumbprint. See PR. Once it is merged, the fix will be available in the following CDK release, at which point a simple deployment will fix the issue in your environment.

---

Originally reported as

When deploying an OpenIdConnectProvider construct using the oidc issuer url retrieved from an EKS cluster (the domain is oidc.eks.us-west-2.amazonaws.com) and no value for the thumbprints property, the resulting auto-obtained thumbprint doesn't match the one I get from following the steps provided here.

Reproduction Steps

const cluster = new eks.Cluster(this, "EKSCluster");

new iam.OpenIdConnectProvider(this, "EKSOIDCProvider", {
    url: cluster.clusterOpenIdConnectIssuerUrl,
    clientIds: ["sts.amazonaws.com"]
});

Error Log

See Other for a related error.

Environment

• CLI Version : 1.45.0
• Framework Version:
• Node.js Version: 14.4.0
• OS : Windows 10 (10.0.18363 Build 18363)
• Language (Version): TypeScript (3.7.2)

Other

If I try to deploy a cluster autoscaler to my EKS cluster using a service account role tied to that provider, the pod enters a CrashLoopBackOff state with the error message:

F0617 20:04:12.561996 1 aws_cloud_provider.go:376] Failed to create AWS Manager: WebIdentityErr: failed to retrieve credentials
caused by: InvalidIdentityToken: OpenIDConnect provider's HTTPS certificate doesn't match configured thumbprint
status code: 400, request id: 00ec5e59-3672-4264-b5d4-c44f573ff50c

If I instead follow the guide to retrieve the correct thumbprint via openssl and provide that to the OpenIdConnectProvider construct, the cluster autoscaler successfully deploys.

---

This is 🐛 Bug Report

[sogos]

Same problem observed in CLI Version: 1.46.0

[eladb]

@bleish @sogos can you use cluster.addServiceAccount() instead of explicitly creating the OIDC provider (see docs)?

[sogos]

Hello @eladb ,

That seem to work perfectly 👍

Tested with alb-ingress-controller:
https://gitlab.com/thibault.cordier1/cdk-eks-fargate/-/blob/master/lib/eks-stack.ts#L68

But i don't understand why the thumbprint on the identity provider is false.
I use the command:

openssl s_client -servername oidc.eks.eu-west-1.amazonaws.com -showcerts -connect oidc.eks.eu-west-1.amazonaws.com:443 | openssl x509  -fingerprint -noout

[eladb]

Glad to hear that addServiceAccount() worked for you. We need to investigate why the default behavior of the iam.OpenIdConnectProvider resource is unable to extract the correct thumbprint from EKS. The current implementation in @aws-cdk/aws-eks actually hard codes the thumbprint (ref).

[eladb]

Our EKS code currently has the following comment:

For some reason EKS isn't validating the root certificate but a intermediat certificate
which is one level up in the tree. Because of the a constant thumbprint value has to be
stated with this OpenID Connect provider. The certificate thumbprint is the same for all the regions.

aws-cdk/packages/@aws-cdk/aws-eks/lib/cluster.ts

Line 711 in 6166a70

thumbprints: [ '9e99a48a9960b14926bb7f3b02e22da2b0ab7280' ],

[eladb]

Routing to IAM

[bleish]

@eladb @sogos

It looks like the lambda is using the tlsSocket.getPeerCertificate method to retrieve the intermediate certificate. By default, that only retrieves the first certificate in the chain. If you pass true to the method it'll return the full chain, linked by the issuerCertificate property. You can follow the chain until you reach the root CA, after which issuerCertificate becomes a circular reference (self-signed). I just tried this and was able to get the proper certificate and thumbprint. In my case (issuer host = oidc.eks.us-west-2.amazonaws.com) the chain was:

*.execute-api.us-west-2.amazonaws.com -> Amazon -> Amazon Root CA 1 -> Starfield Services Root Certificate Authority - G2

The last giving the expected thumbprint of 9e99a48a9960b14926bb7f3b02e22da2b0ab7280.

This is an issue beyond just retrieving the EKS issuer thumbprint. For any issuer, if the first certificate in the chain is self-signed, it'll be the root, but in any other case the intermediate is retrieved instead.

The Node.js TLS documentation for getPeerCertificate gives more info that could be useful in implementing this.

[eladb]

@bleish wow thanks so much for the detailed explanation. Would you say the preferred behavior for the OIDC provider resource is to always use the root CA thumbprint?

[bleish]

@eladb
It seems so. According to their documentation:

When you create an OpenID Connect (OIDC) identity provider in IAM, you must supply a thumbprint. IAM requires the thumbprint for the root certificate authority (CA) that signed the certificate used by the external identity provider (IdP). The thumbprint is a signature for the CA's certificate that was used to issue the certificate for the OIDC-compatible IdP. When you create an IAM OIDC identity provider, you are trusting identities authenticated by that IdP to have access to your AWS account.

One gotcha I ran into when testing the getPeerCertificate(true) method was that the certificate chain contained one more certificate beyond the root CA. When following the openssl steps from the documentation above, I get a chain of 4 certificates, and the last certificate in that chain (Starfield Services Root Certificate Authority - G2) returns the expected 9e99a48a9960b14926bb7f3b02e22da2b0ab7280 thumbprint. However, using getPeerCertificate(true) I get one more certificate beyond that, but it has no Common Name value and is self-signed (so issuerCertificate is circular at that point). I don't know what this certificate is, as the openssl method didn't contain it. Maybe self-signed certs aren't included in the openssl chain? Someone more knowledgable in this area than I am might understand what's going on.

[bleish]

It looks like openssl s_client doesn't always return the full certificate chain. The documentation for the -showcerts flag on s_client states:

Displays the server certificate list as sent by the server: it only consists of certificates the server has sent (in the order the server has sent them). It is not a verified chain.

Using getPeerCertificate(true) does appear to return the full chain, so it seems the Open ID Connect Identity Provider doesn't necessarily expect the absolute root of the chain, but the last certificate provided by the server (Starfield Services Root Certificate Authority - G2 in this case). If that's true, then I'm not sure how, using the tls module, to ensure the correct thumbprint is being obtained.

[chrisjgray]

My organization is also having this same issue. Can this get prioritized?

[comcalvi]

We're working on a fix for this issue right now. If you can't upgrade to the CDK version that contains the fix, you can workaround this issue on an old version by changing any iam.OpenIdConnectProviders that use an EKS Cluster's clusterOpenIdConnectIssuerUrl to use the OpenIdConnectProvider from aws-eks like so:

import * as iam from `aws-cdk-lib/aws-iam`;
import * as eks from `aws-cdk-lib/aws-eks`;

const cluster = new eks.Cluster(this, "EKSCluster", {
  version: eks.KubernetesVersion.V1_21,
});

new eks.OpenIdConnectProvider(this, "OIDCProvider", {
  url: cluster.clusterOpenIdConnectIssuerUrl,
});

Or, if you prefer to use aws-iam, you can hardcode the thumbprint to that of the root certificate (which is exactly what the aws-eks module does), but be aware that EKS will eventually rotate the root certificate as well:

import * as iam from `aws-cdk-lib/aws-iam`;
import * as eks from `aws-cdk-lib/aws-eks`;

const cluster = new eks.Cluster(this, "EKSCluster", {
  version: eks.KubernetesVersion.V1_21,
});

new iam.OpenIdConnectProvider(this, "OIDCProvider", {
  url: cluster.clusterOpenIdConnectIssuerUrl,
    clientIds: ["sts.amazonaws.com"],
    thumbprints: ['9e99a48a9960b14926bb7f3b02e22da2b0ab7280'],
});

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[iliapolo]

Reopening. See #22802

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[drankard]

if this is Closed, shouldn't notice 8607 be removed ? or do i still have to acknowledge it even thoug its fixed by a release?

[mrgrain]

if this is Closed, shouldn't notice 8607 be removed ? or do i still have to acknowledge it even thoug its fixed by a release?

Apologies for this. We had an issue that wrongly showed to notice for non affected versions. This should be fixed now.

[fadinasr]

@mrgrain I have CDK v2.51.1 (build 3d30cdb) and I still got the notice also. I'm currently working on configuring Github actions OpenIdConnectProviders to configure a role for GitHub actions

[mrgrain]

@fadinasr Thanks for letting me know. I can't reproduce this.
We do cache the notice data, and you might have had a version locally with the wrong version ranges. Maybe that's why. Although the cache expires rather quickly.