identity: Proxy fails to validate certificates with name constraints

[olix0r]

Linkerd cannot be deployed when its CA certificate(s) use name constraints. Proxies fail to validate issued certificates.

Repro/Investigation

1. Generate a self-signed certificate with name constraints limiting issued
   certificates to foo.bar:

bar.tpl:

{
    "subject": {{ toJson .Subject }},
    "issuer": {{ toJson .Subject }},
    "keyUsage": ["certSign"],
    "basicConstraints": {
        "isCA": true,
        "maxPathLen": 1
    },
    "nameConstraints": {
        "critical": true,
        "permittedDNSDomains": ["foo.bar"]
    }
}

:; step certificate create foo.bar cert.pem key.pem --template=bar.tpl --not-after=24h --no-password --insecure

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 223894858428966857916253664437917250458 (0xa87094cb1d1a378f60aa362777a4f39a)
    Signature Algorithm: ECDSA-SHA256
        Issuer: CN=foo.bar
        Validity
            Not Before: Aug 30 20:07:42 2022 UTC
            Not After : Aug 31 20:07:42 2022 UTC
        Subject: CN=foo.bar
        Subject Public Key Info:
            Public Key Algorithm: ECDSA
                Public-Key: (256 bit)
                X:
                    10:f9:57:fa:30:30:93:4d:9b:e1:64:20:54:a1:1a:
                    20:76:28:5e:31:c4:6f:19:f6:62:9d:fe:9b:68:e8:
                    14:ee
                Y:
                    58:aa:27:73:08:24:78:a9:57:f1:35:fe:53:d9:2d:
                    83:01:d4:fd:c3:b6:44:26:16:06:a0:fb:6f:d8:4b:
                    6b:c8
                Curve: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Certificate Sign
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:1
            X509v3 Subject Key Identifier:
                A3:E8:20:63:F8:F4:35:D0:0D:47:79:B7:AD:F3:CD:3B:EF:2C:74:AE
            X509v3 Name Constraints: critical
                Permitted:
                  DNS: foo.bar
    Signature Algorithm: ECDSA-SHA256
         30:45:02:20:60:b2:5c:96:44:30:0c:5b:fb:63:98:90:7d:60:
         7e:75:ce:88:81:70:8a:01:97:e8:f5:e4:29:30:bf:87:04:ad:
         02:21:00:ed:fa:f6:71:15:c0:26:99:ec:c7:9a:a3:4d:e6:5f:
         e0:e4:d0:fb:9e:3a:3e:30:26:0f:90:25:ba:0a:17:6f:52

2. Install Linkerd using this certificate:

:; linkerd install --crds |k apply -f -

:; linkerd install \
    --set identityTrustDomain="foo.bar" \
    --set-file identity.issuer.tls.crtPEM=cert.pem \
    --set-file identity.issuer.tls.keyPEM=key.pem \
    --set-file identityTrustAnchorsPEM=cert.pem \
    |k apply -f -

3. The control plane does not start.

The identity controller becomes 'running' and it appears to issue a certificate
to its local proxy:

:; k logs -n linkerd linkerd-identity-5c9d8bbb4c-n8n8n
Defaulted container "identity" out of: identity, linkerd-proxy, linkerd-init (init)
time="2022-08-30T20:13:40Z" level=info msg="running version stable-2.12.0"
time="2022-08-30T20:13:40Z" level=info msg="starting admin server on :9990"
time="2022-08-30T20:13:40Z" level=info msg="starting gRPC server on :8080"
time="2022-08-30T20:13:46Z" level=info msg="issued certificate for linkerd-identity.linkerd.serviceaccount.identity.linkerd.foo.bar until 2022-08-31 20:07:42 +0000 UTC: 4a28173a08c2e9629eeb982e8b2699b6c6007eef8b2377b31eeb2a6149bb371f"

But the proxy fails to validate the certificate:

:; k logs -n linkerd linkerd-identity-5c9d8bbb4c-n8n8n -c linkerd-proxy
...
[     0.011665s] ERROR ThreadId(02) identity: linkerd_proxy_identity_client::certify: Failed to obtain identity error=invalid peer certificate contents: invalid peer certificate: UnknownIssuer

The rest of the control plane fails to start because the identity controller never becomes ready.

---

This process works when name constraints are omitted from the certificate:

{
    "subject": {{ toJson .Subject }},
    "issuer": {{ toJson .Subject }},
    "keyUsage": ["certSign"],
    "basicConstraints": {
        "isCA": true,
        "maxPathLen": 1
    }
}

:; step certificate create foo.bar cert.pem key.pem --template=bar-unconstrained.tpl --not-after=24h --no-password --insecure

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 22516031807147259098312418109158923000 (0x10f06e13bbe38643d7ce95ec4cf91af8)
    Signature Algorithm: ECDSA-SHA256
        Issuer: CN=foo.bar
        Validity
            Not Before: Aug 30 20:33:12 2022 UTC
            Not After : Aug 31 20:33:12 2022 UTC
        Subject: CN=foo.bar
        Subject Public Key Info:
            Public Key Algorithm: ECDSA
                Public-Key: (256 bit)
                X:
                    74:e5:86:db:e6:68:1d:a5:c7:1d:fd:21:29:2d:69:
                    de:f1:0a:57:7c:30:35:f1:9d:c4:c2:d3:61:cf:42:
                    09:19
                Y:
                    ab:00:91:94:2c:df:e5:7f:4b:13:8e:40:e5:04:07:
                    e9:0e:db:c9:03:83:a2:fb:42:80:37:cb:35:df:f4:
                    17:7d
                Curve: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Certificate Sign
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:1
            X509v3 Subject Key Identifier:
                60:42:74:A2:23:17:6A:BD:94:1E:0B:87:4C:A8:70:7B:B8:BD:AD:55
    Signature Algorithm: ECDSA-SHA256
         30:45:02:20:63:24:33:92:8a:71:65:d3:9f:fd:da:44:ab:40:
         1e:74:6e:0c:58:f5:8d:93:98:31:76:be:e4:96:7a:97:a5:e2:
         02:21:00:a1:a0:93:4d:0e:bc:96:0c:af:99:4a:34:0a:40:5f:
         b6:d1:72:d5:ed:16:85:8a:3a:c4:e1:5e:90:43:80:e0:15

:; linkerd identity -n linkerd linkerd-identity-75f7d4df68-wd8km

POD linkerd-identity-75f7d4df68-wd8km (1 of 1)

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
    Signature Algorithm: ECDSA-SHA256
        Issuer: CN=foo.bar
        Validity
            Not Before: Aug 30 20:38:17 2022 UTC
            Not After : Aug 31 20:38:27 2022 UTC
        Subject: CN=linkerd-identity.linkerd.serviceaccount.identity.linkerd.foo.bar
        Subject Public Key Info:
            Public Key Algorithm: ECDSA
                Public-Key: (256 bit)
                X:
                    bb:43:be:97:a7:39:83:6c:e5:71:e1:17:71:c2:ef:
                    fc:64:fd:1e:cb:6f:5c:25:fe:8b:ce:1f:91:93:24:
                    40:bf
                Y:
                    07:89:d1:56:5d:8e:b7:b7:3c:9a:8a:a6:12:e3:19:
                    30:63:63:25:49:4b:8a:61:be:1f:5b:ed:f2:88:6c:
                    b8:79
                Curve: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Authority Key Identifier:
                keyid:5E:A4:79:4B:A0:EA:D3:8A:07:9F:14:8F:2A:77:E8:0E:0B:1A:4B:D7
            X509v3 Subject Alternative Name:
                DNS:linkerd-identity.linkerd.serviceaccount.identity.linkerd.foo.bar

    Signature Algorithm: ECDSA-SHA256
         30:45:02:20:4d:85:c6:af:4b:04:aa:eb:89:8f:c8:64:f3:49:
         42:60:b0:66:6d:1a:22:5d:e9:41:3c:40:9d:54:8f:e0:d8:1c:
         02:21:00:d5:ee:41:06:a1:ca:fb:1c:40:91:e4:35:75:68:ae:
         23:9e:f3:7b:24:b2:4d:14:a6:dd:66:0d:43:7c:5f:79:04

---

Note that we can manually create an end-entity certificate (as used by a proxy) with:

:; step certificate create linkerd-identity.linkerd.serviceaccount.identity.linkerd.foo.bar \
    ee-cert.pem ee-key.pem \
    --san=linkerd-identity.linkerd.serviceaccount.identity.linkerd.foo.bar \
    --ca-key=key.pem --ca=cert.pem \
    --not-after=24h \
    --no-password --insecure

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 55110646295972328840139681094716422534 (0x2975eca5308fa2fde788e681d188f986)
    Signature Algorithm: ECDSA-SHA256
        Issuer: CN=foo.bar
        Validity
            Not Before: Aug 30 20:53:31 2022 UTC
            Not After : Aug 31 20:53:31 2022 UTC
        Subject: CN=linkerd-identity.linkerd.serviceaccount.identity.linkerd.foo.bar
        Subject Public Key Info:
            Public Key Algorithm: ECDSA
                Public-Key: (256 bit)
                X:
                    52:b2:ad:f7:19:87:ef:f0:0c:e3:f4:4a:5c:3f:a1:
                    67:7b:b1:75:31:32:14:ba:42:56:f7:4a:58:b7:af:
                    fe:8c
                Y:
                    ed:fa:80:45:fb:30:51:8a:3e:86:bd:dd:0f:b8:75:
                    01:47:50:d7:c6:0a:24:ca:42:65:02:26:0a:36:db:
                    3d:4f
                Curve: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage:
                Server Authentication, Client Authentication
            X509v3 Subject Key Identifier:
                0B:1B:29:6D:CF:76:77:A7:3E:40:7F:10:E6:0A:BD:69:69:BE:70:7B
            X509v3 Authority Key Identifier:
                keyid:5E:A4:79:4B:A0:EA:D3:8A:07:9F:14:8F:2A:77:E8:0E:0B:1A:4B:D7
            X509v3 Subject Alternative Name:
                DNS:linkerd-identity.linkerd.serviceaccount.identity.linkerd.foo.bar
    Signature Algorithm: ECDSA-SHA256
         30:44:02:20:3d:0a:2b:dd:49:21:a9:95:3c:49:48:17:54:d0:
         99:c9:f7:c7:7b:73:da:39:86:1b:27:8a:e6:9b:5c:c8:68:eb:
         02:20:3e:16:16:bd:ff:82:99:16:c8:f6:60:bb:e5:55:5f:f0:
         de:6e:02:73:37:eb:d8:3a:b4:37:96:31:4e:a8:ce:29

step happily verifies that the end-entity certificate is signed by the CA, even
when name constraints are present:

:; step certificate verify ee-cert.pem \
    --roots=cert.pem \
    --host=linkerd-identity.linkerd.serviceaccount.identity.linkerd.foo.bar

If we generate an end entity certificate for an alternate suffix (idk.lol),
step properly fails to validate the certificate.

:; step certificate create linkerd-identity.linkerd.serviceaccount.identity.linkerd.idk.lol ee-cert.pem ee-key.pem --san=linkerd-identity.linkerd.serviceaccount.identity.linkerd.idk.lol --ca-key=key.pem --ca=cert.pem --not-after=24h --no-password --insecure
...
:; step certificate verify ee-cert.pem \
    --roots=cert.pem \
    --host=linkerd-identity.linkerd.serviceaccount.identity.linkerd.foo.bah
failed to verify certificate: x509: a root or intermediate certificate is not authorized to sign for this name: DNS name "linkerd-identity.linkerd.serviceaccount.identity.linkerd.foo.bah" is not permitted by any constraint

Suggested next steps

Why doesn't the proxy validate its end entity certificates when name constraints are used?

To figure this out, I'd probably try to create a test (or repo) that attempts to
use webpki to validate these certificates. If this works in a standalone repo,
then we can try to figure out how Linkerd's usage differs from a simpler example
setup.

Note that it may be required to alter the format of the private key with, e.g.:

:; openssl pkcs8 -topk8 -inform PEM -outform PEM -nocrypt -in key.pem -out pkcs8.key  

[olix0r]

It seems that there is a known bug in webpki and PR to fix it that's been open without review for >1 year:

• Name constraints reject too many things (www.pki.bayern.de doesn't seem to work) briansmith/webpki#20
• UnknownIssuer with cert from private CA rustls/rustls#841
• Fix name constraints check briansmith/webpki#226

Given that we're already using a fork of webpki, we could consider merging some variation of the proposed fix...

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[wmorgan]

Latest state, AIUI:

1. We need cert-manager to fix Specify Name Constraints in CA Certificate cert-manager/cert-manager#3655
2. We need to move from our own linkerd/webpki fork to rustls/webpki, or to at least apply the fix for webpki does not validate certificates with name constraints rustls/webpki#3 to our fork

If you're watching this issue, a thumbs-up on cert-manager/cert-manager#3655 may help the cert-manager team prioritize that issue.

[whiskeysierra]

We need cert-manager to fix cert-manager/cert-manager#3655

Correct me if I'm wrong, but my understanding is that using cert-manager to automatically rotate the control plane TLS certificates is and always has been an optional thing. So it's more of a nice to have for those who intend to use name constraints, rather than a hard requirement, is it?

We need to move from our own linkerd/webpki fork to rustls/webpki, or to at least apply the fix for rustls/webpki#3 to our fork

I wanted to assess both of these options but I'm missing a lot of background here.
Primarily the motivation and reason behind linkerd/webpki.
Why did you decide to fork it?
Am I correct in assuming that rustls/webpki wasn't a thing back then?
What's the preferred solution here, patching your fork or migrating over to rustls/webpki?
Would you be willing to accept either one of those as a contribution?

[hawkw]

@whiskeysierra

I wanted to assess both of these options but I'm missing a lot of background here.
Primarily the motivation and reason behind linkerd/webpki.
Why did you decide to fork it?
Am I correct in assuming that rustls/webpki wasn't a thing back then?

The linkerd/webpki fork was motivated primarily by needing to include the change linkerd/webpki@1c7b989, which adds APIs for iterating over all the DNS names in a certificate. We were hoping that upstream would merge PR briansmith/webpki#103 in the future and that the fork would be a temporary measure. At the time, the rustls/webpki fork did not yet exist.

What's the preferred solution here, patching your fork or migrating over to rustls/webpki?

The rustls/webpki fork now contains a similar API to the one that we initially forked linkerd/webpki to add in the first place (rustls/webpki#42). Since the linkerd/webpki fork was really only ever intended as a temporary measure, I think depending on the rustls/webpki fork would definitely be preferable, as it's supported by a number of contributors from the Rustls community. The primary reason we haven't adopted this yet is just that it may require some effort to track other API changes from when linkerd/webpki was forked.

Would you be willing to accept either one of those as a contribution?

If you're interested in changing the Linkerd proxy to depend on rustls/webpki rather than linkerd/webpki, I would love to review a PR making that change!

[MrFreezeex]

Hi @hawkw, If I understand correctly this should be fixed with the latest edge release right?

[hawkw]

@MrFreezeex

Hi @hawkw, If I understand correctly this should be fixed with the latest edge release right?

Since we've switched to a version of webpki that supports name constraints, it ought to work --- we'd love to hear back if you end up trying it.

Do note, though, that the cert-manager support for name constraints still doesn't exist (see cert-manager/cert-manager#3655). So, name constraints will probably work on the Linkerd side, but if you're using cert-manager, I don't believe you'll have a way to specify them. This isn't an issue if you're using CAs other than cert-manager.

[whiskeysierra]

I tried to move the name constraint from the root CA to the issuer CA and I end up with the same error as OP:

identity: linkerd_proxy_identity_client::certify: Failed to obtain identity error=invalid peer certificate contents: invalid peer certificate: UnknownIssuer

[whiskeysierra]

I also checked the test suite of restls/webpki and they run all tests from the nameconstraints test suite from bettertls:

https://github.com/rustls/webpki/blob/a86cbb1de087727de926fa2836006305412d8b4e/tests/better_tls.rs

That test suite contains a whopping 9k+ tests:

cat bettertls.tests.json | jq '.suites.nameconstraints.testCases | length'
9491

I'd assume that all interesting cases are covered there.
Not that we're even attempting anything fancy here.

Given that rustls/webpki seems to pass all those tests, I'd suspect that linkerd-proxy's usage of that library somehow prevents nameconstraints from working correctly.

[hawkw]

Hmm, interesting! Thanks for looking into this @whiskeysierra, I guess we'll need to make additional changes in the proxy in order to support name constraints...

[hawkw]

Welp, I think I've figured out what's going on here. The peer certificate verification is being performed through rustls, rather than calling into rustls-webpki directly. The proxy currently depends on rustls v0.20.8, which...still depends on webpki, rather than using rustls-webpki.

So, we will need to update the proxy's rustls dependency to a newer version that actually uses the rustls-webpki fork.

[whiskeysierra]

Wow, that was fast. By the time I read your comment the PR was already merged 🤯

I'll give it another try once the next edge release is available.

[whiskeysierra]

I just tested it with the latest edge release and with a NON-matching name constraint (e.g. .foo.local) on the issuer certificate I now correctly see the linkerd control plane not even starting:

ERROR ThreadId(02) identity: linkerd_proxy_identity_client::certify: Failed to obtain identity error=invalid peer certificate: Other(NameConstraintViolation)

With a matching name constraint (.cluster.local) everything works.

What doesn't work though is having a name constraint on the root CA but not on the issuer CA. It does work with openssl, so I'd expect that to be the correct behavior then.

[hawkw]

I just tested it with the latest edge release and with a NON-matching name constraint (e.g. .foo.local) on the issuer certificate I now correctly see the linkerd control plane not even starting:

ERROR ThreadId(02) identity: linkerd_proxy_identity_client::certify: Failed to obtain identity error=invalid peer certificate: Other(NameConstraintViolation)

With a matching name constraint (.cluster.local) everything works.

Great, thanks for testing it out!

What doesn't work though is having a name constraint on the root CA but not on the issuer CA. It does work with openssl, so I'd expect that to be the correct behavior then.

Hmm, that's interesting. By "doesn't work", do you mean that the name constraint is ignored, or that linkerd doesn't come up even if the name constraint matches?

[whiskeysierra]

It's ignored.

…

On Sat, Sep 23, 2023, 06:39 Eliza Weisman ***@***.***> wrote: I just tested it with the latest edge release and with a NON-matching name constraint (e.g. .foo.local) on the issuer certificate I now correctly see the linkerd control plane not even starting: ERROR ThreadId(02) identity: linkerd_proxy_identity_client::certify: Failed to obtain identity error=invalid peer certificate: Other(NameConstraintViolation) With a matching name constraint (.cluster.local) everything works. Great, thanks for testing it out! What doesn't work though is having a name constraint on the root CA but not on the issuer CA. It does work with openssl, so I'd expect that to be the correct behavior then. Hmm, that's interesting. By "doesn't work", do you mean that the name constraint is ignored, or that linkerd doesn't come up even if the name constraint matches? — Reply to this email directly, view it on GitHub <#9299 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADI7HOKGCPU5OG2JDE457DX3ZRZPANCNFSM6AAAAAAQAZELHM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[whiskeysierra]

I did some more tests and it works as advertised now. The fact that name constraints from root CAs are ignored isn't that big of a thing, to be honest. Given that the root CA(s) should live outside of the cluster anyway, it's all good.

Makes me very happy to have this capability available to me now, because it makes zero-trust multicluster setups a lot more secure now. Thanks for making this happen!

[hawkw]

@whiskeysierra It does seem incorrect to me that the root CA's name constraint is not being checked, I'm trying to figure out why that would be happening. Can I get you to share the name constraint on the root cert in your trust chain, and what the intermediate issuer CA cert was?

[whiskeysierra]

I'm not sure where I went wrong last time, but I just tried - with a fresh set of CAs - to give you a full set of certificates. But now it works as expected.

If I use step to generate a root CA/trust anchor with this:

{
  "subject": "Root CA",
  "keyUsage": [
    "certSign",
    "crlSign"
  ],
  "basicConstraints": {
    "isCA": true,
    "maxPathLen": 10
  },
  "nameConstraints": {
    "critical": true,
    "permittedDNSDomains": [
      ".foo.local"
    ]
  }
}

then it correctly causes the control plane (cluster domain still set to cluster.local) to fail with:

identity: linkerd_proxy_identity_client::certify: Failed to obtain identity  │
│ error=invalid peer certificate: Other(NameConstraintViolation)

[hawkw]

I'm not sure where I went wrong last time, but I just tried - with a fresh set of CAs - to give you a full set of certificates. But now it works as expected.

Oh. Well, nevermind, then! Glad to hear everything works as expected!

I'm going to go ahead and close this issue, but please let us know if you have any other issues. If anything's not working, we can re-open this, or open more specific issues to track anything that comes up. Thanks for all your help testing this stuff out!

[wmorgan]

Now that we support this important capability, we will be adding integration tests to ensure the correct behavior is captured in all future releases.