Enable HTTP caching for Kubernetes OIDC joining

[timothyb89]

This enables HTTP caching for the Kubernetes OIDC join subtype.

This adds a new ValidateTokenWithClient helper that can be used to provide a custom client for anything currently using our shared OIDC validation helper. Clients just need to swap in a caching *http.Client; this PR only updates the Kubernetes OIDC join method for backporting purposes, but it should be straightforward to enable caching on other join methods in the future.

This relies on caching at the HTTP client layer as we otherwise don't have any ability to cache responses without significant tweaks to our OIDC library. This makes use of gregjones/httpcache which is already in our dependency tree; see this thread for some additional library selection notes.

changelog: Cache OIDC responses for Kubernetes OIDC joining

[timothyb89]

Some library usage notes:

• gregjones/httpcache is deprecated and archived, however...
• We already include this library via the Kubernetes client library
• It otherwise appears to be fit for purpose, and no other similarly mature libraries exist (bartventer/httpcache exists but is much less mature)
• It is MIT licensed

I think we can switch to an alternative if/when we find a compelling reason to do so - it should just mean swapping out the implementation we return in NewCachingHTTPClient().

[tigrato]

Do we need to use a third party library for caching jwks?

[timothyb89]

I mean, we certainly could build everything ourselves, but we'd need to rewrite a meaningful portion of zitadel/oidc to support caching natively, and then still build our own RFC 9111-compliant HTTP client anyway since the only way for providers to provide cache hints is through standard HTTP caching headers. We could disregard cache hints and just set a modest value, but then we're still stuck making fairly sweeping changes to our OIDC library.

From what I can tell, just using a caching *http.Client achieves fundamentally the same thing with very little effort required on our end. That said, I'd be very happy to hear suggestions if there's simpler alternatives I've missed!

[tigrato]

My concern is that the memory cache is unbounded, and entries are only cleared upon retries. Since the cache key is based on the URL, if we ever introduce dynamic fields in the JWKS endpoint-such as a timestamp-or encounter an unusually large number of OIDC tokens, memory usage could grow significantly.

[timothyb89]

Hmm, that's a fair point. We could use a LRU cache backend instead, https://github.com/die-net/lrucache is linked directly from gregjones/httpcache - though similar maintenance caveats apply - or I could build one off github.com/hashicorp/golang-lru which we already include.

That said, I don't expect many problems here in practice:

• OIDC endpoints themselves are user specified and cannot change regularly
• JWKS URIs technically could change but zero providers I'm currently aware of do so.

I do agree that an LRU cache of some sort is much better but this trivial implementation is unlikely to actually grow unbounded in practice. In any case, I'll look into the best path for evicting entries.

[timothyb89]

Closing this in favor of a different approach in #57862: caching at the HTTP layer doesn't have desirable "refresh on unknown key ID" behavior and the library doesn't provide any sane way of building that beyond what it already provides.