[v18] Add a caching OIDC validator for use with Kubernetes OIDC (#57862)

[timothyb89]

Backport of #57862 for branch/v18

changelog: Kubernetes OIDC responses are now cached to improve performance and reliability when joining bots and nodes

---

• Add a caching OIDC validator for use with Kubernetes OIDC

This adds a generic caching OIDC validator which is initially used to cache Kubernetes OIDC requests. This takes a simpler approach to the one explored in #57789 and explicitly caches the two resources in question:

• We cache the OIDC discovery configuration for an hour
• We cache the JWKS keyset for 24 hours, and additionally allow the library to cache internally. It currently has no internal method for invalidating old keys, so we purge the oidc.KeySet regularly. However, upon encountering an unknown kid, the library will attempt to fetch new keys on its own.

This validator is designed to be generic and should be easy to apply to other users of our existing OIDC helper.

• Manage validator instances by issuer; use in Kubernetes validator

This adds an indirection layer to automatically manage validator instances as cached data is specific to a particular (issuer, audience) combination.

Additionally, this swaps the new caching validator in in place of the standard non-caching validator.

• Fix comments and add missing mutex lock

• Add tests for the caching validator

• Fix lints

• Add logging

• Fix imports

• Code review suggestions: atomic expiry, misc fixes

This changes validatorExpires to be an atomic int64, which removes a locks from Expires() / IsStale() and ensures an ongoing pruning won't block ValidateToken().

• Replace validator cache and pruning with utils.FnCache

This swaps the hand-rolled cache for validators with utils.FnCache, since it's more robust and its TTL mechanism works properly for our needs.

• Remove commented out test code

• Fix build errors and enable ReloadOnErr

• Code review suggestions