readonly cluster configs

[fspmarshall]

This PR aims to reduce excess CPU/memory usage caused by large numbers of concurrent loads of certain cluster configuration resources. Most notably, values such as auth preference that are loaded for basically any RBAC check performed by a teleport instance. This is typically an inconsequential cost, but on auth servers handling many thousands of concurrent requests, the resource consumption of constantly deserializing these values can be non-trivial.

This PR moves a number of hot paths over to using shared in-memory values rather than loading a separate copy per goroutine. In order to facilitate doing this sharing safely, a new package readonly has been added which provides readonly subsets of certain common cluster configuration interfaces as well as a basic ttl-cache that stores readonly copies in-memory.

Currently, the implementation of readonly.Cache is fairly tightly coupled with the needs of lib/authz and lib/auth/clusterconfig. These two packages were selected as the starting point/proof of concept for this idea. In the long run, we'll likely either want to have a family of different specialized readonly caches, or a way to toggle on and off which resources the cache is configured to handle (much like how the primary cache in lib/cache currently works).

changelog: reduced CPU usage in auth servers experiencing very high concurrent request load.

[github-actions]

The PR changelog entry failed validation: Changelog entry not found in the PR body. Please add a "no-changelog" label to the PR, or changelog lines starting with changelog: followed by the changelog entries for the PR.

[greedy52]

alternatively, the readonly interfaces could be put in api/types and let AuthPreference inherits it. Otherwise, it is easy to miss this interface when adding new functions to AuthPreference. Though even if you miss it, it won't break anything unless you really need the new function. I don't have a strong opinion on this though.

[fspmarshall]

I considered this, but opted against it for the time being mostly because once this is in api it has to obey major version compatibility. This is a somewhat experimental pattern. I don't want it leaking into api until we're confident that this is the right way to handle this kind of problem going forward.

[public-teleport-github-review-bot]

@fspmarshall See the table below for backport results.

Branch	Result
branch/v14	Failed
branch/v15	Failed
branch/v16	Create PR