Use a non-global metrics registry in Teleport

[hugoShaka]

This PR adds a new non-global per-process metrics registry in Teleport.

Using the global registry and global metrics causes conflicts in tests as we are starting multiple Teleport processes and/or other non-teleport processes (tbot, the operator, ...).

Having a new per-process metrics registry will allow Teleport services to register metrics scoped to their Teleport process. This will reduce the conflicts happening in tests.

To ensure backward compatibility, the Teleport metrics server serves both the process-scoped registry and the global registry.

Required for the autoupdate controller metrics PR.

[hugoShaka]

I didn't want to add a metrics RFD, but it would be good to start using the process registry instead of the global one for the next features we build/metrics we add.

[codingllama]

Looks good to me, but I'll let the experts approve first.

[codingllama]

If conflicting metrics are registered I assume they'll be dropped, but unaffected metrics will keep working. Do you know if that's correct?

[hugoShaka]

If conflicting metrics are registered I assume they'll be dropped

Currently, registration conflicts in the global registry can cause:

• hard failure / error returned
• panics
• silent failure (metric does not get registered and we don't know about it)

Adding a local registry will not change the failure modes in case of conflict in the same registry. However, we are adding a new failure mode: metrics conflicting between the local and global registry. In this case, the global will prevail (I did this for backward compatibility reasons as everything is using the global registry today) the local registry will take precedence.

As we start using the local registry more, we might create such hard to detect conflicts. The situation is not strictly worse than today (we already have some racy metric registration with silent failure going on 😬). To ensure no conflict happen we can prefix new metrics by wrapping the registry when passing it to the service.

I think we would benefit from metrics guideline, setting the teleport component in the metric subsystem would reduce the probability of conflict.

[codingllama]

Thanks, very informative.

[codingllama]

Thanks, very informative.

[rosstimothy]

Is it advantageous to move all of our current global metrics to the local registry? If so what kind of migration strategy should we have to eliminate global metrics?

[hugoShaka]

Is it advantageous to move all of our current global metrics to the local registry?

I think so because we will:

• stop having conflicts between tbot, teleport and other programs when running in the same test
• start having accurate metrics when running multiple teleport components together (e.g. in tests, or embedded tbot)
• stop picking up random metrics declared by dependencies we have

If so what kind of migration strategy should we have to eliminate global metrics?

I've not thought this yet, but by supporting both we can take our time with the transition. I'd like to get a few new metrics using the local registry before chosing a recommended pattern. Once we know how we want metrics to be declared and collected, we can write a short metrics RFD and start passing the local registry to the different services.

Migrating might not be trivial because we are heavily relying on package-scoped metrics and global registries. We will need to:

• propagate the registerer from the main process to every service registering metrics
• start putting metrics in structs instead of a package-scoped var
• get rid of the sync.Once and other hacks in place currently avoiding double-registration

I think tbot is a very good starting point because of its limited scope, the conflicts caused by embeddedtbot, and the conflicts it causes in integration tests.

[public-teleport-github-review-bot]

@hugoShaka See the table below for backport results.

Branch	Result
branch/v15	Failed
branch/v16	Failed
branch/v17	Failed