Send Prometheus metrics to Grafana Cloud

[ndr-ds]

Motivation

We're migrating our internal validators to send data to Grafana Cloud.

Proposal

Starting with the local setup. This was useful to test the required changes that we'll likely need for the actual internal validator setup. We probably won't want to send metrics to Grafana Cloud from local runs all the time (specially for cost reasons), so this is disabled by default.

Grafana Cloud credentials are stored in a secret within GCP's Secret Manager. Since Grafana Cloud will be for our internal validators, then I figured storing the secret on GCP made sense.

Test Plan

Ran locally with linera net up --kubernetes, and saw the metrics in our Grafana Cloud instance:

Release Plan

If we want to start sending metrics to Grafana Cloud from our devnet/testnet, then:

• These changes should be backported to the latest devnet branch
• These changes should be backported to the latest testnet branch, then

[ndr-ds]

• Send Prometheus metrics to Grafana Cloud #2994 👈 (View in Graphite)
• Avoid build on linera net up --kubernetes #2986
• RemoteNet -> ExistingNet #2998 : 1 other dependent PR (#2999 )
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[christos-h]

Thanks for the cleanup.

I'm a bit confused on two things:

1. Do we not still want the internal Grafana which is validator specific? In which case why is the dashboard disabled? If you don't want the dashboard why not remove Grafana entirely?
2. The reference to the GCP secret should not be hard-coded in linera-protocol but instead be parameterised and injected by the caller in linera-infra (probably lineractl).

[ndr-ds]

1. The way I did it, either you do Grafana Cloud (writeToGrafanaCloud set to true), or you do the same existing setup we have right now. With Grafana Cloud, we don't need the Grafana pod, or the dashboards or any of that. So the idea is that for internal validators we run it with writeToGrafanaCloud set to true, and for external validators we run it with writeToGrafanaCloud set to false.
2. Ok, sure, I can do that. I thought it was fine since you need to be authenticated with gcloud for the secret to be successfully fetched 🤔