Configure Coder's Prometheus instance to scrape registry-v2

[bcpeinhardt]

Configure Coder's Prometheus instance to scrape registry-v2

https://github.com/coder/dogfood/issues/105

[johnstcn]

Another option is to use Google's managed Prometheus service: https://cloud.google.com/stackdriver/docs/managed-prometheus/cloudrun-sidecar

[bcpeinhardt]

Closing this as complete since we have prometheus scraping the new registry 😎

Scratch that re-opening, I got turned around this still needs doing.

[johnstcn]

The main questions right now are:

1. what should scrape
2. where should the data live

Our options are something like:

1. Use Google-managed prometheus but accept that this will inflate the runtime metrics for both dev and prod. Extremely low-effort.
2. Run a separate Prometheus agent somewhere (potentially in cloud run itself) and send metrics somewhere else. This is very hand-wavy and could take many forms.

[bcpeinhardt]

To start with I've gone the sidecar route. It increased the container memory usage but not terribly, and all the metrics. logging, and alerting are kind of in one place. Unless it somehow changes the performance in a significant way I think we should stick with it.