Support multiple ProxyURLs in http config

[asuffield]

This is primarily intended for use with alertmanager, where
notify.Retrier will retry the http request, so if one proxy is not
responding then it will deliver alerts through the other. This avoids a
SPOF at the proxy for alert delivery.

I wish I had a way to write tests for this, but there's no test infra for http proxies and it's remarkably complicated. Sending this for review while I rig up a test-in-production to verify this does what I need.

[roidelapluie]

Thank you for this pull request.

I am against this change and this kind of magic. If we should support multiple proxy URL's, I would expect to just pass them as a list to the underlying go stdlib. It is not Prometheus' job to decide or implement that kind of retry mechanisms. This is a huge can of worms and I do not want to open it.

[asuffield]

I am against this change and this kind of magic. If we should support multiple proxy URL's, I would expect to just pass them as a list to the underlying go stdlib. It is not Prometheus' job to decide or implement that kind of retry mechanisms. This is a huge can of worms and I do not want to open it.

This is just passing them to the underlying go stdlib, which takes a function rather than a list: https://pkg.go.dev/net/http#Transport.

Prometheus does in fact implement this kind of retry mechanism, it's over here: https://github.com/prometheus/alertmanager/blob/main/notify/util.go#L158-L196

[roidelapluie]

This does not address operability issue. It is hard to know which proxy was used.

as I said, this is a huge can of worms, especially in the alertmanager, when options can be set globally and locally.

To remove the proxy spoc, one would point each alertmanager instance to a different proxy.

[asuffield]

This does not address operability issue. It is hard to know which proxy was used.

as I said, this is a huge can of worms, especially in the alertmanager, when options can be set globally and locally.

I don't think either of these statements are incorrect, they're just statements about the current alertmanager code, not this PR.

To remove the proxy spoc, one would point each alertmanager instance to a different proxy.

That's assuming we have the same number of proxies and alertmanagers, which could be done by deploying N alertmanager instances per location for N proxies, but is a lot of added complexity compared to just hooking up the standard library feature in the config, like this PR does.