reverseproxy: Track dynamic upstreams, enable passive healthchecking

[francislavoie]

I was reviewing #7517 and thinking about how it would handle dynamic upstreams. I was thinking there might be a leak with dynamic upstreams not getting cleaned up since it doesn't happen in the proxy handler's Cleanup(), but that's not the case because we actually drop the upstreams immediately after every request. This also means we don't have support for passive health checks for dynamic upstreams.

This adds separate tracking for dynamic upstreams, which enables passive health checks to work (but not active health checks still, for obvious reasons) and the admin API endpoint can show upstream health as well (with inactive ones getting removed after an hour). This "one hour" lifetime is somewhat arbitrary, but I think it should serve most users well enough, ensures long-term leaks shouldn't happen while making most real-time usecases (checking health and such) still work fine consistently.

Also added tests for dynamic upstream tracking, the admin endpoint, and passive health checks.

AI's Summary of changes

Problem

Dynamic upstreams (from DynamicUpstreams.GetUpstreams) were provisioned into the same UsagePool as static upstreams, but with a defer hosts.Delete(...) at the end of each proxy loop iteration. This caused two bugs:

1. Health state loss — when there was no concurrent traffic, the ref count hit 0 after each request, deleting the *Host entry. The next request created a fresh one, wiping passive fail counts. A backend could fail repeatedly and never be marked down.
2. Churn — every request unnecessarily allocated, stored, and deleted pool entries.

Solution: separate tracking for dynamic hosts

hosts.go

• Added fillDynamicHost() on *Upstream — stores the host in a plain map[string]dynamicHostEntry (protected by sync.RWMutex) instead of the reference-counted UsagePool. Each call refreshes the lastSeen timestamp.
• Added a background cleanup goroutine (started exactly once via sync.Once) that sweeps the map every 5 minutes and evicts entries idle for more than an hour.

reverseproxy.go

• provisionUpstream gains a dynamic bool parameter, routing to fillDynamicHost() vs fillHost() accordingly.
• Static upstreams at Provision time: provisionUpstream(u, false) — unchanged behaviour.
• Dynamic upstreams per-request: provisionUpstream(dUp, true) — no more defer hosts.Delete(...).

admin.go

• The /reverse_proxy/upstreams API endpoint now ranges over both hosts (static) and dynamicHosts (dynamic) so the admin view is complete.

Why passive health checks now work

countFailure increments Host.countFail(1) and schedules a goroutine to decrement it after FailDuration. Since *Host now persists across requests in dynamicHosts, those counts survive between sequential requests. Healthy() reads Host.Fails() < healthCheckPolicy.MaxFails — both of which are correctly set on each fresh *Upstream struct from provisionUpstream — so a dynamic backend that has accumulated enough failures will correctly be skipped by the load balancer.

Assistance Disclosure

Used Github Copilot + Claude Sonnet 4.6

[mholt]

Thanks Francis, this is cool. Looks a little complicated but is probably a pretty decent way to do things. Let's try it?