Unclosed network connections

[joshbenner]

I was running latest (tag 12) today, and noticed a lot (over 10k) of open connections on my node. I restarted the rancher exporter, and all the connections went away. Connections are not being closed.

[joshbenner]

I'm watching netstat inside the prometheus-rancher-exporter, and every time the 3 JSON requests are made, the number of ESTABLISHED connections increment by 3.

[joshbenner]

Took a look at the code, and I think I've tracked the issue to https://github.com/infinityworksltd/prometheus-rancher-exporter/blob/master/utils/utils.go#L24

The HTTP Client caches connections for reuse, and we're creating a new client each time we call any of the GetJson functions. Ideally, a single client should be created for the application (single client is safe to use across multiple goroutines). Alternately, one client per exporter. Finally... we could create/destroy each invocation.

See: https://golang.org/pkg/net/http/#Client

[Rucknar]

Thanks @joshbenner, again well spotted and good write up.

Just having a look at the options now and doing some testing, though i appreciate creating a long-running client would have the least overhead, i'm worried how that would perform over extended periods of time. I'll see what works well locally.

[Rucknar]

Oddly, despite hitting it with apache bench... i'm not seeing the same issues. If i open up 1k connections, they are all recovered within a minute. I'm wondering if testing locally is distorting the issue. I've just pushed up an update, and a new image version (13). Any chance you could test with version 13?

[joshbenner]

I confirm this addresses the issue in both environments where I was seeing the open connection behavior.

[metalmatze]

That might explain the exporter crashing every few days for me. Didn't dig deeper, but it sounds reasonable. 👍

[Rucknar]

Had the same thought @metalmatze. I've done some testing and tag 13 seems to behave okay on my test setup too. Pull request raised for those of you using this as part of the catalog entry: rancher/community-catalog#382

[Rucknar]

Closing this one, feel free to re-open if you think it might still be an issue.