reduce fluentd connection churn during high event-handler load

[fspmarshall]

This PR fixes an issue where the teleport event handler could cause excess CPU usage and connection reset errors in Fluentd when under load. Previously, the teleport event handler's Fluentd client would create unlimited outbound connections, but only ever retain a maximum of 2 idle connections. This lead to frequent creation and destruction of TLS connections when the event handler was trying to send many concurrent events to Fluentd. With these changes, the Fluentd client now both limits the peak connections, and allows all idle connections to persist up to 30s.

We've been aware for a while that the Fluentd client had a tendency toward connection churn, but it never caused any measurable issues in our performance tuning tests. It turns out this was likely due to how our performance testing were being performed. Our performance tuning is generally done with the intent to maximize event throughput for very massive teleport clusters (i.e. clusters with many tens of thousands of events per minute), and for that reason tend to be done with pretty beefy VMs running the exporter and Fluentd. It seems that for whatever reason, it's only when running on smaller more resource-constrained machines that the churn issue really comes to the fore, causing up to a 4x slowdown in overall event throughput.

changelog: fixed issue resulting in excess cpu usage and connection resets when teleport-event-handler is under moderate to high load.

[public-teleport-github-review-bot]

@fspmarshall See the table below for backport results.

Branch	Result
branch/v15	Failed
branch/v16	Failed
branch/v17	Failed