pointSlicePool issues

[woodsaj]

To reduce allocations we use a pointsSlicePool when reading datapoints out of a chunk.
Chunks are read in GetTargetsLocal -> getSeries -> getSeriesFixed
GetTargetsLocal is called when a render request is received and also when a GetData request is received. The GetData call is made when a node does not own a required series and so must call out to a peer for it.

issue 1

The pointSlices used in a render request are returned to the pool after renderMetrics function has completed.
However, if the getTargetsLocal was called from a getData request, the points never get returned to the pool.

issue 2

In a metrictank cluster, the majority of points used on a render request are going to come from remote nodes. However, we are not using the pointsSlicePool when UnMarshalling the byte[] slice returned from the peer.

[woodsaj]

fixing issue 1 is easy. just put the datapoint slices back in the pool at the end of https://github.com/grafana/metrictank/blob/master/api/cluster.go#L194-L204

[shanson7]

I believe sync.Pools are also cleared out on every GC, so that could result in more allocations

[Dieterbe]

related: #958

[Dieterbe]

I believe sync.Pools are also cleared out on every GC

correct

[Dieterbe]

some more analysis needed but this issue has the potential to significantly reduce memory usage.

@woodsaj what brought you to open this issue? observing any particular issue?

[woodsaj]

I was looking over code after a production incident where an MT node suddenly allocated 20+GB of memory. I was just trying to understand the code paths for handling queries and noticed the issues with the pointSlicePool

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Dieterbe]

Issue 1 will be fixed by #1924
Issue 2 has been fixed by #1921