Proposal: make ref cache labels share storage with TSDB

[bboreham]

Looking at the Go heap usage of blocks ingesters, I see a lot of space taken up with label/values, both the strings and the slices.
I see about the same from LoadWAL() created inside Decoder.Series() as I do from v2Push() created inside FromLabelAdaptersToLabelsWithCopy().
For a brand-new ingester that hasn't loaded WAL, I only see the latter.

@pracucci said "In a brand new ingester they may not be duplicated, while in a long running ingester they are, because we don't have access to TSDB labels to reference them and pick up that string for the ref cache."

[pracucci]

I need to take a better look because, if I remember correct label strings are shared between ref cache and TSDB for new label names/values received.

The problem is that when you load the WAL, TSDB loads (among other things) the label strings before we start accepting remote writes.

When a remote write lands, the ref cache is empty, so the label strings are copied to add an entry to the ref cache too and here we have the duplicates because these strings are actually already in TSDB but we have no way to get their reference to use the same string when adding the entry to the ref cache.

Copied from Slack thread.

[sreis]

Sorry to barge in on the conversation.

After digging into why ingesters consumed so much memory after a rollout restart, I just found this issue. The go heap usage I got is consistent with @bboreham's analysis.

The memory usage never goes back to the previous level. Is this expected? Are there any workarounds recommended to handle this memory usage issue during an ingester rollout?

[bboreham]

This idea was implemented in #3951; you can try latest build of master, or wait for 1.9 release in a few weeks.