TSDB unloading

[pstibrany]

In some situations it is possible that ingester stops receiving data for a specific user. For example when user stops sending the data, but also when using shuffle-sharding and ingester stops being part of the shard for given user.

When no series are appended to TSDB for some time (defaults to 1h), Cortex will flush the TSDB head – which creates block and uploads it – but TSDB is still open. This consumes memory.

Ideally Cortex would close such TSDB and release the resources. However there are some problems:

• If TSDB needs to be opened again, this will create a problem with metrics.
• When querying data, ingester will return no results for given user. This may cause invalid results reported back, especially for "fresh" data (within "query store after" duration).
• When ingester restarts, it reopens all TSDBs it finds on the disk.
• More?

Problem with metrics is solveable by using multiple registries per user, and keeping old one around for aggregations to work properly.

Problem with querying can be solved by closing TSDB only after query-store-after period since last TSDB update. Ingester would need to know this value (or have similar flag).

Problem with restarts can be solved by simply deleting local TSDB for given user, when it TSDB is closed. This is safe as long as all blocks have been fully shipped to the long-term storage.

[pracucci]

When querying data, ingester will return no results for given user. This may cause invalid results reported back, especially for "fresh" data (within "query store after" duration).

Given we already close idle TSDBs, isn't this already an issue?

[pracucci]

Problem with restarts can be solved by simply deleting local TSDB for given user, when it TSDB is closed. This is safe as long as all blocks have been fully shipped to the long-term storage.

True it's safe. Only thing worth to note is that wouldn't honor the "retention period" (which is also used as a safety net).

[pstibrany]

When querying data, ingester will return no results for given user. This may cause invalid results reported back, especially for "fresh" data (within "query store after" duration).

Given we already close idle TSDBs, isn't this already an issue?

We don't close idle TSDBs, we only flush them but keep them open.

[pracucci]

We don't close idle TSDBs, we only flush them but keep them open.

Right. An option could be continue to keep them open until the "query ingesters within" period expires.

[pstibrany]

True it's safe. Only thing worth to note is that wouldn't honor the "retention period" (which is also used as a safety net).

It is safety net against not shipping. Another possibility would be to keep “closed” file in TSDB, with timestamp in it. We could then delete TSDB after retention period has passed. It would also allow us to skip opening such TSDB on restart or reopen for more incoming series.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[pstibrany]

Done in #3491