Close TSDB and delete local data when TSDB is idle for a long time. #3491
Conversation
pkg/ingester/ingester_v2.go
Outdated
There was a problem hiding this comment.
I just realised this check has a problem: the lastUpdate is not correct after an ingester restart. An option could be checking the timestamp against max(lastUpdate, TSDB max time).
There was a problem hiding this comment.
When TSDB is opened, we set lastUpdate to current time to avoid premature idle flush and close. This will only be a problem if ingesters never stay up for longer than configured idle timeout.
There was a problem hiding this comment.
Instead of current time, we can set it to last sample time from TSDB when opening, as you suggest.
There was a problem hiding this comment.
When TSDB is opened, we set lastUpdate to current time to avoid premature idle flush and close.
Then we're protected 👍
Instead of current time, we can set it to last sample time from TSDB when opening, as you suggest.
Could make sense.
pkg/ingester/ingester_v2.go
Outdated
There was a problem hiding this comment.
I think would be great to have a couple of metrics (counters) on succeeded and failed closing idle TSDBs. WDYT?
There was a problem hiding this comment.
I've added metrics for number of idle-checks, and various check results. PTAL
Signed-off-by: Peter Štibraný <[email protected]>
Signed-off-by: Peter Štibraný <[email protected]>
Signed-off-by: Peter Štibraný <[email protected]>
Signed-off-by: Peter Štibraný <[email protected]>
|
Thanks @pstibrany for addressing my comments. LGTM 👍 |
4 participants
What this PR does: This PR adds ability for ingester to close idle TSDB and delete local data when TSDB is idle for a long time (no data is appended to it). Closing/deletion only happens if head is empty (possibly thanks to previous force-compaction due to being idle), and all blocks were shipped.
This PR also takes care of hiding user metrics, while keeping the registry to avoid breaking the counters over all users.Support for metrics removal has been extracted to separate PR, #3511.Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]