Blocks should never be deleted by ingester's TSDB retention until shipped to the storage

[pracucci]

When running Cortex blocks storage, the TSDB retention in the ingester is configured to -experimental.tsdb.retention-period. TSDB will then delete any block whose MaxTime is older then that, regardless the block has been successfully shipped to the storage (TSDB has no meaning of the shipper).

There are at least two scenarios where this could happen:

1. Ingester is failing to upload blocks to the storage (eg. auth issue, networking issue, ...)
2. Ingester's TSDB Head compaction is failing (eg. due to an in-memory corruption), ingester is restarted after several hours (more than retention-period), WAL gets replayed, blocks compacted but they're immediately deleted by retention, before they're successfully uploaded to the storage

[pracucci]

An option would be adding another delegate function to TSDB, to have TSDB requesting the delegate if a block is deleteable, in addition to TSDB logic. This way we could forbid the deletion of a block until it's not shipped to the storage.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[pracucci]

Still valid and should be done asap.