Fail to load block when meta.json is empty #8281
Description
Thanos, Prometheus and Golang version used:
Thanos: 0.38.0
Prometheus: 3.3.0
Golang: 1.23
Object Storage Provider:
What happened:
One of my hosts which is running Cortex ingester with Thanos crashed during head compaction resulting in an empty meta.json for a block that was created successfully. When reloading the blocks, the shipper fails to ship the created block because of this empty meta.json.
What you expected to happen:
The content of meta.json should be synced to the file on an atomic write
How to reproduce it (as minimally and precisely as possible):
This is difficult to reproduce as it only happens when there is a host crash and the kernel has not synced the contents of the meta.json.tmp before renaming it to meta.json
Full logs to relevant components:
ts=2024-12-30T15:08:32.577120173Z caller=repair.go:52 level=error ... msg="failed to read meta.json for a block during repair process; skipping" dir=<block_path> err="unexpected end of JSON input"
Anything else we need to know:
A very similar issue was reported in Prometheus previously: prometheus/prometheus#4058 and was resolved by forcing a file sync before the rename operation