Ingester successfully startup even if fail to open an existing TSDB

[pracucci]

Describe the bug
When an ingester starts, it loads all existing TSDBs before joining the ring. If it fails to load 1+ of such TSDBs, it will log an error and continue to startup and join the ring:

level=info ts=2020-10-15T00:49:30.017219611Z caller=ingester_v2.go:1056 msg="opening existing TSDBs"
level=error ts=2020-10-15T00:49:30.062627028Z caller=ingester_v2.go:1103 msg="unable to open user TSDB" err="failed to open TSDB: /data/tsdb/REDACTED: mmap files, file: /data/tsdb/REDACTED/chunks_head/000040: mmap: invalid argument" user=REDACTED
level=info ts=2020-10-15T00:49:30.06268577Z caller=ingester_v2.go:1122 msg="successfully opened existing TSDBs"

The subsequent effect is that whenever the ingester receives a write request, it tries to load the TSDB for every single request. This will shortly lead to an OOM.

Expected behavior
The ingester should better handle the case a TSDB fail to open at startup. I believe the easiest and probably safest solution would be exit the ingester with an error if any TSDB fails to startup.

Storage Engine

• Blocks
• Chunks

[pracucci]

Closed by #3354.