fix: return a corruption error when chunk head out of sequence

[lentil1016]

No description provided.

[codesome]

Thanks for the PR. Can you brief about what you are trying to do exactly? Because what you mentioned in the title of the PR would not fix the the linked issues. (and out of sequence files is not a data corruption)

[codesome]

Also, this PR gave me a hint why #7753 was happening, thanks! When we error out in IterateAllChunks, we are not setting cdm.fileMaxtSet, while we should for every case. Do you think you can do that in this PR?

[codesome]

I have updated the PR description to exclude #5617 as that is unrelated to m-mapping

[lentil1016]

@codesome Hi, yesterday I meet some problem. The log tolds me that it meet some expection on loading head chunk with ref 6778345.

level=info ts=2020-08-24T02:39:28.116Z caller=head.go:641 component=tsdb msg="Replaying on-disk memory mappable chunks if any"
level=error ts=2020-08-24T02:39:32.539Z caller=head.go:646 component=tsdb msg="Loading on-disk chunks failed" err="iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 6778345"
level=info ts=2020-08-24T02:39:32.539Z caller=head.go:757 component=tsdb msg="Deleting mmapped chunk files"
level=info ts=2020-08-24T02:39:32.539Z caller=head.go:760 component=tsdb msg="Deletion of mmap chunk files failed, discarding chunk files completely" err="cannot handle error: iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 6778345"
level=info ts=2020-08-24T02:39:32.539Z caller=head.go:655 component=tsdb msg="On-disk memory mappable chunks replay completed" duration=4.423135929s

It seems that this chunk have an unexpected mint so it cause a return at here with an normal error, then at here, This error is a "cannot handle error", because it's not a CorruptionErr.

Every compact trying after then will failed. Even restart won't help, because this chunk is "cannot handle" and won't be removed. Data will become unvisible 2h by 2h.

level=error ts=2020-08-25T09:00:32.352Z caller=db.go:685 component=tsdb msg="compaction failed" err="reload blocks: head truncate failed: truncate chunks.HeadReadWriter: maxt of the files are not set"
level=info ts=2020-08-25T09:06:14.397Z caller=compact.go:495 component=tsdb msg="write block" mint=1598277600000 maxt=1598284800000 ulid=01EGJDKQT3TGPE2K6NVXWEV93P duration=4m42.041737936s
level=error ts=2020-08-25T09:06:23.632Z caller=db.go:685 component=tsdb msg="compaction failed" err="reload blocks: head truncate failed: truncate chunks.HeadReadWriter: maxt of the files are not set"
level=info ts=2020-08-25T09:11:44.949Z caller=compact.go:495 component=tsdb msg="write block" mint=1598284800000 maxt=1598292000000 ulid=01EGJDYEVX1EMW8M2BG6HJWA9G duration=4m21.303919073s
level=error ts=2020-08-25T09:11:52.854Z caller=db.go:685 component=tsdb msg="compaction failed" err="reload blocks: head truncate failed: truncate chunks.HeadReadWriter: maxt of the files are not set"
level=info ts=2020-08-25T09:17:17.827Z caller=compact.go:495 component=tsdb msg="write block" mint=1598292000000 maxt=1598299200000 ulid=01EGJE8GBPGND1YX2BX28G4WZK duration=4m24.973065568s
level=error ts=2020-08-25T09:17:26.128Z caller=db.go:685 component=tsdb msg="compaction failed" err="reload blocks: head truncate failed: truncate chunks.HeadReadWriter: maxt of the files are not set"
level=info ts=2020-08-25T09:22:29.737Z caller=compact.go:495 component=tsdb msg="write block" mint=1598299200000 maxt=1598306400000 ulid=01EGJEJNTH4718GQQB8A2TQEPB duration=4m3.6077139s
level=error ts=2020-08-25T09:22:37.816Z caller=db.go:685 component=tsdb msg="compaction failed" err="reload blocks: head truncate failed: truncate chunks.HeadReadWriter: maxt of the files are not set"

I'm not sure, but I believe this chunk is corrupted. And I think an iterator function should be able to return an corruptionErr to let the caller know that this chunk have a bad metadata and should be removed.

[codesome]

out of sequence m-mapped chunk for series ref 6778345

Hm, true, its better to delete m-mapped chunks from that point.

we are not setting cdm.fileMaxtSet, while we should for every case.

We should also take care of this

[codesome]

We should be just wrapping this error with corruption error and return and nothing else. You don't need additional argument for f.

[lentil1016]

Ops... sorry, my brainfart :p

[codesome]

cc @roidelapluie, it would be nice to get this into 2.21

[codesome]

LGTM! Thanks