[CI] CorruptedFileIT.testCorruptFileThenSnapshotAndRestore failure

[romseygeek]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=sles/2435/console

After a file is corrupted, snapshot should throw an error, but it seems to succeed instead. Doesn't reproduce locally though.

consoleText.txt

[elasticmachine]

Pinging @elastic/es-distributed

[tlrx]

This test creates an index with 0 replica, merges disabled and a very high value for the flush translog size setting. Then it corrupts a file in a random primary shard (in this failure, it is _2.si), registers a repository and creates a snapshot.

When creating the snapshot, the test expects that the shard snapshot process loads the store metadata and then fails because one of the Lucene files is corrupted, failing the shard snapshot and marking the snapshot as PARTIAL.

In this test failure the snapshot completed as SUCCESS. I looked closely at it and I can't figure why it didn't fail when loading the store metadata. Of course it does not reproduce locally. The test correclty corrupted the file:

[INFO ][test                     ] Corrupting file --  flipping at position 68 from 0 to 1 file: _2.si
[INFO ][test                     ] Checksum before: [2483589937] after: [1016601414] checksum value after corruption: 2483589937] file: _2.si length: 387

I suspect a test bug, or maybe that the file was not part of the snapshot but it should have been. Merges are disabled and flushes are manually executed before corrupting the file.

I pushed 7915b5f to add more debug information, hopefully this error will appear again and we'll be able to grab the shard files and the snapshotted files.

[ywelsch]

@dnhatn With the changes that were made to CombinedDeletionPolicy, do you perhaps know if we can expect all unreferenced Lucene files to be cleaned up after a flush?

[dnhatn]

@ywelsch I have looked at the test.
If the latest value of the global checkpoint has not fsynced before a flush, we will have two commits; then if we corrupt some files which are referenced only by the previous commit, the snapshot of the latest commit will be SUCCESS instead of PARTIAL.

I will follow-up this test with @tlrx.

[ywelsch]

awesome!

[tlrx]

Good catch @ywelsch and @dnhatn. But what could make a file referenced by a commit point and not by the next one, when merges are disabled and the shard manually flushed?

[dnhatn]

@tlrx You're right. My assumption is not correct in this case.