Ensure queries return correctly during rolling upgrades of stateful cluster with RF 3 and only 3 nodes. (#2503) (#2512)

* Use a real ring with mock KV when testing distributor.

This is to teast out errors in the replication logic.

Signed-off-by: Tom Wilkie <[email protected]>

* Extend distributor test to cover the case RF=3 with 2 ingesters.

Signed-off-by: Tom Wilkie <[email protected]>

* Ensure ring correctly calculates the number of allowed failures when RF=3 and #ingesters=2.

Signed-off-by: Tom Wilkie <[email protected]>

* Add changelog and review feedback.

Signed-off-by: Tom Wilkie <[email protected]>

* Refactor some distributor tests to try and get them to pass.

Signed-off-by: Tom Wilkie <[email protected]>

* Speed up tests but polling more frequently.

Signed-off-by: Tom Wilkie <[email protected]>

* Fix same bug on the write path.

Signed-off-by: Tom Wilkie <[email protected]>

* Tidy up the distributor tests.

Signed-off-by: Tom Wilkie <[email protected]>

* Make test correctly handle RF3 and 2 ingesters - previously was succeeding when it shouldn't

Signed-off-by: Tom Wilkie <[email protected]>

* Update pkg/ring/ring.go

Co-Authored-By: Jacob Lisi <[email protected]>
Signed-off-by: Tom Wilkie <[email protected]>

Co-authored-by: Jacob Lisi <[email protected]>
Signed-off-by: Goutham Veeramachaneni <[email protected]>

Co-authored-by: Tom Wilkie <[email protected]>
Co-authored-by: Jacob Lisi <[email protected]>

Author: 3 people

CHANGELOG.md

@@ -101,6 +101,7 @@ This is the first major release of Cortex. We made a lot of **breaking changes**
 * [BUGFIX] Fixed bug in updating last element of FIFO cache. #2270
 * [BUGFIX] Register the metrics of the WAL. #2295
 * [BUXFIX] Experimental TSDB: fixed error handling when ingesting out of bound samples. #2342
+* [BUGFIX] Fix gaps when querying ingesters with replication factor = 3 and 2 ingesters in the cluster. #2503
 
 ### config file breaking changes