Add read-path zone aware replication support

[mattmendick]

Zone aware replication is able to withstand zone reliability issues and help insulate from dataloss. As I understand it, Cortex will still accept writes during this scenario, but reads are not guaranteed to be correct since there is no zone awareness on the read path. I feel adding this would allow Cortex to be more resilient to datacenter outages in that it would not affect users who are querying it.

[pracucci]

After some off-line discussion, the issue could be rephrased as:

Right now the read path reads from all ingesters and allow one failure. In a scenario where we're zone-aware, with 3 zones and 3 replicas in each zone, we can actually allow all the reads from a single zone to fail and still succeed the read. Need to build out the algorithm to support this.

[woodsaj]

Cortex currently supports zone awareness in the distributor ring. If a whole zone is unavailable writes will still succeed. However, reads fail when there are more than (ReplicationFactor/2) ingesters unavailable. So with a replication factor of 3, reads will fail if more than 1 ingester is unavailable.

The reason for this is that on the write path, we query the ring for the list of ingesters for a specific key: https://github.com/cortexproject/cortex/blob/master/pkg/ring/ring.go#L204 , so writes will succeed if the a quorum of ingesters for that specific key are available.

On the read path, we query for all ingesters in the ring: https://github.com/cortexproject/cortex/blob/master/pkg/ring/ring.go#L259. It would be computationally expensive to check that there is a quorum of ingesters available for every token in the ring at query time. If more than ReplicationFactor/2 ingesters have failed, it is possible that for a single token there are insufficient ingesters, so to air on the side of caution, Cortex doesnt even try to execute the query.

[pstibrany]

To be fixed by #3414.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[jtlisi]

Fixed by #3414