Query Frontend should use Round Robin to Choose the Next Tenant to Service

[joe-elliott]

Currently the query frontend randomly chooses a tenant queue and returns it when a querier requests a new query. This can cause exaggerated long tail latency if a tenant is simply repeatedly not chosen.

I propose updating the query frontend to use a simple round robin algorithm to choose the next tenant to increase fairness and eliminate the chance of an "unlucky" tenant.

[cyriltovena]

I read your proposals in #2528 and I think we should go ahead with this simple change, this something that we will benefits right now in Loki where parallelism is higher.

👍

[pracucci]

I also see this an improvement over the current random-based algorithm 👍

[joe-elliott]

I will address this next. Thank you for the feedback.