Do not consider PENDING state as healthy

[pracucci]

What this PR does:
While working on the PR #1818 I've realized that an ingester is considered healthy for read operations while PENDING but not while JOINING. From my perspective, an ingester shouldn't be considered healthy while PENDING, so I'm suggesting to:

1. Do not consider PENDING state as healthy
2. Explicitly enumerate healthy ingester states for the read path

In the read path, there are some API endpoints like LabelNames and LabelValuesForLabelName for which we query all healthy ingesters (see distributor.forAllIngesters()). A PENDING ingester shouldn't hold any data yet, so shouldn't be required to be hit.

A comment I've received in a previous discussion on this topic is that this change may introduce quorum issues during ingesters rollout. However, I can't see a real difference compared to when the ingester switches from PENDING to JOINING, considered that we already consider the JOINING state as unhealthy.

Which issue(s) this PR fixes:
No issue

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[pstibrany]

Personally, I would find it easier to read IsHealthy method, if "true" cases were more explicit. As it is now, each time I read it, I find it confusing. :-(

[gouthamve]

I I think this is better yeah.

[csmarchbanks]

I am concerned that this could break queries after a single ingester failure. If an ingester pod vanishes (node issue or something), it will still be unhealthy in the ring, and now the new pending pod will also be considered unhealthy causing all reads to fail. Would you be able to test that?

[bboreham]

I think Chris is correct, and I think the solution is not to count pending ingesters as an error.
That implies a change to Ring.GetAll().
Maybe we can lose reallyAll at the same time.

In light of new information :)

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[pracucci]

Closing for now