allow tablet picker to exclude specified tablets from its candidate list

[pbibra]

Description

Allow the TabletPicker to take in a list of tablets to exclude from its candidate list.
This is mainly required in the VStreamManager when there is a retriable failure streaming from a tablet.

Example Scenario:
Previously, a VTGate VStream RPC call could fail on a GTID mismatch error. In this case, the tablet is healthy so a client retry may choose the same one and fail again. This can happen if the cell preference for the tablet picker is local and there is only a single tablet in the VTGate's local cell. Instead, we'd like to retry on this error in addition to marking the tablet as unviable for streaming within this request.

Changes:

• Add an ignoreTablets list to TabletPicker properties.
• Update tp.GetMatchingTablets to omit a potential candidate if it also appears in the ignoreTablets list.
• Introduce a shouldRetry() function in the VStreamManager which has two bool return values. The first represents whether the error should be retried, the second determines whether the tablet on which the error occurred should be ignored as a possible candidate on the retry.
• Update vsm.streamFromTablet to check shouldRetry() upon error and pass in a list of ignoreTablets to the TabletPicker.

Retriable Errors:

• Code_UNAVAIALBLE - should retry from before, we will not ignore the tablet on a retry.
• Code_FAILED_PRECONDITION - should retry from before, we will not ignore the tablet on a retry.
• Code_INVALID_ARGUMENT with message GTIDSet Mismatch - should retry, ignore the tablet on a retry.

Related Issue(s)

• Fixes: Feature Request: Add the ability to skip a list of tablets to pick in the TabletPicker #14097

Checklist

• "Backport to:" labels have been added if this change should be back-ported
• Tests were added or are not required
• Did the new or modified tests pass consistently locally and on the CI
• Documentation was added or is not required

[mattlord]

LGTM. Thanks, @pbibra ! I only had a couple of minor comments that we should address before merging.

I also confirmed that the new tests are not flaky by running them with -count 1 -race in a loop for some time.

[mattlord]

I don't think that this is needed, is it? Since there's only one for loop.

[pbibra]

The break statement breaks out of the switch instead of the loop by default, so we had to label the loop