Make sure ConsistentHashingNodeProvider returns unique candidates

[rongrong]

Test plan - unit test

== RELEASE NOTE ==
* Fix a bug where cache performance might be affected when ``CONSISTENT_HASHING`` is used as the scheduling strategy.

[rschlussel]

I see there's no release note. what was the motivation for this change? Does it fix a bug or other issue?

[rschlussel]

Instead of building the node list here, why not add to the unique set no matter what, and then convert the set to an immutable list before returning it?

[rongrong]

ordering is important here.

[rongrong]

I see there's no release note. what was the motivation for this change? Does it fix a bug or other issue?

Welcome back! Yes, it's fixing a bug. The method should return count number of candidates. If they happened to be not unique, it's not satisfying the requirement. Let me check whether we've already had release notes for the feature.

[yuanzhanhku]

Could this be an infinite loop? e.g., when num of workers is < count? Add a test for this case?

It is also possible that this will take a lot of iterations to find next unique candidate if the hash function happens to return a visited virtual node for a lot of times.
How about searching it in the candidates map to find the next unique candidate instead of relying on a random function? Specifically, if we find a duplicate node, we get its iterator/pointer in the map and check the following nodes according to their order in the map until we find a new node or have exhausted all node.

[rongrong]

Hmm, I think the assumption here is that count is small. 2 is used right now. But I agree we should add a check. I think in any realistic scenario the probability of continuously hitting the same virtual nodes is pretty low but that's also assuming count is small. Next physical node on the ring seems better.

[rongrong]

@rschlussel @beinan Can you guys take a look? Thanks!