Uneven partition distributions can lead to inconsistent results

[replay]

While a cluster of Metrictanks is in a state where its partitions are distributed unequally between instances, there can be situations where finds using speculative query execution result in incomplete results.

We're currently using the first partition of each Metrictank as an identifier for its partition group, assuming that all Metrictanks which have the same first partition have all the same partitions

metrictank/cluster/cluster.go

Line 182 in 431b6c9

membersMap[memberStartPartition] = append(membersMap[memberStartPartition], member)

In a situation where the partition distribution is unequal, this can result in partitions being omitted when executing cluster wide finds.
For example:

Metrictank0: 4,5,6,7
Metrictank1: 4,5

It is possible that a /find call would end up on Metrictank1 and the partitions 6,7 would get omitted.

[shanson7]

Yes, this was brought up during the implementation. See #956 (comment)

[replay]

Good point. In the meantime our requirements have changed, so we're now looking for a way to make that work.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Dieterbe]

This is not an actual issue. Just something we'd like to improve