Speculative queries

[shanson7]

For #954

I made speculation configurable, but one thing will still change even with speculation disabled: Local requests are now via HTTP rather than a special case. This is good and bad (good: happens in parallel with peer requests and tracing just works, bad: a bit of overhead)

TODO:

• Figure out partitions / shard group problem
• Some functions (e.g. findSeries) don't use peerQuery; fix that
• Try to more efficiently pre-allocate memory for initial results
• Documentation

[replay]

the metric name in the comment is not right

[replay]

speculationThreshold is configurable now

[replay]

This seems to determine whether a member is already in the map or not based on its first partition. But I can't see where we are sorting the partitions, if two MTs of a shard are configured to have the same partitions but they are specified in a different order wouldn't this break?

[shanson7]

That's true. IMO, it seems like the partitions should be sorted at start up. I could add a sort to the SetPartitions function.

[replay]

I agree that sorting at startup makes the most sense, that way it only needs to be done once

[replay]

I just noticed that in cases where the partition ids are not sorted in the config it is important to first update to this version of MT without activating speculative queries, and only once all MTs are on this version enable speculative queries. Otherwise querying might temporarily be broken until all MTs are updated because the older ones still returned their partition IDs unsorted. But i guess there is nothing we can do about that.

[shanson7]

Well, we could sort the partitions of the cluster peers when we get them. As it stands right now, with or without speculation enabled, the partitions will need to be sorted. That is, unless I change the code as I mentioned in #956 (comment)

[shanson7]

Status update:

I'm using this on our prod setup. We get about 67 render requests/sec. We have 120 shard groups * 2 replicas (240 total peers). This results in about 8000 peer requests/sec.

Speculation kicks in on darn near all requests, as with that many peers, it's highly likely that any is undergoing some GC.

Here's a snapshot with some of the data:
https://snapshot.raintank.io/dashboard/snapshot/Yy5adwlVIpEo7IyoS3QZj7QuikTsCw27

Of note:

1. Speculation "win" percentage is frequently in the high 90's, indicating that most requests are aided by speculation.
2. "Additional" requests to peers (requests that wouldn't be made if speculation was disabled) is frequently less than 5% (Our speculation-threshold is set at 94%, so at most 7 additional HTTP requests are made per speculative query, making our worst case here ~5.8%)
3. "Win %" seems to increase under load (but so does the p90 response time).

You can see in this snapshot when we upped the load:
https://snapshot.raintank.io/dashboard/snapshot/YAifupVWdmQ8nuIDa58qo4ciEmIWAgwk

Here's a comparison of our render response times before and after rollout (under very light load, 4 render reqs/sec):
https://snapshot.raintank.io/dashboard/snapshot/wBWTIbLGkF2GXiQIEjAjyzRFsG5HMqmB

You can see how much smoother and improved the median and p90 response times are.

[replay]

looks great, but could you fix the test please? this might be what you want: afc60d2

[shanson7]

I haven't verified it, but I'm pretty sure that the existing peer query mechanism required peers to have the same partitions as their comrades. Speculative peer queries absolutely requires it. Is it worth supporting something like

if speculation-threshold < 1 {
   peerQuerySpeculatively
} else {
   peerQueryOldWay
}

[replay]

So far I've not heard of any cases where people configured their MT partitions in a way where the partitions are not the same for all instances in a shard (that's what you mean, right?). I think it's probably not worth the additional complexity just to support this very rare edge case. What do you think @woodsaj

[woodsaj]

the docs state that when you have multiple replicas, partitions must be assigned in groups.
https://github.com/grafana/metrictank/blob/master/docs/clustering.md#combining-metrictanks-horizontal-scaling-plus-high-availability

[replay]

LGTM

[Dieterbe]

Here's a comparison of our render response times before and after rollout (under very light load, 4 render reqs/sec):
https://snapshot.raintank.io/dashboard/snapshot/wBWTIbLGkF2GXiQIEjAjyzRFsG5HMqmB
You can see how much smoother and improved the median and p90 response times are.

interestingly:

• win % is way lower than before. probably due to the very light load?
• i see that there were 2 periods wherein speculation was enabled. in the first period, the latencies were still elevated. is this because speculation wasn't fully rolled out across the cluster? (additional HTTP is lower in first period compared to second)

otherwise, all these numbers look great :)

[shanson7]

win % is way lower than before. probably due to the very light load?

Correct. Here's a snapshot under heavier load (with real user queries)
https://snapshot.raintank.io/dashboard/snapshot/IoV6Z91WYgh3VFmlB133fr2xVxnH9B4m

i see that there were 2 periods wherein speculation was enabled. in the first period, the latencies were still elevated. is this because speculation wasn't fully rolled out across the cluster? (additional HTTP is lower in first period compared to second)

Yes, this was still during the rollout.

[shanson7]

Of note, this optimization is likely of more value the larger the cluster. At our 120 shard group cluster, it's been great.