High Server Load on 1 Shard after upgrade of StackExchange.Redis to 2.8.0

[gauravarya18]

We have caches deployed in Azure Redis and we are observing a pattern in multiple caches where after switching client library from 2.0.601 to 2.8.0 -

Shard1 server load has increased from 24th August.

Network Bandwidth has decreased on all shards and increased on shard 1.

Migrated to 2.8.0 on 24th Aug

[mgravell]

Question: what kind of use-case is this? Specifically: is this Redis instance used for data (where we know the data should be sharded by key), or is it actually a pub-sub relay? I'm wondering if this might relate to a change in how we route channel subscriptions. In particular, if you only have a single channel (or pattern) - or at least only one with significant load, I wonder if we're routing pub/sub hooks based on that channel.

You mention caching, so I'm guessing there is also data in play, but I'm wondering if the network bandwidth is being dominated by pub/sub, perhaps cache invalidations. Are you using pub/sub for cache invalidation? or...?

(in short: we don't really get a choice on where to route data requests, so I can't see how we could reduce network load on anything data related; obviously we could increase load if we screwed up the routing and issued things to the wrong server, only fixing via -MOVED redirection, but: that wouldn't case a decrease anywhere)

[mgravell]

I spoke with @philon-msft, and it sounds like my guess was correct. I propose we think about changing the API to make it clear and explicit what routing model the caller would like to use. There are two distinct scenarios here (ignoring "SSUBSCRIBE" and "PSUBSCRIBE" which need to be considered separately):

1. the caller has lots of channels and we can use channel-based routing to distribute load via channels and still get decent distribution
2. the caller has few channels, which could lead to unbalanced distribution

It sounds like you're in the second of these and would benefit from allowing the library to "round-robin" the subscriptions.

Does this sound correct?

The other question, then, is:

3. what should the default be for code using the legacy code-path i.e. whatever you did; I think maybe we should revert this to round-robin, which is probably the option of least-surprise (in terms of: we don't end up with skewed servers)

[gauravarya18]

@mgravell
what kind of use-case is this?
Data involved is here is both kind, i.e., Key-Value pairs and Pubsub.

I think its because of Pubsub only as we see - Pubsub calls has increased on one shard and decreased on remaining

On 2nd part, yes we have a dominant caller which takes up 80-90% of Pubsub traffic and is concentrated with one channel name.
This what we are doing -

There was no issues in 2.0.601, has anything changed in 2.8.0 w.r.t to Pubsub routing, which we need to tweak to make the distribution even again ?

[mgravell]

I think we will need a new API option to make this controllable. That doesn't mean it has to take longer though. Will see what is involved, tomorrow.

[mgravell]

Proposed mitigation: #2958

• default behaviour goes back to round-robin (unless explicitly using SPUBLISH/SSUBSCRIBE)
• a new API allows people to choose to use routing
• when publishing to a channel that we're also subscribed to: use that exact connection
• randomize the seed for round-robin, to avoid repeatable startup patterns

[mgravell]

/cc @philon-msft because you have context

[mgravell]

Closed via 2.9.24; default behaviour reverts to balanced; if you actively want channel routing, there is a new API for that: var channel = RedisChannel.Literal("foo").WithKeyRouting();. Please let me know how you get on!