[FIXED] Gateway RS+/- blocks on account fetch

[MauriceVanVeen]

The methods processGatewayRSub and processGatewayRUnsub can call into updateInterestForAccountOnGateway, which in turn calls s.LookupAccount. If the account is not known it will be fetched if there's an account resolver. But, since this is done while in the read loop of the gateway, this blocks not only receiving the response for the account fetch (when using a NATS resolver) but also any other operation like PING/PONG, which then could lead to a stale connection.

The fix proposed by this PR is to not allow fetching the account inline of the gateway read loop.

Signed-off-by: Maurice van Veen github@mauricevanveen.com

[MauriceVanVeen]

I originally included this comment in the PR description:

However, TestJetStreamSuperClusterMixedModeSwitchToInterestOnlyOperatorConfig fails with Server C3-S1 - outbound gateway connection "C2": no account "A.." found in map. To resolve this, could change the assert to checkGWInterestOnlyModeOrNotPresent with notPresentOk=true if this is expected. But, not sure if this is a proper approach? Don't think blocking the gateway's read loop should be happening in any case though.

Have updated the test to include notPresentOk=true.

Not fetching the account in the gateway's readLoop upon receiving RS+/- seems to be totally fine since it's only even relevant if a leaf node is connected to the server. If none is connected, we can safely skip.

And, I've tested this condition as well where a server gets the RS+ over a gateway for an account that it doesn't know and then also doesn't fetch. When adding a leaf node connection to this server it itself will do the account fetching, which will be run in the client's readLoop as expected and not block the gateway's readLoop. Messages will then still be routed properly from the leaf node, through the gateway server that didn't fetch the account before, then to a gateway server that has a subscription in that account.

[neilalexander]

LGTM