[Fix] Harden the router's resolver

[ljedrz]

While investigating a potential issue with some trusted peers being periodically dropped, I've noticed a lot of instances of Unable to resolve the (...) address in the log extracts from different networks. I believe most of them are triggered unnecessarily, but we need to be sure, and this PR aims to address this.

The proposed changes are as follows:

• 1e8dc49 - changes the dual-lock setup of the resolver to a single-lock one in order to avoid any possibility of mismatch between the address maps; it should also slightly improve its performance
• 260e84b - the inbound method is "fed" from a lower-level queue which doesn't have an awareness of the address resolver, so the entries that fail to resolve there are basically guaranteed to be post-disconnect "stragglers" and may be ignored (instead of triggering potentially many redundant disconnect attempts, which result in further resolver-related warnings)
• 6bb8a74 - this swaps the order of disconnect-related operations, altering the resolver only after a peer is no longer marked as connected; this will avoid situations where an outbound message is greenlit to be sent to a peer (who is marked as connected) only to fail at address resolution right afterwards, triggering a bogus warning
• bccf29a - this is a loosely-related drive-by; we should clear any peer-related cache before marking them as a candidate for connections, in order to avoid a (highly unlikely) scenario where the peer is reconnected to while having outdated cache entries, or even having new and applicable cache entries cleared
• 7ee66b4 - when a peer sends us a Message::Disconnect, we shouldn't report it as a protocol violation; this is mostly a cleanup of one or two misleading logs
• b673d7b - since I've seen some instances of the heartbeat process reporting lingering inactive peers, we should have a fallback cleanup of high-level connection artifacts in case the resolver can't find the physically connected address

Update: the PR was rebased due to a conflict, and while these commit hashes have changed, their contents or order haven't.

[niklaslong]

A nice tightening up of peer tracking! Did a first pass and the current changeset looks good 👍

[howardwu]

@ljedrz Do we need to apply the same changes from Router to Gateway?

[ljedrz]

@howardwu not necessarily; my recommendation would be to first introduce these changes, and then perform a new analysis of the logs, looking for protocol violation false positives and potential connection stability issues. These changes will make the picture a lot more clear.

[joske]

@ljedrz @niklaslong Which logs are you talking about? Were you able to reproduce the issue yourself?

[ljedrz]

@joske I was analyzing the logs of one of the Canarynet clients before and after these changes.

[joske]

Could you share those logs?

[vicsn]

Could you share those logs?

I recall very often seeing the errors Lukasz mentioned. I suggest you can just run your own local canary client as he suggests, if you don't pass any peers you should connect to bootstrap nodes who will connect you to others.

[kaimast]

Let a few comments. Sorry if they are too nitpicky...

Could you resolve the conflicts with staging, as well? Hopefully we can get this merged in the coming days.

[ljedrz]

Rebased (there were no changes to the original commits), and applied the review comments.

[ljedrz]

The devnet CI job has failed, but I can't reproduce it locally; can't restart it either.

Update: it was a deadlock which I avoided locally due to starting the nodes less quickly. Fixed.

[kaimast]

LGTM!