This repository was archived by the owner on Nov 15, 2023. It is now read-only.
This repository was archived by the owner on Nov 15, 2023. It is now read-only.
Grandpa notifications dropping to 0 #7983
Description
We have a node on the master branch of Polkadot. It is connected only to nodes on Polkadot 0.8.27.
It seems that, occasionally, and without any apparent cause, the Grandpa notifications seem to stop being received.
Some information I've gathered from debugging:
- The block announces are still work fine. Most of the code paths of Grandpa messages are shared with block announces.
- We still occasionally attempt to send out Grandpa messages. It's just the receiving that seems to stop.
- The authority-discovery still runs and receives
Event::Dhtevents. In other words, it's most likely not a problem with distributing events to the rest of Substrate. - The problem has started to happen after Rework priority groups, take 2 #7700
- The logs don't show anything suspicious happening when the problem happens. The notifications simply stop being received. There's no substream closing nor disconnect.
- The problem always seems to happen roughly at the same time as a block that takes a long time to import. However, there are a lot of blocks that take a long time to import and that do not necessarily cause the problem to happen.
- Restarting a node connected to the buggy node doesn't fix the problem.