(2.12) NRG: Don't respond success to catchup messages & ignore response if not leader

[MauriceVanVeen]

Desync was observed in the Raft layer in the following scenario:

1. Server 1 becomes leader for term=1 and proposes 100 items, n.pterm=1.
2. Server 2 and 3 only receive the first 50 items and store them in their logs. Server 1 will not know yet these can be committed.
3. Server 2 becomes leader for term=2 and proposes 1 item, stores it in its log but it doesn't get quorum, n.pterm=2.
4. Server 1 becomes leader for term=3, catches up Server 3 for its missing messages from step 1 and gets quorum/commits them. Server 3 does not know yet these can be committed.
5. Server 2 only has 51 items in its log, the initial 50 items from Server 1, and the one item it tried to propose. It tries to become leader with n.pterm=2, n.pindex=51 and Server 3 allows it to become leader because it has n.pterm=1, n.pindex=100. Server 3 votes because the pterm is higher, even though the pindex is lower.
6. Server 2 truncates part of Server 3's log (that Server 1 already committed). This results in desync.

This happened because Server 3 provided quorum to Server 1 during catchup. The issue is resolved if we simply don't respond back to Server 1 and make it commit those entries. We need to fully catchup, and only afterward can we participate in achieving quorum on those entries.

This fixes some cases of desync, for example stream desync with last sequence mismatch.

Signed-off-by: Maurice van Veen github@mauricevanveen.com

[derekcollison]

LGTM - One minor naming nit

[neilalexander]

LGTM in principle but TestNRGSnapshotAndTruncateToApplied looks unhappy.

[MauriceVanVeen]

Yep, seems some thinks broke since the last push. Put in draft for now.

[neilalexander]

LGTM!