Skip to content

fix(consensus): don't propose at historical heights after a bad block-sync handover - #1416

Open
lklimek wants to merge 5 commits into
v1.8-devfrom
fix/1413-propose-after-blocksync-handover
Open

fix(consensus): don't propose at historical heights after a bad block-sync handover#1416
lklimek wants to merge 5 commits into
v1.8-devfrom
fix/1413-propose-after-blocksync-handover

Conversation

@lklimek

@lklimek lklimek commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

TL;DR: A node that hands over from block sync to consensus while still badly behind the network no longer builds proposals at historical heights or leaves its block store ahead of what the application actually processed — the failure mode that wedged a mainnet evonode in a restart-proof crash loop.

User story

As a node operator running a validator that re-syncs from scratch (fresh install, restore, or a long outage), I want my node to keep catching up safely even if block sync hands it to consensus early with a thin or unhelpful peer set, so that it never proposes stale historical blocks, never gets evicted for something that isn't its fault, and never gets stuck in a crash loop that only a full data wipe can fix.

Scenario

Base flow

A node re-syncing from far behind block-syncs from its peers, and once it's judged "close enough," hands over to the normal consensus process to finish catching up and start participating.

Actual behavior

If block sync's peer set stalls (e.g. seed nodes with no blocks to serve) for long enough, the node hands over to consensus even while still thousands of blocks behind. Nothing tells consensus it's still behind, so if the node happens to be selected proposer at one of those historical heights, it builds and proposes a block using present-day application state (e.g. an absurdly-future chain-lock reference). Because of how proposer rotation works, that same validator is then virtually guaranteed to also be the genuine proposer of the very next height — so when the real network commit for that height finally arrives, it collides with the stale block the node already built for itself, and the application panics. Restarting doesn't help: WAL replay reproduces the exact same collision every time, and the only way out is discarding all local Tenderdash data. Separately, if the node crashes mid-apply during block sync, its block store can end up one block ahead of what the application actually finished processing, which also breaks the recovery handshake on the next start.

Expected behavior

A node that hands over to consensus while still meaningfully behind does not build proposals until it has genuinely caught up (voting and following consensus continue normally). Block sync itself is now more reluctant to hand over while there's still a large gap to the tip, since consensus catch-up can only close a small gap at a time. And the block-sync applier no longer persists a block that the application refused to process, so a crash there can't leave the store ahead of the application's own state.

Detailed discussion

What was done

Closes #1413.

Three independent changes, one per commit:

  1. fix(consensus): don't propose after a block-sync handover while still behind — new catchupTracker (internal/consensus/catchup.go), owned by State and consulted by EnterProposeAction next to the existing replay-mode gate; only proposal building is skipped, voting and following are untouched. It arms only when block sync hands over while a peer reported a height above ours (internal/blocksync/reactor.go computes this and threads it through a new behind parameter on SwitchToConsensus), and disarms only on positive evidence — a block committed through consensus while no peer claims a higher height, or a peer directly reporting a height at or below ours. A peer that simply hasn't reported anything yet counts as "unknown," not "caught up," since every peer is in that state right after a handover — exactly when the node is most behind. Once disarmed, only a new handover re-arms it, so a single lying peer can hold a window the node already entered but can never open one on a node that wasn't behind.

  2. fix(blocksync): keep syncing when the tip is out of consensus catch-up's reach — most of the "don't hand over while a servable peer exists" logic already existed on this branch's base. The remaining gap: the 10-minute wall-clock stall backstop (maxSyncStall) used to hand over regardless of how far behind the node was. It now only fires within maxCatchupGap (10 blocks) of the highest height any peer claims, since consensus catch-up closes roughly one block per gossip cycle and can't meaningfully close a gap of thousands. Beyond that gap, block sync keeps retrying and logs why on every interval, rather than handing a hopeless case to consensus.

  3. fix(blocksync): don't persist a block the application refused — the block-sync applier used to call SaveBlock before the application had finished processing the block. The literal fix suggested in the issue (save only after the whole ApplyBlock succeeds) would actually be a regression: it would move the crash window from a store/state mismatch the replay handshake explicitly supports recovering from (storeHeight == stateHeight+1) onto one it treats as fatal. Instead, the applier now runs ApplyBlock's own two constituent calls itself — ProcessProposal (the application either accepts or refuses the block) and FinalizeBlock (the application commits it) — with SaveBlock in between. This exactly mirrors the sequence the consensus commit path (ApplyCommitAction) has always used; block sync was the outlier. A block the application refuses now leaves nothing persisted, closing the actual gap the issue describes without weakening crash recovery anywhere else.

Not in scope, deliberately: the ABCI-app-side sanity check on chain-lock age mentioned in the issue is a dashpay/platform concern, already addressed there (platform#4462). A pre-existing, unrelated hazard was also identified and intentionally left untouched — internal/blocksync.Reactor.poolRoutine stops the synchronizer and switches to consensus without joining any block application still in flight, so in principle both could briefly talk to the ABCI app during handover. This exists identically before and after this PR; flagging it here for visibility, not fixing it as part of #1413.

Testing

New/updated tests, all passing:

  • internal/consensus: TestCatchupTrackerMayPropose (7-case table), TestCatchupTrackerDoesNotRearm, TestCatchupTrackerNotWired, TestEnterProposeSuppressedWhileCatchingUp, TestEnterProposeResumesOnceCaughtUp.
  • internal/blocksync: TestBlockApplierDoesNotSaveBlockRejectedByApp, TestBlockApplierSavesBlockBeforeFinalize, TestBlockApplierApply (updated), TestStallVerdictFor (extended with the behind-by-how-much dimension), TestWaitForSyncHandsOverAfterMaxSyncStall (retargeted to a near-tip peer), TestWaitForSyncKeepsSyncingWhileFarBehind (new), TestStallSnapshotIsOneObservation (extended).

All new tests were verified to fail against the pre-fix behavior before the corresponding fix landed. go test ./internal/consensus/... ./internal/blocksync/... and go vet ./... pass; gofmt/golangci-lint clean on the changed lines (pre-existing findings in untouched files are unaffected).

Breaking changes

consensusReactor.SwitchToConsensus gained a behind bool parameter (internal interface, not part of any public API).

Checklist

  • Self-review performed (3 passes)
  • Code commented, particularly the non-obvious parts (why height-0 peers don't count as "caught up", why the applier split isn't a novel interleaving)
  • Unit tests added/updated
  • Documentation — no user-facing docs affected

Prior work

Attribution

🤖 Co-authored by Claudius the Magnificent AI Agent

lklimek and others added 3 commits August 24, 2026 22:44
The applier saved a block to the block store and only then handed it to
the application. An application that refuses the block - and one holding
a stale execution context for that height does refuse it - left the store
one height ahead of a block it never processed, and every later start had
the handshake re-process that block through the same application, so the
node could not restart its way out (#1413).

ApplyBlock is now run as its two halves with the store advanced between
them: ProcessProposal, where an application refuses a block and nothing
has been persisted yet, then SaveBlock, then FinalizeBlock, where the
application commits. Saving before the commit is what keeps the store
from falling behind the application or the state, a case the handshake
rejects outright, while a store one ahead is one it recovers from.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…p's reach

The wall-clock backstop handed a node over to consensus after ten minutes
without progress however far behind it was. What follows that handover is
consensus catch-up, which moves about one block per gossip cycle: it
closes a gap of a few blocks in seconds and a gap of thousands never, and
in the meantime the node sits in the validator set at heights the network
committed long ago (#1413).

The backstop now applies only within maxCatchupGap of the highest height
any peer claims. Further back, block sync keeps retrying and logs the
distance every interval - the only route to the tip that exists. A stall
on a block no peer can serve still ends block sync as before, so a node
that genuinely has nowhere to fetch from is unaffected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… behind

Block sync hands over to consensus even when it never reached the tip,
and nothing told consensus about it. A long-lived validator handed a
historical height is very likely the genuine proposer of the next height
too, so it proposed a block built from present-day application state; the
real block for that height then collided with it, and the node could not
restart its way out (#1413).

A handover that leaves the node provably behind - block sync stopped
while a peer claimed a height above ours - now holds back proposals until
something says the node reached the network: a peer reporting a height no
higher than its own, or a block committed through consensus while no peer
reports one. Voting and following consensus are untouched, and a
suppressed round times out and rolls on like any other silent proposer.

Only a handover can start the window, and it closes for good, so a peer
lying about its height can neither open one nor reopen one - it can only
hold a window the node already entered open while it stays connected. A
node no peer claims to be ahead of, a solo validator or a fresh network,
never enters it at all.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 449ea7c5-2959-4c91-89b5-bd96f8f653d4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@lklimek
lklimek marked this pull request as ready for review August 25, 2026 09:13
@thepastaclaw

thepastaclaw commented Aug 25, 2026

Copy link
Copy Markdown

🕓 Queued for automated review — 16th in line, estimated start in ~2.7 h (commit b95760d)
Estimated review time once started: ~20 min (two-phase automated review; median of recent runs).

  • Request priority review — tick this box and the review moves to the front of the queue.

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Preliminary review — Codex only

The PR improves block-sync handover and application rejection ordering, but three in-scope fail-open paths still permit the historical-height/store-ahead failure modes it is intended to prevent. Proposal suppression is neither armed for the normal one-block-behind handover nor tied to the block-sync high-water mark, and an incoherent accepted ProcessProposal response is still detected only after persistence. Focused tests for the newly added catch-up and block-sync cases pass; the full consensus package run also encountered an unrelated timing-sensitive load test failure.
Source: Codex general and consensus-security reviewers (exact backend model IDs were not present in the supplied evidence); Claude Agent SDK final verifier (exact backend model ID was not exposed to this runtime). openclaw-agent/cliproxy/gpt-5.6-sol was orchestration-only and is not reviewer evidence.

Validated blockers were found in the Codex precheck. Opus is deferred until a fresh Codex revalidation clears the blocker gate.

Review provenance

  • Codex reviewers: gpt-5.6-sol — general (completed), gpt-5.6-sol — tenderdash-consensus-security (completed)
  • Verifier: gpt-5.6-sol — verifier
  • Sonnet: not run (deferred by blocker gate)

🔴 3 blocking

🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.

In `internal/blocksync/reactor.go`:
- [BLOCKING] internal/blocksync/reactor.go:311: Arm proposal suppression whenever the observed peer tip is above persisted state
  `Synchronizer.height` is the next unapplied height, while `state.LastBlockHeight` is the last applied height. Therefore, when a peer advertises `state.LastBlockHeight+1`, `IsCaughtUp` returns true even though that peer's tip block has not been applied, and the `!caughtUp` condition forces `behind=false`. An ordinary peer can advertise exactly that range, cause the handover, and delay its consensus height/commit traffic until the validator enters propose at that already-committed historical height. This bypasses the new suppression gate despite positive evidence that the peer is ahead. Whether block sync considers the remaining gap suitable for consensus catch-up and whether proposals must remain suppressed are separate predicates; arm the tracker whenever the observed block-sync tip exceeds the persisted state height.

In `internal/consensus/catchup.go`:
- [BLOCKING] internal/consensus/catchup.go:77-81: Do not discard the block-sync high-water mark based on mutable peer state
  The tracker retains only a callback to current consensus `PeerState` heights, not the block-sync tip that proved the node was behind. Those heights are untrusted and do not exclusively represent an advertised tip: `handleVoteMessage` calls `SetHasCommit`, whose `setHasCommit` raises `PRS.Height` to the catch-up commit's own historical height, and an unsigned `NewRoundStep` can establish a similarly low initial height. If that value is at or below the local proposal height, this code permanently clears the tracker—even before any block is committed—and a later report of the peer's actual far-higher tip cannot re-arm it. Retain the block-sync target height as a monotonic high-water mark and suppress proposals until committed local progress reaches that target; lower claims, inferred commit heights, and peer removal must not reduce or clear it.

In `internal/blocksync/applier.go`:
- [BLOCKING] internal/blocksync/applier.go:96-103: Validate ProcessProposal coherence before saving the block
  With `verify=false`, `ProcessProposal` checks the application's status, response structure, transaction-result shape, and app-hash length, but it does not verify that the returned app hash, results hash, validator updates, or consensus parameters match the committed block. `SaveBlock` persists the block before `FinalizeBlock` performs that coherence check through `ValidateBlockWithRoundState`. If the application returns ACCEPT with a stale but structurally valid response, finalization returns `ErrInvalidBlock` and the applier panics after advancing the block store while neither Tenderdash state nor the application was committed. Handshake replay deterministically encounters the same mismatch, preserving the restart loop this PR is intended to close. Perform the coherence validation before `SaveBlock`; if duplicate last-commit signature verification is too expensive, split that verification from the response/header coherence checks rather than delaying the first coherence gate until after persistence.

// behind: a peer claiming a height above ours is the one thing that says so.
// Where no peer claims one - a solo validator, a network with no peers at all
// - the node is not held back at all.
behind := !caughtUp && r.synchronizer.MaxPeerHeight() > state.LastBlockHeight

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Blocking: Arm proposal suppression whenever the observed peer tip is above persisted state

Synchronizer.height is the next unapplied height, while state.LastBlockHeight is the last applied height. Therefore, when a peer advertises state.LastBlockHeight+1, IsCaughtUp returns true even though that peer's tip block has not been applied, and the !caughtUp condition forces behind=false. An ordinary peer can advertise exactly that range, cause the handover, and delay its consensus height/commit traffic until the validator enters propose at that already-committed historical height. This bypasses the new suppression gate despite positive evidence that the peer is ahead. Whether block sync considers the remaining gap suitable for consensus catch-up and whether proposals must remain suppressed are separate predicates; arm the tracker whenever the observed block-sync tip exceeds the persisted state height.

Suggested change
behind := !caughtUp && r.synchronizer.MaxPeerHeight() > state.LastBlockHeight
behind := r.synchronizer.MaxPeerHeight() > state.LastBlockHeight

source: ['codex']

Comment on lines +77 to +81
peerHeight := t.peerHeight()
if peerHeight > height || (peerHeight == 0 && !t.committed) {
return false
}
t.peerHeight = nil

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Blocking: Do not discard the block-sync high-water mark based on mutable peer state

The tracker retains only a callback to current consensus PeerState heights, not the block-sync tip that proved the node was behind. Those heights are untrusted and do not exclusively represent an advertised tip: handleVoteMessage calls SetHasCommit, whose setHasCommit raises PRS.Height to the catch-up commit's own historical height, and an unsigned NewRoundStep can establish a similarly low initial height. If that value is at or below the local proposal height, this code permanently clears the tracker—even before any block is committed—and a later report of the peer's actual far-higher tip cannot re-arm it. Retain the block-sync target height as a monotonic high-water mark and suppress proposals until committed local progress reaches that target; lower claims, inferred commit heights, and peer removal must not reduce or clear it.

source: ['codex']

Comment on lines +96 to 103
uncommittedState, err := e.blockExec.ProcessProposal(ctx, block, commit.Round, e.state, false)
if err != nil {
panic(fmt.Sprintf("failed to process committed block (%d:%X): %v", block.Height, block.Hash(), err))
}
processTime := time.Since(start)

start = time.Now()
e.store.SaveBlock(block, blockParts, commit)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Blocking: Validate ProcessProposal coherence before saving the block

With verify=false, ProcessProposal checks the application's status, response structure, transaction-result shape, and app-hash length, but it does not verify that the returned app hash, results hash, validator updates, or consensus parameters match the committed block. SaveBlock persists the block before FinalizeBlock performs that coherence check through ValidateBlockWithRoundState. If the application returns ACCEPT with a stale but structurally valid response, finalization returns ErrInvalidBlock and the applier panics after advancing the block store while neither Tenderdash state nor the application was committed. Handshake replay deterministically encounters the same mismatch, preserving the restart loop this PR is intended to close. Perform the coherence validation before SaveBlock; if duplicate last-commit signature verification is too expensive, split that verification from the response/header coherence checks rather than delaying the first coherence gate until after persistence.

Suggested change
uncommittedState, err := e.blockExec.ProcessProposal(ctx, block, commit.Round, e.state, false)
if err != nil {
panic(fmt.Sprintf("failed to process committed block (%d:%X): %v", block.Height, block.Hash(), err))
}
processTime := time.Since(start)
start = time.Now()
e.store.SaveBlock(block, blockParts, commit)
uncommittedState, err := e.blockExec.ProcessProposal(ctx, block, commit.Round, e.state, true)

source: ['codex']

@thepastaclaw thepastaclaw left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final validation — Phase 2 only (queue backlog)

Four blocking issues remain: unsigned peer heights can indefinitely suppress proposals, the handover loses a post-stop state synchronization barrier, and two paths bypass the intended historical-proposal protection. The consensus and blocksync package suites pass with the race detector and deadlock tag, but their existing tests do not cover these failure scenarios. The worktree is unchanged.

Source: reviewer 1: gpt-6-astra (agent: phase2-reviewer, role: general); reviewer 2: gpt-6-astra (agent: phase2-reviewer, role: tenderdash-consensus-security); final verifier: gpt-6-astra (agent: astra-verifier, role: final-verifier)

Review provenance

  • Triage: critical by gpt-6-astra (effort low) — The changes affect consensus proposal eligibility, block-sync handover, and block persistence ordering, where incorrect height tracking or recovery behavior could stall validators, produce invalid historical proposals, or leave nodes in restart-proof crash loops.
  • Phase 1 reviewers: not run (skipped for throughput: 55 PRs queued, above the 10 limit)
  • Fresh verifier: gpt-6-astra — final-verifier; agent astra-verifier
  • Phase 2 reviewers: gpt-6-astra — general (completed, effort xhigh); agent phase2-reviewer, gpt-6-astra — tenderdash-consensus-security (completed, effort xhigh); agent phase2-reviewer

🔴 4 blocking

🤖 Prompt for all review comments with AI agents
These findings are from an automated code review. Verify each finding against the current code and only fix it if needed.

In `internal/consensus/catchup.go`:
- [BLOCKING] internal/consensus/catchup.go:77-79: Do not give unsigned height claims an indefinite veto over proposals
  A peer without validator keys can both arm and indefinitely hold this gate, even when the node is at the real tip. During a multi-validator restart at committed height H, the peer can advertise StatusResponse{Base: H + 2, Height: 1 << 60}; this passes blocksync validation. Honest peers hold blocks only through H, so nobody advertises a usable block at H+1. After syncTimeout, stopNothingToFetch hands over with behind=true. An unsigned NewRoundStep advertising height 1<<60 also passes consensus validation and makes this condition reject every proposal attempt. Honest peer reports and blockCommitted do not override that maximum, and there is no expiry while the attacker remains connected. The does-not-rearm protection does not help because the attacker controls the evidence used for initial arming. Repeating this against restarting validators can leave every proposer suppressed without any Byzantine voting power. Require verified or corroborated catch-up evidence, or a bounded mechanism for discarding unproven ahead-height claims, rather than granting one peer a permanent proposal veto.
- [BLOCKING] internal/consensus/catchup.go:77-81: Preserve the known handover target before accepting lower peer heights
  The handover transfers only a boolean and discards the blocksync height that established the node was behind. This gate then permanently disarms on any nonzero consensus maximum at or below the local proposal height. For example, a node can hand over after height 99 knowing a blocksync peer advertises height 5000, while that peer has not announced a consensus height; peers still syncing defer their consensus gossip until ready. If a seed reports consensus height 1, mayPropose(100) immediately clears the tracker despite the known gap. A later announcement of 5000 cannot re-arm it. A historical catch-up commit can similarly populate PeerState.Height through SetHasCommit without reporting the sender's actual tip. Preserve the handover target until catch-up evidence establishes that it has been reached or legitimately superseded; a lower consensus height from the currently reporting peers is not sufficient proof.

In `internal/blocksync/reactor.go`:
- [BLOCKING] internal/blocksync/reactor.go:306: Restore the post-stop applier-state read before consensus handover
  Caching executor.State() before Stop introduces a stale-state window and removes an existing synchronization barrier. State() and Apply() use the same applier mutex. Previously, the post-Stop State() call waited for an Apply already holding that mutex; now a valid H+1 application can start after this read and consensus receives H without waiting for it. If application finalization has completed but the state-store write has not, State.OnStart can reload H and resume ABCI processing at a height already processed by the application. There is also a completed-apply failure: if H+1 finishes during Stop and requests RetainHeight=H+1, LoadSeenCommit contains H+1 and LoadBlockCommit(H) has been pruned. SwitchToConsensus then panics loading the stale H commit before its startup reload can help. Capture peer-height evidence before shutdown if needed, but read the state passed to consensus after Stop. This restores the barrier removed by this PR without requiring a fix for the separate, pre-existing lack of an apply-loop join.
- [BLOCKING] internal/blocksync/reactor.go:311: Determine proposal suppression independently of the sync verdict
  IsCaughtUp compares the synchronizer's next block height against peers' committed block heights, so caughtUp=true does not establish that the actual handover state has applied their tip. With LastBlockHeight=99, synchronizer.height=100, and a peer advertising committed height 100, WaitForSync returns true and this expression forces behind=false. Consensus starts at height 100 without arming proposal suppression, although the network has already committed that height. A selected proposer can therefore still build the historical proposal this PR is intended to prevent. The same short-circuit ignores a higher peer learned between WaitForSync and this read. Keep caughtUp for the existing skipWAL decision, but determine behind independently by comparing peer-height evidence with the actual post-stop handover state's LastBlockHeight.

Comment on lines +77 to +79
peerHeight := t.peerHeight()
if peerHeight > height || (peerHeight == 0 && !t.committed) {
return false

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Blocking: Do not give unsigned height claims an indefinite veto over proposals

A peer without validator keys can both arm and indefinitely hold this gate, even when the node is at the real tip. During a multi-validator restart at committed height H, the peer can advertise StatusResponse{Base: H + 2, Height: 1 << 60}; this passes blocksync validation. Honest peers hold blocks only through H, so nobody advertises a usable block at H+1. After syncTimeout, stopNothingToFetch hands over with behind=true. An unsigned NewRoundStep advertising height 1<<60 also passes consensus validation and makes this condition reject every proposal attempt. Honest peer reports and blockCommitted do not override that maximum, and there is no expiry while the attacker remains connected. The does-not-rearm protection does not help because the attacker controls the evidence used for initial arming. Repeating this against restarting validators can leave every proposer suppressed without any Byzantine voting power. Require verified or corroborated catch-up evidence, or a bounded mechanism for discarding unproven ahead-height claims, rather than granting one peer a permanent proposal veto.

source: ['claude']

// NOTE: Don't sleep in the FOR_LOOP or otherwise slow it down!
func (r *Reactor) poolRoutine(ctx context.Context, stateSynced bool) {
caughtUp := r.synchronizer.WaitForSync(ctx)
state := r.executor.State()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Blocking: Restore the post-stop applier-state read before consensus handover

Caching executor.State() before Stop introduces a stale-state window and removes an existing synchronization barrier. State() and Apply() use the same applier mutex. Previously, the post-Stop State() call waited for an Apply already holding that mutex; now a valid H+1 application can start after this read and consensus receives H without waiting for it. If application finalization has completed but the state-store write has not, State.OnStart can reload H and resume ABCI processing at a height already processed by the application. There is also a completed-apply failure: if H+1 finishes during Stop and requests RetainHeight=H+1, LoadSeenCommit contains H+1 and LoadBlockCommit(H) has been pruned. SwitchToConsensus then panics loading the stale H commit before its startup reload can help. Capture peer-height evidence before shutdown if needed, but read the state passed to consensus after Stop. This restores the barrier removed by this PR without requiring a fix for the separate, pre-existing lack of an apply-loop join.

source: ['claude']

// behind: a peer claiming a height above ours is the one thing that says so.
// Where no peer claims one - a solo validator, a network with no peers at all
// - the node is not held back at all.
behind := !caughtUp && r.synchronizer.MaxPeerHeight() > state.LastBlockHeight

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Blocking: Determine proposal suppression independently of the sync verdict

IsCaughtUp compares the synchronizer's next block height against peers' committed block heights, so caughtUp=true does not establish that the actual handover state has applied their tip. With LastBlockHeight=99, synchronizer.height=100, and a peer advertising committed height 100, WaitForSync returns true and this expression forces behind=false. Consensus starts at height 100 without arming proposal suppression, although the network has already committed that height. A selected proposer can therefore still build the historical proposal this PR is intended to prevent. The same short-circuit ignores a higher peer learned between WaitForSync and this read. Keep caughtUp for the existing skipWAL decision, but determine behind independently by comparing peer-height evidence with the actual post-stop handover state's LastBlockHeight.

source: ['claude']

Comment on lines +77 to +81
peerHeight := t.peerHeight()
if peerHeight > height || (peerHeight == 0 && !t.committed) {
return false
}
t.peerHeight = nil

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Blocking: Preserve the known handover target before accepting lower peer heights

The handover transfers only a boolean and discards the blocksync height that established the node was behind. This gate then permanently disarms on any nonzero consensus maximum at or below the local proposal height. For example, a node can hand over after height 99 knowing a blocksync peer advertises height 5000, while that peer has not announced a consensus height; peers still syncing defer their consensus gossip until ready. If a seed reports consensus height 1, mayPropose(100) immediately clears the tracker despite the known gap. A later announcement of 5000 cannot re-arm it. A historical catch-up commit can similarly populate PeerState.Height through SetHasCommit without reporting the sender's actual tip. Preserve the handover target until catch-up evidence establishes that it has been reached or legitimately superseded; a lower consensus height from the currently reporting peers is not sufficient proof.

source: ['claude']

Preserve processing before block persistence and finalization afterward,
while recording both execution phases in the upstream exec histogram.
Retain handover safeguards and both sets of applier regression tests.

<sub>🤖 Co-authored by [Claudius the Magnificent](https://github.com/lklimek/claudius) AI Agent</sub>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

consensus: after a block-sync handover while still behind, a validator proposes at historical heights and wedges itself permanently

2 participants