unroll: Reorg-safe unilateral-exit subsystem

[ellemouton]

Summary

Makes the client-side unilateral-exit subsystem reorg-safe end-to-end. Today the unroll actor treats every confirmation as monotonic: if a proof-graph anchor or the sweep tx is reorged out (or vanishes during daemon downtime), the planner happily progresses off stale state and can broadcast a sweep against a target the chain no longer holds. This PR makes the entire stack reversible:

• chainsource gains reorg-aware conf/spend watches with reorg + finality events (ConfReorgedEvent, ConfDoneEvent, spend equivalents). The conf/spend sub-actors are multi-shot, and finality is synthesized height-based (FinalityDepth) so backends that drop the underlying signal (lndclient) still produce Done.
• chainbackends forwards lndclient reorg pings into the new event surface; bare correlation IDs only, since gRPC transport drops the depth/block-hash payloads.
• txconfirm confirmation watches become reorg-aware end-to-end with a new Finalized terminal state; Confirmed is no longer terminal until finality.
• unroll FSM gains:
  • Rollback semantics for TxReorgedEvent / SpendReorgedEvent that prune the reorged subtree from ConfirmedTxids + InFlightTxids and downgrade the sweep when the target is lost.
  • A new AwaitingExternalSpendFinality state with a persisted ProvisionalExternalSpend anchor so an external spend doesn't terminally fail the job until it's finalized.
  • PhaseCompleted is provisional until the sweep is Finalized; the registry only fires UnrollTerminatedMsg after finality.
  • Restart reconciliation against the canonical chain via a new ChainReconciler interface. The chainsource-backed implementation probes via future-mode RegisterConf/RegisterSpend with bounded timeouts, baked per-actor caller-IDs (target outpoint) so two restored actors probing a shared proof-graph ancestor cannot collide on chainsource service keys.

Tests

• Unit: chainsource reorg-aware conf/spend actor tests, lndclient forwarder tests, txconfirm reorg lifecycle tests, unroll reorg-safety integration tests (proof-root reorg blocks downstream materialization, target reorg clears CSV maturity, sweep-confirmation reorg reverses Completed, external-spend reorg resumes actor, sweep stays provisional until finalized), restart-reconciliation tests (target gone, sweep gone, external-spend gone, target height refresh, combined offline-reorg restart), concurrent-probe service-key collision test.
• Systest (against bitcoind + lnd in regtest): full chainsource conf reorg round-trip, full txconfirm reorg round-trip including the new TxReorged event.

Stacked on

Targets `recipient-fraud-watcher-client` because the two stacks both extend `unrollplan.State`, the unroll FSM, and the actor checkpoint codec. No functional dependency, but rebasing onto main would require manual conflict resolution in 7-8 files; keeping the stack until the fraud PR lands.

Test plan

• CI green on unroll, chainsource, chainbackends, txconfirm, darepod unit suites
• CI green on systest (`TestChainSourceConfReorgRoundTrip`, `TestTxConfirmReorgRoundTrip`)
• `make lint-native` clean
• Reviewer sanity-check that the restart-reconciliation ordering invariant (reconcile runs before FSM session is bound; before block sub / spend watch / ResumeEvent reissue) holds in `actor.go:ensureLoaded`

---

• Closes unroller: add re-org awareness #207 (unroller re-org awareness — full implementation lands in this PR)

[gemini-code-assist]

Code Review

This pull request introduces comprehensive reorg-awareness across the chain notification pipeline, extending from the LND backend through the chainsource and txconfirm subsystems to the unroll registry. Key changes include the implementation of multi-shot confirmation and spend watches, height-based finality synthesis for gRPC-based backends, and a restart reconciliation mechanism to prune stale anchors. My feedback focuses on several resource leaks identified in the txconfirm actor, where the underlying chainsource watch is not explicitly unregistered before evicting terminal entries during cancellation, notification retries, or asynchronous completion.

[gemini-code-assist]

The chainsource confirmation watch should be explicitly unregistered when a terminal entry is cancelled. If the entry reached a terminal state (Finalized or Failed) but was waiting for notification completion or retry, the watch might still be active in the background. Unregistering here ensures resource hygiene.

[gemini-code-assist]

The chainsource confirmation watch is leaked when a terminal notification is successfully delivered via the retry path in handleConfirmationObserved. While handleConfirmationDone correctly unregisters the watch upon immediate successful notification, this retry branch calls evictTerminal without first invoking unregisterConfWatch. This results in the underlying chainsource sub-actor remaining active indefinitely for finalized transactions that required notification retries.

[gemini-code-assist]

The chainsource confirmation watch is leaked when a terminal notification is successfully delivered via the retry path in handleBlockObserved. Similar to the issue in handleConfirmationObserved, this branch calls evictTerminal without first invoking unregisterConfWatch, leading to a resource leak in the chainsource actor system.

[gemini-code-assist]

The chainsource confirmation watch is leaked when an asynchronous terminal notification completes. If handleConfirmationDone (or handleConfirmationFailed) defers notification to a goroutine, it returns early without unregistering the watch. When the result eventually arrives here, the entry is evicted but the watch is never cleaned up. Adding an explicit unregister call before eviction ensures the sub-actor is released.

[claude]

Claude finished @ellemouton's task in 6m 7s —— View job

---

PR Review: unroll: Reorg-safe unilateral-exit subsystem

• Gather context: read PR diff and key files
• Review chainsource reorg-aware conf/spend actors
• Review txconfirm reorg lifecycle and resource cleanup
• Review unroll FSM rollback semantics and new states
• Review restart reconciliation ordering invariants
• Review chainbackends lndclient forwarding
• Synthesize findings and post full review

---

Overall Assessment

The design is sound end-to-end. The reorg rollback semantics, finality synthesis, provisional-state promotion, and restart-reconciliation ordering all hold up under scrutiny. A few targeted notes below.

---

Gemini's Resource-Leak Findings Are False Positives

All four flagged call-sites (actor.go:535, actor.go:593, actor.go:756, actor.go:1325) pass through evictTerminal, which already guards the unregister:

// txconfirm/actor.go:1240-1246
func (a *TxBroadcasterActor) evictTerminal(ctx context.Context, entry *trackedTx) {
    if entry.confWatchRegistered {
        if err := a.unregisterConfWatch(ctx, entry); err != nil { ... }
    }
    ...
}

The confWatchRegistered sentinel means double-unregisters are impossible and the watch is freed exactly once regardless of which eviction path wins. No leak.

---

Reconciliation Ordering Invariant (requested in test plan)

The ordering in ensureLoaded (unroll/actor.go:611–688) is correct:

1. Load proof
2. Load descriptor
3. Construct planner
4. reconcileOnRestart() — mutates b.pending in place before any session exists
5. Build FSM session from post-reconciliation checkpoint
6. ensureBlockSubscription
7. ensureSpendWatch

Reconciliation runs before the FSM session is bound, and therefore before any ResumeEvent reissue path can fire (which lives inside driveEvent → routeOutbox). No stale sweep can be re-broadcast off a checkpoint the reconciler has not yet validated.

---

Issue 1: reconcileCheckpoint probes already-pruned txids — potential slow restart

reconcileCheckpoint (unroll/reconcile.go:117–150) iterates over a snapshot of confirmedIn, but pruneReorgedSubtree removes descendants from the live slice while the loop runs. If the slice contains both an ancestor A and descendant B (sorted by txid hash, not topology), and A is processed first and its subtree (including B) is pruned, the loop will still issue reconciler.ConfirmedTx(B) when it reaches B — a 10-second probe that can never return a useful answer.

With a deep confirmation chain and multiple reorged ancestors, each redundant probe burns up to defaultReconcileProbeTimeout (10s). A 5-deep proof graph with 2 reorged subtrees could add 30+ seconds to restart.

Suggested fix — skip txids already absent from checkpoint.State.ConfirmedTxids:

for _, txid := range confirmedIn {
    // Skip: already pruned by an earlier ancestor's subtree eviction.
    if !containsHash(checkpoint.State.ConfirmedTxids, txid) {
        continue
    }
    anchor, err := reconciler.ConfirmedTx(ctx, txid)
    ...
}

Fix this →

---

Issue 2: cleanupConfWatch fires unconditionally — wasted unregister per successful probe

In unroll/reconcile_chainsource.go:179–180:

defer r.cleanupConfWatch(callerID, &probeTxid, probePkScript)
event, err := confResp.Future.Await(probeCtx).Unpack()

The defer runs even on the success path, sending an UnregisterConfRequest for a service key whose ConfActor already self-exited after delivering the future-mode event. The request is harmless (chainsource no-ops on unknown keys) but redundant — one extra round-trip per confirmed txid per restart. The same applies to cleanupSpendWatch (reconcile_chainsource.go:254–255).

Consider returning early from the deferred cleanup when the future succeeded:

probeSucceeded := false
defer func() {
    if !probeSucceeded {
        r.cleanupConfWatch(callerID, &probeTxid, probePkScript)
    }
}()
event, err := confResp.Future.Await(probeCtx).Unpack()
if err == nil {
    probeSucceeded = true
    return fn.Some(...), nil
}

Fix this →

---

Issue 3: Stale comment in notifyRegistryIfTerminal

unroll/actor.go:1322:

// The chainsource transport (lndclient over gRPC) does not surface a finality
// signal today, so production deployments may retain the child indefinitely after
// a sweep confirms; height-based finality gating is a Phase 7 follow-up.

This was accurate before this PR stack, but darepod: Enable height-based finality on the chainsource actor (commit 6f18ca9) wires FinalityDepth > 0 in production, so TxFinalizedMsg will now arrive for confirmed sweeps and sweepFinalized will be latched. The "Phase 7 follow-up" reference is now stale and could mislead future reviewers. Suggest updating to note that height-based finality is active when FinalityDepth > 0 is configured (production default), with the lndclient-no-Done caveat still applying to deployments that have not set it.

---

Correctness Notes (all confirmed good)

maybeLatchSweepFinalized ordering (actor.go:235–242): The latch runs after ensureLoaded but before handleEvent, so notifyRegistryIfTerminal inside driveEvent sees sweepFinalized = true and fires UnrollTerminatedMsg on the same message turn. Correct.

ConfActor height-based finality (chainsource/conf_actor.go:318–334): Block subscription arms on first confirmation (a.blockReg == nil guard), resets confirmHeight = 0 on reorg, and resumes depth counting from the next reconfirmation height. The single block registration correctly services multiple confirm→reorg→reconfirm cycles. Correct.

handleConfirmationDone state guard (txconfirm/actor.go:683–685): Drops ConfDoneEvent when state is not Confirmed (e.g. after a reorg moved it back to AwaitingConfirmation). This is safe because the chainsource ConfActor continues watching and will fire a fresh Confirmed then Done sequence after re-confirmation. Correct.

reconcileExternalSpend ordering vs sweep reconciliation (unroll/reconcile.go:193–272): Appears unsafe at first glance — if the sweep was Broadcasted its txid is cleared by the sweep reconciliation block, so the benign-spender check Sweep.Txid.IsSome() would miss. However, SpentOutpoint only fires for confirmed spends; a sweep that is only in the mempool does not produce a spend event within the probe timeout. And a confirmed sweep would have ConfirmedTx = Some, so its txid would not be cleared in the first place. Ordering is safe.

lndclient Done channel (chainbackends/lndclient_adapters.go:279–287): Correctly allocated but never written to, with FinalityDepth acting as the production substitute. The comment is accurate. The chainntnfs.ConfirmationEvent.Done and SpendEvent.Done being live-but-never-signalled channels never race because downstream code (ConfActor/SpendActor) checks them in select arms that have competing height-based finality arms — whichever fires first wins and the goroutine returns.

---

[levmi]

Triage linkage check — verified this PR's live-rollback reducer implements #207's full proposed solution (reorg notifications, roll back ConfirmedTxids, re-register conf watches, reset CSV-maturity baseline, backward state transition) and then some.

Suggest adding Closes #207 so merge auto-closes it. (Heads-up: this is a draft stacked on a non-main base — the auto-close will only fire once it lands on the default branch.)

[ellemouton]

Superseded by #897 as part of condensing the reorg-safety client stack (epic lightninglabs/darepo#454) from 12 PRs into 3. The commits are carried over unchanged; see #897. Branch retained as a backup.