[Router] Ask the fleet when a graft's splice goes unwitnessed (10/13)

[Kangyan-Zhou]

Stack 10 of 13. Base: router-peer-bootstrap-9-shared-fetch. 5 files changed, 833 insertions(+), 11 deletions(-)

The stack

#	PR	lines	what it adds
1	#40687	962	the sharded tree gains export_snapshot / restore_snapshot; nothing calls them yet
2	#40688	2213	GET /internal/kv_snapshot — the producer half; nothing consumes it yet
3	#40689	536	--kv-peer-selector, the peer registry, the _peers gauges; no behaviour change
4	#40690	1150	the EndpointSlice watch that fills the registry; still no consumer
5	#40691	1177	BootstrapTracker + the fetch client; the consumer's state and transport
6	#40692	933	VettedSnapshot::from_wire — the only bridge from wire bytes to the tree
7	#40693	2750	the pump holds a Pending rank's batches and grafts a snapshot handed to it
8	#40694	2478	the sweep: ask siblings, vet, deliver — one sweep per discovered worker
9	#40695	1211	fold a discovery burst into one fleet-wide fetch, plus the gap retry it enables
10	#40696	844	a graft nothing witnessed asks the fleet instead of guessing ← this PR
11	#40697	646	component test over the real transport: same match_prefix answers as the source
12	#40698	943	kind-cluster proof, the KV-publishing fake worker, and the RBAC/downward-API manifest
13	#40699	869	/readyz holds until bootstrap settles; --kv-bootstrap-seed-required; the metrics

Each PR's base is the branch below it, so every diff shown here is that
PR's own change. Review bottom-up; GitHub retargets each child to main as
its parent merges.

What the series does

A router replica subscribes to each worker's KV topic mid-stream, so every
block already resident in the engine's radix cache is invisible to it — and
engines publish BlockStored only as they insert, so a prefix cached hours
ago is never re-announced. A cache-blind replica then scatters the prefixes the
warm replicas were keeping consolidated, degrading the engines' locality for
the whole fleet; a rolling update does that to every replica in turn. This
series makes a booting replica pull a tree snapshot from a warm sibling over
HTTP and graft it beneath its live delta stream. Off unless --kv-peer-selector
is set.

Supersedes #39750, which carried the same work as one branch on a stale base.

What this change does

A graft whose held queue was empty leaves its watermark unchecked until a live
batch arrives, and nothing guarantees one ever does. An idle rank would serve
grafted state forever with its continuity unproven, and a BlockRemoved lost in
the subscribe window would survive as a permanent false cache hit that no later
event corrects.

Silence is not evidence, so the bounded wait ends in a QUESTION, not a verdict.
Reaching the deferred path means nothing arrived between subscribing and
grafting, and the subscriber is live before the fetch — so silence is far more
often "this rank published nothing" than "we lost a delta". Discarding on a timer
would cost every quiet fleet its warm tree, which is the regression this feature
exists to prevent.

So the pump asks the fleet whether the publisher moved past the watermark. Any
peer's cursor is admissible: sequence numbers are the publisher's, so a peer
reporting one above ours proves a batch was emitted that we never saw. A peer too
cold to bootstrap from is still a valid witness, which is why the probe reads the
wire cursor directly instead of vetting. It asks for the cursor table alone, not
a snapshot — the question is answered completely by one integer per rank, and
fetching a tree to read it made the proof cost scale with the tree, so a fleet
large enough to need bootstrap was also the fleet that could not afford to prove
it.

A peer counts as a witness for a rank only when its table names that rank AT OR
ABOVE the watermark: a peer that never saw the rank, or whose cursor is below
the watermark (a stalled subscription, a reset we missed), cannot say what came
after, and counting it would manufacture a "no advance" out of ignorance. A
peer whose table carries a SNAPSHOT_FORMAT this build does not recognise is
skipped, the same rule vetting applies. A witness ABOVE the watermark resolves
the rank as Advanced, which goes through the same resolve_gap as the
first-batch check — discard the graft, and one retry if the tracker allows it.
A graft no batch and no peer can speak to is KEPT, tallied warm_unwitnessed
rather than warm, after MAX_UNKNOWN_PROBES unanswerable rounds — without a
stop, a single-replica deployment would re-probe forever and its verdict would
never resolve in the metrics.

The probe runs off-pump because it does network I/O; the verdicts come back over
the pump's own control channel so the tree write stays on the single writer. It
runs as ONE pass per sweep tick for every due rank, fetching each peer's cursor
table at most once and judging every rank against it, so idle ranks do not
multiply the fleet's work. A pass is capped by SPLICE_PROBE_BUDGET, one tick
short of the proof timeout, so passes never overlap and a hung peer cannot
stack up concurrent passes; a peer not reached inside the budget is simply not
a witness this pass. Each verdict carries the graft's watermark as well as its
epoch: a gap retry re-grafts under the same incarnation, so a stale verdict
about the previous graft is recognised and ignored. A verdict that never lands
(a panicked probe task, say) does not freeze the rank: it is re-probed once
the timeout since the last launch elapses.

Fresh-engine fix

Rebased onto the fixed #40693–#40695; the only code conflict was placing demote_unproven_rank next to resolve_from_origin. The pump's batch-0 restart arm removes the rank's awaiting_splice_proof entry, so a probe verdict still in flight for the replaced graft is dropped. probe::tests::pump_drops_a_probe_verdict_about_a_graft_a_restart_replaced pins that.

Tests

cargo fmt --check, cargo clippy --all-targets -- -D warnings, and the lib +
component + proxy suites all pass on this branch on its own, not only on the
tip of the stack.

Review pass

Reviewed with /code-review --fix and /simplify; fixes were folded into this PR's own commit and the stack was re-verified tier by tier (cargo fmt --check, cargo clippy --all-targets -D warnings, lib + component + proxy tests on every branch).

• Fixed: a stale verdict about a previous graft is ignored (watermark gate), and probe passes can no longer overlap.
• File layout: this PR's code lives in state/kv_events/index/probe.rs (new), plus index.rs; no file the stack creates exceeds ~1,300 lines.

🤖 Generated with Claude Code

https://claude.ai/code/session_016HmJvHV7QDPk3qAjQYzthd

---

CI States

Latest PR Test (Base): ✅ Run #36700533783
Latest PR Test (Extra): ❌ Run #36700533552
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.