Repository navigation
[Router] Prewarm a booting replica's cache-aware tree from a warm sibling - #39750
Closed
Kangyan-Zhou wants to merge 10 commits into
Closed
Kangyan-Zhou wants to merge 10 commits into
Kangyan-Zhou wants to merge 10 commits into
Conversation
A router replica subscribes to each worker's KV topic mid-stream, so every block already resident in the engine's radix cache is invisible to it. The recovery paths for that — pulling tree state from a warm sibling, or later from the worker itself — all need the same thing first: a way to get the tree out and back in without losing what it knows. Add that surface and nothing else. `export_snapshot` emits a flat, parent-linked node list in DFS pre-order; `restore_snapshot` grafts one back, creating nodes that are absent and unioning carriers into nodes that are present, so restoring onto a live tree is well defined. Records are parent-linked by INDEX, not by block hash. The same hash legitimately occupies several tree positions, so a hash-keyed replay would land in `resolve_parent`'s ambiguous branch and could graft a chain under the wrong node. Full root-to-node hash paths would be unambiguous too, and quadratic in depth against chains that run thousands of blocks deep. Tiers ride along per carrier, parallel to `workers` by index, and `SnapshotNode::retain_carriers` is the only correct way to filter that pair — filtering `workers` alone silently re-pairs every later carrier with another carrier's tiers. An absent `tiers` list reads as device, which is what a carrier list meant before tiering existed, so the format needs no version bump in either direction. A tier entry naming only bits this build does not rank is DROPPED rather than folded onto device: folding would make a future tier's carrier a preferred device owner here, which is the trade `Tiers::for_store` already refuses to make for an unknown `medium`. A snapshot arrives over the network, so shape is validated before any mutation: backward parent references, in-range worker indices, and a tier list that either pairs one-to-one with its carriers or is absent. `restore_snapshot` stays `pub(super)` — the ids in a worker table must be resolved against the live worker set, never deserialized straight off the wire. Both passes release the tree lock every 4096 records. The tree is one lock, so an unchunked walk of a fleet-sized tree would stall the event pump — and with it every publisher cursor — for the walk's whole duration. The result is a point-in-time aggregate rather than one consistent instant, which is what a consumer grafting beneath a live delta stream already tolerates. Restore is safe across the seam because it runs on the pump, the only writer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YZorgAox1CpLNHSjzpcxdb
github-actions
Bot
added
the
dependencies
Kangyan-Zhou
force-pushed
the
router-peer-bootstrap
branch
from
September 16, 2026 17:38
fc94583 to
2c65af5
Compare
Nothing consumes this yet. It is the producer half of peer bootstrap: a replica that already holds a warm tree makes it fetchable, so a sibling starting later can graft it instead of routing cache-blind until traffic re-stores every block the engines already hold. `GET /internal/kv_snapshot` answers a `PeerSnapshot` — worker table, per-rank cursors, and the tree. The cursors are half the payload's purpose: a graft is only sound if the consumer can prove its own live stream resumes at `cursor + 1`, so the body has to say where the producer's stream stopped. `404` when this router maintains no local tree (an external Indexer is the routing signal, or cache-aware routing is off). A consumer reads that exactly as it reads an unreachable peer, which is also what an older router image answers for an unknown path, so a mixed-version fleet degrades to cold boots rather than errors. Three things the endpoint refuses to claim. `producer_ready` is false unless the tree is non-empty AND the hashing config is fully established: an empty tree would let two replicas in a rolling update bootstrap off each other and both inherit nothing, and a half-published config — `block_size` visible before the bigram flag, a real window in `add_worker` — would stamp unigram hashing on a bigram fleet and hand the recipient blocks it can never match. `BlockSizeOracle::hash_config` is the both-or-neither read that closes it. `?max_age_ms=N` lets the CONSUMER state its freshness requirement, because only the consumer knows when its ranks began holding. Staleness is a correctness input here, not a preference: an export older than the consumer's subscribe leaves the events in between in neither place, and the graft has to be thrown away. The build is single-flighted behind an async mutex and the encoded body is cached, so a boot herd shares one tree walk; the entry is stamped BEFORE the cursors are read, so it never claims to cover more of the publisher's stream than it does. `?cursors_only=true` answers the cursor table alone, read live with no cache. A splice probe asks one question — is your cursor for this rank above my watermark? — and paying a full tree export for it costs a walk, a multi-megabyte serialise and a compress, per peer per probe. That table is deliberately a SUPERSET of the export's, which is filtered to ranks still carrying nodes: a graft recipient needs a block source, but a witness only needs to have seen the publisher's stream. Its `nodes` is always empty, which makes the body ungraftable by construction rather than by discipline. The route carries its own gzip layer rather than the app carrying one. It is the only response here big enough to be worth compressing, and route-scoping means the proxy path pays no per-response predicate check and does not depend on tower-http continuing to exempt `text/event-stream`. Quality is `Fastest`: the body is highly repetitive, so level 1 already shrinks it several-fold, and this runs on a replica serving traffic during a boot herd. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YZorgAox1CpLNHSjzpcxdb
The config surface for peer bootstrap, and the registry a discovered peer set lands in. Nothing watches EndpointSlices yet and nothing fetches a snapshot, so an operator who sets the selector gets a validated config, a warning-free boot and `sgl_router_kv_bootstrap_peers 0` — no behaviour changes here. `PeerRegistry` exists because an empty peer set has two meanings and conflating them is the failure mode. "The watch has not delivered yet" is routine — worker discovery regularly wins that race — and treating it as "I have no siblings" makes a joining replica give up before its peers are even known. It tracks `synced` for that, and `ever_had_peers` for the other direction: an EndpointSlice repack deletes the last slice before its replacement arrives, and during a rolling update every sibling can be notReady at once, so a dip to empty after siblings were seen is transient. Candidates come back shuffled so simultaneous boots do not all stampede whichever peer sorts first. The selector is rejected unless the router maintains its own radix tree and finds workers through Kubernetes: with an external Indexer there is nothing to graft into, and without EndpointSlices there is no way to find a sibling. Both would otherwise be accepted and silently ignored, leaving cold boots with no explanation. `--kv-bootstrap-timeout-ms` is bounded at 600s because `Instant::now() + Duration::from_millis(n)` panics on overflow — an unbounded value would abort the process at startup instead of failing to parse. `sgl_router_kv_bootstrap_peers` plus its `_synced` companion make both readings of a zero visible. An RBAC gap on the router's own Service — the likeliest misconfiguration, since the existing worker Role already covers it namespace-wide — shows up as `synced 0` that never becomes 1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YZorgAox1CpLNHSjzpcxdb
The other half of peer discovery: a second EndpointSlice watch, scoped by `--kv-peer-selector`, keeps `PeerRegistry` current. Still nothing consumes a snapshot — the consumer lands next — but with this an operator can set the selector and confirm discovery works before any behaviour depends on it. A replica must not offer itself as a bootstrap source, and POD_IP alone cannot do that. It carries only the pod's PRIMARY address, so on a dual-stack Service the pod's secondary-family entry survives an IP filter — which then latches "siblings exist" and stops a genuinely lone replica from ever concluding it is alone. POD_NAME matched against each endpoint's `targetRef` excludes the whole pod; POD_IP stays as the fallback for endpoints without one. With neither set the replica stays in its own peer list, which is wasteful rather than wrong, and says so at WARN. The watch buffers a relist and publishes only on `InitDone`. Publishing each `InitApply` would expose a partially rebuilt set, and an empty partial marks the registry synced — the exact state that turns a bootstrap into a cold boot. The regression test asserts mid-relist, because checking only the settled state passes with the bug present. It starts before worker discovery, because a bootstrap consults the peer set the moment the first worker appears: an empty set there means that worker's ranks skip bootstrap and run cold. Failure to start is non-fatal — routing does not depend on peer discovery, and losing it means replicas boot cold, which is the pre-existing behaviour. Deployments that want self-exclusion need POD_NAME/POD_NAMESPACE/POD_IP from the downward API; the manifest that wires them ships with the e2e test in the consumer change, where there is something for it to exercise. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YZorgAox1CpLNHSjzpcxdb
The consumer's half of the wire contract, with no consumer yet: the type that turns a peer's `PeerSnapshot` into something graftable, the per-rank state that tracks a bootstrap, and the client that fetches a body. The pump that uses them lands next. The trust boundary is enforced by types rather than by discipline. A snapshot arrives over the network as `PeerSnapshot`, carrying `WireWorker` rather than `KvWorkerId`; `VettedSnapshot::from_wire` is the only bridge, and it resolves every wire identity against the locally discovered worker set, so a peer cannot introduce a `KvWorkerId` the local registry does not already know — which would otherwise let routing resolve to a non-registered endpoint. `VettedSnapshot`'s fields are private and `graft_into` is the only route to the tree's restore path from outside the module, so format, block-size and parent-bound checks cannot be skipped by a caller that assembles nodes itself. That is also what retires `restore_snapshot`'s `#[allow(dead_code)]`. Vetting rejects rather than repairs: an unknown format, a producer that is itself cold or not ready, an out-of-range or forward parent reference, tiers that do not pair with workers, a block-size mismatch, or a snapshot that prunes to nothing. A rank whose snapshot is rejected runs cold, which is slow and visible; a snapshot grafted on a hope can hide a missed `BlockRemoved` as a permanent false cache hit, which is silent. `BootstrapTracker` holds per-rank state and the outcome tallies. Its epoch is per rank, not process-wide: a process-wide counter would let one worker's re-add invalidate another worker's in-flight task. `RankOutcome` and `SnapshotOutcome` are deliberately separate tallies — one counts ranks that finished bootstrapping, the other counts peers that were asked — because a single peer answers for many ranks and summing them together would make neither readable. Three bounds on a fetch, not one. `reqwest`'s `timeout` is a TOTAL request bound, so a single value has to serve two opposite jobs: cutting a peer that is gone, and allowing a large body to finish. Sized for the first, it rejects every peer whose snapshot is big enough to be worth having — the failure gets MORE likely the warmer, i.e. the more useful, the fleet's tree becomes. So `connect_timeout` cuts a peer that is gone, `read_timeout` cuts one that stopped sending mid-body, and the total bounds only a transfer that is progressing. `flate2` moves from dev-dependencies to dependencies for the inflate path, used directly rather than through reqwest's `gzip` feature: that feature is crate-wide, so enabling it would make every client in the process advertise gzip and auto-decode — including the proxy client carrying SSE. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YZorgAox1CpLNHSjzpcxdb
…tream A booting replica now holds each rank's batches while it pulls a snapshot from a warm sibling, grafts that snapshot underneath them, and releases the queue — so it starts with the fleet's prefixes instead of relearning them from traffic. Ordering is the design. The subscriber starts FIRST, before any fetch, so no delta can fall into the gap between the peer's export and our first received batch. The peer's cursor then does two jobs: as a FILTER it drops the held batches the snapshot already reflects, and as a WATERMARK it proves the stream resumes at `cursor + 1`. A hole there means a delta was lost, and a lost `BlockRemoved` is a permanent false cache hit — silent, where a cold rank is merely slow and visible in the metrics. So every edge resolves to `Failed` for the affected rank: unreachable peer, an older image's 404, a peer still cold, a block-size mismatch, an unknown worker, a sequence gap. Run cold, never wrong. Hold-back overflow deliberately breaks that symmetry: it discards the queued prefix, because replaying a discontiguous one is worse than resuming live. One sweep at a time, not one per worker. A snapshot is fleet-wide, so sweeping from `add_worker` re-downloads the same document once per engine — on a fleet of any size that is one fetch of a multi-megabyte body per engine per booting replica, most of them timing out. A coordinator takes whatever is queued, awaits ONE sweep, and merges everything that arrived while it ran. The fetch IS the batching window, so this costs no latency, and it is safe because the invariant is a sequence condition adjudicated per rank — a rank whose publisher advanced in between is discarded to `gap`, never spliced over a hole. Freshness travels with the batch: the strictest `holding_since` in it becomes the `max_age_ms` the fetch asks for, so a peer serving from cache rebuilds rather than handing back an export older than some rank in the batch. A gap is the costliest failure — a snapshot fetched, grafted, then thrown away because the live stream did not join its watermark — so a gapped rank is handed back for one more sweep, capped at one retry, stamped NOW and refused a ride-along on a sweep whose request predates it. Without both stamps the retry is adjudicated against a snapshot taken before the gap and re-gaps by construction. Grafts run on the pump, like every other tree write. `remove_worker`'s teardown moves there too: `apply_snapshot` samples its gates once and then does O(nodes) of work, so off-pump teardown could let a graft finish after `clear_worker` and re-insert carriers nothing would ever reclaim. Three gates reject a stale graft — wrong incarnation, no longer live, no longer pending — and the epoch is per rank, because a process-wide counter would let one worker's re-add invalidate another's in-flight task. When the held queue is empty at graft time there is nothing to check the watermark against, so the check is deferred to whichever batch arrives first. A rank that then goes quiet leaves the question open; the follow-up commit asks the fleet instead of guessing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YZorgAox1CpLNHSjzpcxdb
A graft whose held queue was empty leaves its watermark unchecked until a live batch arrives, and nothing guarantees one ever does. An idle rank would serve grafted state forever with its continuity unproven, and a `BlockRemoved` lost in the subscribe window would survive as a permanent false cache hit that no later event corrects. Silence is not evidence, so the bounded wait ends in a QUESTION, not a verdict. Reaching the deferred path means nothing arrived between subscribing and grafting, and the subscriber is live before the fetch — so silence is far more often "this rank published nothing" than "we lost a delta". Discarding on a timer would cost every quiet fleet its warm tree, which is the regression this feature exists to prevent. So the pump asks the fleet whether the publisher moved past the watermark. Any peer's cursor is admissible: sequence numbers are the publisher's, so a peer reporting one above ours proves a batch was emitted that we never saw. A peer too cold to bootstrap from is still a valid witness, which is why the probe reads the wire cursor directly instead of vetting. It asks for the cursor table alone, not a snapshot — the question is answered completely by one integer per rank, and fetching a tree to read it made the proof cost scale with the tree, so a fleet large enough to need bootstrap was also the fleet that could not afford to prove it. Only a witness ABOVE the watermark discards. A graft no batch and no peer can speak to is KEPT, tallied `warm_unwitnessed` rather than `warm`, after `MAX_UNKNOWN_PROBES` unanswerable rounds — without a stop, a single-replica deployment would re-probe forever and its verdict would never resolve in the metrics. The probe runs off-pump because it does network I/O; the verdict comes back over the pump's own control channel so the tree write stays on the single writer. An outstanding probe blocks relaunch inside its timeout and stops blocking past it: a verdict that never lands (a panicked probe task, say) would otherwise freeze the rank in `Recovered` with no outcome ever recorded. Also re-attaches `demote_unproven_rank`'s doc comment, which sat above `requeue_gapped_rank` describing the wrong function. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YZorgAox1CpLNHSjzpcxdb
…t copied Component-level proof over the real transport: an axum server serving `/internal/kv_snapshot`, a reqwest client fetching it, JSON over a loopback socket. The interesting failure modes live in the wire format and the vetting step, not in the tree algebra — that is already covered by the unit tests on `export_snapshot` / `restore_snapshot`. The equivalence assertion is deliberately behavioural: for a large query set, `match_prefix` must return the same matched length and the same carrier set on both replicas. That is the property routing actually depends on; comparing node counts alone would pass while routing diverged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YZorgAox1CpLNHSjzpcxdb
A kind-cluster proof of the two questions the feature exists to answer: a replica joining a warm fleet ends up with the same cache-aware view as the replicas it bootstrapped from, and it keeps up with new engine events afterwards — the snapshot is spliced UNDER the live stream, not substituted for it. Plus the rolling-update hazard: new replicas must not bootstrap from each other and inherit an empty tree. Ships the KV-publishing fake worker the test drives, and the manifest that wires `POD_NAME`/`POD_NAMESPACE`/`POD_IP` from the downward API and the EndpointSlice RBAC the peer watch needs — deferred from the discovery commit to here, where there is something for it to exercise. Three anti-flake decisions, because the naive version of this test is flaky: events are driven by a POST rather than a timer, so nothing is ever in flight the test did not ask for; views are compared only after every replica's cursor has provably caught up to the worker's `last_seq`, never on a fixed sleep; and nothing asserts on a transient mid-rollout state — the scale-up case is asserted directly and the rollout case only after `rollout status` completes. Views are compared canonically, since snapshot node order follows per-shard hash-map iteration. Not yet run: this needs a kind cluster, so it is verified by construction only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YZorgAox1CpLNHSjzpcxdb
…vable A replica that boots with an empty KV tree does not just route cache-blind itself — its dispatches scatter the prefixes the warm replicas were keeping consolidated, so it degrades the engines' locality for the whole fleet. A rolling update does that to every replica in turn, and with nothing in the metrics to attribute the hit-rate drop to. `/readyz` grows two conditions. The third holds 503 until peer bootstrap has settled: a snapshot grafted, or every sibling having proved it has nothing, or `--kv-bootstrap-timeout-ms` giving up. It latches, so a later scale-up can never drag an already-serving replica back out of the Service. It is inert unless both cache-aware routing and a peer selector are configured, so nothing changes for a fleet that has not opted in. The fourth is `--kv-bootstrap-seed-required`, off by default, and it exists because condition 3 settles either way once the deadline expires — that escape is load-bearing, and it is also why `settled()` cannot tell "seeded" from "gave up". This one can: it refuses readiness when a sweep proved siblings were there and their tree could not be pulled. A NotReady pod leaves the Service, so a failed seed then STALLS a rolling update with the previous generation still serving, instead of completing it with cache-blind replicas. Which verdict may gate is the entire safety argument. `NoPeers` (first deploy, single replica) and `FleetCold` (every sibling PROVED it holds nothing) are legitimate nothing-to-inherit outcomes; gating on either bricks a boot that has nothing to inherit — and because an unready replica leaves its own EndpointSlice, every sibling would then see an empty peer set: a fleet-wide deadlock with no automatic exit. Only `TimedOut` over a non-empty candidate set counts. Two further rails cover the case the verdict genuinely cannot distinguish, a simultaneous fleet-wide restart where every peer is booting: the hold is bounded at three times the bootstrap budget and then opens with a WARN, and it latches open the first time `/readyz` answers 200. The tests assert over the verdict class rather than by example, because the wrong answer in either direction is silent — gate too little and a bad rollout ships, gate too much and the fleet deadlocks. The metrics make the whole path attributable. `bootstrap_state` per rank, so a rank stuck pending (holding its events back, heading for overflow) is visible; `peer_snapshot_total` by fetch outcome, where a fleet pinned at `unreachable` with a warm tree means the per-fetch bound is too small for the body rather than a network fault; `bootstrap_rank_total` once per rank at its final verdict; `bootstrap_sweep_total` per sweep, which is what separates a healthy early settle on a cold fleet from burning the whole deadline — the rank outcomes fold both into `abandoned`. `seed_failed` is emitted whether or not the gate is armed: with it off, that gauge is the only thing that says the seed did not land. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YZorgAox1CpLNHSjzpcxdb
Kangyan-Zhou
force-pushed
the
router-peer-bootstrap
branch
from
September 16, 2026 17:58
2c65af5 to
a9c44dc
Compare
This was referenced Sep 22, 2026
ShangmingCai
marked this pull request as ready for review
October 1, 2026 03:56
Collaborator
|
Closed as all split PRs have been fixed and merged. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A router replica subscribes to each worker's KV topic mid-stream. ZMQ SUB
delivers deltas from wherever the publisher has reached, so every block already
resident in the engine's radix cache is invisible to that replica — and engines
publish
BlockStoredonly as they insert, so a prefix cached hours ago isnever re-announced. A fresh replica gets the deep blocks and none of the roots.
It is worse than one slow replica. A cache-blind router scatters prefixes the
warm replicas were keeping consolidated, so it degrades the engines' locality
for everyone. A rolling update does that to every replica in turn.
The effect is not subtle. After a router rollout the average prefix overlap per
routing decision falls to near zero and the engines' prefix-cache hit rate
follows it down, while an oracle replay of the same teed traffic shows the
workload's reuse is exactly where it was. The routers simply cannot find it.
What
A booting replica pulls a tree snapshot from a warm sibling over HTTP and
grafts it beneath its live delta stream, holding
/readyzuntil the sweepsettles. Off unless
--kv-peer-selectoris set.export_snapshot/restore_snapshotGET /internal/kv_snapshotproducer + cache + gzip +cursors_only--kv-peer-selector, the peer registry,_peersmetricsVettedSnapshot,BootstrapTracker, the fetch client/readyzgate, seed-required gate, bootstrap metricsCommit 6 is the one unit that does not divide further: producer and consumer are
joined by the pump's
PumpControlchannel, so any cut between the sweep and thegraft would have to ship an intermediate state that is not in the design.
Ordering is the design
The subscriber starts first, before any fetch, so no delta falls into the
gap between the peer's export and our first received batch. The peer's cursor
then does two jobs: as a FILTER it drops the held batches the snapshot already
reflects, and as a WATERMARK it proves the stream resumes at
cursor + 1.A hole there means a delta was lost, and a lost
BlockRemovedis a permanentfalse cache hit — silent, where a cold rank is merely slow and visible in the
metrics. So every edge resolves to
Failedfor the affected rank: unreachablepeer, an older image's 404, a peer still cold, a block-size mismatch, an
unknown worker, a sequence gap. Run cold, never run wrong.
Two paths deliberately break that symmetry, both bounded. Hold-back overflow
discards the queued prefix (replaying a discontiguous one is worse than
resuming live). And a graft that no batch and no peer can speak to is KEPT,
tallied
warm_unwitnessedrather thanwarm— discarding on a timer wouldcost every quiet fleet its warm tree, which is the regression this prevents.
Silence is not evidence, so an unproven rank asks the fleet: any peer's
cursor above the watermark proves a batch was published that we never saw.
One sweep, not one per worker
A snapshot is fleet-wide, so sweeping from
add_workerre-downloads the samedocument once per engine — on a large fleet that is one fetch of a
multi-megabyte body per engine per booting replica, most of them timing out. A
coordinator takes whatever is queued, awaits ONE sweep, and merges everything
that arrived while it ran. The fetch IS the batching window, so it costs no
latency, and it is safe because the invariant is a sequence condition
adjudicated per rank.
Three fetch bounds, not one
reqwest'stimeoutis a TOTAL request bound, so one value had to serve twoopposite jobs. Sized to cut a peer that has gone away, it also cuts a large
snapshot that is transferring perfectly well: every fetch times out, every peer
is booked
unreachable, and the sweep abandons every rank — more likely thewarmer, i.e. the more useful, the tree becomes. Raising the deadline only buys
more timeouts, because the per-fetch value is
min(deadline / 4, cap). Nowconnect_timeoutcuts a peer that is gone,read_timeoutcuts one that stoppedsending mid-body, and the total bounds only a transfer that is progressing.
Readiness
/readyzholds 503 until bootstrap settles, bounded by--kv-bootstrap-timeout-msand latching so a later scale-up cannot un-ready aserving replica.
--kv-bootstrap-seed-required(off by default) additionallyrefuses readiness when a sweep proved siblings were present and their tree
could not be pulled, so a failed seed stalls a rolling update instead of
completing it with cache-blind replicas.
Which verdict may gate is the whole safety argument:
NoPeers(first deploy)and
FleetCold(every sibling proved it holds nothing) must never gate —an unready replica leaves its own EndpointSlice, so gating there deadlocks the
fleet with no automatic exit. Only
TimedOutover a non-empty candidate setcounts, the hold is bounded at 3× the budget, and it latches open on the first
ready 200.
Configuration
Needs
get/list/watchon EndpointSlices for the router's own Service, andPOD_NAME/POD_NAMESPACE/POD_IPfrom the downward API for self-exclusion(see
tests/e2e/k8s_integration/manifests/kv-bootstrap.yaml).New series:
sgl_router_kv_tree_nodes,..._bootstrap_settled,..._bootstrap_seed_failed,..._bootstrap_state,..._peer_snapshot_total,..._bootstrap_rank_total,..._bootstrap_sweep_total,..._bootstrap_peers{,_synced}.Tests
cargo test: 828 lib + 4 + 74 component + 101 proxy. Every commit passes onits own —
cargo fmt --check,cargo clippy --all-targets -- -D warningsandthe lib suite were run at each of the ten commits, not only on the tip. New k8s
e2e (
test_kv_peer_bootstrap.py+ a KV-publishing fake worker) asserts a bootingreplica reaches the same
match_prefixanswers as the warm replica it copiedfrom.
Not yet run: the k8s e2e needs a kind cluster; it is verified by
construction only.
Review note
A review of the finished branch found 8 defects, all fixed in the commit that
introduced them. The one worth naming:
settled()latched on a deadline armedat construction, and
/metricsor any peer probe reads it — so on a fleetwith slow worker discovery the readiness gate died before the first rank
registered and the feature was silently a no-op. Reproduced with a failing test
before fixing.
🤖 Generated with Claude Code
https://claude.ai/code/session_01YZorgAox1CpLNHSjzpcxdb
CI States
Latest PR Test (Base): ❌ Missing
run-cilabel -- add it to run CI tests.Latest PR Test (Extra): ❌ Blocked --
run-ciis required first.Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.