[Router] Watch EndpointSlices for sibling router replicas (4/13)

[Kangyan-Zhou]

Stack 4 of 13. Base: router-peer-bootstrap-3-peer-selector. 3 files changed, 1129 insertions(+), 21 deletions(-)

The stack

#	PR	lines	what it adds
1	#40687	962	the sharded tree gains export_snapshot / restore_snapshot; nothing calls them yet
2	#40688	2213	GET /internal/kv_snapshot — the producer half; nothing consumes it yet
3	#40689	536	--kv-peer-selector, the peer registry, the _peers gauges; no behaviour change
4	#40690	1150	the EndpointSlice watch that fills the registry; still no consumer ← this PR
5	#40691	1177	BootstrapTracker + the fetch client; the consumer's state and transport
6	#40692	933	VettedSnapshot::from_wire — the only bridge from wire bytes to the tree
7	#40693	2750	the pump holds a Pending rank's batches and grafts a snapshot handed to it
8	#40694	2478	the sweep: ask siblings, vet, deliver — one sweep per discovered worker
9	#40695	1211	fold a discovery burst into one fleet-wide fetch, plus the gap retry it enables
10	#40696	844	a graft nothing witnessed asks the fleet instead of guessing
11	#40697	646	component test over the real transport: same match_prefix answers as the source
12	#40698	943	kind-cluster proof, the KV-publishing fake worker, and the RBAC/downward-API manifest
13	#40699	869	/readyz holds until bootstrap settles; --kv-bootstrap-seed-required; the metrics

Each PR's base is the branch below it, so every diff shown here is that
PR's own change. Review bottom-up; GitHub retargets each child to main as
its parent merges.

What the series does

A router replica subscribes to each worker's KV topic mid-stream, so every
block already resident in the engine's radix cache is invisible to it — and
engines publish BlockStored only as they insert, so a prefix cached hours
ago is never re-announced. A cache-blind replica then scatters the prefixes the
warm replicas were keeping consolidated, degrading the engines' locality for
the whole fleet; a rolling update does that to every replica in turn. This
series makes a booting replica pull a tree snapshot from a warm sibling over
HTTP and graft it beneath its live delta stream. Off unless --kv-peer-selector
is set.

Supersedes #39750, which carried the same work as one branch on a stale base.

What this change does

The other half of peer discovery: a second EndpointSlice watch, scoped by
--kv-peer-selector, keeps PeerRegistry current. Still nothing consumes a
snapshot — the consumer lands next — but with this an operator can set the
selector and confirm discovery works before any behaviour depends on it.

A replica must not offer itself as a bootstrap source, and POD_IP alone cannot
do that. It carries only the pod's PRIMARY address, so on a dual-stack Service
the pod's secondary-family entry survives an IP filter — which then latches
"siblings exist" and stops a genuinely lone replica from ever concluding it is
alone. POD_NAME matched against each endpoint's targetRef excludes the whole
pod (POD_NAME falls back to HOSTNAME, which Kubernetes defaults to the pod
name); POD_IP stays as the fallback for endpoints without one. With neither a
name nor an IP known the replica stays in its own peer list, which is wasteful
rather than wrong, and says so at WARN.

Only slices of the address family this router listens on are kept: 0.0.0.0
opens an AF_INET socket, so it is IPv4-only however "unspecified" it looks,
while :: (dual-stack on Linux) and a hostname keep both. Each ready endpoint
yields its first address only, and a sibling listed once per family is offered
once — one URL per backend.

The watch buffers a relist and publishes only on InitDone. Publishing each
InitApply would expose a partially rebuilt set, and an empty partial marks
the registry synced — the exact state that turns a bootstrap into a cold boot.
The regression test asserts mid-relist, because checking only the settled state
passes with the bug present. Watcher errors back off (default_backoff), so a
persistent 403 from missing RBAC is not a tight LIST loop against the API
server, and they leave the peer set untouched rather than clearing it.

It starts before worker discovery, because a bootstrap consults the peer set the
moment the first worker appears: an empty set there means that worker's ranks
skip bootstrap and run cold. Failure to start is non-fatal — routing does not
depend on peer discovery, and losing it means replicas boot cold, which is the
pre-existing behaviour.

Deployments that want self-exclusion need POD_NAME/POD_NAMESPACE/POD_IP from the
downward API; the manifest that wires them ships with the e2e test in the
consumer change, where there is something for it to exercise.

Tests

cargo fmt --check, cargo clippy --all-targets -- -D warnings, and the lib +
component + proxy suites all pass on this branch on its own, not only on the
tip of the stack.

Review pass

Reviewed with /code-review --fix and /simplify; fixes were folded into this PR's own commit and the stack was re-verified tier by tier (cargo fmt --check, cargo clippy --all-targets -D warnings, lib + component + proxy tests on every branch).

• Fixed: the watcher backs off between restarts, 0.0.0.0 is treated as an IPv4-only listener when picking the address family, and each endpoint yields one URL.
• File layout: this PR's code lives in discovery/k8s/peers.rs (new), a child module of discovery/k8s.rs, with the startup wiring in main.rs; no file the stack creates exceeds ~1,300 lines.

🤖 Generated with Claude Code

https://claude.ai/code/session_016HmJvHV7QDPk3qAjQYzthd

---

CI States

Latest PR Test (Base): ✅ Run #36411968288
Latest PR Test (Extra): ❌ Run #36411967730
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.

[ShangmingCai]

Looks good.

[ShangmingCai]

/tag-and-rerun-ci