[isytwu]

Make the ranged-get key handle survive the access pattern a chunked reader actually produces

What this is

BatchGetRanges used to resend the whole key list on every call. #644 originally
added a wire key handle so a repeat call could name its keys instead, and
stopped PoolClient::ResolveLocalBatch from rebuilding the key vector for each
batch. Both were validated only on a host-only microbench.

This PR adds the instrumentation that says whether they fire, fixes a cliff the
instrumentation exposed, fixes two real bugs found along the way, and prices all
of it end to end.

Headline, measured end to end on DeepSeek-V4-Pro conc=128 agentic replay:

	control	this PR	Δ	run-to-run noise
Input throughput	248,740 tok/s	251,559 tok/s	+1.13%	0.02%
TTFT p50	1935.88 ms	1835.14 ms	−5.20%	0.7%
Inter-token latency	53.63 ms	51.83 ms	−3.36%	0.6%

The control is the branch point, which contains no key handle at all — it
resends the whole key list on every ranged get. Under this PR, 86% of 178,578
ranged-get calls send an 8-byte handle instead. One paired run; the noise column
is the spread between two runs of the control itself, on different nodes.

Two caveats stated up front rather than buried. TTFT mean and p99 are not
measurable here — their run-to-run spread (5.9% and 16.7%) exceeds the effect,
so the −1.33% and +10.36% in the full table below are both noise. And an earlier
revision of this description claimed "0% hit rate in the real path"; that came
from a microbench with a deliberately constructed access cycle, and direct
measurement of production says otherwise — ~99% of key-set reuses there are
already inside the old capacity. The 0% is a real worst case a workload can
land on, not a description of the one we run.

What the production workload actually does

One instrumented run of DeepSeek-V4-Pro conc=128 agentic replay, eight ranks,
~19–20k key-set reuses each, UMBP_KEY_HANDLE_DEBUG=1. This is a count of the
access pattern, not a timing, so it has no noise floor to argue with:

rank	reuses	<8	<16	<64	<512	<1024
umbp-1768	20,158	99.2%	99.8%	99.8%	99.8%	100%
umbp-1770	19,154	98.5%	99.8%	99.9%	99.9%	100%
umbp-1773	19,179	99.4%	100%	100%	100%	100%
umbp-1775	19,119	99.5%	99.9%	99.9%	99.9%	100%

<N is the share of reuses an LRU of capacity N would have caught. The old
capacities were 8 (client) and 64 (server).

So on this workload the old capacity was already catching 98.5–99.5% of
reuses, and the new one catches ~99.9%. The curve is nearly flat from 8 to
512; raising capacity buys about one percentage point here. The last 0.2%
needs 1024.

Server side, same run: hits=153835 misses=277 mints=24466 hit_rate=99.8% held=512. The mechanism is working — 153k calls named their keys instead of
resending them — and it was working at the old capacity too.

The reader's shape, for anyone sizing this differently:
distinct_batch_sizes=426, top: 2048keys:577 12keys:515 2keys:448 7keys:384.

The cliff: LRU inverts under a fixed-order cycle

_run_layer_wise_batch chunks each layer group's keys into
step = RANGES_PER_CALL // ranges_per_object pieces and walks those chunks in
the same order, once per layer group. That is a cycle of C distinct key sets.

An LRU of capacity < C evicts precisely the entry it is about to need next. It
does not degrade gracefully; it inverts. The old capacities were 8 (client) and
64 (server), and C in production is far larger than either.

Measured on two independent machines, capacity pinned to 8:

cycle length	hits	mints	remints
C = 8	440 (100%)	8	0
C = 9	0	504	495
C = 16	0	896	880

The cliff is one entry wide. C=16 produced exactly 176 hits on both machines
independently once the fix was in, which is what gave confidence this is
structural rather than a timing artifact.

This is a constructed cycle, not the production one. In a strict cycle of C
sets the reuse distance is exactly C−1, so capacity 8 and C=9 sit one entry
apart and every single reuse misses. The measured workload above does not sit
there — its distances are almost all below 8. Both facts are real: LRU's failure
here is a cliff rather than a slope, and the workload we run happens to stand
next to it rather than on it. What the fix buys is that the cliff stops
existing, not a large gain on today's traffic.

Commits

commit	what
e36d3c9	bench: what a layer-wise restore costs on the standalone wire
0ab7423	perf: resolve a whole batch without rebuilding its key vector
bf6ef2f	perf: let a ranged get name its keys instead of resending them
645dd17	feat: reuse-distance instrumentation — the diagnosis, kept regardless of the fix
33b25e3	test: bench the shape a chunked reader actually produces (--chunk, --ranges-per-key, --threads)
943e118	perf: size the key handles for a chunked reader, and stop aging them
3c944d2	fix: let a zero key-handle capacity actually turn the handles off
8d69559	test: guard the capacity against the cycle that broke it
6647871	fix: let the client's switch also stop the server remembering

The instrumentation is the load-bearing part

645dd17 adds a reuse-distance histogram keyed by key-set fingerprint, behind
UMBP_KEY_HANDLE_DEBUG. It reports, per client, hits/misses/mints/remints and a
CDF of "what hit rate would capacity N have given". At the broken configuration
it printed:

would-hit-at-capacity: <8:0.0% <16:100.0% <32:100.0% ...

That names the fix instead of inviting a guess at it, and it stays in the tree.

The fix is the eviction policy, not just the capacity

943e118 raises capacity (client 8 → 128, server 64 → 512, both env-tunable via
UMBP_KEY_HANDLE_SLOTS / UMBP_KEY_HANDLE_CAPACITY) and replaces LRU with
random replacement. Capacity alone would only move the cliff: any fixed
capacity has a cycle length that defeats LRU exactly. Random replacement has no
worst case here — it degrades smoothly as capacity/cycle falls, so a workload
that outgrows the table loses proportionally instead of falling to zero.

The client-side move-to-front std::rotate goes with it: with random
replacement there is nothing to age.

What the new defaults cost. One entry holds one key set — roughly 160 KiB
for the thousand-odd 128-byte keys a layer group asks about — so the server's
512 entries are about 80 MiB, on a node whose tier is measured in terabytes.
The client's 128 slots are per worker process. Both are env-tunable
(UMBP_KEY_HANDLE_CAPACITY, UMBP_KEY_HANDLE_SLOTS) and both are allocated
lazily, so a deployment that never repeats a key set pays nothing. This is the
main resource trade in the PR and is stated here so it can be weighed rather
than discovered.

Two real bugs the new tests found

3c944d2 — UMBP_KEY_HANDLE_CAPACITY=0 was documented as "handles off" but the
eviction loop while (held_.size() >= capacity) spins forever at capacity 0
after the first insert. Found by trying to use the disable switch as a
benchmark arm.

6647871 — UMBP_KEY_HANDLE_SLOTS=0 turned off the client's cache and left the
bill with the server. LookupKeyHandle fingerprinted the keys on every miss
without consulting the slot count, and a non-zero fingerprint is exactly the
server's instruction to remember the key set, so a client that would never name
any of it still filled the table to capacity — about 80 MiB at the default — and
paid to hash every key on every call for an answer it discards. Handled the way
the empty-key case already is: send a zero fingerprint. Guarded by a test that
asserts through the handle counter (handles are issued from 1 and never reused,
so if the client's whole run minted nothing, the first handle a raw stub can get
is still 1), verified to fail without the change and pass with it.

This also corrects the microbench numbers below. tools/bench_key_handle_arms.sh
uses UMBP_KEY_HANDLE_SLOTS=0 as its "off" arm, so that arm was paying for a
table it never used. The gap it reported is therefore slightly wider than the
mechanism actually earns.

Nothing to gate for in-process deployments

An embedded deployment never reaches this code: CreateUMBPClient only builds a
StandaloneProcessClient when config.standalone_process is set, and every
key-handle symbol lives in the standalone client and server. The key_handle
fields are in umbp_standalone.proto — the standalone wire protocol only. There
is no runtime state to gate, so no gate was added; a guard on an unreachable
path would only suggest the path is reachable.

Results

1. Microbench — bench_umbp_standalone_ranged_wire, chunked-reader shape

Alternating arms, 5 reps, median, handle-off vs handle-on:

shape	machine A	machine B
8192 keys, 512-chunk	−38.0%	—
8192 keys, 1024-chunk	−32.1%	−28.4%
+ 8 ranges/key (realistic layer group)	−20.4%	−17.3%
4096 keys, 256-chunk, 8 workers (server table)	−52.0%	—

Server-table capacity sweep: 64 → 22.3% hit / 111.5 ms; 512 → 100% hit / 53.6 ms.
After the fix, C = 9 / 16 / 32 all hit 100% with zero remints, on both machines.

Read these as an upper bound, for the 6647871 reason above.

2. End to end — DeepSeek-V4-Pro, conc=128, agentic replay

Two paired runs, each arm back to back on one node, same corpus, same container,
one variable (the wheel).

metric	control	this branch	Δ
TTFT mean	3715.94 ms	3666.47 ms	−1.33%
TTFT p50	1935.88 ms	1835.14 ms	−5.20%
TTFT p90	6171.25 ms	5747.30 ms	−6.87%
TTFT p99	29602.73 ms	32669.23 ms	+10.36%
Input throughput	248,740 tok/s	251,559 tok/s	+1.13%
ISL mean	102,993	103,036	+0.04%
prefix_cache_hit	95.5%	95.5%	0

The second pair is what makes this readable. Its control arm is a repeat of
the first pair's control arm — same wheel, same configuration — so the two bound
what this benchmark can resolve:

metric	pair 1 control	pair 2 control	spread between two identical runs
Input throughput	248,740	248,689	0.02%
TTFT p50	1935.88	1922.34	0.7%
ITL mean	53.63	53.33	0.6%
TTFT mean	3715.94	3934.36	5.9%
TTFT p99	29602.73	34549.62	16.7%

Putting the two tables together, effect against its own noise floor:

metric	Δ	noise	ratio
Input throughput	+1.13%	0.02%	57×
TTFT p50	−5.20%	0.7%	7.4×
ITL mean	−3.36%	0.6%	5.6×
TTFT mean	−1.33%	5.9%	0.2× — not measurable
TTFT p99	+10.36%	16.7%	0.6× — not measurable

So this benchmark resolves throughput, p50 and ITL well, and does not resolve
TTFT mean or p99 at all. The three it resolves all move the right way by five to
fifty-seven times their noise. The +10.36% p99 — the one number that looked like
a tail regression — is inside a 16.7% control-to-control spread; I raised it as a
possible cost when I had only one pair, and the second pair says it was noise.

That the two control runs agreed on throughput to 0.02% on different nodes is
what makes the throughput number worth quoting from a single pair.

The second pair's test arm did not complete: on that node, GPU IPC buffer
registration collapsed from 968 buffers in 0.65 s to roughly one per 83 s, from
its second tenancy onward. Same wheel, same tier, and the same node's first
tenancy an hour earlier registered in milliseconds — a node-level pathology, not
the build. It is called out here because the control-to-control comparison above
is what that run was worth, and it is worth a lot.

3. TTFT paired A/B — topology 3, 256K prompt, full hit

The measurement built for exactly this question: alternating base/opt so both
arms see the same machine drift, one fresh stack per rep, paired differences
with a t-statistic.

base: n=6  median 288.4 ms
opt:  n=7  median 288.6 ms

complete pairs: 6
  rep3  288.0 -> 289.1  (+1.1)
  rep4  288.7 -> 288.6  (-0.1)
  rep5  290.7 -> 285.2  (-5.5)
  rep6  292.3 -> 281.7  (-10.6)
  rep7  284.2 -> 290.4  (+6.2)
  rep8  286.9 -> 279.4  (-7.5)

TTFT paired diff: median -2.8 ms, mean -2.7 ms, 4/6 improved, sd 6.2, t=-1.08

−2.7 ms on ~288 ms is −0.94%, in the direction the change predicts, and not
significant: |t| = 1.08 against the 2.571 that df=5 needs at p<0.05.

What I think this adds up to

Stating it the way the measurements support, rather than the way I first wrote
it:

The key handle mechanism works, and worked before this PR. 153,835 hits
against 24,466 mints on the production workload — those calls named their keys
instead of resending them, which is what #644 set out to do.

The capacity and eviction change removes a cliff that today's traffic does not
stand on. Measured: 98.5–99.5% of reuses were already inside the old capacity,
so the change is worth about one percentage point of handle hit rate here. It is
worth much more on a workload whose chunk cycle happens to exceed the capacity,
where LRU goes to zero rather than degrading — but I have only produced that
workload deliberately, in a bench.

The end-to-end gain is real and comes almost entirely from the mechanism, not
from the capacity change. +1.13% throughput, −5.20% TTFT p50, −3.36% ITL, each
five to fifty-seven times its own run-to-run noise. Attribute it correctly: the
control has no handle and resends every key list, so what is being measured is
86% of calls dropping from tens or hundreds of kilobytes to eight bytes. The
capacity change contributes about 0.65 pp of handle hits on top of that — real,
and small.

TTFT mean and p99 stay unmeasurable on this benchmark, and the single-prompt
256K TTFT A/B is separately uninformative: 1023 keys against a 1024-key chunk
step is one chunk, so it exercises neither the cycle nor much of the wire
volume. Its −0.94% at t=−1.08 is a null result about the wrong workload rather
than evidence about this one.

The instrumentation is what makes any of this checkable, and is the part I
would most argue to keep. It is what turned "we believe the handle cache works"
into eight ranks, twenty thousand reuses each, and a capacity curve — including
the part that contradicted my own initial claim.

Two genuine bugs were found on the way (3c944d2, 6647871), both in the
disable path, both invisible until something tried to use it.

Correctness

• C++ unit tests pass, including test_standalone_shm_ipc. The new
  RangedGetKeyHandlesSurviveALongerCycleThanTheOldCapacity case mints a 40-set
  cycle and asks for all of them back in mint order; verified to fail under
  UMBP_KEY_HANDLE_CAPACITY=8. The new
  DisablingKeyHandlesAlsoStopsTheServerRemembering case likewise fails without
  6647871.
• test_umbp_prim_gate.py / umbp_prim_ci.py batch_exists 1→8 rank scaling
  gate: no regression.
• pre-commit passes on every commit individually, not just the squashed diff.
• The A/B provenance was checked rather than assumed: the control wheel's base
  has src/umbp and tests/cpp/umbp byte-identical to this branch's base, and
  the test wheel differs from the control in exactly the five source files these
  commits touch.
• Unchanged on purpose: fingerprint validation, the client-side full key
  comparison that is the actual source of correctness, and the
  key_handle_unknown → resend retry. The handle stays a hint.

Merge status

Merges cleanly into current main (f9697e1). The two files that overlap with
upstream work since the branch point — src/umbp/distributed/pool_client.cpp
and tests/cpp/umbp/distributed/CMakeLists.txt — auto-merge with no conflicts.

Not in this PR

• The resolve cache / shared page list. ResolvedEntry::pages is read at
  154 sites across 8 non-test files; making it shared is a MediumBackend
  interface change, and folding it in would have made the e2e arm
  multi-variable. A previous attempt (kept on perf/umbp-resolve-cache-rejected)
  measured only −2.2%/−3.0% because it cached the map probe while
  BatchResolve still returned by value and paid every per-key page-vector copy
  — the copy is the cost, and removing it is the follow-up worth doing. Not
  attempted here, as distinct from measured and found worthless.
• The sglang-side tail-group misalignment. Worth about 2.5% of restore time
  by the earlier estimate, which is under every noise floor measured above, and
  I have no measurement of my own that says otherwise.