[ZLkanyo009]

Why

State images (Qwen3.8 / Kimi-K3 recurrent state, StateByteCodec) share LMCache's LocalCPU pool and its LRU with paged KV. A KV lookup refreshes the chunks it hits, but a state hit is answered by ATOM's StateOffloadIndex, so LMCache never saw an image used: a prefix that hit every turn kept its KV young and its image old, pressure evicted the image first, and the index kept advertising it until a joint KV + state load parked on it, missed, and recomputed the whole prompt. Before this PR, AgentX c64 with a 224 GiB pool recomputed 2.8% of all input tokens that way (88 of 3,786 state loads failed; https://github.com/wuhuikx/LLM-Booster/issues/157).

What

1. Refresh on every hit. Each hit the resumable scan votes for is sent to the workers as a touch (LMCacheOffloadMetadata.state_touches, one StateTouchId(prefix_hash, generation) each). Every rank moves the image to the young end of LocalCPU's LRU (StateByteCodec.touch: cache_policy.update_on_hit under cpu_lock). get refreshes too, so an offered load needs no touch, and a re-store of an image that is still hot is skipped.
2. Keep the index in step. Each rank reports on k3_state_touch whether the image is still there, failure-dominant across TP like a joint load. A gone image leaves the index. An admission that would load an image not confirmed within tier_verified_for_s (0.25 s per GiB of LMCACHE_MAX_LOCAL_CPU_SIZE, clamped to 5–60 s) is held one pass for that verdict (StateOffloadIndex.hold_admission, via BlockManager.record_allocation(defer_unverified=True)), at most the existing OFFLOAD_LOOKUP_DEFER_S. It then plans a lower boundary instead of parking on a miss. Why that window: a confirmed image is at LocalCPU's young end, so it survives until a whole pool of new entries arrives. AgentX c64 filled about 0.25 GiB/s per GPU; the window assumes 1 GiB/s and trusts a quarter of that turnover. 5 s is well above a touch round trip; 60 s is the largest window tried at c48 / c64. It cannot ride lookup_pending (the lmcache_mp deferral that shares this bound): that check runs before the tier lookup and the joint decision that pick the hash to verify.

No new environment variables. If LMCache's LocalCPU internals are missing, or an image is held only by another backend (LocalDisk), a touch only confirms presence, warns once and is counted in the K3 connector's get_statistics() as state_touches_unrefreshed; loads still work. Touch ids are not tombstoned by KVOutputAggregator, so they cannot push store / source tombstones out of its ring.

The tier logic lives in state_offload.py (index) and kimi_k3/ (worker). block_manager.py and scheduler.py only call it: record_allocation gains a keyword-only defer_unverified=False (other callers are unchanged) and asks hold_admission about the hash _state_resume_hash picks (the same helper allocate uses), and the scheduler calls publish_touches / collect_touch_verdicts.

The vLLM plugin also uses StateByteCodec (plugin/vllm/kv_transfer/connector.py), so it gets the codec-level changes only: get refreshes the LRU, a hot image is not stored twice, and binding warns once if LMCache's LocalCPU internals are missing. It has no engine-side touches or admission hold.

Not changed

Seen, left as on main:

• OFFLOAD_STATE_CPU_SIZE stays a cap with no reservation and no startup check against the pool size. The state_tier_cpu_bytes docstring still says the cap is "carved from" the pool.
• StateByteCodec.get() keeps a missed key in its own cap LRU until the next put or touch drops it.
• StateOffloadFace gets no touch methods; the K3 scheduler is the only implementer, and the shell and multi connectors forward with getattr.
• The vLLM plugin's state hits are also answered by StateOffloadIndex (plugin/vllm/kv_transfer/kda_state.py), so the LRU asymmetry remains there. Fixing it needs touches in the plugin's metadata, run on the plugin worker, with verdicts carried back through vLLM's connector output, and tests/plugin does not run in CI. Left for a separate change.

Tests

In the existing component files: test_state_offload_index.py (touch throttling, verdicts, the hold and its bound, the window), test_lmcache_offload_connector.py (codec refresh against a fake LocalCPU LRU, connector round trip, TP quorum), test_state_checkpoint.py (admission hold, voters, resume hash, scheduler retry), test_offload_shell.py (the scheduler shells' forwards), test_block_manager.py (window from the pool), test_kv_aggregator.py (tombstones). 34 mutations of these mechanisms (the cleanup that drops a hash's touch state with it, and the wiring: a touch-only step is work for the worker, start_load_kv touches, the scheduler applies verdicts, the plain and multi scheduler shells forward touches and verdicts) were each checked to turn a test red.

Accuracy and performance

TP2 + EP2 on MI355X (-tp 2 --enable-expert-parallel, MTP 3, FP8 KV, LMCACHE_MAX_LOCAL_CPU_SIZE=208.6, OFFLOAD_STATE_CPU_SIZE=104.3, --max-num-seqs 64), AgentX c64 for 1800 s, same cards. Main is the deployment stack without this PR.

• Throughput: 170,192 → 174,709 tok/s/GPU.
• TTFT p50: 0.9 → 0.8 s.
• Recomputed share of input: 0.065 → 0.059.
• State loads failed: 23 of 111 → 0 of 102.
• Full GSM8K: 0.9803 → 0.981 on 3ccc2e9fe, and 0.978 (stderr 0.004) on e9481f9bd, the simplification pass before its review fixes, which change no behaviour. 0.9773 on the previous revision (paired stderr 0.0026). GSM8K makes no state loads, so these differences are run-to-run noise.

Earlier, single GPU (MI355X, MTP 3, FP8 KV, LMCACHE_MAX_LOCAL_CPU_SIZE=208.6): GSM8K 1319 questions 0.9795 (before: 0.9795). Split CPU retrieval (pass 2 answers every question again with its prefix loaded from the CPU tier: 90% of prompt tokens and 1,273 state images, 0 failed): 0.9735 against 0.9750 in pass 1, 16 of 1319 answers differ (paired stderr about 0.003). AgentX c64, 3600 s, 4 card-swapped rounds against the previous revision: 128,070 → 128,694 tok/s/GPU, TTFT p50 3.28 → 3.35 s, 0 failed state loads either way. Against the reservation revision: c64 +2.0% tok/s/GPU, 0 of 3,894 state loads failed.

[github-actions]

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

• ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

• ✅ Run after the PR is approved and Pre Checkin passes
• ✅ Run immediately when an approval review is submitted
• ✅ Can be requested before approval with labels

Label	Tests
ci:full	Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom	Run native ATOM model accuracy tests
ci:vllm	Run ATOM vLLM OOT model accuracy tests
ci:sglang	Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2668 --add-label <label>

[ZhangLirong-amd]

Thanks for the investigation; the failure mode and the measurements are clear. My concern is with the fix itself: pinning the images turns the shared pool into a static KV/state split, and the root cause stays in place.

Root cause: state images never get an LRU refresh

In LMCache's LocalCPUBackend, every lookup refreshes the KV keys it touches: contains(key, pin=True) queues the key, and touch_cache() calls cache_policy.update_on_hit on it. State keys never go through that lookup. StateOffloadIndex answers "does the tier hold this state" on the engine side, so LMCache never learns that an image was just matched. A prefix that is hit on every turn keeps its KV at the hot end and its state image at the cold end. Under pressure, the image goes first. It is not cold; it is just never touched. That is the asymmetry described in this PR's motivation.

Pinning works around this, but it costs:

• A static split that needs manual tuning. Your own c64 data show this: 32 GiB and 96 GiB are both clearly worse than 64 GiB. The right value will change with the model, the concurrency and the pool size.
• Dead reservations. A joint load needs both legs. A pinned image whose KV has been evicted can never be used again, but it keeps its share of the reservation until the codec's own LRU retires it.
• The index can still be stale. It still does not track what LMCache holds. Pinning only stops LMCache from removing state.

Preferred fix

1. Refresh state together with KV. When StateOffloadIndex reports a tier state hit for a prefix (at lookup or admission), touch that image's key in LMCache the same way the KV keys are touched (contains(key, pin=True) + touch_cache(), or cache_policy.update_on_hit). State and KV then compete on real recency in one pool, with no fixed reservation and no split to tune. max_cache_bytes can stay as a loose cap.
2. Keep the index consistent when an image is evicted anyway. Either hook LMCache's eviction, or check contains before parking a joint load. If the image is gone, drop it from the index and fall back to a KV-only load or a plain prefill, instead of discovering the miss after the park and recomputing the whole prompt.

Fallback, only if this needs to land first as a stopgap

If you need this merged before the refresh is done, at minimum:

• Do not make the reservation the default. With OFFLOAD_STATE_CPU_SIZE defaulting to 32 GiB, every state-tier deployment silently gets a hard 32 GiB reservation. Put the pinning behind an explicit opt-in switch.
• Validate the budget against the pool at startup. If the state budget is at or above LMCACHE_MAX_LOCAL_CPU_SIZE (or leaves too little for KV), refuse to start or warn loudly. Otherwise pinned images can fill the whole pool and every KV save fails silently. #2562's accuracy protocol (LMCACHE_MAX_LOCAL_CPU_SIZE=32) is exactly that case.

Please treat the LRU refresh as the main fix and the opt-in reservation as the fallback.

[ZLkanyo009]

@ZhangLirong-amd @valarLip Thanks for the review. I reworked the PR along the lines you suggested and pushed 39dc5dda1 and 4d8798f1c on top of 4204a9e4. The description is updated.

LRU refresh on every tier hit. StateOffloadIndex.could_serve is the one predicate every tier voter goes through, so every hash it votes for becomes a touch (at most one per hash per 0.5 s). Touches travel to the workers as LMCacheOffloadMetadata.state_touches, and each rank calls StateByteCodec.touch, which runs cache_policy.update_on_hit(key, hot_cache) under LocalCPU's cpu_lock (your second option). I did not use contains(pin=True) + touch_cache(): that needs a paired unpin, and keys_in_request is shared with the KV lookup. get and a re-store of a resident image refresh too.

Index consistency. Every touch reports resident / gone on a new k3_state_touch completion channel, failure-dominant across TP. A gone image is dropped from the index. To fall back before parking rather than miss after it, an admission about to load an image not confirmed in the last 60 s waits one scheduler pass for its verdict, bounded by the existing OFFLOAD_LOOKUP_DEFER_S. A resident image then loads; a gone one has already left the index, so the retry plans a lower boundary (HBM checkpoint, older image, or plain prefill). I didn't hook LMCache eviction: LocalCPU has no eviction callback, and under TP one rank's eviction can't reach quorum.

Reservation removed. No extra MemoryObj reference any more. OFFLOAD_STATE_CPU_SIZE is only a cap again, and the codec's own LRU now follows use. Measured below: no reservation is at least as fast as the reservation, so I did not keep it as an opt-in. No new environment variables.

Startup check. When the worker builds the state tier, it warns if the cap is ≥ LMCACHE_MAX_LOCAL_CPU_SIZE (bounds nothing) or above half of it (images may crowd out KV).

Unit tests. tests/test_state_tier_lru_refresh.py has 45 tests and replaces the reservation tests. It covers the codec against a fake LocalCPU with LMCache's LRU, touch throttling and verdicts, TP quorum, scheduler plumbing, the admission hold (held / not held / timed out / retried), and the budget warning. The state-tier, offload, block-manager and scheduler test files all pass. The full CPU suite has 3 failures, the same ones main has on this host (test_graph_marker ×2, test_sparse_indexer_side_effect_op).

Accuracy (in-process lmcache_offload, MTP 3, FP8 KV):

• GSM8K LIMIT=200: 0.985 strict / flexible.
• Split CPU retrieval over 1319 questions: pass 1 0.9750, pass 2 0.9735. Pass 2 loaded 90% of prompt tokens and 1,273 state images from CPU with 0 failures; 9 questions were right only in pass 1 and 7 only in pass 2. Reference range is 0.975–0.98.

Performance, AgentX, 224 GB CPU pool, state cap 59.6 GiB, --max-num-seqs 64. Each pair ran at the same time on the same node; "before" is the reservation (4204a9e4):

conc	before	after	TTFT avg / p50 s, before → after	failed state loads, before → after
64	110,662	112,921 (+2.0%)	20.1 / 19.0 → 19.2 / 18.7	0 → 0 (25 evicted images caught by touch)
48	131,579	132,188 (+0.5%)	2.8 / 1.1 → 2.6 / 1.0	1 → 0

ITL is unchanged. For reference, the earlier runs of 4204a9e4 on other nodes gave c64 111,298 and c48 133,816. Since images and KV now share one LRU by use, the cap can also be raised: at c64, a cap of half the pool (104.3 GiB) gave 113,973 (+2.5% over 59.6 GiB in the same tree), with 0 failed loads.

The confirmation window was 5 s in 39dc5dda1. That held almost every AgentX load and cost +0.3 s p50 TTFT on cache-hit requests at c48, so 4d8798f1c raises it to 60 s, still far inside the pool's roughly 13-minute LRU turnover at c64. The table above is with 60 s.

One heads-up: #2562 changes record_allocation to take an AdmissionProbe. Whichever lands second needs a small adaptation (keep *, defer_unverified=False -> bool and run the hold check after _fit_joint), and the test tree was built that way.

Could you take another look?

[ZhangLirong-amd]

Thanks, the rework addresses the root cause: state images now get refreshed in LMCache's LRU together with their KV, evicted images leave the index, and the reservation is gone. I have two issues on 4d8798f1c.

1. The verify hold can last indefinitely (state_offload.py:260)

needs_verify measures the hold from the latest touch send, not from when the admission was first held:

sent = self._touch_sent_at.get(h)
...
return now - sent <= defer_s

A held request is re-probed on every pass. Each probe goes through could_serve, which puts h back in _touch_wanted. Every 0.5 s (_TIER_TOUCH_INTERVAL_S) take_touches sends h again and resets _touch_sent_at[h], so now - sent never reaches OFFLOAD_LOOKUP_DEFER_S (2 s by default). In normal operation the verdict comes back within a step or two, so this does not show up. But the 2 s bound is meant as the fallback for the case where it doesn't come back: a dropped metadata field, a nested MultiConnector path, or a rank that never reports. In that case the request is held forever, with nothing in the logs.

Please measure the bound from the first hold of h, for example a _hold_started_at[h] set on the first True and cleared when a verdict arrives or the hash is forgotten. Resending a touch should not extend it. A test where verdicts never arrive, asserting that the admission proceeds within defer_s, would lock this in.

2. Touch completions churn the shared tombstone ring (aggregator.py:161)

KVOutputAggregator._connector_completions tombstones every connector completion (lambda _key: True) in one ring of terminal_tombstone_limit (4096). That ring is shared by k3_state_source, the store reports and k3_state_load_disposition. Each touch is a new StateTouchId(hash, round). At c64, with every voted hash re-touched every 0.5 s, that is on the order of a hundred entries per second, which cycles the ring in well under a minute. The tombstones that guard store and source completions against late duplicate reports get evicted along with them.

round already makes every touch identity unique, so a touch cannot be reported twice and needs no tombstone. Please exclude StateTouchId from tombstoning, or give touches their own completion group.

[ZLkanyo009]

@ZhangLirong-amd Thanks, both are real. Fixed in eff79dcf9 on top of 4d8798f1c (no rebase).

1. The verify hold now has a fixed bound. StateOffloadIndex records _hold_started_at[h] on the first hold of h (setdefault), and needs_verify releases the admission once now - started > OFFLOAD_LOOKUP_DEFER_S. Resending a touch no longer affects the timer: _touch_sent_at only drives the 0.5 s resend throttle. The timer is cleared by any evidence about h (a touch verdict, a completed store or a completed load, all through a new _mark_verified) or when h is forgotten, and stop_refreshing clears everything. After a hold expires, h is admitted without waiting until new evidence arrives, so a lost verdict costs one 2 s wait, not 2 s on every admission. Expiries are counted in state_offload_verify_holds_expired (in the periodic stats line), and the first one logs a WARNING naming the setting, so a lost verdict path shows up in the logs.

Tests in tests/test_state_tier_lru_refresh.py:

• test_a_lost_verdict_holds_no_longer_than_the_lookup_defer is now your scenario: verdicts never arrive, the held admission is re-probed and touches are taken every 0.05 s (the touch is really resent several times), and the admission proceeds within defer_s, with verify_holds_expired == 1 and no further hold.
• test_evidence_after_an_expired_hold_rearms_it: after an expiry, a verdict re-arms the hold, so once that confirmation goes stale the next load waits again, timed from the new hold.
• test_a_verdict_ends_the_hold_timer: time held before a verdict does not count toward the next hold.

2. Touches no longer take tombstones. The _connector_completions predicate in KVOutputAggregator is now not isinstance(key[1], StateTouchId). Store, source and load-disposition completions are tombstoned as before. I didn't add a separate group: round already makes every touch unique, so touches need no dedup memory at all. The StateTouchId / StateStoreOperationId docstrings say so. New test test_state_touches_do_not_evict_store_and_source_tombstones in tests/test_kv_aggregator.py: with a ring of 4, a store and a source completion followed by 100 touches still leave both tombstones, and late duplicate store/source reports are still dropped.

All four new tests fail on 4d8798f1c and pass on eff79dcf9.

Validation (same setup as last round):

• Full CPU suite: 9625 passed. The 3 failures are the same ones main has on this host (test_graph_marker ×2, test_sparse_indexer_side_effect_op). Black and ruff are clean.
• GPU unit tests (state tier, offload, aggregator, PP KV status, block manager, scheduler; 13 files): 1202 passed.
• GSM8K LIMIT=200, in-process lmcache_offload: 0.985 strict / flexible.
• AgentX c64, 3600 s, same node at the same time, 4d8798f1c vs eff79dcf9:

	4d8798f1c	eff79dcf9
throughput, tok/s/GPU	111,842	111,354 (−0.4%)
TTFT avg / p50 / p90, s	19.3 / 18.0 / 29.3	19.5 / 18.9 / 28.3
recomputed, share of input	0.166	0.169
state loads from CPU (failed)	3,900 (0)	3,854 (0)
touches sent (images found gone)	5,918 (43)	5,940 (30)
admissions held for a verdict	1,826	1,896
holds that hit the 2 s bound	–	0

The difference is within run-to-run noise: the same 4d8798f1c tree gave 112,921 tok/s and 18.7 s p50 TTFT in last round's c64 run. State hits are unchanged, with no failed loads on either side. No hold reached the bound: verdicts come back within a pass or two, so the new timer only matters when one is lost.

On merge order: this conflicts with #2562 in block_manager.py (record_allocation) and scheduler.py (the admission call site). I've rehearsed the merge on main + #2562 + #2636 + #2655. record_allocation takes the AdmissionProbe and keeps *, defer_unverified=False -> bool, with the hold checked after _fit_joint, so it judges the joint the fit actually kept. With that resolution, the state-tier, block-manager, scheduler and offload tests pass. I'll merge main into this branch once #2562 lands.

Could you take another look?

[ZLkanyo009]

@ZhangLirong-amd @valarLip Merged latest main (035a78a, which now includes #2562) into this branch; no rebase, no force push. New merge commit: 881aacf.

Conflicts (both with #2562, both in the admission path):

• atom/model_engine/block_manager.py record_allocation: [ATOM] accelerate lm cache: load the CPU prefix when the Qwen3.5 HBM hit is off the chunk grid #2562 changed the signature to record_allocation(seq, probe: AdmissionProbe) and added _fit_joint(decision, probe) before committing. Combined as record_allocation(seq, probe, *, defer_unverified=False) -> bool. The verify hold (_tier_state_unverified) now runs after _fit_joint and before _commit_joint_boundary. That way the hold judges the joint that will actually be loaded: _fit_joint may drop the joint, e.g. partial_chunk_no_room. _fit_joint is pure, so a held request has committed nothing and moved no counters, and the retry starts clean. [ATOM] accelerate lm cache: load the CPU prefix when the Qwen3.5 HBM hit is off the chunk grid #2562's internal record=True caller doesn't pass defer_unverified, so it always returns True as before.
• atom/model_engine/scheduler.py admission loop: passes [ATOM] accelerate lm cache: load the CPU prefix when the Qwen3.5 HBM hit is off the chunk grid #2562's hbm_probe with defer_unverified=True; on False the request goes back to prefix_waiters and the loop continues, as before.
• tests/test_state_tier_lru_refresh.py adapted to the new signature (AdmissionProbe(2, [1, 2]), _fit_joint stubbed to identity in the hold tests). Added test_the_hold_judges_the_joint_fit_kept: when _fit_joint drops a stale joint, the request is not held and the dropped-joint decision is committed. It fails if the hold check is moved before _fit_joint.

Verification
The merged tree is byte-identical to a pre-merge I validated on a simulated main (ebd5f84f + #2562 squash), whose tree equals the real 035a78a4.

• black 26.10.1 / ruff: clean on the 13 Python files this PR touches.
• CPU unit suite (.github/scripts/run_unit_tests.sh): 10097 passed, 3 failed. The same 3 also fail on plain main on this machine: test_graph_marker::test_compiled_marker_preserves_inplace_aliases[False/True] and test_sparse_indexer_side_effect_op::test_side_effect_op_stays_ordered_without_a_synthetic_output, all inductor g++ compile errors.
• GPU unit tests on MI355, 13 files (test_state_tier_lru_refresh, test_lmcache_offload_connector, test_state_checkpoint, test_block_manager, test_scheduler, test_state_offload_index, test_admission_host_overhead, test_lmcache_offload_v4_page_slot, test_lmcache_offload_gpu_disk_e2e, test_multi_connector, test_offload_shell, test_kv_aggregator, test_pp_kv_status): 1195 passed. main + [ATOM] accelerate lm cache: load the CPU prefix when the Qwen3.5 HBM hit is off the chunk grid #2562 alone gives 1146, one test file fewer.
• End-to-end with Qwen3.8 MXFP4 TP1, BF16 KV, in-process lmcache_offload, MTP 3. Qwen offload needs [Qwen 3.8] LMCache offload for QSA + GDN + PLE state #2636, which isn't in main yet, so this ran on main + [ATOM] accelerate lm cache: load the CPU prefix when the Qwen3.5 HBM hit is off the chunk grid #2562 + [Qwen 3.8] LMCache offload for QSA + GDN + PLE state #2636 + the identical fix(offload): refresh state images in LMCache's LRU on every tier hit and keep the index consistent #2668 merge delta:
  • GSM8K LIMIT=200: 0.985 on pass 1, 0.99 on pass 2.
  • GSM8K questions 0-439 with a small HBM pool, so pass 2 restores state from the CPU tier: 0.9773 on both passes. All 440/440 pass-2 requests hit the CPU tier, with 435 state loads and 0 failures.
  • No tracebacks.

[ZhangLirong-amd]

Thanks for the fix. A few comments:

1. block_manager.py:1570: the hash selection logic is now duplicated outside allocate. Could it be extracted into a shared helper? Otherwise the two copies are likely to drift apart in future changes.

2. state_offload.py:24: _TIER_VERIFIED_FOR_S = 60 s is hard-coded. Per the comment, it was derived from AgentX c64 with a 208 GiB pool turning over roughly every 13 minutes; with a different pool size or concurrency that may no longer hold. Suggest making it configurable through envs, or deriving it from the pool size.

3. state_offload.py:200: could_serve used to be a pure check, but with refresh_hits enabled it now also adds entries to _touch_wanted, which is a side effect. Suggest splitting "can this be served" from "register this for a touch".

4. state_offload.py:406: needs_verify is on the admission path and reads the env var and parses the float on every call. Reading it once at module load should be enough.

[ZLkanyo009]

@ZhangLirong-amd thanks for the review. All four points are addressed in c54eda2 (one commit on top of 881aacf, no new environment variable). cc @valarLip

1. Duplicated hash selection in _tier_state_unverified and allocate. Both now call one helper, BlockManager._state_resume_hash(seq, joint_hash, hit_hash). It returns the joint boundary hash if there is one; otherwise -1 (nothing restored) for a replayed hit; otherwise the HBM hit hash. _tier_state_unverified asks it which state the admission would restore and holds only on that hash. allocate passes the same result to both _attach_state_slots calls. The replay branch in allocate now only clears replay_kv_load, since the hash choice moved into the helper. Tests: TestResumeHash in tests/test_state_tier_lru_refresh.py checks the priority order and that a replayed hit is never held. It also has an end-to-end case asserting that the hash the hold asks about is the hash allocate hands to _attach_state_slots.

2. Hard-coded _TIER_VERIFIED_FOR_S = 60. This window is now derived from the LMCache CPU pool, so it needs no new environment variable. The block manager already reads chunk_size from build_lmcache_config(...). state_tier_lmcache_sizes now returns max_local_cpu_size (the existing LMCACHE_MAX_LOCAL_CPU_SIZE) from the same config, and passes it to StateOffloadIndex(pool_bytes=...). tier_verified_for_s(pool_bytes) then works as follows:

• The pool is LRU, so an image can only be evicted after a whole pool of newer entries has arrived. The pool's turnover time is therefore the window in which a confirmation stays valid.
• The fill rate depends on the workload and is not known at startup. The formula assumes a ceiling of 1 GiB/s per worker, which is 4x what AgentX c64 stores on one GPU (~0.25 GiB/s), and trusts a quarter of the turnover at that rate.
• That gives window = pool_GiB / 1 GiB/s * 0.25, clamped to [5 s, 60 s]:

pool	window
208.6 GiB (AgentX)	52 s
100 GiB	25 s
20 GiB or less, or size unknown	5 s
240 GiB or more	60 s

• The 5 s floor keeps the window well above a touch round trip. The 60 s ceiling is the largest window we measured.
• At 208.6 GiB the real turnover at c64 is ~14 min, so 52 s is conservative.
• A wrong window only costs performance, never correctness. Too long admits a load that may miss and get recomputed (which is what every load did before touches existed). Too short holds a load for one extra pass.

I didn't add a knob because the project doesn't take new environment variables, and the pool size already determines the window. A follow-up could measure the fill rate at runtime instead of assuming a ceiling, but that change is larger than this PR needs. Tests: TestVerifiedWindow covers the 52 s value at 208.6 GiB, monotonic growth between the bounds, the floor for an unknown pool, and that the index holds by its own window. It also checks that BlockManager feeds the LMCache pool size into the index (100 GiB gives 25 s).

3. could_serve has a side effect. could_serve is now a pure check (can_load and h in hashes). Registering a touch is a separate method, want_touch(h), which does nothing unless refresh_hits is set. The two voters that count a tier hit call it explicitly:

• PagedStateCheckpointCoordinator._reachable, on a tier hit only. An HBM hit does not register a touch.
• BlockManager._request_state_load, after request_load accepts the load.

_tier_can_serve only asks and registers nothing. Tests: TestIndexTouches::test_could_serve_queues_nothing, test_a_registered_hit_queues_a_touch, and TestVotersRegisterTouches (one case per voter, plus the HBM-hit and _tier_can_serve negatives).

4. needs_verify reads the env on every call. OFFLOAD_LOOKUP_DEFER_S is now read once, when StateOffloadIndex is built, and stored as self.defer_s. It is not read at all when refresh is off. The constructor also accepts defer_s= for tests. Tests: TestNeedsVerify::test_the_lookup_defer_is_read_once_at_construction and test_an_index_without_refresh_never_reads_the_defer.

Testing

• tests/test_state_tier_lru_refresh.py: 63 passed (was 47). The 20 new or changed cases fail on 881aacf because they use the new interface. black / ruff clean.
• Full CPU suite (.github/scripts/run_unit_tests.sh set): 10112 passed, 3 failed on c54eda2 vs 10097 passed, 3 failed on 881aacf. The same 3 tests fail on both (test_graph_marker x2 and test_sparse_indexer_side_effect_op, because the inductor C++ build fails on the CPU host). The +15 passed are the new tests.
• GPU (MI355X): 23 state-tier / offload / block-manager / scheduler / checkpoint test files, 1851 passed, 2 failed. The 2 failures are in tests/model_ops/engram/test_host_path.py (page-lock registration is refused in the container) and fail the same way on the stack without this commit (1836 passed, same 2 failed).
• AgentX c64, same node, both servers started together after a shared barrier, 3600 s, 4 rounds with the cards swapped every round, 208.6 GiB pool / 104.3 GiB state cap. both run main + [Qwen 3.8] LMCache offload for QSA + GDN + PLE state #2636 + perf(moe): Triton softmax top-k routing, bit-identical to AITER #2665 + feat(spec): capture the Qwen3.8 MTP draft steps in CUDA graphs (MRoPE) #2663 + fix(qwen4-exp): register the QSA FP8 scale planes for KV offload #2655 + this PR + perf(qwen4_exp): keep the PLE n-gram table in page-locked host memory, read over UVA #2771, with this PR at 881aacf ("before") or c54eda2 ("after"):

round (cards: after / before)	tok/s/GPU, before → after	TTFT avg / p50 / p90 s, before → after	state loads (failed), before → after	holds expired
1 (GPU 2 / 3)	127,285 → 129,111	6.5 / 3.3 / 15.6 → 6.1 / 3.5 / 13.4	1,166 (0) → 1,159 (0)	0 → 0
2 (GPU 3 / 2)	128,998 → 128,067	5.8 / 2.7 / 13.0 → 6.2 / 3.2 / 14.6	1,148 (0) → 1,165 (0)	0 → 0
3 (GPU 2 / 3)	127,652 → 129,170	6.3 / 3.4 / 13.8 → 6.1 / 3.1 / 13.4	1,161 (0) → 1,125 (0)	0 → 0
4 (GPU 3 / 2)	128,346 → 128,428	6.5 / 3.7 / 14.2 → 6.5 / 3.6 / 14.1	1,214 (0) → 1,121 (0)	0 → 0
mean	128,070 → 128,694 (+0.5%)	6.28 / 3.28 / 14.15 → 6.23 / 3.35 / 13.88	4,689 (0) → 4,570 (0)	0 → 0

Throughput and TTFT are unchanged within noise. After wins on throughput in 3 of 4 rounds, and p50 TTFT splits 2-2: rounds 1-2 favoured before and rounds 3-4 favoured after. Recomputed input is 0.142 → 0.141. Neither side had a failed state load in any round. The 52 s window holds 5% more admissions than the old 60 s (4,823 vs 4,577 over 4 rounds), as expected for a shorter window, with no measurable cost.

• Accuracy on the new tree: GSM8K 1319 questions 0.9795 (stack without this commit: 0.9795). Split CPU retrieval on questions 0–439: pass 1 0.9795, pass 2 0.9795 (stack without this commit: 0.9750 / 0.9818), with pass 2's cache hits identical to that run (state-hit 99.36%).
• CI on c54eda2: Pre Checkin, ATOM Test, ATOM vLLM Test and ATOM SGLang Test all green.

The PR description is updated to match (could_serve / want_touch, the derived window, test counts, these numbers).

[ZLkanyo009]

@valar-cop review this pr

[valar-cop]

LGTM, minor robustness points only

any perf number? gsm8k? and LMCache interface not verified here

ci green.

[valar-cop]

the presence-only fallback silently no-ops the touch but still marks verified.. add a counter and warn once?

[ZLkanyo009]

@valar-cop Followed in 9219a8d and refined in the two later rounds (1b311b6, 4eff581). This reply was held back and is posted together with those rounds.

How the presence-only case works now:

• A touch returns TouchResult:
  • REFRESHED: the image was in LocalCPU's hot_cache, and update_on_hit moved it.
  • PRESENT: outside the hot cache but still held by another backend, e.g. LocalDisk, or LMCache lacks the LocalCPU internals; see the line-46 thread.
  • GONE.
• PRESENT is counted. The worker warns once, and each such touch reaches the engine on a k3_state_touch_refresh completion, failure-dominant across TP. It shows up as state_offload_touches_unrefreshed on the existing state checkpoints: stats line.
• put's skip-the-store fast path applies only to REFRESHED. A disk-only image is stored again, which puts it back into the CPU tier.

I kept the "resident" verdict for PRESENT on purpose. The verdict answers "can get read this image right now", and presence answers that correctly, because get reads every backend and promotes the image into LocalCPU. Being marked verified only lets later admissions within verified_for_s skip the one-pass hold. If the image is evicted in that window, the load misses and the request recomputes (fail_load -> forget), which is how every load behaved before touches existed. So a stale confirmation costs a recompute, never wrong state. The two alternatives are both worse:

• Reporting it gone would drop a loadable image from the index.
• Not ending the hold would re-hold every admission on that hash until OFFLOAD_LOOKUP_DEFER_S expires.

In our setup (in-process LocalCPU only) presence-only never happened: the GPU smoke on 1b311b6 shows state_offload_touches_unrefreshed=0 over 757 touches. With a LocalDisk backend it does happen; numbers are in the summary.

Tests in tests/test_state_tier_lru_refresh.py:

• test_a_disk_only_image_is_counted_and_warned_once
• test_a_refreshed_or_gone_image_is_not_counted
• test_a_disk_only_image_is_stored_back_into_the_cpu_tier

cc @valarLip

[valar-cop]

presence-only counting as resident is fine by me: a stale confirmation costs a recompute, never wrong state. the disk run and the real hot_cache reorder cover what i asked.

lgtm. one thing: log the derived verify window once at startup, the stats line shows the holds but not the window they run against.

[valar-cop]

want_touch now a hard dep on the admission path.. getattr like the rest of the file instead?

[ZLkanyo009]

@valar-cop Followed in 9219a8d (this reply was held back and is posted with the later rounds). _reachable now looks up want_touch with getattr(self.offload, "want_touch", None) and calls it only if it is callable, like stats in the same file. An attached offload without want_touch still has its tier hits counted; it just doesn't get the LRU refresh. I left BlockManager._request_state_load calling it directly, because the block manager's state_offload is always the StateOffloadIndex it builds itself.

Test: TestVotersRegisterTouches::test_an_offload_without_touches_still_counts_a_tier_hit attaches an offload that only has hashes / could_serve. It fails on c54eda2 with AttributeError.

cc @valarLip

[valar-cop]

ok, and the block manager's state_offload is indeed only the index built at block_manager.py:508, so the asymmetry is safe. but then the getattr exists just for the test fake (test_an_offload_without_touches_still_counts_a_tier_hit).. give the fake a want_touch and call it directly in both voters? don't want two conventions for the same method.

also 3 ci jobs still pending (atom test / vllm / sglang), hold until those are green.

[valar-cop]

some changes needed

any evidence touches actually reorder the lru? unrefreshed_touches == 0? and a disk backend run for the put fast path?

ci green.

[valar-cop]

these lmcache internals are all getattr, if a name changes we silently fall back to contains and every load still 'succeeds'.. can we resolve once in bind_storage_manager and fail loudly?

[ZLkanyo009]

@valar-cop The first half is done (1b311b6): bind_storage_manager resolves the LocalCPU LRU once (_local_cpu_lru), and touches use the resolved backend with no per-touch getattr.

On "fail loudly": 1b311b6 raised at bind, but your round-3 comment on line 46 asks not to take the engine down on an LMCache bump. 4eff581 settles on loud but not fatal:

• When an internal is missing, bind logs one WARNING naming it.
• Touches then only confirm presence.
• Every touch is counted in state_offload_touches_unrefreshed on the existing stats line.

That rules out the silent fallback you were worried about: a drifted LMCache shows touches_unrefreshed growing with touches_sent, and it is 0 on our runs. I didn't keep the raise because a lost refresh only costs a recompute on a miss, which is how loads behaved before this PR. The full reasoning and the tests are in the line-46 thread.

Checked on real LMCache in the container (0.5.6.dev139+gf22dec28, the real LocalCPUBackend with its default LRUCachePolicy; the script drives the PR's codec). get_evict_candidates(hot_cache, 1) is what LocalCPUBackend.allocate calls under memory pressure:

order before touch [1, 2, 3]
evict candidate before touch [1]
touch(1) -> REFRESHED
order after touch [2, 3, 1]
evict candidate after touch [2]

cc @valarLip

[valar-cop]

loud-not-fatal is the right call, and the real-lmcache run answers my earlier question.

one gap in the "drift shows up in touches_unrefreshed" story: the except at connector.py:333 maps a raising touch to REFRESHED and logs a traceback per touch, so a behavioral drift floods the log and leaves the counter at 0. count it as unrefreshed, or warn once like state_object.py:225?

[valar-cop]

fast path is true for any backend, so a disk-only image never gets re-promoted into cpu tier now? base it on hot_cache residency instead

[ZLkanyo009]

@valar-cop You're right; I confirmed it against the LMCache 0.5.6 source. With a disk backend, batched_put writes to every backend. LocalCPU then evicts from its hot_cache while LocalDisk keeps the file, so StorageManager.contains still returns true. The fast path therefore skipped the store, and the image stayed out of the CPU tier until a load promoted it back through get.

Fixed in 1b311b6. touch now returns TouchResult (GONE / PRESENT / REFRESHED), and put skips the store only on REFRESHED, meaning the image was in LocalCPU's hot_cache and its LRU entry was just moved. A disk-only image (PRESENT) goes through batched_put again. LocalCPU takes it, since the key is not in hot_cache, and LocalDisk skips it because key in self.dict, so the re-store costs one D2H and no second disk write.

Test: test_a_disk_only_image_is_stored_back_into_the_cpu_tier. The mock spills an image out of the hot cache into a disk dict, and the test checks that the second put reaches batched_put again (2 puts, not 1) and the image is back in hot_cache.

Disk-backend run: I ran it on 4eff581 with LMCache's own LMCACHE_LOCAL_DISK (1000 GB, 442 GB used) on top of the same 208.6 GiB LocalCPU pool: MI355X, AgentX c64, 900 s. LocalDisk kept every image LocalCPU evicted, so 93 of 723 touches found their image only on disk (state_offload_touches_unrefreshed=93, touch_misses=0, and the one-time warning fired). All 480 state loads completed, 0 failed, 0 holds expired, Traceback 0. There is no counter for how many of the 3,858 stores were such re-stores (stores_attempted does not split them), so the re-store itself is covered by the unit test above, not by a number from this run.

cc @valarLip

[valar-cop]

makes sense, and the code matches what you describe: the fast path only returns early on REFRESHED (atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py:177), PRESENT falls back into batched_put.

93 of 723 touches finding their image only on disk is exactly the case that broke, so that run plus the unit test is good enough for me. one thing: those re-stores are invisible in the stats line, so if they ever grow we wouldn't see it .. maybe count them next to touches_unrefreshed? rest lgtm.

[ZLkanyo009]

Left as is for this PR. The disk-only case already shows up as state_touches_unrefreshed, counted on each touch that finds such an image. A separate re-store count would be one more stats key, and like state_touches_unrefreshed it would only reach Prometheus through the shared llm_engine whitelist.

[valar-cop]

unrefreshed_touches only shows once via the warning, no ongoing signal in stats.. can we expose it so we can tell refresh from presence-only?

[ZLkanyo009]

@valar-cop Done in 1b311b6. The count is now in the existing state checkpoints: stats line as state_offload_touches_unrefreshed, next to state_offload_touches_sent and state_offload_touch_misses. That line now separates three outcomes:

• Refreshed: the image was in LocalCPU's hot cache and its LRU entry was moved.
• Presence-only: the image is resident on another backend but its eviction order was not touched. These are counted in touches_unrefreshed.
• Gone: counted in touch_misses.

How it is plumbed:

• Each worker reports, per touch, a second completion on a new k3_state_touch_refresh channel with the same StateTouchId, next to the existing resident / gone report on k3_state_touch.
• KVOutputAggregator applies the same failure-dominant quorum, so a touch counts as unrefreshed if any TP shard only found the image present.
• The scheduler connector counts the failures. take_state_touches_unrefreshed is forwarded through LMCacheOffloadConnectorScheduler / MultiConnectorScheduler like take_state_touch_verdicts.
• Scheduler._apply_state_touch_verdicts adds the count to StateOffloadIndex.touches_unrefreshed.
• The one-time WARNING on the worker stays.
• The refresh report is not a residency verdict: it never forgets a hash or marks one verified.

Tests:

• test_a_presence_only_shard_counts_the_touch_unrefreshed.
• test_both_touch_reports_reach_quorum_across_tp: two ranks go through a real KVOutputAggregator, one REFRESHED and one PRESENT. The touch stays resident and counts one unrefreshed.
• test_presence_only_touches_reach_the_stats_line.
• test_the_shells_forward_the_unrefreshed_count.

On the GPU smoke with in-process LocalCPU only: 1b311b6 (MI355X, AgentX c64, 900 s, 208.6 / 104.3 GiB) shows state_offload_touches_sent=757 state_offload_touch_misses=126 state_offload_touches_unrefreshed=0, so no touch fell back to presence. With a LocalDisk backend added (4eff581), it shows 93 of 723: those touches found their image only on disk. Details are in the summary.

cc @valarLip

[valar-cop]

lgtm, went through the plumbing: stats line, failure-dominant quorum, shell + multi forwarding are all wired as described.

one thing in connector.py:343: result is not TouchResult.PRESENT reports refreshed=True for a GONE image, so that flag really means "not unrefreshed".. rename it (or a comment), it reads like a residency signal next to resident.

[ZLkanyo009]

@valar-cop Added the comment in bdef7ef, next to the (resident, refreshed) verdict map: a GONE verdict counts as refreshed, since nothing was left unrefreshed. Kept the name to avoid touching the emit site and its tests.

[valar-cop]

checked, the comment sits on the verdict map (connector.py:130) and matches the emit site: GONE gives (False, True) and never reaches state_touches_unrefreshed at connector.py:960. not renaming is fine. resolved on my side.

[valar-cop]

some changes needed, the hold is base-scheduler only and the hard raise worries me

any real lmcache run showing touch reorders hot_cache? and perf number?

ci green.

[valar-cop]

allocate_waiting still admits with record=True, so the hold never applies there? can we route it through the probe api too

[ZLkanyo009]

@valar-cop I checked this, and I'd leave it as is. Routing it through the probe API would make that path worse, not better.

• allocate_waiting belongs to DecodeScheduler only: the decode process of RapidServe's shared-cache P/D split (DecodeEngineCore). The base scheduler's admission at line 2382 already goes through record_allocation(..., defer_unverified=True).
• DecodeScheduler overrides _schedule and never calls _publish_state_loads. Touches are published only from there (_publish_state_loads -> _publish_state_touches). So on this scheduler a touch is never sent and its verdict never comes back. A defer_unverified hold there could only wait out OFFLOAD_LOOKUP_DEFER_S (2 s) on every admission with a state hit, then admit as it does today.
• The same missing _publish_state_loads means a CPU-tier state load cannot be carried on this scheduler at all. The state tier is a base-scheduler feature; the prefill side of this split (PrefillScheduler) has no BlockManager.
• The can_allocate(seq) call in allocate_waiting is identical on main; this PR doesn't change it.

So "the hold is base-scheduler only" is true, and it's by construction: it is scoped to the only scheduler that publishes the touches the hold waits for. Making DecodeScheduler carry state-tier loads and touches would be a separate feature. I've left it out of this PR.

cc @valarLip

[valar-cop]

ok, checked — allocate_waiting calls allocate(seq) with no cached blocks, so no state load or touch gets queued there at all, and the only other publisher is the engine's idle drain (engine_core.py:440), which won't run under load. agree, leave it as is. add one line at the call site saying the verify hold is base-scheduler only, so the next reader doesn't ask again?

[valar-cop]

missing can_load check here? on kv_producer we hold for a load that never happens

[ZLkanyo009]

@valar-cop Yes, real; fixed in 4eff581. _tier_state_unverified ends in StateOffloadIndex.needs_verify(h), which only checked h in self.hashes. A kv_producer (can_load=False) fills hashes from its own stores, so a stale hash held the admission for a pass, up to OFFLOAD_LOOKUP_DEFER_S, while _attach_state_slots (through _tier_can_serve -> could_serve) would never offer the load. needs_verify now asks could_serve(h), the same capability + membership predicate request_load and both voters use, so the hold and the load cannot disagree. I put the check in the index, not at block_manager.py:1592, so every caller of needs_verify gets it.

Test: TestNeedsVerify::test_a_store_only_role_holds_nothing. A can_load=False index with a stale stored hash neither holds nor queues a touch. It fails on 1b311b6.

cc @valarLip

[valar-cop]

checked: needs_verify now goes through could_serve (atom/model_engine/state_offload.py:296), same predicate as _tier_can_serve (atom/model_engine/block_manager.py:2098) and request_load, so a store-only index can't hold. one gap left in the stale-gone guard: it compares verified_at to the latest touch send, so a gone verdict from an older touch landing after a newer store plus a newer touch still forgets a live hash.. only costs one hit, no park, so fine to leave. LGTM.

[valar-cop]

stale gone verdict can erase an image we just stored.. check _verified_at vs _touch_sent_at before forgetting?

[ZLkanyo009]

@valar-cop Fixed in 4eff581, as you suggested: on a gone verdict, apply_touch_verdict now first checks whether _verified_at[h] > _touch_sent_at[h]. If so, a store of h landed (note_stored marks it verified) after the latest touch was sent, so the touch looked before those bytes were there, and the verdict is dropped without counting a miss. This also covers a hash that was forgotten and stored again while an older touch was in flight: forget clears _touch_sent_at, so any later confirmation wins.

How likely it is end to end: within one scheduler pass, _apply_state_touch_verdicts runs before take_state_reports. A verdict and a store report from the same worker step therefore already land in the safe order. The window is a store report applied in an earlier pass than a verdict from a touch that ran before the store landed. That's narrow, but the index can't rule it out from what it sees, and a wrong forget throws away a CPU hit (the next request over that prefix recomputes it). The guard is two dict reads.

Trade-off: if two touches of h are in flight (they're throttled to one per 0.5 s), a resident verdict for the older one can make a gone verdict for the newer one look stale. The image is then kept until the next touch. That case costs at most a load that misses and recomputes, the pre-PR behavior, never wrong state.

Tests in TestIndexTouches, built with a hand-driven clock:

• test_a_gone_verdict_older_than_a_store_is_ignored: touch sent, then the store lands, then gone arrives; the hash stays. Fails on 1b311b6.
• test_a_gone_verdict_for_a_forgotten_then_restored_hash_is_ignored: fails on 1b311b6.
• test_a_gone_verdict_after_a_store_and_a_newer_touch_applies: store, then a newer touch, then gone; the hash is dropped.

cc @valarLip

[valar-cop]

lgtm. checked the guard and the ordering it leans on: _apply_state_touch_verdicts runs before take_state_reports in the pass (scheduler.py:4956 vs 4959), and forget clears _touch_sent_at so a re-store always wins. the two-touches-in-flight swallow is the only cost and it's a recompute, not wrong state.

one small thing: a dropped stale verdict is silent, no counter. if you want it visible in state checkpoints: later, fine, not a blocker.

[ZLkanyo009]

Left as is. A dropped stale verdict means a store landed after the touch looked, so the index already holds the newer fact; a counter would only show a race that is resolved correctly. I can add it if it turns out to be useful.

[valar-cop]

why raise here? a lmcache version bump should not take the engine down, degrade like the backend is None case

[ZLkanyo009]

@valar-cop Agreed; changed in 4eff581. This also revises my answer to your earlier comment on line 184 (fail loudly), so I'm explaining the trade-off in both threads.

What it does now:

• bind_storage_manager still resolves the LocalCPU LRU once (_local_cpu_lru); touches never getattr again.
• If local_cpu_backend, cpu_lock, hot_cache, cache_policy or a callable cache_policy.update_on_hit is missing, it no longer raises. It degrades exactly like the local_cpu_backend is None case: touches only confirm presence through StorageManager.contains, and loads and stores are unaffected.
• To keep it from being silent, which was your round-2 concern:
  • bind logs one WARNING naming what is missing, e.g. LMCache's storage manager has no local_cpu_backend.hot_cache, so state touches only confirm presence ... (every touch is counted in state_offload_touches_unrefreshed);
  • every touch then lands in state_offload_touches_unrefreshed on the state checkpoints: stats line. A drifted LMCache therefore shows up as touches_unrefreshed growing with touches_sent, where it should stay at 0.

Why degrade rather than raise: the touch only buys performance (keeping a hot image from being evicted). If it is lost, a load that misses recomputes, which is how every load behaved before this PR. That is not worth refusing to start the engine over on an LMCache bump. The two rounds asked for different things; this keeps both: the internals are resolved once at bind, as you asked in round 2, and the failure is loud in the log and the stats without taking the engine down.

Tests in tests/test_state_tier_lru_refresh.py (each fails on 1b311b6):

• test_a_local_cpu_without_the_lru_internals_degrades_to_presence[cpu_lock|hot_cache|cache_policy]
• test_a_policy_without_update_on_hit_degrades_to_presence
• test_a_storage_manager_without_local_cpu_backend_degrades_to_presence

Each checks for a single bind warning that names the missing attribute, PRESENT touches, and that every touch is counted.

cc @valarLip

[valar-cop]

degrade instead of raise is fine by me.

but the stat is not a clean drift signal: on your own disk run 93 of 723 touches were unrefreshed in normal operation (state_object.py:220, image only on LocalDisk), so a drifted lmcache and a disk tier look identical in touches_unrefreshed. your bind warning is the real signal then, don't lean on "should stay at 0".

minor: without hot_cache the put fast path (state_object.py:177) never matches, so a re-store of a resident image packs and D2Hs again and LocalCPU refuses the duplicate. rare, but fine if you're ok with that on a drifted lmcache.

[ZLkanyo009]

Left as is. This only happens on an LMCache whose LocalCPU internals moved, which bind already warns about once. The cost is a duplicate pack and D2H that LocalCPU refuses, not wrong state.

[ZLkanyo009]

@valar-cop thanks for the three rounds. This one comment answers all of them, because my round-1 and round-2 replies were held back and are posted now together with round 3. cc @valarLip

What changed since c54eda2 (head is now 4eff581; no new env vars, merge-only):

• 9219a8d (round 1): presence-only touches are counted and warned; _reachable tolerates an offload without want_touch.
• 1b311b6 (round 2):
  • The LocalCPU LRU is resolved once in bind_storage_manager.
  • put's fast path applies only when the image is in LocalCPU's hot_cache, so a disk-only image goes back into the CPU tier.
  • state_offload_touches_unrefreshed appears on the state checkpoints: stats line, through a new k3_state_touch_refresh completion (failure-dominant across TP).
• 4eff581 (round 3):
  • A missing LMCache internal degrades to presence-only touches instead of raising; bind warns once and every touch is counted.
  • A gone verdict older than the image's latest store is ignored.
  • needs_verify asks could_serve, so a kv_producer holds nothing.
• atom/kv_transfer/offload/README.md: the touch section covers the refresh channel and stat, and the "what to re-check when porting LMCache" table has a row for the internals the touch uses.

Every inline comment has its own reply. The only one I did not change is allocate_waiting (scheduler.py:2382). It belongs to RapidServe's DecodeScheduler, which never publishes state loads or touches, so a hold there would only time out; details in that thread. The two "raise" threads conflicted (round 2 asked to fail loudly, round 3 not to take the engine down), so the code now does both: internals are resolved once at bind, and a missing one is reported in the log and the stats without stopping the engine.

Does a touch reorder LMCache's hot_cache on real LMCache?

• Directly, in the container (lmcache 0.5.6.dev139+gf22dec28 from rocm/atom-dev:nightly_202610221638): the real LocalCPUBackend with its default LRUCachePolicy and OrderedDict hot_cache, driven by the PR's StateByteCodec. get_evict_candidates(hot_cache, 1) is what LocalCPUBackend.allocate calls under memory pressure.

  order before touch [1, 2, 3]   evict candidate [1]
  touch(1) -> REFRESHED
  order after touch  [2, 3, 1]   evict candidate [2]
  

• In the serving runs: LMCache's own stats only count evictions. They expose neither the LRU order nor a per-hit refresh, so I can't show the reorder from LMCache-side counters. The ATOM-side evidence is touches_unrefreshed. A touch counts as refreshed only if the key was found in hot_cache under cpu_lock and update_on_hit was called on it, which for LRU is move_to_end. On the in-process LocalCPU smoke below, 757 touches went out and touches_unrefreshed=0, with no touch of state hash ... failed warning in the log. So every touch either found its image gone (126 of them reached the index as misses) or moved it in hot_cache.

Disk backend (LMCACHE_LOCAL_DISK, LMCache's own config): one run on 4eff581. It adds LMCACHE_LOCAL_DISK=/tmp/a11_disk/, LMCACHE_MAX_LOCAL_DISK_SIZE=1000 (442 GB used) on top of the same 208.6 GiB LocalCPU pool; AgentX c64, 900 s.

• LocalDisk kept every image LocalCPU evicted. 93 of 723 touches found their image only on disk: touches_unrefreshed=93, touch_misses=0, and the one-time warning fired.
• 480 state loads, 0 failed; 0 holds expired; no touch errors; Traceback 0.
• No counter splits out how many stores were re-stores of disk-only images, so the fast-path fix is covered by test_a_disk_only_image_is_stored_back_into_the_cpu_tier rather than by a number from this run.
• This run's throughput (153,536 tok/s/GPU) is not a PR number. The disk tier also keeps evicted KV: recomputed input falls from 0.160 to 0.071 of input.
• Not tested with a disk backend: TP > 1, and long runs where the disk itself evicts.

Perf.

• A/B on c54eda2 against 881aacf, 4 rounds of 3600 s, cards swapped every round. Full table: fix(offload): refresh state images in LMCache's LRU on every tier hit and keep the index consistent #2668 (comment)
  • Throughput: 128,070 -> 128,694 tok/s/GPU (+0.5%).
  • TTFT avg 6.28 -> 6.23 s.
  • State loads: 4,570 with 0 failed.
• The later commits only change counters, verdict handling and the bind-time fallback; the touch itself is the same call. I didn't repeat the 4-round A/B. Single-GPU smokes, same config (900 s, so a noisier number than the 3600 s A/B):
  • 9219a8d: 112,617 tok/s/GPU; 471 loads, 0 failed.
  • 1b311b6: 111,973 tok/s/GPU (-0.6%, within run-to-run noise); 413 loads, 0 failed; 0 holds expired.
  • 4eff581: only smoked with the disk backend (below), so there is no like-for-like throughput number. Its changes do not touch a hot path. The hold is unchanged in this config: 370 admissions deferred for a verdict, against 375 on 1b311b6, with 0 holds expired. The can_load check only matters on kv_producer, and the stale-verdict guard only skips a forget.
• Machine and config for all runs:
  • One MI355X (gfx950) per server.
  • amd/Qwen3.8-Flash-Next-Quark-MXFP4, TP1, MTP 3 with forced acceptance 2.32, FP8 KV, in-process lmcache_offload.
  • LMCACHE_MAX_LOCAL_CPU_SIZE=208.6, OFFLOAD_STATE_CPU_SIZE=104.3, --max-num-seqs 64.
  • AgentX (inferencex-agentx-mvp, seed 42), c64.
  • Stack: main + [Qwen 3.8] LMCache offload for QSA + GDN + PLE state #2636 + perf(moe): Triton softmax top-k routing, bit-identical to AITER #2665 + feat(spec): capture the Qwen3.8 MTP draft steps in CUDA graphs (MRoPE) #2663 + fix(qwen4-exp): register the QSA FP8 scale planes for KV offload #2655 + this PR + perf(qwen4_exp): keep the PLE n-gram table in page-locked host memory, read over UVA #2771.
• These are author runs on our MI355X node. I did not run gfx942, TP > 1 or DP attention.

GSM8K (c54eda2; nothing numeric changed since). Same stack and node, real MTP 3 verification, FP8 KV, 208.6 / 104.3 GiB:

• Full 1319 questions, 5-shot: 0.9795, the same as with 881aacf.
• Split CPU retrieval on questions 0–439 (pass 1 stores, pass 2 reloads every prefix from the CPU tier): 0.9795 / 0.9795, against 0.9750 / 0.9818 for 881aacf. Pass 2's state-hit rate was 99.36%, as in that run.

LMCache interface (0.5.6):

• Touch: StorageManager.local_cpu_backend, resolved once at bind; then cache_policy.update_on_hit(key, hot_cache) under LocalCPUBackend.cpu_lock. That is the call touch_cache() makes for KV hits; for LRU it is move_to_end. I don't use contains(pin=True) + touch_cache(): the pin would need a matching unpin, and keys_in_request is the list the concurrent KV lookup is filling.
• Outside the hot cache: StorageManager.contains(key) checks presence and the touch is counted unrefreshed.
• Store, load and cap use batched_put, get and remove.
• Non-LRU policies: the touch makes the same update_on_hit call as a KV hit, so LFU bumps the frequency and FIFO is a no-op.
• Not verified: the LMCache MP server, remote backends, and other LMCache versions. The internals the touch uses are not public API; if they change, the bind warning and touches_unrefreshed show it.
• The unit-test mock models hot_cache, the LRU policy, refcounts, duplicate-key refusal, oldest-first eviction that skips referenced objects, and (new) a disk dict behind the hot cache. It does not model the asynchronous put task, pin counts, allocator-driven eviction or per-rank pools under TP; only the end-to-end runs cover those.

Tests / CI on 4eff581.

• tests/test_state_tier_lru_refresh.py: 81 passed (66 on 9219a8d). The 9 new or changed round-3 tests fail on 1b311b6. The round-2 tests are written against the TouchResult API that round introduced.
• Full CPU suite: 10130 passed, 3 failed (10126 / 3 on 1b311b6, 10115 / 3 on 9219a8d). The same 3 fail on every head and on main, because the inductor C++ build fails on the CPU host.
• black 26.10.1 / ruff clean locally.
• CI on 4eff581: Black, Ruff, non-GPU unit tests and the offline inference smoke pass. Check Pre Checkin Signal and the accuracy jobs were skipped by the Heavy CI Gate.

[valar-cop]

@ZLkanyo009 ok, the direct hot_cache test plus touches_unrefreshed 0 / 93 is the evidence i wanted, and degrading instead of raising on a missing internal is the right call.

two things left: tp > 1 and gfx942 never ran, and ci is still pending (atom test, vllm, sglang). can we get at least one tp>1 smoke before merge?

lgtm apart from that.

[valar-cop]

not ready yet.. need tp>1 evidence and clarify the codec doc

any tp>1 run? gfx942? and please re-run ATOM Test, it was cancelled

ci green.

[valar-cop]

single timestamp cannot attribute the verdict to the touch.. carry a round id instead of comparing wall clock?

[ZLkanyo009]

@valar-cop Leaving the timestamp guard as is. The case it misses is the one you described in r4426150719: a gone verdict from an older touch that lands after a newer store and a newer touch. The index then forgets a hash whose image is still there, and the next request over that prefix recomputes instead of loading. Nothing parks and no wrong state is restored, so the cost is one lost hit.

Attributing a verdict to its send needs the touch's generation to come back with it. The worker side already has one (StateTouchId(prefix_hash, generation), renamed from round to match the sibling ids), but the scheduler reduces the TP quorum to {hash: resident} in take_state_touch_verdicts, and StateOffloadIndex would have to keep a generation per hash next to _touch_sent_at. That changes the verdict interface across the connector, the scheduler and the index for a case that costs one recompute, so I'm not doing it in this PR.

In the TP2 + EP2 run (summary comment) the PR side had 55 touch misses in total, and a load that missed after a resident verdict would show up as loads_failed, which stayed at 0.

cc @valarLip

[valar-cop]

fine by me, that's the case i already called one lost hit in r4426150719, and the connector reduces to {hash: resident} at connector.py:726, so a generation would have to ride through it. not worth that for one recompute, and loads_failed at 0 covers the park case. leave it.

[valar-cop]

docstring says get promotes it back, but get never inserts into hot_cache? which is true...

[ZLkanyo009]

@valar-cop The production comment was right about the behavior, but it didn't say where the promotion happens, and the fake's comment was wrong. Both are reworded in 8e9e0c9.

• StateByteCodec.get calls storage_manager.get(key). In LMCache 0.5.6, StorageManager.get (lmcache/v1/storage_backend/storage_manager.py, lines 437-461) walks the active backends. On a hit from any backend other than LocalCPU, PD or Maru, it calls LocalCPUBackend.submit_put_task(key, memory_obj), which inserts the key into hot_cache and calls cache_policy.update_on_put. So a disk-only image that is read does go back into LocalCPU. LMCache does it, not ATOM, and the TouchResult.PRESENT comment now says that StorageManager.get is what writes it back.
• The fake's disk only stores presence (self.disk[key] = True) and its get reads only the hot cache, so it never modeled the read-back that its comment claimed. The comment now says it is presence only, used by contains; no test loads a disk-only image.

No behavior change.

cc @valarLip

[valar-cop]

lgtm, the PRESENT comment now names the write-back (StorageManager.get -> submit_put_task) and nothing claims ATOM does it, and the fake comment matches what the fake actually models. can't read lmcache source from this checkout so i'm taking your file/lines as the citation. the code is consistent: get re-touches after unpack at atom/kv_transfer/offload/hybrid/kimi_k3/state_object.py:340, so a promoted image does get refreshed.

[valarLip]

lmcache stauff should be here

[ZLkanyo009]

@valarLip I read this as: LMCache-specific code shouldn't live in block_manager.py. If you meant something else, please correct me.

Done in 77d081e. BlockManager no longer reads anything LMCache-specific for this PR:

• lmcache_pool_bytes and the state_tier_lmcache_sizes(config) call are gone. The chunk-size probe is back to main's one line, self._joint_chunk_tokens = state_tier_chunk_tokens(config).
• StateOffloadIndex now takes the confirmation window itself, verified_for_s=..., in seconds, instead of LMCache's pool_bytes. BlockManager passes state_tier_verified_for_s(config) and knows nothing about the pool.
• The LMCache read lives in atom/model_engine/state_offload.py, next to main's state_tier_chunk_tokens. Both now share one reader, _lmcache_engine_config(config), which does the build_lmcache_config call and the multi sub-config unwrap that was inline in main's state_tier_chunk_tokens. state_tier_verified_for_s turns LMCACHE_MAX_LOCAL_CPU_SIZE into the window through the existing tier_verified_for_s.
• Behaviour is unchanged:
  • Same values: 256 tokens and 25 s for a 100 GiB pool in the test; 52 s at 208.6 GiB.
  • Same gate: both reads happen only when a state tier is hosted (kv_offload_enabled). An engine without offload, or without LMCache installed, still neither imports LMCache nor logs a traceback; main's TestJointChunkProbeIsGated tests pass unchanged.
  • With an unreadable LMCache config, the chunk probe still logs its one traceback, and the window falls back to the 5 s floor.
• What's left in block_manager.py from this PR is one import and that verified_for_s= argument.

LMCache-specific code that was already in block_manager.py on main (99ebf63), and that this PR does not touch:

• The chunk-size probe in __init__ (state_tier_chunk_tokens(config) into _joint_chunk_tokens), and the kv_offload_enabled gate around it.
• The joint KV boundary (_joint_kv_boundary and its helpers), which aligns to LMCache's chunk grid and reports lmcache_within_hbm.
• The tier-hit reach that reads LMCache's seq.offload_joint.kv_prefix_tokens, around compressed_hit.
• Assorted comments that name LMCache.

Moving those behind a backend-neutral interface would change main's code and the joint-load path, so I left them out of this PR. Do you want that done in a separate PR?

Tests:

• tests/test_state_tier_lru_refresh.py::TestVerifiedWindow::test_the_block_manager_sizes_the_window_from_the_lmcache_pool: now stubs the LMCache config reader, so both values go through the real helpers.
• New test_an_unreadable_lmcache_config_gets_the_floor.
• Full CPU suite: 10131 passed, 3 failed. The same 3 fail on main: the inductor C++ build fails on the CPU host.
• black 26.10.1 / ruff clean.
• No GPU run: nothing on the admission or scheduling path changed.

[ZLkanyo009]

@valarLip done in 8d40782. This PR adds nothing LMCache-specific to block_manager.py: one import, the verified_for_s= argument and the admission hold. The pool-size read lives in state_offload.py, and it now shares one reader, _lmcache_engine_config, with main's state_tier_chunk_tokens, as described above. That reader had been dropped in a later round and is back. Main's existing LMCache code in block_manager.py is unchanged; I can move it in a separate PR if you want.

[valar-cop]

ci triage for 8d40782:

ATOM vLLM Test passed. https://github.com/ROCm/ATOM/actions/runs/42164740143

ATOM SGLang Test passed. https://github.com/ROCm/ATOM/actions/runs/42164740116

ATOM Test passed. https://github.com/ROCm/ATOM/actions/runs/42164740067

Pre Checkin passed. https://github.com/ROCm/ATOM/actions/runs/42164718528

[valar-cop]

a few small things..

tp>1 run of the touch/verdict path?

ci green.

[valar-cop]

log this window once at startup? otherwise the holds/expiries line has no reference

[ZLkanyo009]

@valar-cop Not adding a startup line for this. The window is a fixed function of one value LMCache already logs at startup: max_local_cpu_size appears in LMCache's Creating LMCacheEngine with config line. The window is clamp(0.25 s × max_local_cpu_size GiB, 5, 60), and the offload README gives that formula; 208.6 GiB gives 52 s.

The other input is the hold bound, OFFLOAD_LOOKUP_DEFER_S: 2.0 s by default, documented in docs/environment_variables.md. So the holds / expiries counters can already be read against both numbers from the startup log and the config. Nothing is wrong or unsafe without the extra line, so I'm keeping this PR to fixes only.

For reference, the TP2 + EP2 run in the summary used 208.6 GiB per worker, which gives 52.15 s.

cc @valarLip

[valar-cop]

ok, taking lmcache's config log line as given since i can't read lmcache here. formula is in the README and the pool size is in the config it prints, so the window is derivable, and the unreadable-pool path already logs the 5 s floor at atom/model_engine/state_offload.py:664. dropping it.

[valar-cop]

a raising touch counts as refreshed here, so drift looks healthy.. warn once and count it unrefreshed?

[ZLkanyo009]

@valar-cop Fixed in 8e9e0c9. A touch whose codec call raises is now reported as resident but unrefreshed, the same verdict as TouchResult.PRESENT:

• The image stays indexed, because the exception says nothing about the bytes.
• The touch is counted as unrefreshed, so a drifted LMCache that raises on every touch shows up instead of looking healthy. After the independent review the count is state_touches_unrefreshed in the K3 connector's get_statistics() (024e3e8) rather than a separate take method on three connectors.
• Only the first failure on each worker logs a traceback, instead of one per touch.

Test: TestConnectorTouches::test_a_failing_touch_is_unrefreshed_and_logged_once (three failing touches over two steps: all (True, False), one record with exc_info). It fails on 77d081e. The offload README's touch paragraph says it too.

This is the same gap you raised in r4426148581.

cc @valarLip

[valar-cop]

nothing here promotes disk-only back to hot_cache, reword to what get actually guarantees? same for the fake

[ZLkanyo009]

@valar-cop Same point as r4426260647, answered there: LMCache's own StorageManager.get writes a hit from LocalDisk back into LocalCPU via LocalCPUBackend.submit_put_task (0.5.6, storage_manager.py lines 437-461), and nothing in ATOM does. 8e9e0c9 says that in the TouchResult.PRESENT comment. It also fixes the fake's comment, which claimed a read-back the fake does not model: its disk only answers contains.

cc @valarLip

[valar-cop]

checked, both comments now say what the code does: _refresh only returns PRESENT off contains and get goes straight to storage_manager.get (state_object.py:232, :313), so naming LMCache's write-back instead of ours at state_object.py:30 is right, and the fake's comment no longer claims a read-back it doesn't model.

resolved on my side.

[ZLkanyo009]

@valarLip This push is an overall cleanup of the PR, not another point fix. Since the version you reviewed on 11-01 (77d081e83) there are five commits:

• 8e9e0c9bd: a raising touch counts as unrefreshed.
• 0dda9c9d3: where the code lives.
• b1582b46b: trim to what the fix needs.
• 024e3e8c3: the changes from an independent quality review of that result.
• caf4ea8f5: moves a test block so it merges cleanly with [Qwen 3.8] LMCache offload for QSA + GDN + PLE state #2636.

Behaviour is unchanged apart from the raising-touch fix and one dropped duplicate touch (below).

Size (added lines against the merge base, blank lines not counted): source 464 → 438, tests 1052 → 903, comments and docs 312 → 164. In total +2103/−62 → +1709/−69.

Moved out of the shared files

• block_manager.py no longer decides anything about the tier. _tier_state_unverified is split: the hold decision and its counter are StateOffloadIndex.hold_admission(h). The block manager only reports which off-HBM hash an admission would restore from (_resume_hash_off_hbm). That is built on _state_resume_hash, which allocate also uses and which reuses tier_resume_replays. The index takes one argument, verified_for_s=....
• scheduler.py no longer has _publish_state_touches / _apply_state_touch_verdicts. They are StateOffloadIndex.publish_touches / collect_touch_verdicts, and the scheduler keeps two call sites.
• tests/test_state_tier_lru_refresh.py is gone. Its tests are in the existing files for each component: test_state_offload_index.py, test_lmcache_offload_connector.py, test_state_checkpoint.py and test_block_manager.py. The shared clock fixture is in conftest.py.

Removed or folded

• The unrefreshed-touch count had its own take method on the K3 scheduler, the shell and multi. It is now state_touches_unrefreshed in the K3 connector's existing get_statistics(), which the shell and multi already forward.
• An offered load no longer queues a touch of its own. The load is its own verdict (complete_load / fail_load), and the codec's get refreshes the LRU.
• Single-caller helpers folded into their caller: needs_verify, verify_defer_s() and stop_refreshing(). The same goes for a getattr shim in _reachable that only test fakes needed.
• Test-only or unreachable inputs:
  • the index's defer_s argument (the index reads OFFLOAD_LOOKUP_DEFER_S itself, still only when the refresh is on);
  • a StateTouchId round check;
  • an empty-list guard in enqueue_state_touches;
  • an unreachable state_bytes <= 0 branch.
• Unit round trips: the window and the budget check take GiB directly, and the window is one 0.25 s/GiB constant.
• A missing LocalCPU internal now warns once, not twice with a wrong reason.
• StateTouchId.round is renamed generation, like its sibling ids.
• The touch methods joined StateOffloadFace.
• Comments, docstrings and the offload README now state the constraint, not the derivation.
• Copy-pasted tests are parametrized, and duplicates are merged.

Reviewed and kept

• The kimi_k3 connector.py / state_object.py changes. Qwen3.8's state tier is the kimi_k3 hybrid offload, and the worker side of a touch can only live there.
• The try/except around the codec touch, which wraps LMCache internals rather than repo code.
• The refresh channel next to the residency channel: the TP quorum is what lets the scheduler count each touch once.
• The startup budget warning (item 4 of the description).

Verification

• black / ruff clean.
• CPU full suite: 10125 passed; the 3 failures (test_graph_marker ×2, test_sparse_indexer_side_effect_op) fail on main too.
• GPU unit tests (23 files: the changed tests plus the block manager, scheduler and state-offload ones): 1864 passed. The 2 failures are the engram/test_host_path.py environment cases, which fail the same way on main.
• 27 mutations, one per mechanism (including the cleanup that drops a hash's touch state with it), were applied in turn. Each one turns a test red.
• TP2 + EP2 (-tp 2 --enable-expert-parallel, MI355X, MTP 3, FP8 KV, LMCACHE_MAX_LOCAL_CPU_SIZE=208.6, OFFLOAD_STATE_CPU_SIZE=104.3, --max-num-seqs 64). Main is the deployment stack without this PR, and both runs used the same cards. AgentX c64, 1800 s, then full GSM8K:

run	tok/s/GPU	TTFT avg / p50 / p90 s	recomputed share of input	state loads (failed)	touches (found gone)	admissions held (hold expired)	requests	GSM8K strict / flexible
main	170,192	2.2 / 0.9 / 6.3	0.065	111 (23)	–	–	3,392	0.9803 / 0.9803
this PR (caf4ea8f5)	174,709	2.0 / 0.8 / 5.9	0.059	102 (0)	177 (55)	265 (0)	3,466	0.9773 / 0.9773

• On main, 23 of 111 state loads missed in LMCache and recomputed. With the PR none failed: 55 touches found an image already gone, and those hashes left the index before an admission planned a load on them.
• Neither side logged a traceback. No touch was unrefreshed: the once-only warning never fired.
• GSM8K never reaches the state tier: all three GSM8K runs had 0 state loads, 0 touches and 0 holds. The PR-minus-main gap is −0.0030, with a paired stderr of 0.0026 over 1319 questions. The same PR before the review round scored 0.9795, and the two PR runs differ on 11 questions with the same code path, so this is run-to-run noise.

The PR description is rewritten to say why, and its file list matches this diff.

cc @ZhangLirong-amd

[valar-cop]

a few small things..

ci green.

[valar-cop]

please add one comment here or at allocate_waiting, the hold is base-scheduler only since DecodeScheduler never publishes touches

[ZLkanyo009]

@valar-cop Done in 3ccc2e9: the comment at the hold now says it is base-scheduler only, since DecodeScheduler never publishes touches and so never holds.

[valar-cop]

checked, the comment is there at scheduler.py:2387 and the hold claim is right: _schedule is overridden (scheduler.py:5390) and allocate_waiting never passes defer_unverified, so a decode admission can't hold. resolved on my side.

one wording question: the idle drain still calls _publish_state_loads on a decode engine (engine_core.py:441), so can a decode engine publish touches when idle? if so, 'never publishes touches' is a bit strong, 'never on this path' covers it.

ci still in progress on this head, hold until green.

[ZLkanyo009]

@valar-cop Yes, it can: the idle drain reaches _publish_state_loads and so publish_touches, and allocate_waiting -> can_allocate -> _reachable registers touches. Fixed in bdef7ef: the comment now gives the real reason, that allocate_waiting never passes defer_unverified, so a decode engine never holds.

[valar-cop]

checked, the comment is right now: on a decode engine allocate_waiting goes can_allocate -> _reachable -> want_touch (page_unit_checkpoint.py:1081) and the idle drain's _publish_state_loads does publish those touches (scheduler.py:3871), but neither path calls record_allocation, so the hold can't fire. resolved on my side.

pre checkin still in progress, hold until it's green.

[ZLkanyo009]

Update caf4ea8f5..3ccc2e9fe: trim the diff to what the hit refresh and the admission hold need. Behaviour is unchanged, apart from dropping the startup budget warning.

Diff vs. main (merge-base 035a78a4d): 21 files, +1709/−69 → 19 files, +1558/−25. Non-blank lines: source +438/−37 → +387/−18, tests +903/−6 → +842/−2, comments and docs +164/−25 → +133/−5. The remaining removed lines are:

• signatures (record_allocation, connector_completion);
• the call lines the feature changes;
• imports reformatted for new names;
• one appended docs row;
• allocate calling the shared _state_resume_hash, kept as @ZhangLirong-amd asked (point 1 of the earlier review) so allocate and the hold cannot drift apart.

Reverted, by file:

• disaggregation/types.py: the StateStoreOperationId docstring reflow.
• offload/_offload_common.py: back to main. StateOffloadFace gets no new abstract methods; the shell and multi connectors already forward with getattr.
• kimi_k3/state_object.py:
  • the _cache_keys comment;
  • the get() miss-pop, with its test (get() keeps only the LRU refresh);
  • the move of put()'s key line.
• kimi_k3/connector.py:
  • max_cache_bytes=state_tier_cpu_bytes() as on main;
  • the startup warning when the state cap exceeds half the pool. It came with the removed reservation and is not part of the hit refresh.
• model_engine/state_offload.py:
  • the original class, forget, could_serve and state_tier_cpu_bytes docstrings (the class docstring only gains a paragraph);
  • state_tier_chunk_tokens is no longer refactored;
  • state_tier_budget_warning is removed.
• model_engine/block_manager.py: record_allocation keeps its original shape and only gains the hold, behind a keyword-only defer_unverified=False, so other callers are unchanged.
• model_engine/scheduler.py: the original _publish_state_loads docstring and lines. The touch calls are insertions only.
• offload/README.md: the budget-warning sentence.
• Tests:
  • conftest.py and _FakeStorage back to main;
  • an existing module docstring restored;
  • the new block moved above the existing section header;
  • dropped the budget-warning cases and four codec tests that pass unchanged on main (they test _put, which this PR does not touch).

Also, per @valar-cop's note on scheduler.py, a comment saying the hold is base-scheduler only, because DecodeScheduler never publishes touches.

Not changed (seen, left as on main):

• OFFLOAD_STATE_CPU_SIZE stays a cap, with no startup check against the pool; its docstring still says "carved from".
• get() still leaves a missed key in the codec's cap LRU until the next put or touch.
• state_tier_chunk_tokens and state_tier_verified_for_s each unwrap the multi sub-config.
• The vLLM plugin gets only the codec-level changes, with no engine-side touches.

Verification on 3ccc2e9fe:

• black / ruff clean.
• CPU full: 10115 passed. The 3 failures also fail on main (test_graph_marker ×2, test_sparse_indexer_side_effect_op).
• GPU unit tests (MI355X): 1854 passed. The 2 failures are engram/test_host_path, which fails the same way on main.
• 27 mutations of the touch and hold mechanisms each turn a test red.
• TP2 + EP2 full GSM8K: 0.981 (previous revision 0.9773, main 0.9803).

Pending:

• CI on the new head.
• Reviewer sign-off.
• [Qwen 3.8] LMCache offload for QSA + GDN + PLE state #2636 will conflict here in record_allocation only: keep its four lines after _commit_joint_boundary, then this PR's return True.

@ZhangLirong-amd @valar-cop cc @valarLip

[ZLkanyo009]

Update 3ccc2e9fe..85f74ff40: merge origin/main at 8a07708e1 (37 new commits since the previous merge-base, including #2906). The PR's own code is unchanged.

Conflict: only docs/environment_variables.md. Main added four OFFLOAD_MP_* rows after OFFLOAD_LOOKUP_DEFER_S and left that row untouched. The resolution keeps this PR's OFFLOAD_LOOKUP_DEFER_S row as it was, followed by main's four rows.

The diff against the new main is identical to the previous one: 19 files, +1558/−25, with the same files and the same changed lines. Nothing from main leaks into it.

Checked against #2906 (lmcache_mp lookup thread, transfer thread, same-step saves); nothing here needed to change:

• scheduler.py: before both build_connector_meta calls, _publish_state_loads(), which queues this PR's touches, still runs ahead of the new _note_connector_scheduled_tokens, so touches still reach the metadata. The admission hold (record_allocation(defer_unverified=True)) and the verdict collection beside the state store reports are untouched by main.
• The new lookup and transfer threads belong to lmcache_mp only. They never reach StateOffloadIndex, _reachable or the block manager. offer_lookups is implemented only by the MP scheduler and the multi forwarder, and the K3 tier on lmcache_offload never goes through it. Touches, verdicts and the hold therefore stay on the scheduler thread.
• metadata.py: main's after_forward is on LMCacheReqMeta; state_touches is on LMCacheOffloadMetadata.
• offload/connector.py, multi_connector.py: main's new after_forward, note_scheduled_tokens and offer_lookups forwarders sit next to the touch forwarders. K3 defines no after_forward.

Verification on 85f74ff40, CPU only (no GPU was available this round, so no GPU unit tests or GSM8K):

• black 26.10.1 / ruff clean.
• Related tests (75 kv_transfer / offload / state tier / scheduler / block manager files): 2612 passed. The 4 errors in test_lmcache_mp_builder_views.py ('atom.utils' is not a package) also occur on main alone.
• CPU full: 10722 passed against 10655 on main 8a07708e1. The same 3 tests fail on both (test_graph_marker ×2, test_sparse_indexer_side_effect_op).

Pending:

• CI on the new head.
• Re-review: @valarLip's change request from 11-01 is still open.
• [Qwen 3.8] LMCache offload for QSA + GDN + PLE state #2636 still conflicts with this PR in one block of record_allocation; the resolution is unchanged.

@valarLip @ZhangLirong-amd

[ZLkanyo009]

Update 85f74ff40..bdef7efb4: comment and wording fixes from a review-pr self-review, plus tidier test imports. No behaviour change.

Diff vs. main (merge-base 8a07708e1): 19 files, +1558/−25 → +1549/−25. This commit touches 4 files (+12/−21). No conflict with current main (69d7931b0).

The review-pr pass found 0 must-change, 0 worth-addressing and 4 smaller items. No correctness problems.

Changed:

• scheduler.py: the hold comment had the wrong reason. DecodeScheduler does publish touches when idle; it never holds because allocate_waiting never passes defer_unverified. This corrects the comment from r4439370699.
• kimi_k3/connector.py: the (resident, refreshed) comment now says a GONE verdict counts as refreshed (nothing was left unrefreshed).
• state_offload.py: the stats comment names loads_deferred_for_verify (admission passes held). The expired-hold warning and comment now say verdicts are "lost or late (e.g. a stalled step)" rather than "being lost"; the one expiry seen in smoke runs was a JIT stall, not a lost verdict.
• tests/test_lmcache_offload_connector.py: the touch channels are imported at the top with the other kimi_k3 names; three in-function imports of names already imported there are removed.
• PR description: one sentence on why the hold cannot reuse lmcache_mp's lookup_pending deferral.

Follow-ups, not in this PR:

• One scheduler-level hook for both "skip until the tier answers" timers (lookup_pending per sequence, hold_admission per hash). This needs lmcache_mp changes.
• Export state_touches_unrefreshed to Prometheus. This needs the shared llm_engine whitelist and exporter.
• Moving main's existing LMCache code out of block_manager.py, as asked on @valarLip's thread.

Verification on bdef7efb4, CPU only (no GPU this round; comments and test imports only):

• black / ruff clean.
• Related tests (76 kv_transfer / offload / state tier / scheduler / block manager files): 2612 passed. The 4 errors in test_lmcache_mp_builder_views.py also occur on main.
• The review's own run of affected_tests.py on 85f74ff40: 2626 passed, 0 failures.

Pending:

• CI on the new head.
• @valarLip's re-review of the change request from 11-01.

@valarLip @ZhangLirong-amd

[ZLkanyo009]

Update bdef7efb4..8d4078222: a simplification pass over this PR's own code and tests, then an independent review of it (0 must-change, 3 worth-addressing, 7 smaller; all fixed here except one). No behaviour change.

Diff vs. main (merge-base 8a07708e1): 19 files, +1549/−25 → 21 files, +1624/−38.

• Source: +387/−18 → +395/−25. The extra deletions are main's config-reading lines in state_tier_chunk_tokens, which moved into the shared reader.
• Tests: +832/−2 → +887/−2, mostly the new wiring tests.
• Comments and docs: +137/−5 → +142/−10.
• New files touched: tests/conftest.py and tests/test_offload_shell.py.
• No conflict with current main (e302f8796).

What changed, by kind:

• One fact in one place.
  • state_tier_chunk_tokens and state_tier_verified_for_s share one reader, _lmcache_engine_config; main's import, multi unwrap and build moved into it unchanged. This is the shared reader my 11-01 reply on the block_manager.py thread described; it had been dropped in a later round.
  • The offload README no longer repeats the window constants.
  • offload_clock is one fixture in tests/conftest.py.
• One value instead of a tuple. The K3 worker keeps each touch's TouchResult, now a plain Enum compared by identity. _emit_touch_verdicts, next to _emit_load_dispositions, turns it into the residency and refresh completions. With no tier, every touch is GONE before the loop.
• Conversions done once. Touch ids become ints where enqueue_state_touches creates them. The later int() / bool() / float() calls are removed, and a -1e18 sentinel is now -inf.
• Comments state constraints only. The AgentX c64 measurement behind the window constants moved into the PR description. A hold_admission docstring paragraph that repeated the field comments is gone.
• Tests check behaviour, not internals.
  • The worker touch tests check the completions the worker emits.
  • The codec tests check the pool instead of _cache_keys.
  • The window bounds come from tier_verified_for_s instead of private constants.
  • Two index tests that the scheduler tests already cover are removed.
  • No function-local imports.
• Wiring tests. A touch-only step counts as work for the worker, start_load_kv touches, the scheduler applies verdicts, and the plain and multi scheduler shells forward touches and verdicts. Removing any of those lines used to leave the CPU tests green. 34 mutations now each turn a test red (was 27).

Not changed:

• StateOffloadIndex(verified_for_s=None) means an index whose connector carries no touches.
• record_allocation(..., defer_unverified=False): only the admission loop can retry a request a pass later.
• The forwards use getattr, like their siblings on main.
• The vLLM plugin's state hits are also answered by StateOffloadIndex, so the LRU asymmetry remains there. Fixing it needs touches through the plugin's metadata and verdicts back through vLLM's connector output, so it is left for a separate change.
• The three optional valar-cop suggestions are left as is; the reasons are in each thread.

Verification on 8d4078222:

• Lint (black 26.10.1, ruff): clean.
• Related test files (8): 863 passed, 1 skipped.
• Mutation check: 34/34 mutants killed.
• CPU full suite: 10725 passed, 3 failed. The 3 failures (test_graph_marker ×2, test_sparse_indexer_side_effect_op) fail the same way on main.
• GPU unit tests (MI355X, smoke tree with the same patch): 1863 passed, 2 failed; the 2 engram failures are on main too.
• Kimi-K3 TP2+EP2 GSM8K: 0.978 (stderr 0.004), run on the previous head e9481f9bd. Not rerun here because this update changes no behaviour; that run also sent no touches.

Pending:

• CI on the new head (running).
• @valarLip's re-review of the change request from 11-01.

@ZhangLirong-amd @valar-cop cc @valarLip