Skip to content
Merged
Show file tree
Hide file tree
Changes from 52 commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
f0610f5
DeepSeek-V4.1-Flash: LMCache tier offload through the MP server (vLLM…
zgplvyou Oct 17, 2026
26f5c0c
dsv41 offload: serve checkpoint-backed local hits; drop debug instrum…
zgplvyou Oct 19, 2026
d10a698
dsv41 offload: address PR review (preemption fence, tail saves, ride …
zgplvyou Oct 19, 2026
9c04c16
recipe(dsv41): HBM-then-tier results, DSpark LMCache on/off, theory v…
zgplvyou Oct 19, 2026
96927f7
test(dsv41): sort the decode-keep test's imports (ruff I001)
zgplvyou Oct 19, 2026
95c687d
dsv41 offload: look up only the extent a load can end on (#2642)
linsun12 Oct 20, 2026
08d6753
style(dsv41): black after rebasing onto main
zgplvyou Oct 26, 2026
807d20b
mp worker: release finished transfers on the step thread, while its s…
zgplvyou Oct 26, 2026
13bae6f
dsv41: restore checkpoints when the cursor read is deferred
zgplvyou Oct 26, 2026
6fdb823
dsv41 checkpoints: store decode keeps at the commit, from the build's…
zgplvyou Oct 26, 2026
a7d323e
mp worker: destroy IPC resources on the step thread once nobody else …
zgplvyou Oct 26, 2026
bece476
Merge main 481c2bfd4 into the V4.1 LMCache offload branch
zgplvyou Oct 27, 2026
0dcb360
test(lmcache-mp): skip the submit_request fallback test without its p…
zgplvyou Oct 27, 2026
b845e89
dsv41: keep the plugin's checkpoint restore and warmup exemption out …
zgplvyou Oct 27, 2026
ac8bcbd
dsv41 offload: address review (vLLM 0.33 hit cap, save retirement, fe…
zgplvyou Oct 28, 2026
a6ed491
dsv41 offload: address follow-up review (MP stalled saves, layout ove…
zgplvyou Oct 29, 2026
428fd67
dsv41 offload: give the plugin test fixtures the connector's new fields
zgplvyou Oct 29, 2026
dd4a59a
deepseek_v41: keep vLLM plugin rules out of the model validator
zgplvyou Oct 29, 2026
52cbd1e
deepseek_v41: drop the plugin check from the model validator
zgplvyou Oct 29, 2026
65cd127
dsv41 offload: keep vLLM's knowledge out of the shared offload code
zgplvyou Oct 29, 2026
6e7ba5f
docs: describe what the V4.1 plugin admits now
zgplvyou Oct 29, 2026
fee2de2
dsv41 offload: address review round 9
zgplvyou Oct 29, 2026
9b2cdf9
tests: let the index-plane parity test's QuantizationConfig stub take…
zgplvyou Oct 29, 2026
4c07290
Address round 10 review: V4.1 plugin guards and per-step lookup budget
zgplvyou Oct 31, 2026
57f8757
plugin state checkpoints: one defaults helper and one runner wrapper
zgplvyou Oct 31, 2026
69e2ebc
mp lookup: check the client once instead of adapting every request
zgplvyou Oct 31, 2026
7512caf
mp worker: take one last look at pending loads in get_finished
zgplvyou Oct 31, 2026
22021a6
mp worker: reuse step events instead of creating and destroying one p…
zgplvyou Oct 31, 2026
831f5c8
plugin V2 runner: read checkpoint block ids from host rows, not a tab…
zgplvyou Oct 31, 2026
71b69db
V4.1 plugin: one owner for the prefill tail and one prefill-chunk cli…
zgplvyou Oct 31, 2026
0763613
V4.1 plugin: let V2 profile CUDA-graph memory instead of skipping by …
zgplvyou Oct 31, 2026
9cc401a
mp worker: one native-state registration and block-id path for every …
zgplvyou Oct 31, 2026
c4721c2
V4.1 plugin: carry the state checkpoint as a native-state image
zgplvyou Oct 31, 2026
d8e827d
V4.1 plugin: a finished request saves to its last checkpointed boundary
zgplvyou Oct 31, 2026
c4569ad
V4.1 plugin: say which side enforces the tier hit grain
zgplvyou Oct 31, 2026
8eaf726
vLLM plugin offload: one BoundaryPlanner base for the KDA and V4.1 pl…
zgplvyou Oct 31, 2026
69fbd74
vLLM plugin offload: drive every boundary planner through one loop pe…
zgplvyou Oct 31, 2026
9da531f
vLLM plugin offload: build each planner next to its class, and refuse…
zgplvyou Oct 31, 2026
de2c4dd
recipe: DeepSeek-V4.1 nvfp4 does not start on vLLM 0.31.1
zgplvyou Oct 31, 2026
ee65d81
Merge origin/main into the DeepSeek-V4.1 LMCache offload branch
zgplvyou Oct 31, 2026
414de75
plugin V2 host rows: replace a resumed row, arm only when kept; CPU-s…
zgplvyou Oct 31, 2026
65e2907
tests: use list for the empty runner-class stub (ruff PIE807)
zgplvyou Oct 31, 2026
105730d
mp worker: record the step event only for a step that reaches the server
zgplvyou Oct 31, 2026
0687f4b
V4.1 plugin: store decode keeps at the next build, before it stages
zgplvyou Nov 1, 2026
6da7026
plugin V2 host rows: keep them only on runners that read them
zgplvyou Nov 1, 2026
3587050
test: give the proxy factory contract the decode-keep store site
zgplvyou Nov 1, 2026
7cb4bf9
V4.1 plugin: build the step from the host where the host is exact
zgplvyou Nov 1, 2026
f5d9788
V4.1 plugin: defer the cursor read only while serving synchronously
zgplvyou Nov 1, 2026
b6f6685
offload: address the core review of 11-02
zgplvyou Nov 2, 2026
53e32bd
offload: report native saves that land during the forward that step
zgplvyou Nov 2, 2026
413df1b
V4.1: build the pool geometry beside the model config
zgplvyou Nov 2, 2026
3593a34
offload: run native ATOM's MP transfers on the step thread again
zgplvyou Nov 2, 2026
7081137
offload: address the second pass of the 11-02 core review
zgplvyou Nov 3, 2026
9e70cbc
Merge main 498133ed9 into the V4.1 LMCache offload branch
zgplvyou Nov 3, 2026
97c8dde
offload: native ATOM drops finished MP futures as main does
zgplvyou Nov 3, 2026
85d55d8
offload: keep native V4.1's LMCache namespace as main computes it
zgplvyou Nov 3, 2026
0132de1
offload: one MP transfer thread for both frontends, and scheduler fixes
zgplvyou Nov 3, 2026
4b589d8
offload: drop handed-back MP futures while the stream is idle
zgplvyou Nov 3, 2026
45355d2
offload: the plugin's polling worker as a constructor flag, and revie…
zgplvyou Nov 4, 2026
a278334
offload: swap the transfer thread's error under its lock
zgplvyou Nov 4, 2026
6a08af2
offload: native MP paths as main, one floor record, and the 11-05 review
zgplvyou Nov 5, 2026
d7619fe
tests: wire two plugin tests to the plugin's dense scheduler
zgplvyou Nov 5, 2026
ba157dd
offload: settle the polling thread's shutdown, failure and stalls
zgplvyou Nov 6, 2026
ce7dd45
offload: deadlines from the forward thread, and the 11-06 review
zgplvyou Nov 6, 2026
4b0dc75
offload: never fail-stop a finished transfer from the forward thread
zgplvyou Nov 6, 2026
ce1187d
offload: remove the forward thread's deadline check
zgplvyou Nov 9, 2026
f259bbb
Merge origin/main into perzhang/dsv41-lmcache-031
zgplvyou Nov 9, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions atom/kv_transfer/offload/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,8 +148,12 @@ blocks before the remote store becomes terminal.
### Running the MP server

The matching LMCache build must support the global `--null-block-id` setting
and sparse null handling. Run the MP server on the same host, with GPU IPC
access to the ATOM worker allocations:
and sparse null handling. `--null-block-id` must equal the engine's
`lmcache.mp.null_block_id` (default `-1`); native ATOM and the vLLM plugin
read the same option, so one server can serve both. No RPC reports the
server's value, so a mismatch is not caught: a chunk the engine marks empty
would be read as a real block. Run the MP server on the same host, with GPU
IPC access to the ATOM worker allocations:

```bash
lmcache server --host 127.0.0.1 --port 5555 \
Expand Down
57 changes: 47 additions & 10 deletions atom/kv_transfer/offload/chunked_scheduler.py
Original file line number Diff line number Diff line change
Expand Up @@ -663,6 +663,46 @@ def _build_save_request(
save_operation=operation,
)

def _unbuilt_save_is_final(self) -> bool:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_unbuilt_save_is_final defaults to True in the base class, the opposite of the pre-PR behaviour (keep the tracker and retry next step). In core, the only scheduler whose _build_save_request returns None is NativeState, which has to override it back to False (native_state_scheduler.py:388); the only one that needs True is the plugin's V4.1. Please flip the default to False and let V4.1 opt in. Also, _retire_finished_save (L706-735) is a fourth hand-written "pop the tracker, return the lease" — and it deducts still_read, while _retry_or_retire_failed_save's finished branch (L1441-1455) returns the whole lease; one concept, two rules — fold them into one helper. And _save_source_is_backed (L737-750) is unreachable in core (core's block_table is an int array, so is not None is always true) yet runs for every model, with a WARNING and stats.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b6f6685: the default is False and V4.1 opts in. One helper retires a finished save on every path and keeps blocks another operation still reads. The unbacked-save check moved to the plugin's schedulers.

"""Whether a save `_build_save_request` declined stays unbuildable once
its request has finished, so its tracker is retired rather than kept
for a retry. False, as before this hook existed: a layout that only
declines for reasons that clear (a full pin budget, a source still
being written) retries. A layout whose declines are final opts in."""
return False

def _retire_finished_save(self, sid: str, seq) -> None:
"""Retire a finished request's save that will never be emitted.

Gives back the suffix leased at teardown, except blocks another
operation of this request may still read; `_release_operation_lease`
frees those when that operation becomes source-safe. The one rule for
every path that retires a finished request's save.
"""
entry = self._save_tracker.get(sid)
if entry is None or entry[0] is not seq:
return
self._save_tracker.pop(sid, None)
lease_key = id(seq)
leased = self._save_lease_blocks.get(lease_key)
if not leased:
return
still_read: set[int] = set()
for operation, owner in self._save_operation_owner.items():
if owner is seq:
still_read.update(
self._save_operation_blocks.get(operation, {}).values()
)
free = leased - still_read
if free:
leased.difference_update(free)
self._pending_source_safe_releases.append(frozenset(free))
self.total_source_safe_released_blocks += len(free)
if not leased:
self._save_lease_blocks.pop(lease_key, None)
self._save_lease_at.pop(lease_key, None)
self._save_lease_owner.pop(lease_key, None)

def _build_load_request(
self,
seq,
Expand Down Expand Up @@ -896,6 +936,12 @@ def build_connector_meta(self) -> LMCacheOffloadMetadata:
if request is None:
if late_acquired:
self._block_manager.free_leased_blocks(late_acquired)
if (
getattr(seq, "_offload_finished", False)
and self._unbuilt_save_is_final()
):
# A tracker left behind would hold its lease forever.
self._retire_finished_save(sid, seq)
continue
self._track_save_statistics(save_operation, aligned - saved)
meta.add_request(request)
Expand Down Expand Up @@ -1306,16 +1352,7 @@ def _retry_or_retire_failed_save(self, operation: SaveOperationId) -> bool:
if getattr(entry[0], "_offload_finished", False):
# Teardown may already have returned the source blocks to the pool.
# A missing cache entry is safer than a retry against recycled KV.
self._save_tracker.pop(sid, None)
lease_key = id(entry[0])
remaining = self._save_lease_blocks.pop(lease_key, None)
self._save_lease_at.pop(lease_key, None)
self._save_lease_owner.pop(lease_key, None)
if remaining:
# Unbound (plugin) teardown also leased the not-yet-emitted
# suffix; no further save reads it once the request retires.
self._pending_source_safe_releases.append(frozenset(remaining))
self.total_source_safe_released_blocks += len(remaining)
self._retire_finished_save(sid, entry[0])
return False
entry[1] = min(int(entry[1]), previous)
logger.warning(
Expand Down
70 changes: 58 additions & 12 deletions atom/kv_transfer/offload/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@
"kimi_k3": "kimi_k3",
"qwen": "qwen",
"m3": "m3",
"dsv41": "dsv41",
# Compatibility with the names used while the layout split was developed.
"terminal_unit": "hybrid",
"page_slot": "hybrid",
Expand All @@ -54,6 +55,11 @@
"indexer_dtype",
"indexer_types",
)
# Hashed for the `dsv41` layout only, so no other model's namespace moves:
# `sliding_window` sizes V4.1's per-request window ring and
# `candidate_block_size` pages its index plane, so either one moves the bytes
# a stored object holds while every field above stays put.
_DSV41_PAGE_FIELDS = ("sliding_window", "candidate_block_size")
_HF_INTEGER_GEOMETRY_FIELDS = frozenset(_HF_PAGE_FIELDS) - {
"compress_ratios",
"indexer_dtype",
Expand All @@ -80,12 +86,23 @@

# Layouts that own a tier for a model's per-request *recurrent* state.
# `kimi_k3` is the kimi_linear KDA state; `qwen` is the same tier over a GDN
# PAGE image. `hybrid` (DSV4 sparse-attention checkpoints) is not
# PAGE image; `dsv41` is DeepSeek-V4.1's CSA2 window rings, compressor rings
# and Engram cursor. `hybrid` (DSV4 sparse-attention checkpoints) is not
# recurrent-state and dense<->hybrid is a legitimate operator override for
# namespace separation. An explicit `offload_layout` override may not
# downgrade a recurrent-state model to a layout that owns no tier for it
# (silent wrong output) -- see `select_offload_layout`.
_STATE_OWNING_LAYOUTS = frozenset({"kimi_k3", "qwen"})
_STATE_OWNING_LAYOUTS = frozenset({"kimi_k3", "qwen", "dsv41"})

# Every spelling of the V4.1 family, mirroring `atom.utils.selector._V41_TYPES`.
# Matching only the text config's own type left the wrapper and DSpark
# spellings to fall through to the `compress_ratios` probe below and resolve to
# `hybrid` -- outside `_STATE_OWNING_LAYOUTS`, so an `offload_layout: dense`
# override would be honoured on a model that carries per-request state, and
# sharing DSV4's key space besides.
_DSV41_MODEL_TYPES = frozenset(
{"deepseek_v41", "deepseek_v41_text", "deepseek_v41_dspark"}
)

logger = logging.getLogger("atom")

Expand Down Expand Up @@ -155,17 +172,28 @@ def select_offload_layout(config) -> str:
"does not preserve the NSA index cache. Drop the override or set "
"it to 'm3'."
)
# An override may pick between compatible layouts, but it may not strip a
# state-owning model down to a layout with no tier for its per-request state
# -- the same silent-wrong-output the GDN refusal pre-empts, reached by a
# different door (an explicit `dense` rather than a fall-through).
if natural in _STATE_OWNING_LAYOUTS and mapped not in _STATE_OWNING_LAYOUTS:
# An override may pick between compatible layouts, but a state-owning
# model's state tier is its own: any other layout, state-owning or not,
# has no tier for that state -- the same silent-wrong-output the GDN
# refusal pre-empts, reached by a different door (an explicit override
# rather than a fall-through). `dsv41` on a K3 model, say, reaches the
# dense connector with the state tier off.
if natural in _STATE_OWNING_LAYOUTS and mapped != natural:
raise ValueError(
f"lmcache_offload: offload_layout={override!r} would run a "
f"{natural!r}-family model (per-request recurrent state) on the "
f"{mapped!r} layout, which owns no tier for that state -- its KV "
"prefix would be restored over stale state (silent wrong output). "
"Drop the override, or disable offload for this model."
f"Drop the override, set it to {natural!r}, or disable offload for "
"this model."
)
# The reverse for the one layout this alias table gained for a single
# family: `dsv41` names DeepSeek-V4.1's PAGE+STATE pool, and any other
# model would run the dense codec under its namespace with collapse off.
if mapped == "dsv41" and natural != "dsv41":
raise ValueError(
f"lmcache_offload: offload_layout={override!r} is DeepSeek-V4.1's "
f"layout; this model resolves to {natural!r}. Drop the override."
)
return mapped

Expand Down Expand Up @@ -207,6 +235,16 @@ def _layout_from_model(config) -> str:
# KDA per-request state that a reusable prefix also needs.
if model_type == "kimi_linear":
return "kimi_k3"
# DeepSeek-V4.1 before the `compress_ratios` probe below, which it would
# otherwise answer: V4.1 declares compress ratios exactly as DSV4 does, but
# it also carries per-request CSA2 state (window ring per layer, the
# compressor's incomplete group, the Engram cursor) that the `hybrid`
# PAGE+SLOT codec owns no tier for. Resolving it to `hybrid` would both
# share DSV4's key space and leave it outside `_STATE_OWNING_LAYOUTS`, so
# an `offload_layout: dense` override would silently restore a KV prefix
# over a window ring that never existed.
if model_type in _DSV41_MODEL_TYPES:
return "dsv41"
if getattr(hf_config, "compress_ratios", None):
return "hybrid"
# GDN publishes the same PAGE-image contract the kimi_k3 tier already
Expand Down Expand Up @@ -342,14 +380,16 @@ def build_page_namespace(
base_model_name = str(
getattr(config, "model_tag", None) or getattr(config, "model", "atom-model")
)
layout = select_offload_layout(config)
document = {
"schema": "atom-page-namespace",
"layout_version": layout_version,
"model": base_model_name,
"page_mode": {
"hybrid": "dsv4-page-regions",
"m3": "m3-page-regions",
}.get(select_offload_layout(config), "dense-opaque-block"),
"dsv41": "dsv41-page-state",
}.get(layout, "dense-opaque-block"),
"kv_cache_dtype": str(getattr(config, "kv_cache_dtype", "auto")),
"index_cache_dtype": str(getattr(config, "index_cache_dtype", "auto")),
"block_size": _strict_integer(
Expand All @@ -375,10 +415,16 @@ def build_page_namespace(
# The config-time snapshot when there is one; see `Config.__post_init__`.
"hf_geometry": getattr(config, "offload_page_hf_geometry", None)
or _stable_hf_geometry(hf),
"speculative_config": _stable_config_value(
getattr(config, "speculative_config", None)
),
# A frontend's own stable form when it supplies one, as for the HF
# geometry above: only it knows which of its fields are per-process.
"speculative_config": getattr(config, "offload_speculative_config", None)
or _stable_config_value(getattr(config, "speculative_config", None)),
}
if layout == "dsv41":
document["dsv41_geometry"] = {
name: _strict_integer(f"PAGE {name}", getattr(hf, name))
for name in _DSV41_PAGE_FIELDS
}
canonical = json.dumps(
document,
sort_keys=True,
Expand Down
11 changes: 10 additions & 1 deletion atom/kv_transfer/offload/metadata.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,18 @@ def __init__(
if self.atom_bytes_per_block <= 0:
raise ValueError("ATOM raw-byte metadata: bytes_per_block must be > 0")
if chunk_size % self.atom_block_size != 0:
# The block size is not always the configured one: on a hybrid
# model it can be raised until one attention page holds one
# recurrent-state page (Kimi-K3: 128 -> 1536), so the chunk size
# that does work cannot be read off the command line. Name it here
# rather than only the two values that do not work.
raise ValueError(
"LMCache chunk size must be divisible by ATOM KV block size: "
f"chunk_size={chunk_size}, block_size={self.atom_block_size}"
f"chunk_size={chunk_size}, block_size={self.atom_block_size}. "
"Set LMCACHE_CHUNK_SIZE to a multiple of "
f"{self.atom_block_size}. On a hybrid model the block size "
"can be derived from the recurrent state page size, so it can "
"differ from the configured one."
)

def __getattr__(self, name: str) -> Any:
Expand Down
27 changes: 23 additions & 4 deletions atom/kv_transfer/offload/mp/deployment.py
Original file line number Diff line number Diff line change
Expand Up @@ -158,10 +158,11 @@ def _config_has_fully_replicated_tp_pages(config: Any) -> bool:
if offcfg._is_minimax_m3(hf_config):
return False
hf_config = getattr(hf_config, "text_config", hf_config)
# Kimi-K3's MLA KV is replicated, but its KDA checkpoint images -- stored
# in those same PAGE units -- hold TP-sharded heads, so every rank keeps
# its own copy.
if getattr(hf_config, "model_type", None) == "kimi_linear":
# A state-owning layout's recurrent checkpoints ride inside the PAGE unit.
# Kimi-K3's hold TP-sharded heads, so the unit as a whole is not replicated
# even though the MLA KV in it is; the others are left one copy per rank
# until their backend declares otherwise.
if offcfg.select_offload_layout(config) in offcfg._STATE_OWNING_LAYOUTS:
return False
return getattr(hf_config, "kv_lora_rank", None) is not None

Expand All @@ -182,6 +183,24 @@ def _config_has_own_pool_draft(config: Any) -> bool:
return bool(get_attn_backend(attn_family(draft_hf)).DRAFT_OWNS_KV_POOL)


def _null_block_id(config: Any) -> int:
"""The MP server's ``--null-block-id``, which marks a chunk with no image.

A native-state layout names it for every chunk before the one holding a
checkpoint image. It has to equal the server's: a value the server reads as
a real block makes a later hit restore whatever that block holds. No RPC
reports the server's value, so it is configured here
(``lmcache.mp.null_block_id``) and defaults to the ``-1`` the offload
README starts the server with; every frontend sharing a server fills the
same one.
"""
return offcfg._strict_integer(
"lmcache.mp.null_block_id",
_extra_config(config).get("lmcache.mp.null_block_id", -1),
minimum=-1,
)


def _tp_replication_factor(config: Any) -> int:
"""Return the PAGE rank-collapse factor selected before workers start.

Expand Down
Loading