[donghyq]

Motivation

An active hybrid-model request contains more than a reusable prefix: it also owns Mamba recurrent state, generated tokens/logprobs, and native decoding progress. This draft adds an opt-in, default-disabled snapshot path to continue that request in a fresh SGLang process after the source exits, without recomputing its committed history.

The supported contract is deliberately bounded: identical model weights/version, source build, TP size, dtype, page/layer geometry, and the same device pair, using trusted shared local storage. Local pool capacity may differ when the actual request allocations fit. Exact sampling-RNG trajectory equivalence, cross-host/weight recovery, crash-atomicity, HA, Trainer integration, and performance are outside this draft's demonstrated scope.

Modifications

• Export Attention KV plus Mamba conv/temporal state at a committed native decoding boundary; require every TP rank's immutable payload and manifest before source-slot release.
• Reconstruct a new native request with its token/logprob history, normalized sampling parameters, reasoning/stream progress, and remaining generation budget; validate identity and payload integrity before allocating/importing into fresh slots.
• Separate per-request state geometry from local Mamba pool capacity, preserving strict dtype, layer, version, source-build, and TP checks.
• Include public export/resume/reference/negative clients, a public-input reproduction guide, and focused CPU storage/request contracts. No platform launcher, private model data, prompt, logs, or service endpoints are included.
• Align the CPU tests with CustomTestCase; handle empty CPU-only CUDA_VISIBLE_DEVICES in the test runner's port default without changing nonempty-device behavior.

Relationship to existing work

• verl#8273 preserves resident vLLM requests through weight updates. This draft provides SGLang fresh-process continuation at the same weight version; it does not claim to complete the whole verl roadmap item.
• verl#8168 restores serving processes/weights and replays requests to rebuild KV. This draft persists the actual hybrid request state.
• #41694 and #39511 provide related hybrid prefix/checkpoint storage work. Generic hybrid cache storage is not a novelty claim here. Feedback on the active-request API, ownership, and reuse of existing storage abstractions is particularly useful; the duplicate search was bounded, not exhaustive.

Accuracy Tests

GPU qualification on the original tested base ed0b8f433f887995b1ce361b15aaf9f6ca93483f, before refreshing this draft onto 127808733667e7b5b9be38f9dbb72acc6464bab4:

• Qwen3.5-35B-A3B, the same two A100 80GB GPUs, TP2; sequential source exit then fresh target, not two simultaneous engines.
• Actual source/target Mamba pool capacities were 219 and 281 per rank. New target slots received 32,737,280 bytes per rank in total across Attention and Mamba conv/temporal state, with matching digests and no retained source-process record.
• All 128 token IDs and 128 selected-token logprobs matched an uninterrupted greedy reference exactly (max difference 0): 10 preserved outputs plus 118 new outputs.
• Both TP ranks reported 53 committed history positions, zero historical positions recomputed, 118 completed native decoding inputs, and zero extend inputs. These are completed-forward observations, not initial zero counters or cache-hit inference. They do not imply zero prefill across a whole workload.
• Owned source exit, an empty GPU compute inventory before target startup, and owned target cleanup were verified separately from the public clients.

CPU and rejection evidence:

• The tested capacity revision passed five targeted Linux tests with full module imports and CUDA uninitialized.
• A prior revision passed 27 Linux CPU tests and four real invalid restore requests (version/layout/source binding/manifest reference): HTTP 200 headers followed by SSE error 400, no outputs/import side effects. Those four HTTP cases were not rerun on this refreshed draft.
• Current-base local checks: all applicable pre-commit hooks passed for the changed files; 14 real filesystem-store component cases passed in an isolated harness (CI imports/base projected out, not a full module run); 10 cases exercised the actual runner-port conditional, including reproducing and fixing empty-device visibility; all 13 changed Python files parsed; the three public CLI help commands exited successfully. Complete current-base Linux imports/CI and GPU validation remain pending. Local full-module collection is blocked by unavailable Triton on macOS; no pass is claimed for it.

The GPU prompt is not distributed. The reproduction guide supplies a public input and parameterized commands; that input is not claimed as the previously measured replay. The public clients never start or stop engines.

CUDA_VISIBLE_DEVICES='' PYTHONPATH=python python test/registered/unit/mem_cache/test_token_kv_snapshot_store.py
CUDA_VISIBLE_DEVICES='' PYTHONPATH=python python test/registered/unit/managers/test_token_kv_snapshot.py

Speed Tests and Profiling

Not run. This is a correctness feature; no latency, throughput, GPU-hour savings, or cross-weight equivalence is claimed. Normal serving behavior remains default-disabled, but enabled-path overhead still needs measurement before any performance claim.

Checklist

• Applicable changed-file pre-commit hooks
• Current-base Linux CPU CI and related engine CI
• Human submitter line-by-line review and test review/execution
• Maintainer agreement on API/lifecycle and storage overlap
• Current-base GPU qualification before marking ready for merge
• Public reproduction clients and scope documentation

AI assistance (Codex and GLM-5.3) was used for implementation, tests, and documentation. Human submitter line-by-line review remains pending while this PR is a draft; no human review or test attestation is implied.

---

CI States

Latest PR Test (Base): ❌ Run #39576461085
Latest PR Test (Extra): ❌ Run #39576460849
Latest PR Test (AMD ROCm 10): ❌ Run #39576460860

[ifishland]

This PR is too large.