[number-eleven11]

Motivation

This PR extends the existing Weight Cache Daemon feature introduced in
#27139 and described in
Fast Engine Recovery: Sub-Second Engine Restart for SGLang via Weight Cache Daemon
to DeepSeek V4.1, with Engram in either GPU or host memory.

The article addresses the repeated checkpoint I/O and weight processing paid on
every engine restart, which prolong service outages during worker failures and
rolling updates. It retains processed weights in a persistent daemon and lets
replacement engines reuse them through CUDA IPC. This PR builds on that feature
and addresses the gaps that prevent the same recovery path from working for V4.1.

Why this extension is needed:

• V4.1 needs more than tensor import. Its FP8 dense / FP4 expert path has
  post-load layouts, quantization metadata and cached tensor references that a
  replacement engine must restore consistently. Related failures are documented
  in #39794. This PR restores
  the processed state, rebinds mHC and TopK references, and checks backend/layout
  compatibility instead of merely relaxing the IPC admission check or repeating
  transforms on shared weights.
• Host Engram needs its own resident recovery path. GPU tensor IPC cannot
  share its host backing. This PR preserves daemon-owned host tables and imports
  them in replacement workers; optional HOST_NUMA VMM reuses allocation handles
  to avoid repeated cudaHostRegister on unchanged tables.
• Faster checkpoint reads still pay checkpoint I/O on restart. The
  InstantTensor and positioned-read PRs below accelerate loading from storage.
  This PR complements them by reusing already processed GPU weights and host
  Engram after a worker restart, while the daemon survives. Their cold-load
  optimizations can still benefit initial daemon startup, but are not included
  in this patch.

The result is a V4.1-specific implementation of the article's resident-weight
recovery path, plus an experimental prewarmed standby for its active-standby
scenario. Complete weight reload and service recovery results are reported below;
full long-context acceptance and other remaining gates are still pending.

Related work

• #27139 — Support fast engine recovery through weight cache: the original feature this PR extends.
• #38306 — Support post-load lifecycle and MXFP4 IPC: related model-support work describing K3/DSV3 validation; this PR adds V4.1-specific state restoration and host Engram recovery.
• #39794 — Reject routed MXFP4 experts for IPC: documents why accepting a base FP8 configuration is insufficient for routed-expert post-load layouts.
• #40453 — Add InstantTensor loader support, and #42919 — Load V4.1 with the Engram host table: accelerate checkpoint loading while keeping host-resident tables off the GPU stream.
• #42920 — Parallel positioned reads for Engram host tables: accelerates storage-to-host loading with parallel preadv; this PR addresses resident recovery and mapping reuse instead.

Speed Tests and Profiling

Engram in GPU memory

Disk reload compared with resident CUDA IPC recovery:

Metric	Before	After	Speedup
Complete main-model weight reload	50.56 s	1.02 s	49.6×
Complete service recovery: process launch → healthy service	95.09 s	36.53 s	2.60×

Values are medians: two final matched disk runs and three recovery runs.
Compilation/autotuning caches were warm. The weight reload covers the entire
main model, including Engram and post-load state; service recovery also includes
communication, KV and runtime initialization.

Engram in shared host memory

Host layout and comparison	Metric	Before	After	Speedup
Shared table: disk reload → resident IPC + memfd	Complete main-model weight reload	85.45 s	14.44 s	5.9×
Per-rank shards: existing memfd recovery → HOST_NUMA VMM	Complete main-model weight reload	4.77 s	3.36 s	1.42×
Per-rank shards: existing memfd recovery → HOST_NUMA VMM	Launch-helper return → polled service readiness	45.54 s	40.53 s	1.12×

Host recovery values are medians of three runs; the shared-table disk baseline
has one sample. Shared tables and per-rank shards are separate experiments:
85.45 s is not a matched baseline for 3.36 s. Exact process-launch to healthy
service time was not measured for the host VMM comparison; its readiness timer
starts after the launch helper returns.

Both placements require an already loaded, surviving daemon for optimized
recovery; initial daemon startup is excluded.

An already prewarmed standby provided its first response token 0.205–0.422 s after active-launcher SIGKILL in the memfd experiment. This is standby takeover, not a new worker reload. VMM plus standby has not yet had a combined full-model fault-injection run.

Setup: four NVIDIA L20D GPUs, TP4/EP1/attention-DP1, native FP8 dense/FP4 expert, flashinfer_mxfp4 / dsv4, vision disabled, context 4096, maximum four requests, CUDA Graph and overlap disabled. The VMM experiment used private KV capacity 8192 per worker.

Modifications

• Add explicit V4.1 recovery opt-in, a versioned post-load state contract and compatibility fingerprint covering model, daemon, IPC loader, load orchestration and relevant quantization source. Restore processed tensor aliases, quantization metadata, mHC references and TopK correction bias without transforming shared weights again.
• Add daemon-owned host Engram FD sharing, with memfd and optional cuda_vmm backends. VMM clients map GPU views of the same physical host allocation without copying it or repeating host registration.
• Align daemon/client backend initialization and isolate draft loading from target IPC loading.
• Add an experimental two-worker standby gateway with generation warmup, promotion and replenishment. Failed requests are not replayed, and session state is rejected.
• Add registered CPU tests and configuration/usage documentation.

Upgrade note: restart the weight daemon after upgrading recovery source, then start clients from the same source/configuration. Clients reject a cache produced by different recovery code; the old daemon's processed state is not silently accepted.

Accuracy Tests

72 CPU tests pass (8 host VMM, 6 host FD, 17 model-state/compatibility, 32 protocol, 9 standby). The new regression test verifies that changes to recovery lifecycle and quantization source invalidate the compatibility fingerprint. Earlier revisions passed upstream lint; the latest revision is submitted to CI. GPU timings are from the frozen experiments and were not rerun after the final compatibility-check hardening.

For the host VMM comparison, all 30 prompt comparisons matched text and input/output token IDs; 24/30 also matched logprobs exactly. The 1401-token prompt differed even in a memfd same-service repeat, so strict long-context numerical acceptance remains unresolved. Short prompts and the 385-token case were exact. Host gather and shared-table CUDA Graph component checks passed; full-model CUDA Graph/shared-layout validation is pending.

Limitations

The feature remains explicitly opt-in and experimental. The experiment used sglang-kernel 0.4.7 with the source's expected 0.4.9 version check bypassed; aligned-dependency validation is pending. Full-capacity throughput, draft/speculative execution and other topologies remain unvalidated. The daemon must survive, and same-GPU standby workers do not isolate GPU faults. In-flight requests/KV state are not recovered. Initial daemon host allocation costs remain; this does not implement #42920's parallel checkpoint reads.

Checklist

• Format changed code and pass upstream lint.
• Add registered tests and pass 72 related CPU cases.
• Document configuration and before/after recovery results.
• Complete long-context and aligned-dependency validation.
• Run full GPU CI with the required maintainer approval / run-ci label.

---

CI States

Latest PR Test (Base): ❌ Run #38537918009
Latest PR Test (Extra): ❌ Run #38537918119
Latest PR Test (AMD ROCm 10): ❌ Run #38537917934