[ormandj]

Builds on #40453 (InstantTensor loader by YitaoYuan), which is now on main.

Motivation

--load-format instanttensor (#40453) streams every checkpoint file to the GPU, and each weight loader then copies its part. With --engram-config '{"placement": "cpu"}', DeepSeek-V4.1 keeps its Engram tables in pinned host memory: 188.8 GiB of rows and scales in 2 of the 48 checkpoint files. Under InstantTensor, every rank would first materialize each full table on its GPU. The FP8 rows of one table are about 91.5 GiB. That does not fit on a 96 GB GPU, where the rest of the model already takes about 73 GiB per rank at TP4. So the host table currently has to use the default loader.

Modifications

• DefaultModelLoader.Source.host_resident_weight_patterns: a model can list substrings of checkpoint tensor names that it keeps in host memory. Source.init_new reads this attribute from the model, like allow_patterns_overrides.
• Under --load-format instanttensor, when the model declares host-resident patterns, the loader reads the safetensors headers and splits the file list. Files with no matching tensor stream through InstantTensor as before. Files that contain a matching tensor load through the existing multi-threaded CPU safetensors iterator. The matching tensors stay on the host. The other tensors in those files are moved to the current device, so every tensor that is not host-resident arrives on the GPU, as with InstantTensor. When nothing matches, InstantTensor gets the full file list, as before. Non-safetensors files are not opened by the split; InstantTensor still reports them. As with the default loader, --weight-loader-disable-mmap cannot read the F8_E8M0 scales in the Engram files (safetensors.torch.load in safetensors 0.8.0 has no such dtype); a follow-up PR avoids that path.
• DeepseekV4ForCausalLM returns (".engram.embed.",) when --engram-config places the Engram tables on the host ("placement": "cpu") and None otherwise. The DSpark draft shares the property because it reads the same checkpoint and drops the Engram tensors. In the DeepSeek-V4.1 checkpoint, the two Engram files contain only Engram tensors.

Accuracy Tests

Load-path change only: the same values reach the same weight loaders. New tests in test/registered/unit/model_loader/test_instanttensor_loader.py use a three-file checkpoint with DeepSeek-V4.1 tensor names and an InstantTensor stub. The stub records the files it receives and places tensors on a stand-in device:

• With the host table enabled, the Engram file never reaches InstantTensor. Only the two Engram table tensors stay on the host, with exact values. The Engram file's other tensor arrives on the device. Every checkpoint tensor is yielded exactly once.
• With the host table disabled, all files go to InstantTensor as before.
• A checkpoint with only host-resident files never calls InstantTensor.
• The split does not open non-safetensors files.
• The DSpark draft declares the same patterns as the target.

On this branch (current main plus #40453), the file passes (14 tests) on macOS (CPU) and on Linux aarch64 (CPU, PyTorch 2.14.1). With the production change reverted, 4 of the 5 new tests fail. The rest of test/registered/unit/model_loader/ also passes on macOS (132 passed, 1 skipped in total; test_modelopt_loader.py was not collected in the local environment because tvm_ffi is not installed).

Earlier, on #40453 at the same head with an older main, a separate check used the real InstantTensor library on one RTX PRO 6000 Blackwell GPU. The dense files and the Engram file's other tensor arrived on cuda:0, and the Engram tables stayed on the CPU with exact values. The production diff of this PR is unchanged since that check.

2026-10-29: rebased onto main 974831643f, where #40453 has merged (the branch now carries only this commit) and #46893 replaced SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE with --engram-config, so DeepseekV4ForCausalLM now declares the Engram patterns when --engram-config sets "placement": "cpu"; test_instanttensor_loader.py passes on the new head (14 tests, macOS CPU), and the SM120 GPU results in this description are from the previous head and will be re-run.

Speed Tests and Profiling

DeepSeek-V4.1-Flash, TP4/EP4, 4x RTX PRO 6000 Blackwell (SM120, PCIe), host Engram table (per-rank layout), checkpoint on local NVMe (ZFS), one server start each, on an older main with the same source otherwise. Times are the Load weight elapsed time on TP0 for the target model:

Loader	Target model load
default safetensors (mmap)	264.2 s
--load-format instanttensor (#40453) + this change	119.9 s

Without this change, InstantTensor cannot be used with the host table on this setup (see Motivation), so the default loader is the only comparison. In the InstantTensor run, the 46 device files took about 42 s. The 2 Engram files then took about 68 s through the CPU safetensors path. A follow-up PR replaces that path with positioned reads.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

Developed with AI assistance.

---

CI States

Latest PR Test (Base): ❌ Run #40961104686
Latest PR Test (Extra): ❌ Run #40961104852
Latest PR Test (AMD ROCm 10): ❌ Run #40961104822