[YitaoYuan]

Motivation

Add InstantTensor as an optional high-performance loader for Safetensors checkpoints.

InstantTensor supports distributed weight loading and pipelined prefetching. This integration lets users select it with --load-format instanttensor while preserving SGLang's existing checkpoint discovery and model-loading flow.

It has already been integrated into several other projects, such as vLLM. https://docs.vllm.ai/en/latest/models/extensions/instanttensor/

Modifications

• Add instanttensor to the supported load formats and CLI choices.
• Add an InstantTensor weight iterator based on instanttensor.safe_open.
• Require PyTorch's cuda device interface (including CUDA and ROCm). Reject unsupported devices during server-argument validation and at the iterator entry, before importing InstantTensor or accessing the current CUDA device.
• Reuse the existing Safetensors file discovery, shard index filtering, and checkpoint selection logic.
• Validate selected filenames at the InstantTensor iterator entry and reject non-.safetensors files before opening any checkpoint. Keep SGLang's existing discovery, download, cache, and pattern-priority behavior unchanged; unsupported selections fail explicitly rather than being automatically replaced.
• Pass SGLang's initialized device process group to InstantTensor for distributed loading.
• Support tuning through --model-loader-extra-config, including buffer_size, chunk_size, io_depth, and backend.
• Convert JSON backend names or ordered candidate lists to InstantTensor backend/policy enums. Skip SGLang's extra-config validation for InstantTensor and forward optional parameters without filtering. Pass filename, framework, device, process group, and copy=True explicitly; Python rejects duplicate arguments, and InstantTensor handles unsupported parameters and values.
• Add the optional sglang[instanttensor] dependency with instanttensor>=0.1.9 to support these tuning options.
• Document core tuning options in the shared loader configuration table and link to InstantTensor for the latest options.
• Extend SRTRunner to select the load format and pass extra loader configuration in model-loading tests.
• Add focused CPU unit tests and a weekly single-GPU output-consistency test comparing Safetensors with explicitly configured InstantTensor loading.

Configuration example

pip install "sglang[instanttensor]"

python -m sglang.launch_server \
  --model-path Qwen/Qwen3.5-4B \
  --load-format instanttensor \
  --model-loader-extra-config '{"backend":"URING","chunk_size":8388608,"io_depth":64}'

For the tuning options shown above, explicit non-null values override the corresponding INSTANTTENSOR_* environment variables. Omitted or null values retain InstantTensor's environment-variable and automatic defaults. Backend names are case-sensitive; an ordered list such as {"backend":["URING","AIO"]} selects the first available candidate. Additional optional parameters are forwarded unchanged, except that filename, framework, device, process_group, and copy cannot be overridden. Unsupported parameters, including generic loader settings such as enable_multithread_load and num_threads, are not silently dropped; they are left for InstantTensor to reject.

Accuracy Tests

CPU unit tests

All 8 InstantTensor-related CPU tests passed: 6 iterator tests and 2 server-argument hardware-validation tests.

These tests cover configuration forwarding (including copy=True), backend conversion, current-device and process-group selection, and rejection of unsupported filename suffixes at the iterator entry. Hardware tests accept the CUDA and ROCm platform classes, reject non-CUDA device types before importing InstantTensor or calling CUDA APIs, and check that other load formats remain usable with CPU devices. These are CPU-only interface tests, not ROCm hardware validation; real NCCL behavior is not covered by the mocks.

Related regression tests, CI registration checks, and applicable pre-commit checks also passed.

GPU generation consistency

The single GPU E2E test passed. It runs weekly, defaults to Qwen/Qwen2-0.5B, and launches two engines: a Safetensors baseline and InstantTensor with backend="MMAP", chunk_size=8388608, and io_depth=2. It clears inherited INSTANTTENSOR_* variables while the engines run. MMAP avoids requiring io_uring support in the CI environment.

The latest local CUDA run used Qwen/Qwen3.5-4B with PyTorch 2.13.0 and InstantTensor 0.1.9, using FP16 inference. An earlier run also passed with deliberately conflicting InstantTensor environment variables set before launch:

Ran 1 test in 46.021s
OK

InstantTensor produced identical generated text to the Safetensors baseline. The maximum differences in prefill and decode log probabilities were both zero. This is an output-consistency test, not a byte-for-byte comparison of the final model parameters or a loading-speed benchmark. Registered GPU CI covers this single-GPU MMAP path, not real TP2 loading or a backend matrix.

Previously validated serving configurations

The original end-to-end serving validation also covered:

• Qwen3-30B-A3B, TP1
• DeepSeek-R1, TP8+EP8

For every configuration, Safetensors and InstantTensor produced the same response and token IDs.

Speed Tests and Profiling

Test on 50GB/s storage

Model	GPU	Backend	Load Time (s)	Throughput (GB/s)	Speedup
Qwen3-30B-A3B	1*H200	Safetensors	57.4	1.07	1x
		fastsafetensors	22.2	2.75	2.59x
		RMS	9.29	6.57	6.18x
		InstantTensor	1.77	34.5	32.4x
DeepSeek-R1	8*H200	Safetensors	160	4.30	1x
		fastsafetensors	75.8	9.08	2.11x
		RMS	101	6.82	1.58x
		RMS (distributed mode)	192	3.59	0.83x
		InstantTensor	15.3	45.0	10.5x

Test on in-memory file system (tmpfs)

Model	GPU	Backend	Load Time (s)	Throughput (GB/s)	Speedup
Qwen3-30B-A3B	1*H200	Safetensors	12.7	4.81	1x
		fastsafetensors	16.7	3.66	0.76x
		RMS	8.21	7.44	1.55x
		InstantTensor	1.26	48.5	10.1x
DeepSeek-R1	8*H200	Safetensors	22.6	30.5	1x
		fastsafetensors	63.4	10.9	0.36x
		RMS	79.8	8.63	0.28x
		RMS (distributed mode)	156	4.41	0.14x
		InstantTensor	12.1	56.9	1.87x

See this document for details.

The loading comparison uses SGLang's Load weight elapsed time. Server ready time is included for completeness but also covers model initialization, kernel compilation, and cache setup, so it is not used to calculate loading throughput.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

---

CI States

Latest PR Test (Base): ⏳ Run #37907615463
Latest PR Test (Extra): ❌ Run #37907615336
Latest PR Test (AMD ROCm 10): ❌ Run #37907615675

[YitaoYuan]

I am one of the lead developers of InstantTensor.
Full benchmark results.

[shuwang21]

Nice work. Since SGLang already ships --load-format fastsafetensors, have you also compared with that in SGLang? benchmark.md does have those rows — could they be included in the PR description as well? The file also has a vllm 0.13.0 comment, so I'm not sure whether these were measured with vLLM or SGLang. Could you clarify? Thanks.

[YitaoYuan]

@shuwang21
vLLM and SGLang share an almost identical loading pipeline. In our tests, the two achieve the same loading speed — it only depends on the choice of backend. We recently reran the benchmark on another platform, where the overall bandwidth was lower but the tail IO latency was also lower than in our earlier tests, which to some extent narrows the gap between the baselines. We used the latest versions of all baselines. Here are the results based on the latest version of SGLang:

Model	Load format	TP	Weight loading time
Qwen3.5-35B-A3B	Safetensors	1	139.97 s
Qwen3.5-35B-A3B	FastSafetensors	1	49.34 s
Qwen3.5-35B-A3B	RunAI Model Streamer	1	7.09 s
Qwen3.5-35B-A3B	InstantTensor	1	6.73 s
Qwen3.5-35B-A3B	Safetensors	2	125.56 s
Qwen3.5-35B-A3B	FastSafetensors	2	32.93 s
Qwen3.5-35B-A3B	RunAI Model Streamer (distributed)	2	6.96 s
Qwen3.5-35B-A3B	InstantTensor	2	6.91 s

Note that although Runai achieves performance comparable to InstantTensor, it uses ~40 GB of memory per process — far more than InstantTensor. Since this was an ad-hoc test, we didn't cover larger scales this time (e.g., 8 GPUs); at larger scales the gap between InstantTensor and the baselines becomes even more pronounced.

By the way, I've updated the PR description directly, and it now includes our previous full comparison. The code will be updated within a day to add the extra parameter passing.

[YitaoYuan]

@shuwang21 Added tuning options via --model-loader-extra-config, with unit tests and documentation. Updated the minimum InstantTensor version to 0.1.9.
All 32 loader-related unit tests passed, and both default and custom configurations matched the Safetensors baseline in generated text and log probabilities.

[shuwang21]

Thanks. LGTM.

[YitaoYuan]

@shuwang21 Addressed both comments: moved the options into the existing “Model loader extra config” table and updated the test dependency to instanttensor>=0.1.9.
Extra options are now forwarded directly to InstantTensor without a fixed allowlist, allowing newer options without SGLang changes. The docs list the core parameters and link to InstantTensor for the latest options. Experimental parameters are omitted for now; we’ll keep the documentation updated.

[alexnails]

general comments, and then also please take a second pass at the tests.

[YitaoYuan]

We’ll further review and validate the test scripts, and update them accordingly.

[YitaoYuan]

/rerun-failed-ci

[YitaoYuan]

/rerun-failed-ci

[Fridge003]

Extra tests passed https://github.com/sgl-project/sglang/actions/runs/37670994030

[YitaoYuan]

Appreciated!

[whybeyoung]

LGTM!!!