[Feature] Add InstantTensor weight loader

[arlo-scitix]

PR for this RFC.

Purpose

Speed up model loading and fully utilize the bandwidth of high-speed storage (e.g., 400 Gbps networked storage).

Test Plan

Load any model with any parallelism setting, on H20 141G for example:

vllm serve Qwen/Qwen3-30B-A3B --load-format instanttensor

vllm serve deepseek-ai/DeepSeek-R1 --load-format instanttensor --tensor-parallel-size 8 --enable-expert-parallel

Test Result

See README of our repo.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[mergify]

Documentation preview: https://vllm--36139.org.readthedocs.build/en/36139/

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

[gemini-code-assist]

Code Review

This pull request introduces support for InstantTensor to accelerate model weight loading. The changes span dependency management, configuration, model loading logic, and documentation. My review focuses on ensuring the CUDA-specific nature of this new feature is made clear to users. I've provided suggestions to add an explicit check for CUDA availability in the code and to update the associated docstrings and documentation to mention this requirement. These changes will help prevent runtime errors and improve the user experience for those on non-CUDA platforms.

[robertgshaw2-redhat]

Is it possible to make this the default? What considerations should we have for its usage?

[arlo-scitix]

@robertgshaw2-redhat, thanks. We’ve responded in the RFC.

[mgoin]

LGTM for the initial integration, thank you!

[mergify]

Hi @arlo-aisys, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[arlo-scitix]

@mgoin
We’ve fixed all issues that were causing the CI tests to fail. The remaining test failures don’t appear to be caused by InstantTensor. Who should we contact next among the maintainers to help move this PR forward?

[bbartels]

@mgoin
We’ve fixed all issues that were causing the CI tests to fail. The remaining test failures don’t appear to be caused by InstantTensor. Who should we contact next among the maintainers to help move this PR forward?

You can post in the PR reviews channel on the vllm slack

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @arlo-aisys.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mgoin]

LGTM, thanks a lot!