[Release 2.10] Update to Torch 2.10 - final release by atalman · Pull Request #30525 · vllm-project/vllm

atalman · 2025-12-11T23:52:48Z

Note

^{Cursor Bugbot is generating a summary for commit 37af14c52963e6d528866dded7f51985a1fcef7e. Configure here.}

Note

Upgrade to PyTorch 2.10.0 across the project

Bumps torch/torchaudio/torchvision to 2.10.0 (and torchvision 0.25.0), updates CMake supported torch versions to 2.10.0, and refreshes related deps (triton 3.6.0, nvidia-nvshmem-cu12 3.4.5).
Docker: switch PyTorch index to .../whl/test, enable --prerelease=allow for installs, and plumb test indexes in CUDA and CPU images; add extra-index usage to the python-only compile test.
CI/tooling: pre-commit pip-compile now uses test cu129 index; Prime-RL script force-reinstalls torch/vision from test cu129.
Tests/compile paths: replace version gates from 2.10.0.dev to 2.10.0 and update decorators/env checks accordingly.
Requirements files (CUDA/ROCm/build/test) updated to the new versions and indexes for cu129 and rocm7.

^{Written by Cursor Bugbot for commit 37af14c52963e6d528866dded7f51985a1fcef7e. This will update automatically on new commits. Configure here.}

FIX #29595
FIX #33888

gemini-code-assist

Code Review

This pull request updates various dependencies to test the upcoming Torch 2.10 release candidate, primarily by bumping the version of torch and related packages like torchaudio, torchvision, and torchao across multiple configuration files. While the version updates are consistent, I've found a critical issue with how version-specific workarounds are handled. The logic for applying monkey-patches for PyTorch 2.9 bugs has been changed to apply to all future versions (>=2.9.0), which poses a significant forward-compatibility risk. My review provides suggestions to scope these patches to a more limited version range to prevent them from causing issues in future PyTorch releases where the original bugs may be fixed.

gemini-code-assist · 2025-12-11T23:54:20Z

vllm/env_override.py

@@ -363,7 +363,7 @@ def _update_scheduler_patched(self) -> None:
        self.scheduler = Scheduler(self.operations)


-if is_torch_equal("2.9.0"):
+if is_torch_equal_or_newer("2.9.0"):


The condition is_torch_equal_or_newer("2.9.0") is too broad. These monkey-patches are workarounds for specific bugs in PyTorch 2.9.0. While they might be necessary for 2.10.0, applying them to all future versions is risky and can lead to conflicts when the bugs are fixed upstream. It's better to scope this to the versions where the patch is known to be needed. A safer approach would be to specify an upper bound, for example, to include versions 2.9.x and 2.10.x but not 2.11.x and newer.

Suggested change

if is_torch_equal_or_newer("2.9.0"):

if is_torch_equal_or_newer("2.9.0") and not is_torch_equal_or_newer("2.11.0"):

gemini-code-assist · 2025-12-11T23:54:20Z

vllm/model_executor/layers/conv.py

@@ -251,6 +251,6 @@ def forward_cuda(self, x: torch.Tensor) -> torch.Tensor:
        # See: https://github.com/vllm-project/vllm/issues/27406
        # and https://github.com/pytorch/pytorch/issues/166122
        # By default, we use CUDNN's convolution ops with optimization.
-        if self.enable_linear and is_torch_equal("2.9.0"):
+        if self.enable_linear and is_torch_equal_or_newer("2.9.0"):


The condition is_torch_equal_or_newer("2.9.0") is too broad. This workaround is for a performance regression in PyTorch 2.9.0. Applying it to all future versions is risky, as the fix might be included in a future release, and this workaround could become detrimental or incorrect. It's better to scope this to the versions where the workaround is known to be needed. A safer approach would be to specify an upper bound, for example, to include versions 2.9.x and 2.10.x but not 2.11.x and newer.

Suggested change

if self.enable_linear and is_torch_equal_or_newer("2.9.0"):

if self.enable_linear and is_torch_equal_or_newer("2.9.0") and not is_torch_equal_or_newer("2.11.0"):

mergify · 2025-12-11T23:56:59Z

Hi @atalman, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

mergify · 2025-12-12T00:38:19Z

Hi @atalman, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

mergify · 2025-12-12T15:36:06Z

Hi @atalman, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

mergify · 2025-12-12T21:30:37Z

Hi @atalman, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

mergify · 2025-12-13T14:56:46Z

Hi @atalman, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

mergify · 2025-12-15T14:34:41Z

Hi @atalman, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

mergify · 2025-12-15T15:38:31Z

Hi @atalman, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

mergify · 2025-12-16T00:30:05Z

Hi @atalman, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

mergify · 2025-12-16T03:36:29Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @atalman.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

mergify · 2026-02-06T14:45:27Z

Hi @atalman, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

mgoin

@atalman this looks close to green, maybe all the way in CI. Is this close to merge?

atalman · 2026-02-08T16:22:45Z

@mgoin I believe it is. However I don't have the ability to merge in OSS, if someone can assist in merging this would be nice cc @simon-mo

mgoin · 2026-02-08T21:50:49Z

Let's go for it, I can't find any clearly failing tests from the logs

DarkLight1337 · 2026-02-09T08:58:12Z

After upgrading to torch==2.10.0 / triton==3.6.0, I'm getting this log when starting up vLLM:

ERROR 02-09 08:55:51 [gpt_oss_triton_kernels_moe.py:46] Failed to import Triton kernels. Please make sure your triton version is compatible. Error: cannot import name 'SparseMatrix' from 'triton_kernels.tensor' (/home/cyrus/miniconda3/envs/vllm/lib/python3.10/site-packages/triton_kernels/tensor.py)

It's not blocking for me (I don't use GPT-OSS) but this is worth looking into and fixing.

johnnynunez · 2026-02-09T09:32:02Z

After upgrading to torch==2.10.0 / triton==3.6.0, I'm getting this log when starting up:
ERROR 02-09 08:55:51 [gpt_oss_triton_kernels_moe.py:46] Failed to import Triton kernels. Please make sure your triton version is compatible. Error: cannot import name 'SparseMatrix' from 'triton_kernels.tensor' (/home/cyrus/miniconda3/envs/vllm/lib/python3.10/site-packages/triton_kernels/tensor.py)
It's not blocking for me (I don't use GPT-OSS) but this is worth looking into and fixing.

Because on that version it was moved i think so, that error sounds to me familiar when dgx spark was launched. Anyways triton 3.6.0 has important fixes for dgx spark
Could anyone investigate that? @mgoin @simon-mo

robertgshaw2-redhat · 2026-02-09T15:05:58Z

After upgrading to torch==2.10.0 / triton==3.6.0, I'm getting this log when starting up:
ERROR 02-09 08:55:51 [gpt_oss_triton_kernels_moe.py:46] Failed to import Triton kernels. Please make sure your triton version is compatible. Error: cannot import name 'SparseMatrix' from 'triton_kernels.tensor' (/home/cyrus/miniconda3/envs/vllm/lib/python3.10/site-packages/triton_kernels/tensor.py)
It's not blocking for me (I don't use GPT-OSS) but this is worth looking into and fixing.
Because on that version it was moved i think so, that error sounds to me familiar when dgx spark was launched. Anyways triton 3.6.0 has important fixes for dgx spark Could anyone investigate that? @mgoin @simon-mo

This is a problem for Hopper, since these are the best kernel for gpt-oss

mgoin · 2026-02-09T15:34:25Z

@DarkLight1337 I just tried running myself on H200 with a fresh environment and it worked fine using the triton backend. I think your install might have had some old state if you upgraded in place, such as not rebuilding triton_kernels in our cmake

eugr · 2026-02-09T16:53:28Z

Yes, works well in my builds too with Triton 3.6.0 and triton-kernels built from the release branch. No errors.
It does not work with Triton built from main branch though because triton-kernels in main are missing matmul_ogs module.

DarkLight1337 · 2026-02-09T16:57:50Z

@DarkLight1337 I just tried running myself on H200 with a fresh environment and it worked fine using the triton backend. I think your install might have had some old state if you upgraded in place, such as not rebuilding triton_kernels in our cmake

Ok let me try rebuilding from scratch

varun-sundar-rabindranath · 2026-02-09T17:12:53Z

I verified running gpt-oss-20b on B200 and H100 as well. Didn't run into any issues - good eval scores.

mikaylagawarecki · 2026-02-10T01:10:50Z

@atalman Was the rocm version used in the AMD docker: build image job bumped by this PR? I still seem to see 2.9.1 in the build on my PR

Pytorch version >= 2.10.0 expected for ROCm build, saw 2.9.1 instead.

https://buildkite.com/vllm/ci/builds/50790/steps/canvas?jid=019c4507-3016-40e5-b3e1-8d50e3fb9ee7&tab=output#019c4507-3016-40e5-b3e1-8d50e3fb9ee7/L5762

atalman requested review from LucasWilkinson, tjtanaa and tlrmchlsmth as code owners December 11, 2025 23:52

mergify bot added ci/build nvidia rocm Related to AMD ROCm labels Dec 11, 2025

github-project-automation bot added this to NVIDIA Dec 11, 2025

gemini-code-assist bot reviewed Dec 11, 2025

View reviewed changes

atalman requested a review from bigPYJ1151 as a code owner December 12, 2025 00:34

atalman requested a review from hmellor as a code owner December 13, 2025 14:49

atalman force-pushed the update_to_210 branch from 1b1e207 to 6a98b4f Compare December 13, 2025 14:51

atalman force-pushed the update_to_210 branch from 6a98b4f to 5e8a504 Compare December 15, 2025 14:29

atalman requested review from ProExpertProg, yewentao256, youkaichao and zou3519 as code owners December 15, 2025 15:25

atalman mentioned this pull request Dec 16, 2025

[Release 2.10] umbrella issue - vLLM CI failures pytorch/pytorch#170433

Closed

18 tasks

atalman force-pushed the update_to_210 branch from e873a83 to 22dff7b Compare December 16, 2025 00:24

mergify bot added the needs-rebase label Dec 16, 2025

atalman mentioned this pull request Dec 16, 2025

[Release 2.10] Test Torch 2.10 RC - with skipped test #30790

Closed

atalman force-pushed the update_to_210 branch from 22dff7b to 44fe379 Compare December 16, 2025 15:24

atalman added 8 commits February 6, 2026 06:41

release_210_larger_timeout

d08182e

release_210_testing

9664b2f

release_210_bump_triton_kernels

f38c1b5

triton_kernels

a52278c

fix

c6ee36b

fix

b887fe4

lint

d413837

fix

c41f59c

fix

099f19a

mgoin approved these changes Feb 6, 2026

View reviewed changes

gshtras mentioned this pull request Feb 9, 2026

[Bugfix][ROCm][GPT-OSS] Use old triton_kernels implementation on ROCm if the new API is not available #34153

Merged

This was referenced Feb 9, 2026

[ROCm][Bugfix] Resolve Dynamo tracing crash from amdsmi calls in on_gfx* arch detection #34108

Merged

[CI Failure]: mi325_1: Entrypoints Unit Tests #34160

Closed

[CI Failure]: mi325_1: Entrypoints Integration Test (API Server 1) #29541

Closed

bnellnm mentioned this pull request Feb 9, 2026

[CI Failure]: tests/integration/test_rl.py: RuntimeError: operator torchvision::nms does not exist #34166

Open

3 tasks

mikaylagawarecki mentioned this pull request Feb 24, 2026

[Bug]: AMD docker image still using torch 2.9 despite 2.10.0 in requirements/rocm-build.txt #35163

Open

1 task

chamwen mentioned this pull request Mar 3, 2026

[Bug]: branch v0.16.0 still rely on torch 2.9.1, not 2.10 #35823

Closed

1 task

danigarciaoca mentioned this pull request Mar 3, 2026

[Bug]: Qwen3-VL-235B-A22B-Instruct Grounding Accuracy Issue in vLLM (>= v0.11.1) #29595

Closed

1 task

qiching mentioned this pull request Mar 20, 2026

[Bugfix] Add early detection for CUDA < 13.0 on sm_103+ GPUs (GB300) #37630

Open

	if is_torch_equal_or_newer("2.9.0"):
	if is_torch_equal_or_newer("2.9.0") and not is_torch_equal_or_newer("2.11.0"):

	if self.enable_linear and is_torch_equal_or_newer("2.9.0"):
	if self.enable_linear and is_torch_equal_or_newer("2.9.0") and not is_torch_equal_or_newer("2.11.0"):

Uh oh!

Conversation

atalman commented Dec 11, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

mergify bot commented Dec 11, 2025

Uh oh!

mergify bot commented Dec 12, 2025

Uh oh!

mergify bot commented Dec 12, 2025

Uh oh!

mergify bot commented Dec 12, 2025

Uh oh!

mergify bot commented Dec 13, 2025

Uh oh!

mergify bot commented Dec 15, 2025

Uh oh!

mergify bot commented Dec 15, 2025

Uh oh!

mergify bot commented Dec 16, 2025

Uh oh!

mergify bot commented Dec 16, 2025

Uh oh!

mergify bot commented Feb 6, 2026

Uh oh!

mgoin left a comment

Choose a reason for hiding this comment

Uh oh!

atalman commented Feb 8, 2026

Uh oh!

mgoin commented Feb 8, 2026

Uh oh!

DarkLight1337 commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

johnnynunez commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

robertgshaw2-redhat commented Feb 9, 2026

Uh oh!

mgoin commented Feb 9, 2026

Uh oh!

eugr commented Feb 9, 2026

Uh oh!

DarkLight1337 commented Feb 9, 2026

Uh oh!

varun-sundar-rabindranath commented Feb 9, 2026

Uh oh!

mikaylagawarecki commented Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

15 participants

atalman commented Dec 11, 2025 •

edited by github-actions bot

Loading

DarkLight1337 commented Feb 9, 2026 •

edited

Loading

johnnynunez commented Feb 9, 2026 •

edited

Loading

mikaylagawarecki commented Feb 10, 2026 •

edited

Loading