[Core] Update PyTorch to 2.14.0, torchvision to 0.29.0, triton to 3.8.0 (test channel) by atalman · Pull Request #52183 · vllm-project/vllm

atalman · 2026-08-13T14:56:49Z

Purpose

Update the PyTorch ecosystem to the 2.14.0 release candidate, resolving wheels from the PyTorch test channel (download.pytorch.org/whl/test/...):

torch: → 2.14.0
torchvision: → 0.29.0
triton: → 3.8.0

2.14.0 is published on download.pytorch.org/whl/test/ but not yet on the release index / PyPI, so the index URLs point at the test channel. This mirrors the 2.13.0 test PR #45731.

Verified availability (test channel) before pinning

torch-2.14.0+cu130: present
torchvision-0.29.0: present
triton==3.8.0: present, and is exactly what pytorch release/2.14 pins in .ci/docker/triton_version.txt
torchaudio: newest published is still 2.11.0 on every channel, so that pin is unchanged

ROCm

requirements/build/rocm.txt moves from the rocm7.1 index to test/rocm7.2. rocm7.1 has no torch 2.14.0 wheels — 2.14 is built for rocm7.2 and rocm7.14 only. That file was also still on torch 2.11.0 / triton 3.6.0, so this brings it in line with CMakeLists.txt's TORCH_SUPPORTED_VERSION_ROCM.

Ray dependency-compatibility check

.buildkite/scripts/check-ray-compatibility.sh now offers uv both the stable and test channels (it already runs with --index-strategy unsafe-best-match), so an RC torch pin coming from a transitive dep resolves during the RC window instead of failing against a stable-only index. Same fix as #45731.

Deliberately not carried over from #45731

That PR also carried fixes reacting to breakages found during 2.12/2.13 CI. Those are not copied here, since 2.14 should get whatever it actually needs once CI reports:

xfails for pytorch#184431, pytorch#187735, pytorch#187727
the ROCm source-build override + install_flash_attn_rocm.sh (needed because ROCm/pytorch had no release/2.13 branch)
verify_torch_wheel.py / UV_NO_VERIFY_HASHES RC-respin handling

requirements/build/tpu.txt is also left alone — TPU torch is bumped separately (#50412).

Test Plan

Full CI (apply the ready label to trigger the Buildkite run).

BC-breaking?

No — test-channel version bump only.

Authored with AI assistance; modeled on #45731.

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Harry-Chen · 2026-08-13T15:00:13Z

Could you please include a revert of #51185 in this PR? cc @janeyx99

atalman · 2026-08-13T17:57:44Z

Pushed a follow-up commit aligning the NVIDIA pins with torch 2.14.0.

torch pins these with == in its wheel metadata (PYTORCH_EXTRA_INSTALL_REQUIREMENTS in pytorch release/2.14), so the values requirements/test/cuda.txt was compiled against for 2.13 made the lockfile unresolvable alongside the new torch — not a silent drift, an actual resolution failure on the CUDA test path.

package	was	now	torch 2.14 (cu130)
`nvidia-cudnn-cu13`	9.20.0.48	9.24.0.43	`==9.24.0.43`
`nvidia-nccl-cu13`	2.29.7	2.30.7	`==2.30.7`

Unchanged because they already satisfy 2.14: cuda-toolkit (13.0.3.0 normalises to ==13.0.3), cuda-bindings (13.0.3 within >=13.0.3,<14), nvidia-cusparselt-cu13 (0.8.1), nvidia-nvshmem-cu13 (3.4.5).

Both new versions are published on the test channel (test/cu130) and on PyPI.

Checked the rest of the tree for other NVIDIA pins that might need the same treatment — there are none. requirements/cuda.txt only has nvidia-cudnn-frontend>=1.19.1 and nvidia-cutlass-dsl[cu13]==4.6.2, neither torch-derived; no nccl/cudnn/cusparselt/nvshmem pins in any Dockerfile, pyproject.toml, .in file or buildkite config; and requirements/test/{cpu,xpu}.txt carry no nvidia-* pins.

Caveat for reviewers: this was hand-edited, not recompiled — I don't have a working uv pip compile path against the test index. Worth regenerating the lockfile properly to pick up any transitive changes these two bumps imply.

mergify · 2026-08-17T04:18:27Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @atalman.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

**Summary:** torchao's v1 configs were deprecated as of v0.17.0 and officially removed in v0.18.0. These tests are testing functionality that no longer exist upstream and are causing breakages in vllm-project#52183. Fixes vllm-project#52947. **Tese Plan:** ``` pytest -v -s tests/quantization/test_torchao.py ```

**Summary:** torchao's v1 configs were deprecated as of v0.17.0 and officially removed in v0.18.0. These tests are testing functionality that no longer exist upstream and are causing breakages in vllm-project#52183. Fixes vllm-project#52947. **Tese Plan:** ``` pytest -v -s tests/quantization/test_torchao.py ``` Signed-off-by: andrewor14 <andrewor14@gmail.com>

mergify · 2026-08-31T20:25:36Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @atalman.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

….0 (test channel) Update the PyTorch ecosystem to the 2.14.0 release candidate, resolving wheels from the PyTorch test channel (download.pytorch.org/whl/test/...). Mirrors the 2.13.0 test PR vllm-project#45731. - torch: 2.13.0 -> 2.14.0 - torchvision: 0.28.0 -> 0.29.0 - triton: 3.7.1 -> 3.8.0 2.14.0 is published on download.pytorch.org/whl/test/ but not yet on the release index or PyPI, so the index URLs point at the test channel. Verified availability on the test channel before pinning: - torch 2.14.0+cu130, torchvision 0.29.0, triton 3.8.0 all present - triton 3.8.0 is exactly what pytorch release/2.14 pins in .ci/docker/triton_version.txt - torchaudio's newest published version is still 2.11.0 on every channel, so that pin is unchanged ROCm: requirements/build/rocm.txt moves from the rocm7.1 index to test/rocm7.2. rocm7.1 has no torch 2.14.0 wheels; 2.14 is built for rocm7.2 and rocm7.14 only. That file was also still on torch 2.11.0 / triton 3.6.0, so this brings it in line with CMakeLists' TORCH_SUPPORTED_VERSION_ROCM. check-ray-compatibility.sh now offers uv both the stable and test channels (it already runs with --index-strategy unsafe-best-match), so an RC torch pin coming from a transitive dep resolves during the RC window instead of failing against a stable-only index. Same fix as vllm-project#45731. Deliberately not carried over from vllm-project#45731: the xfails and workarounds that were specific to breakages found during 2.12/2.13 CI (pytorch#184431, #187735, reactions to that RC's failures; 2.14 should get whatever it actually needs once CI reports. requirements/build/tpu.txt is also left alone, as TPU torch is bumped separately (vllm-project#50412). Test Plan: full CI (apply the `ready` label to trigger Buildkite). BC-breaking? No - test-channel version bump only. Authored with AI assistance; modeled on vllm-project#45731.

torch 2.14.0 pins these with '==' in its wheel metadata (PYTORCH_EXTRA_INSTALL_REQUIREMENTS in pytorch release/2.14), so the values compiled against 2.13 make requirements/test/cuda.txt unresolvable alongside the new torch: nvidia-cudnn-cu13 9.20.0.48 -> 9.24.0.43 nvidia-nccl-cu13 2.29.7 -> 2.30.7 cuda-toolkit, cuda-bindings, nvidia-cusparselt-cu13 and nvidia-nvshmem-cu13 already satisfy 2.14's requirements and are unchanged. Both new versions are published on the test channel (test/cu130) and on PyPI. Hand-edited rather than recompiled: I could not run 'uv pip compile' here, so a reviewer with a working resolver should regenerate the lockfile to pick up any transitive changes these two bumps imply.

…backend Every CPU job in the 2.14.0 build fails in the image build: × No solution found when resolving dependencies: ╰─▶ Because there is no version of torch{...}==2.14.0+cpu and you require torch{...}==2.14.0+cpu, we can conclude that your requirements are unsatisfiable. uv's --torch-backend only knows the stable channels, so it pins the index to download.pytorch.org/whl/cpu and ignores the --extra-index-url that points at the test channel. torch 2.14.0+cpu only exists under whl/test/cpu, so nothing resolves. Replace --torch-backend cpu with an explicit --extra-index-url built from a PYTORCH_CPU_INDEX_BASE_URL build-arg, mirroring how Dockerfile handles PYTORCH_CUDA_INDEX_BASE_URL. Switching channels is now a one-line change on both the CPU and CUDA sides. UV_INDEX_STRATEGY=unsafe-best-match is already set in base-common and is inherited by every stage touched here. torch 2.14.0+cpu, torchaudio 2.11.0+cpu, torchvision 0.29.0+cpu, torchcodec 0.14.0+cpu and triton 3.8.0 are all present on whl/test/cpu, so the pinned set in requirements/test/cpu.txt resolves from this one index.

…slowdown torch 2.14.0 makes the CPU model-execution shards run 2-3x slower than 2.13, so `CPU-Language Generation and Pooling` and `CPU-Multi-Modal Model Tests` 1 and 2 are killed by the `timeout` in run-cpu-test.sh (`exit 124`) partway through and produce no signal at all. Measured wall clock on the same `queue=intel-cpu` runners, with an identical test selection (`195 items / 59 deselected / 136 selected` for Multi-Modal 1): | shard | 2.13 | 2.14 | | Language Generation + Pooling | ~30m | killed | | Multi-Modal 1 | 17-21m | killed | | Multi-Modal 2 | ~20m | killed | Raise both steps to 90m, roughly 4x the 2.13 baseline, so the shards run to completion. That tells us the real 2.14 duration and whether they pass at all once they are allowed to finish. This is deliberately a measurement change, not a fix -- it hides a real regression and must be reverted before merge. Tracked upstream at pytorch/pytorch#193951.

With the 90m limit, two of the three affected shards now complete and give us a real measurement of the torch 2.14 CPU slowdown. Comparing pytest-phase durations (not job wall clock, which includes the image build) against the pinned-torch baselines 84265 and 84342, on an identical test selection: | shard | 2.13 | 2.14 | factor | | Multi-Modal 2 | 711s / 781s | 4924s | ~6.6x | | Multi-Modal 1 | 996s / 1286s | >5400s | >4.2x | | Language Gen+Pool | 1473s / 1593s | 3969s | ~2.6x | `CPU-Multi-Modal Model Tests 1` is the heaviest shard and still hit the 90m ceiling, so raise the Multi-Modal step to 120m. The step is templated with `parallelism: 4`, so this applies to all four shards; 2, 3 and 4 finish well under it. Language Generation stays at 90m -- it completes in ~66m. Still a measurement change, not a fix. Revert before merge. Tracked at pytorch/pytorch#193951.

torchao ships per-torch-version builds, and 0.17.0 was built against 2.13. The previous commit in this stack only moved the CUDA pin to the test channel so it would resolve at all; the version itself stayed on 0.17.0. 0.18.0 is the torch 2.14 pairing and is available on both channels this PR needs: - download.pytorch.org/whl/test/cu130 (CUDA quantization lane) - PyPI (the two ROCm quantization lanes, which install torchao without an --index-url; a cu130 index would be wrong there) Updates all three pin sites, including the two in test-amd.yaml that the earlier index-url change missed.

RELEASE-ONLY. The 2.14.0 RC is republished to the PyTorch test channel under an unchanging version string. torch is installed in the `base-common` stage of `docker/Dockerfile.cpu`, which is upstream of the vLLM source COPY, so its layer cache key depends only on the base image and `requirements/cpu.txt` -- neither of which changes when an RC is respun or when the branch is rebased. The ECR registry cache (`--cache-from type=registry`, keyed on the PR/branch rather than the commit) therefore restores the torch layer on every build. In build 84705 the CPU image build reported 28 CACHED layers and never resolved or downloaded torch at all, while the CUDA build in the same run downloaded torch fresh. Add `--no-cache` so the CPU image always installs torch and triton from the test channel. Revert once 2.14.0 is final and published to PyPI.

RELEASE-ONLY. The CPU test shards do not pull the image published by image_build_cpu.sh -- run-cpu-test.sh builds its own image with a plain `docker build`, using the agent's local BuildKit cache. So the `--no-cache` added to image_build_cpu.sh does not cover them. torch is installed in the `base-common` stage of docker/Dockerfile.cpu, which is upstream of the vLLM source COPY, so its layer key depends only on the base image and `requirements/cpu.txt`. Neither changes when the 2.14.0 RC is respun under the same version string, so the cached layer is restored every time. In build 84719 the `CPU-Multi-Modal Model Tests 1` shard reported steps vllm-project#16, vllm-project#6, wheel and could not have exercised the ideep/oneDNN revert. Add `--no-cache` to the shard build, and set `UV_NO_CACHE=1` in the Dockerfile base stage: `--no-cache` re-executes the layer but does not clear the `RUN --mount=type=cache,target=/root/.cache/uv` mount, so uv would still serve the stale same-named wheel from its own cache. Revert both once 2.14.0 is final and published to PyPI.

**Summary:** torchao's v1 configs were deprecated as of v0.17.0 and officially removed in v0.18.0. These tests are testing functionality that no longer exist upstream and are causing breakages in vllm-project#52183. Fixes vllm-project#52947. **Tese Plan:** ``` pytest -v -s tests/quantization/test_torchao.py ``` Signed-off-by: andrewor14 <andrewor14@gmail.com>

mergify · 2026-09-17T10:14:37Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @atalman.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

atalman requested review from Harry-Chen, LucasWilkinson, khluu and tlrmchlsmth as code owners August 13, 2026 14:56

claude Bot reviewed Aug 13, 2026

View reviewed changes

mergify Bot added ci/build nvidia labels Aug 13, 2026

github-project-automation Bot added this to NVIDIA Aug 13, 2026

mergify Bot added the cpu Related to CPU backends label Aug 13, 2026

atalman requested review from bigPYJ1151 and xuechendi as code owners August 15, 2026 12:16

mergify Bot added the needs-rebase label Aug 17, 2026

atalman force-pushed the test-pytorch-2.14.0-test branch from 7496ef0 to d934d15 Compare August 17, 2026 23:31

mergify Bot removed the needs-rebase label Aug 17, 2026

atalman mentioned this pull request Aug 18, 2026

[vllm] [2.14 regression][CPU] CPU model-execution shards up to ~6.6x slower than 2.13, and one shard hangs after engine shutdown pytorch/pytorch#193951

Closed

atalman force-pushed the test-pytorch-2.14.0-test branch from 77bd157 to 0e797b5 Compare August 19, 2026 00:06

686f6c61 mentioned this pull request Aug 19, 2026

[BugFix][Kernel] Pass kv_cache_dtype as int to avoid stable-ABI host leak #50870

Closed

oops-oom mentioned this pull request Aug 19, 2026

[Core] Update PyTorch to 2.14.0 (torchvision 0.29.0, triton 3.8.0) #52918

Closed

4 tasks

andrewor14 mentioned this pull request Aug 19, 2026

[Tests] Delete deprecate torchao tests for v1 configs #52956

Merged

3 tasks

atalman force-pushed the test-pytorch-2.14.0-test branch 6 times, most recently from 731afae to 1c91ac7 Compare August 26, 2026 14:54

atalman force-pushed the test-pytorch-2.14.0-test branch 2 times, most recently from 47d9f16 to ab79934 Compare August 27, 2026 18:48

mergify Bot added the needs-rebase label Aug 31, 2026

atalman and others added 8 commits September 15, 2026 11:32

atalman force-pushed the test-pytorch-2.14.0-test branch from ab79934 to 002c42b Compare September 15, 2026 18:36

mergify Bot removed the needs-rebase label Sep 16, 2026

mergify Bot added the needs-rebase label Sep 17, 2026

Navigation Menu

Uh oh!

[Core] Update PyTorch to 2.14.0, torchvision to 0.29.0, triton to 3.8.0 (test channel) - #52183

[Core] Update PyTorch to 2.14.0, torchvision to 0.29.0, triton to 3.8.0 (test channel)#52183
atalman wants to merge 8 commits into
vllm-project:mainfrom
atalman:test-pytorch-2.14.0-test

atalman commented Aug 13, 2026

Uh oh!

claude Bot left a comment

Uh oh!

Harry-Chen commented Aug 13, 2026

Uh oh!

atalman commented Aug 13, 2026

Uh oh!

mergify Bot commented Aug 17, 2026

Uh oh!

mergify Bot commented Aug 31, 2026

Uh oh!

mergify Bot commented Sep 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

atalman commented Aug 13, 2026

Purpose

Verified availability (test channel) before pinning

ROCm

Ray dependency-compatibility check

Deliberately not carried over from #45731

Test Plan

BC-breaking?

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

Harry-Chen commented Aug 13, 2026

Uh oh!

atalman commented Aug 13, 2026

Uh oh!

mergify Bot commented Aug 17, 2026

Uh oh!

mergify Bot commented Aug 31, 2026

Uh oh!

mergify Bot commented Sep 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants