Skip to content

[Core] Update PyTorch to 2.13.0, torchvision to 0.28.0, triton to 3.7.1 (test channel) - #45731

Closed
atalman wants to merge 14 commits into
vllm-project:mainfrom
atalman:test-pytorch-2.13.0-test
Closed

[Core] Update PyTorch to 2.13.0, torchvision to 0.28.0, triton to 3.7.1 (test channel)#45731
atalman wants to merge 14 commits into
vllm-project:mainfrom
atalman:test-pytorch-2.13.0-test

Conversation

@atalman

@atalman atalman commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

Purpose

Update the PyTorch ecosystem to the 2.13.0 release candidate, resolving wheels from the PyTorch test channel (download.pytorch.org/whl/test/...):

  • torch: → 2.13.0
  • torchvision: → 0.28.0
  • triton: → 3.7.1 (matches torch 2.13.0's Requires-Dist: triton==3.7.1)

2.13.0 is currently published on download.pytorch.org/whl/test/ but not yet on the release index / PyPI, so the index URLs point at the test channel. This mirrors the 2.12.1 test PR #45082.

Verified availability (test channel, cu130)

  • torch-2.13.0+cu130: present
  • torchvision-0.28.0+cu130: present
  • triton==3.7.1: present and is exactly what torch 2.13.0's wheel metadata requires

Ray dependency-compatibility check

This PR includes the fix from #45082: .buildkite/scripts/check-ray-compatibility.sh offers both the stable and test channels to uv (which runs with --index-strategy unsafe-best-match), so the RC torch pin (e.g. from nixl-cuNN) resolves from the test channel during the RC window instead of failing against a stable-only index.

Test Plan

Full CI (apply the ready label to trigger the Buildkite run).

BC-breaking?

No — test-channel version bump only.

Authored with AI assistance; modeled on #45082.

@mergify mergify Bot added the cpu Related to CPU backends label Jun 15, 2026
This was referenced Jun 16, 2026
Closed
Closed
@mergify

mergify Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @atalman.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Jun 18, 2026
This was referenced Jun 19, 2026
Closed
Closed
Closed
Closed
Closed
@mergify mergify Bot added v1 kv-connector multi-modality Related to multi-modality (#4194) and removed needs-rebase labels Jun 19, 2026
@atalman atalman mentioned this pull request Jun 23, 2026
Closed
3 tasks
@mergify

mergify Bot commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @atalman.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Jun 23, 2026
@atalman atalman mentioned this pull request Jun 23, 2026
Closed
@atalman
atalman force-pushed the test-pytorch-2.13.0-test branch from d9e72ee to 53e79d7 Compare June 23, 2026 22:33
@mergify mergify Bot removed the needs-rebase label Jun 23, 2026
@atalman atalman mentioned this pull request Jun 25, 2026
Closed
@atalman
atalman force-pushed the test-pytorch-2.13.0-test branch from e1c6ee2 to 241e630 Compare June 25, 2026 19:58
@mergify mergify Bot added the rocm Related to AMD ROCm label Jun 26, 2026
atalman added 3 commits July 4, 2026 06:18
@atalman
[CI][CPU] Copy common.txt into CPU test-deps stage so cpu.in include …
7ac7226 
…resolves

The CPU image build compiles cpu.txt from cuda.in (seeded as cpu.in) at build
time because torch 2.13.0 is test-channel-only and can't use the pre-committed
cpu.txt path. But cuda.in begins with `-r ../common.txt`, and the vllm-test-deps
stage only copied cuda.in -> cpu.in, so `uv pip compile` failed:

    error: Error parsing included file in `requirements/test/cpu.in`
      Caused by: failed to read `requirements/test/../common.txt`:
                 No such file or directory (os error 2)

breaking both the x86 and arm64 CPU image builds and every downstream CPU test.
COPY requirements/common.txt into the stage (it has no further -r includes) so
the include resolves. Unlike the 2.12.1 branch (production channel), 2.13 must
keep the build-time compile, so this is the minimal correct fix.

Test Plan: reproduced the failing layout locally and confirmed the `-r ../common.txt`
include resolves after the copy; full validation is the CPU image build in CI.
@atalman
[CI][ROCm] Rebuild flash-attn against installed torch for ABI compati…
09d26ea 
…bility

The rocm/vllm-dev base image ships a flash-attn wheel compiled against its
pinned PyTorch (ROCm/pytorch release/2.11, per Dockerfile.rocm_base). This image
then reinstalls the torch 2.13 test-channel ROCm wheels on top, so the prebuilt
flash_attn_2_cuda.so no longer matches the torch C++ ABI and fails to import:

    ImportError: .../flash_attn_2_cuda...so: undefined symbol:
      _ZN3c104impl3cow23materialize_cow_storageERNS_11StorageImplE
    (c10::impl::cow::materialize_cow_storage, changed by pytorch #179063)

The import happens on model load via the Llama RoPE apply_rotary_emb_flash_attn
path, so EngineCore fails to start and every AMD GPU test that loads a model
fails (Basic Correctness, sleep-mode mem tests, etc.). This is the same ABI
break already tracked for nixl_ep (pytorch/pytorch#187727), on a different
prebuilt extension.

Fix: add tools/install_flash_attn_rocm.sh (mirrors install_torchcodec_rocm.sh
and the base image's FA build recipe) and run it in the ci_base stage right
after the torchcodec rebuild, recompiling Dao-AILab/flash-attention@0e60e394
(the base image's FA pin) against the installed torch. The script is idempotent
(skips when flash-attn already imports) and caches the built wheel.

Test Plan:
- `bash -n tools/install_flash_attn_rocm.sh` (syntax).
- Full validation is the AMD CPU/GPU image build + AMD GPU tests in CI on the
  2.13 branch; the flash_attn_2_cuda import error should no longer occur.
@atalman
[CI][CPU] Cap qwen2_vl max_tokens on CPU for torch 2.13 logprob drift
06e2ffe 
On torch 2.13 the qwen2_vl multi-image generation accumulates CPU numerical
drift: HF and vLLM agree for a long prefix (~69 tokens) then a token flips
outside vLLM's top-N logprobs only near the end of the 128-token generation,
failing the top-N check (test_multi_image_models[qwen2_vl-...], Test2). vLLM's
max_logprobs is capped at 20, so the earlier num_logprobs 10->20 widening is
exhausted.

This is acceptable numerical drift, not a correctness bug, so cap max_tokens to
64 on CPU (well before the ~token-69 divergence) instead of dropping the test.
The multi-image path stays under test on CPU; GPU keeps max_tokens=128. The
CPU num_logprobs=20 window is retained for the compared prefix.

See pytorch/pytorch#187735.

Test Plan: `python3 -m py_compile tests/models/multimodal/generation/test_common.py`;
full validation is CPU-Multi-Modal Model Tests 1 in CI (qwen2_vl multi-image now
generates 64 tokens on CPU and matches HF within the top-N window).
@atalman
atalman force-pushed the test-pytorch-2.13.0-test branch from 8d7f288 to 06e2ffe Compare July 4, 2026 13:18
@mergify

mergify Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @atalman.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Jul 6, 2026
@atalman atalman mentioned this pull request Jul 9, 2026
Merged
@atalman

atalman commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

closing PyTorch was released

@atalman atalman closed this Jul 9, 2026
@github-project-automation github-project-automation Bot moved this from Todo to Done in AMD Jul 9, 2026
@github-project-automation github-project-automation Bot moved this to Done in NVIDIA Jul 9, 2026
atalman added a commit to atalman/vllm that referenced this pull request Jul 14, 2026
@atalman
[Core] Update PyTorch to 2.13.0, torchvision to 0.28.0, triton to 3.7.1
ea724aa 
PyTorch 2.13.0 is published to the production PyTorch index
(download.pytorch.org/whl/...), so this is a pure production-channel
version bump -- no test-channel index URLs -- same approach as vllm-project#45082.

- torch:       2.11.0 -> 2.13.0
- torchvision: 0.26.0 -> 0.28.0
- triton:      3.6.0  -> 3.7.1  (matches torch 2.13.0 Requires-Dist: triton==3.7.1)
- torchaudio:  stays at 2.11.0 (latest published torchaudio)

torch-2.13 test adaptations ported from the test-channel PR vllm-project#45731
(production-adjusted; ROCm excluded):
- CPU compat test: use --enforce-eager instead of TORCH_COMPILE_DISABLE.
- xfail qwen2audio text-then-audio prompt_embeds (pytorch/pytorch#184431).
- qwen2_vl multi-image: cap CPU max_tokens / widen logprobs for torch 2.13
  numerical drift (pytorch/pytorch#187735).
- nixl_ep import: xfail on ABI mismatch pending rebuild (pytorch/pytorch#187727).

Production-channel follow-up to vllm-project#45731.

Co-authored-by: Claude
Signed-off-by: Andrey Talman <atalman@users.noreply.github.com>
@atalman atalman mentioned this pull request Aug 13, 2026
Open
atalman added a commit to atalman/vllm that referenced this pull request Aug 19, 2026
@atalman
[Core] Update PyTorch to 2.14.0, torchvision to 0.29.0, triton to 3.8…
1a2d1be 
….0 (test channel)

Update the PyTorch ecosystem to the 2.14.0 release candidate, resolving wheels
from the PyTorch test channel (download.pytorch.org/whl/test/...). Mirrors the
2.13.0 test PR vllm-project#45731.

- torch:       2.13.0 -> 2.14.0
- torchvision: 0.28.0 -> 0.29.0
- triton:      3.7.1  -> 3.8.0

2.14.0 is published on download.pytorch.org/whl/test/ but not yet on the release
index or PyPI, so the index URLs point at the test channel.

Verified availability on the test channel before pinning:
- torch 2.14.0+cu130, torchvision 0.29.0, triton 3.8.0 all present
- triton 3.8.0 is exactly what pytorch release/2.14 pins in
  .ci/docker/triton_version.txt
- torchaudio's newest published version is still 2.11.0 on every channel, so
  that pin is unchanged

ROCm: requirements/build/rocm.txt moves from the rocm7.1 index to
test/rocm7.2. rocm7.1 has no torch 2.14.0 wheels; 2.14 is built for rocm7.2 and
rocm7.14 only. That file was also still on torch 2.11.0 / triton 3.6.0, so this
brings it in line with CMakeLists' TORCH_SUPPORTED_VERSION_ROCM.

check-ray-compatibility.sh now offers uv both the stable and test channels
(it already runs with --index-strategy unsafe-best-match), so an RC torch pin
coming from a transitive dep resolves during the RC window instead of failing
against a stable-only index. Same fix as vllm-project#45731.

Deliberately not carried over from vllm-project#45731: the xfails and workarounds that were
specific to breakages found during 2.12/2.13 CI (pytorch#184431, #187735,
reactions to that RC's failures; 2.14 should get whatever it actually needs once
CI reports. requirements/build/tpu.txt is also left alone, as TPU torch is
bumped separately (vllm-project#50412).

Test Plan: full CI (apply the `ready` label to trigger Buildkite).

BC-breaking? No - test-channel version bump only.

Authored with AI assistance; modeled on vllm-project#45731.
atalman added a commit to atalman/vllm that referenced this pull request Sep 15, 2026
@atalman
[Core] Update PyTorch to 2.14.0, torchvision to 0.29.0, triton to 3.8…
54cb309 
….0 (test channel)

Update the PyTorch ecosystem to the 2.14.0 release candidate, resolving wheels
from the PyTorch test channel (download.pytorch.org/whl/test/...). Mirrors the
2.13.0 test PR vllm-project#45731.

- torch:       2.13.0 -> 2.14.0
- torchvision: 0.28.0 -> 0.29.0
- triton:      3.7.1  -> 3.8.0

2.14.0 is published on download.pytorch.org/whl/test/ but not yet on the release
index or PyPI, so the index URLs point at the test channel.

Verified availability on the test channel before pinning:
- torch 2.14.0+cu130, torchvision 0.29.0, triton 3.8.0 all present
- triton 3.8.0 is exactly what pytorch release/2.14 pins in
  .ci/docker/triton_version.txt
- torchaudio's newest published version is still 2.11.0 on every channel, so
  that pin is unchanged

ROCm: requirements/build/rocm.txt moves from the rocm7.1 index to
test/rocm7.2. rocm7.1 has no torch 2.14.0 wheels; 2.14 is built for rocm7.2 and
rocm7.14 only. That file was also still on torch 2.11.0 / triton 3.6.0, so this
brings it in line with CMakeLists' TORCH_SUPPORTED_VERSION_ROCM.

check-ray-compatibility.sh now offers uv both the stable and test channels
(it already runs with --index-strategy unsafe-best-match), so an RC torch pin
coming from a transitive dep resolves during the RC window instead of failing
against a stable-only index. Same fix as vllm-project#45731.

Deliberately not carried over from vllm-project#45731: the xfails and workarounds that were
specific to breakages found during 2.12/2.13 CI (pytorch#184431, #187735,
reactions to that RC's failures; 2.14 should get whatever it actually needs once
CI reports. requirements/build/tpu.txt is also left alone, as TPU torch is
bumped separately (vllm-project#50412).

Test Plan: full CI (apply the `ready` label to trigger Buildkite).

BC-breaking? No - test-channel version bump only.

Authored with AI assistance; modeled on vllm-project#45731.
atalman added a commit to atalman/vllm that referenced this pull request Sep 24, 2026
@atalman
[Core] Update PyTorch to 2.14.0, torchvision to 0.29.0, triton to 3.8…
ed2711a 
….0 (test channel)

Update the PyTorch ecosystem to the 2.14.0 release candidate, resolving wheels
from the PyTorch test channel (download.pytorch.org/whl/test/...). Mirrors the
2.13.0 test PR vllm-project#45731.

- torch:       2.13.0 -> 2.14.0
- torchvision: 0.28.0 -> 0.29.0
- triton:      3.7.1  -> 3.8.0

2.14.0 is published on download.pytorch.org/whl/test/ but not yet on the release
index or PyPI, so the index URLs point at the test channel.

Verified availability on the test channel before pinning:
- torch 2.14.0+cu130, torchvision 0.29.0, triton 3.8.0 all present
- triton 3.8.0 is exactly what pytorch release/2.14 pins in
  .ci/docker/triton_version.txt
- torchaudio's newest published version is still 2.11.0 on every channel, so
  that pin is unchanged

ROCm: requirements/build/rocm.txt moves from the rocm7.1 index to
test/rocm7.2. rocm7.1 has no torch 2.14.0 wheels; 2.14 is built for rocm7.2 and
rocm7.14 only. That file was also still on torch 2.11.0 / triton 3.6.0, so this
brings it in line with CMakeLists' TORCH_SUPPORTED_VERSION_ROCM.

check-ray-compatibility.sh now offers uv both the stable and test channels
(it already runs with --index-strategy unsafe-best-match), so an RC torch pin
coming from a transitive dep resolves during the RC window instead of failing
against a stable-only index. Same fix as vllm-project#45731.

Deliberately not carried over from vllm-project#45731: the xfails and workarounds that were
specific to breakages found during 2.12/2.13 CI (pytorch#184431, #187735,
reactions to that RC's failures; 2.14 should get whatever it actually needs once
CI reports. requirements/build/tpu.txt is also left alone, as TPU torch is
bumped separately (vllm-project#50412).

Test Plan: full CI (apply the `ready` label to trigger Buildkite).

BC-breaking? No - test-channel version bump only.

Authored with AI assistance; modeled on vllm-project#45731.
atalman added a commit to atalman/vllm that referenced this pull request Sep 24, 2026
@atalman
[Core] Update PyTorch to 2.14.0, torchvision to 0.29.0, triton to 3.8…
935860c 
….0 (test channel)

Update the PyTorch ecosystem to the 2.14.0 release candidate, resolving wheels
from the PyTorch test channel (download.pytorch.org/whl/test/...). Mirrors the
2.13.0 test PR vllm-project#45731.

- torch:       2.13.0 -> 2.14.0
- torchvision: 0.28.0 -> 0.29.0
- triton:      3.7.1  -> 3.8.0

2.14.0 is published on download.pytorch.org/whl/test/ but not yet on the release
index or PyPI, so the index URLs point at the test channel.

Verified availability on the test channel before pinning:
- torch 2.14.0+cu130, torchvision 0.29.0, triton 3.8.0 all present
- triton 3.8.0 is exactly what pytorch release/2.14 pins in
  .ci/docker/triton_version.txt
- torchaudio's newest published version is still 2.11.0 on every channel, so
  that pin is unchanged

ROCm: requirements/build/rocm.txt moves from the rocm7.1 index to
test/rocm7.2. rocm7.1 has no torch 2.14.0 wheels; 2.14 is built for rocm7.2 and
rocm7.14 only. That file was also still on torch 2.11.0 / triton 3.6.0, so this
brings it in line with CMakeLists' TORCH_SUPPORTED_VERSION_ROCM.

check-ray-compatibility.sh now offers uv both the stable and test channels
(it already runs with --index-strategy unsafe-best-match), so an RC torch pin
coming from a transitive dep resolves during the RC window instead of failing
against a stable-only index. Same fix as vllm-project#45731.

Deliberately not carried over from vllm-project#45731: the xfails and workarounds that were
specific to breakages found during 2.12/2.13 CI (pytorch#184431, #187735,
reactions to that RC's failures; 2.14 should get whatever it actually needs once
CI reports. requirements/build/tpu.txt is also left alone, as TPU torch is
bumped separately (vllm-project#50412).

Test Plan: full CI (apply the `ready` label to trigger Buildkite).

BC-breaking? No - test-channel version bump only.

Authored with AI assistance; modeled on vllm-project#45731.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tjtanaa tjtanaa Awaiting requested review from tjtanaa tjtanaa will be requested when the pull request is marked ready for review tjtanaa is a code owner
@dllehr-amd dllehr-amd Awaiting requested review from dllehr-amd dllehr-amd will be requested when the pull request is marked ready for review dllehr-amd is a code owner
@AndreasKaratzas AndreasKaratzas Awaiting requested review from AndreasKaratzas AndreasKaratzas will be requested when the pull request is marked ready for review AndreasKaratzas is a code owner
@bigPYJ1151 bigPYJ1151 Awaiting requested review from bigPYJ1151 bigPYJ1151 will be requested when the pull request is marked ready for review bigPYJ1151 is a code owner
@xuechendi xuechendi Awaiting requested review from xuechendi xuechendi will be requested when the pull request is marked ready for review xuechendi is a code owner
@ApostaC ApostaC Awaiting requested review from ApostaC ApostaC will be requested when the pull request is marked ready for review ApostaC is a code owner
@orozery orozery Awaiting requested review from orozery orozery will be requested when the pull request is marked ready for review orozery is a code owner
@ivanium ivanium Awaiting requested review from ivanium ivanium will be requested when the pull request is marked ready for review ivanium is a code owner
@DarkLight1337 DarkLight1337 Awaiting requested review from DarkLight1337 DarkLight1337 will be requested when the pull request is marked ready for review DarkLight1337 is a code owner
@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 will be requested when the pull request is marked ready for review ywang96 is a code owner
@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat will be requested when the pull request is marked ready for review robertgshaw2-redhat is a code owner
@aarnphm aarnphm Awaiting requested review from aarnphm aarnphm will be requested when the pull request is marked ready for review aarnphm is a code owner
@NickLucche NickLucche Awaiting requested review from NickLucche NickLucche will be requested when the pull request is marked ready for review NickLucche is a code owner
@khluu khluu Awaiting requested review from khluu khluu will be requested when the pull request is marked ready for review khluu is a code owner
@Harry-Chen Harry-Chen Awaiting requested review from Harry-Chen Harry-Chen will be requested when the pull request is marked ready for review Harry-Chen is a code owner
@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth will be requested when the pull request is marked ready for review tlrmchlsmth is a code owner
@LucasWilkinson LucasWilkinson Awaiting requested review from LucasWilkinson LucasWilkinson will be requested when the pull request is marked ready for review LucasWilkinson is a code owner
1 more reviewer
@hai19851221-lgtm hai19851221-lgtm hai19851221-lgtm approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ci/build cpu Related to CPU backends kv-connector multi-modality Related to multi-modality (#4194) needs-rebase nvidia rocm Related to AMD ROCm v1

Projects

AMD
Status: Done
NVIDIA
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@atalman @hai19851221-lgtm