Skip to content

test(ci): GPU-tests speedups (OPS-8033, OPS-8113, OPS-8117) - #12964

Merged
pvijayakrish merged 3 commits into
release/1.4.0from
dtokarev/release-1.4.0-ci-gpu-parallel-backports
Aug 11, 2026
Merged

test(ci): GPU-tests speedups (OPS-8033, OPS-8113, OPS-8117)#12964
pvijayakrish merged 3 commits into
release/1.4.0from
dtokarev/release-1.4.0-ci-gpu-parallel-backports

Conversation

@dmitry-tokarev-nv

@dmitry-tokarev-nv dmitry-tokarev-nv commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Summary

Cherry-pick 3 merged CI test-scheduling changes from main onto release/1.4.0 as one PR. All three shrink the pre_merge GPU stage by moving CPU-only work off the VRAM-budgeted lane or by correcting an over-stated VRAM reservation; none of them change product code.

Cherry-pick Origin Effect on the GPU stage
test(ci): reclassify CPU-only sglang unit tests from gpu_1 to gpu_0 (OPS-8033) #12718 (8d4a014873) 8 CPU-only sglang unit files leave the VRAM stage (~8,400s cumulative); loadscope runs them in ~21s
test(ci): reclassify CPU-only vllm unit tests from gpu_1 to gpu_0 (OPS-8113) #12826 (a274300453) 322 vLLM tests / ~12,024s of 19,562s cumulative leave the VRAM stage
test(ci): correct the mm_router VRAM profile from 18.7 to the measured 7.6 GiB (OPS-8117) #12842 (3405438c75) 18.7 GiB exceeded the multi-process budget and made every test in those two files exclusive; 7.6 GiB is the measured solo peak

Each commit is cherry-picked individually with -x, so the origin commit is recorded in each message and per-PR attribution is preserved.

Linear: OPS-8033, OPS-8113, OPS-8117

Conflict resolution

Only #12718 conflicted, in two files, and both conflicts were main-vs-release/1.4.0 drift rather than anything in the backported change. Resolved to apply the gpu_1gpu_0 intent and nothing else:

  • test_sglang_multimodal_embedding_cache.py — took gpu_0, kept this branch's timeout(60) block, and did not pull in main's pytest.mark.multimodal (not present on release/1.4.0).
  • test_sglang_unit.py — took the _cpu_engine_when_no_accelerator autouse fixture (the actual mechanism behind the reclassification) and dropped test_configured_engine_route_cannot_replace_built_in_route, which is pre-existing main context unrelated to this change.

The other two cherry-picks applied cleanly.

Validation

  • Combined diff is 19 files (8 + 9 + 2), matching the sum of the three source PRs — no extra files rode along.
  • Both conflict-resolved files parse; no conflict markers remain anywhere in the tree.
  • pre-commit run --all-files passes on this branch with no autofixes.
  • Verified the mm_router comment's cross-reference still holds here: tests/mm_router/test_vllm_mm_router_e2e.py is also at profiled_vram_gib(7.6) on release/1.4.0.
  • The one remaining pytest.mark.gpu_1 under the touched vLLM test dirs (omni/test_omni_nixl_connector.py) is intentional — test(ci): reclassify CPU-only vllm unit tests from gpu_1 to gpu_0 (OPS-8113) #12826 left it at gpu_1 on main too.
  • All three commits are GPG-signed and DCO signed-off.

CI on the release branch still needs a maintainer /ok to test b2ba74f959.

🤖 Generated with Claude Code


Open in Devin Review

dmitry-tokarev-nv and others added 3 commits August 10, 2026 16:08
…OPS-8033) (#12718)

Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
(cherry picked from commit 8d4a014)
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
…S-8113) (#12826)

Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit a274300)
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
…d 7.6 GiB (OPS-8117) (#12842)

Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 3405438)
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
@dmitry-tokarev-nv
dmitry-tokarev-nv requested review from a team as code owners August 10, 2026 20:11
@github-actions github-actions Bot added test backend::vllm Relates to the vllm backend backend::sglang Relates to the sglang backend frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` labels Aug 10, 2026

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

Open in Devin Review

Comment thread components/src/dynamo/sglang/tests/test_sglang_unit.py
@dmitry-tokarev-nv dmitry-tokarev-nv changed the title test(ci): backport pre_merge GPU-stage speedups to release/1.4.0 (OPS-8033, OPS-8113, OPS-8117) test(ci): GPU-tests speedups (OPS-8033, OPS-8113, OPS-8117) Aug 10, 2026
@datadog-official

datadog-official Bot commented Aug 10, 2026

Copy link
Copy Markdown

🎯 Code Coverage (details)
Patch Coverage: 89.66%
Overall Coverage: 44.16% (+3.98%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: b2ba74f | Docs | Datadog PR Page | Give us feedback!

@pvijayakrish
pvijayakrish merged commit 62dfee5 into release/1.4.0 Aug 11, 2026
182 of 187 checks passed
@pvijayakrish
pvijayakrish deleted the dtokarev/release-1.4.0-ci-gpu-parallel-backports branch August 11, 2026 00:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::sglang Relates to the sglang backend backend::vllm Relates to the vllm backend frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` size/L test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants