[Bugfix] Max-load throughput cliff when max_num_seqs is not a multiple of 8

[andylolu2]

Why

By default, cudagraphs are captured in (mostly) multiples of 8. When max_num_seqs is not set to a multiple of 8, it has the unfortunate side effect that the last bucket of batch sizes doesn't fall into a FULL cuda graph. For example, with max_num_seqs=100 (and no spec decode), we get:

Num decodes	Cudagraph mode
95	FULL
96	FULL
97	PIECEWISE
98	PIECEWISE
99	PIECEWISE
100	PIECEWISE

This is quite bad practically, because servers can get stuck in the slow PIECEWISE mode when it's overloaded (num_running_reqs = max_num_seqs).

In a TP4/B200 reproduction this was not a small boundary effect: batch 97 started behind batch 96 and progressively slowed over a 2,000-token decode, eventually falling below 1,000 tok/s/server. Adding the legal FULL-100 shape removed the cliff.

Fix

This is actually already correctly handled when spec decode is turned on by #50488, but not when spec dec is off. The fix is to simply piggy-back off the PR's changes: when the inferred default is used without speculative decoding, add max_num_seqs as the widest uniform-decode shape if it fits under the existing capture ceiling:

elif max_num_seqs <= max_cudagraph_capture_size:
    uniform_decode_sizes = [max_num_seqs]

For the example above, the ladder becomes:

..., 88, 96, 100, 104, 112, ...

The unchanged graph manager then produces FULL descriptors through 100, while later sizes remain PIECEWISE-only.

Benchmark evidence

The reproducer used Mistral-Small-4-119B-2603 with TP4 on B200, max_num_seqs=100, logprobs enabled, 2,000 generated tokens, and a production-shaped prompt distribution with roughly equal mass around 8k, 16k, 32k, 64k, and 127k input tokens.

Batch	Before	With FULL 100
96	3,753 tok/s	3,760.78 tok/s
97	1,523 tok/s	3,876.85 tok/s

Batch 97 improved 2.55× (+154.6%) and became comparable to batch 96. Before the fix its server-throughput windows decayed from 2,922 tok/s near output token 302 to 829 tok/s in the final window. After the fix they stayed flat at 3,923, 3,890, 3,862, and 3,824 tok/s. Total time fell from 129.72 s to 52.41 s.

An Nsight trace showed that the PIECEWISE path accumulated tensor-parallel rank drift: attention remained around 0.58 ms, while graph-launch gaps grew from about 0.69 ms to 5.5 ms and MNNVL all-reduce wait medians grew from about 10 µs to 3.6–4.7 ms. This PR avoids that path for a legal decode batch; it does not attempt to independently repair PIECEWISE rank drift.

Test results

VLLM_TARGET_DEVICE=cpu .venv/bin/python -m pytest -q \
  tests/compile/test_config.py \
  -k default_cudagraph_capture_size_respects_platform_ceiling
11 passed, 61 deselected

VLLM_TARGET_DEVICE=cpu .venv/bin/python -m pytest -q \
  tests/v1/cudagraph/test_cudagraph_manager.py \
  -k 'uniform_decode_pads_up_to_full_graph or uniform_decode_exact_match_is_not_over_padded or uniform_decode_beyond_capture_ladder_falls_back or divisor_query_len_dispatch_is_unchanged'
6 passed, 5 deselected

.venv/bin/pre-commit run --files \
  vllm/config/vllm.py tests/compile/test_config.py
all hooks passed, including Ruff, mypy, SPDX, and repository checks

Revert check: with the production hunk reverted while retaining the new test, both cases fail (2 failed): neither 100 nor 101 is present in the inferred capture sizes.

No model evaluation was run because this changes only which execution graph serves an already-valid shape; model operations and outputs are unchanged.

Not a duplicate

I searched open PRs for max_num_seqs, cudagraph, capture sizes, off-stride, and uniform decode. The nearby work addresses different contracts:

• [Bugfix][Spec Decode] Capture the widest uniform decode batch by default #50488 introduced uniform_decode_sizes for speculative decoding and deliberately kept non-speculative defaults unchanged. This PR closes that remaining non-speculative boundary hole using its abstraction.
• Fix cudagraph max capture size upper bound #37865 appends max_cudagraph_capture_size; here that value is 200, so it still does not add the required 100-request boundary.
• [Bugfix] Cover uniform decode sizes under an explicit max_cudagraph_capture_size #55004 covers uniform decode under an explicitly configured scalar maximum, primarily with speculative decoding. This PR changes only the inferred default and leaves explicit configuration untouched.
• [Bugfix][MRV2] Dispatch uniform decode to a padded FULL cudagraph #53407 lets a uniform decode pad to a wider compatible FULL graph, but it requires such a graph to exist. In this case 104 requests exceeds the scheduler limit, so there is no compatible FULL graph until 100 is captured.
• [Bugfix][Spec Decode] Ladder cudagraph candidates per uniform query length #56136 concerns candidate ladders for multiple dynamic-speculation query lengths, not the non-speculative capture set.
• Revert "[Bugfix][Spec Decode] Capture the widest uniform decode batch by default" (#50488) #54352 is a draft auto-revert for the memory regression caused when [Bugfix][Spec Decode] Capture the widest uniform decode batch by default #50488 raised capture ceilings. This patch never raises the ceiling and adds only a shape already below it.

No open PR covers an off-stride, non-speculative max_num_seqs boundary.

Reviewer guide

• vllm/config/vllm.py: the two-line behavior change reuses the uniform-decode capture-size path; the nearby comment is updated to match its broadened scope.
• tests/compile/test_config.py: one new parameter exercises the existing capture-boundary contract at max_num_seqs=100.

There are no moves, generated files, or mechanical propagation changes.

AI assistance disclosure

OpenAI Codex assisted with the investigation, patch, tests, duplicate-work audit, and PR write-up. I reviewed every changed line and the benchmark evidence before submission.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[benchislett]

LGTM

[github-actions]

✅ @andylolu2, CI is now available for this PR.

• /ci run starts upstream CI; /amd-ci run starts AMD CI only.
• Your branch must contain every commit currently on its upstream target branch. Merge or rebase onto the latest target branch, then rerun the command. Append --allow-stale to a run command to test an outdated branch at your own risk.
• /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
• /amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
• /ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

[benchislett]

/ci run

[github-actions]

❌ This PR is 17 commits behind upstream main. Your branch must contain every commit currently on upstream main. No new CI build was started. Merge or rebase onto the latest main, then rerun /ci run. To test this branch at your own risk, use /ci run --allow-stale.

[andylolu2]

/ci run

[github-actions]

✅ Triggered Buildkite CI #89668 for commit 91d92cc82081.