Skip to content

docs(vllm): document current benchmark sampling flags, demote the legacy aliases #12979

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dagil-nvidia wants to merge 1 commit into from
Closed

docs(vllm): document current benchmark sampling flags, demote the legacy aliases #12979

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 57 additions & 9 deletions docs/fern/pages/reference/backends/vllm-configuration.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -191,22 +191,40 @@ These flags control the self-benchmark sweep that runs on startup before the wor
Environment variable: `DYN_BENCHMARK_MODE`
</ParamField>

<ParamField path="--benchmark-prefill-granularity" type="integer" default="16">
Number of ISL sample points for the prefill sweep.
<ParamField path="--benchmark-points-file" type="string" default="null">
JSON file of explicit pure prefill/decode benchmark points, applied uniformly to every data-parallel rank. The file completely replaces generated grid sampling for the phases selected by `--benchmark-mode`, so the sampling limits below are ignored when it is set. It is read and normalized once before vLLM workers start, then the same contents are forwarded to every rank.
Comment on lines +194 to +195

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge State that benchmark mode is required

Using --benchmark-points-file without explicitly setting --benchmark-mode does not merely leave the phase unspecified: DynamoVllmConfig._load_explicit_benchmark_points raises ValueError("--benchmark-points-file requires --benchmark-mode"), so the worker fails during argument validation. The new flag description should identify this required companion option so a command assembled from the reference can start successfully.

Useful? React with 👍 / 👎.

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Document the benchmark-points JSON schema

This newly documented flag is not usable from the reference because it never specifies or links to the strict manifest schema. A repo-wide search finds no other user-facing schema or example, while BenchmarkPoints requires schema_version, prefill, and decode, with different required fields for prefill and decode entries and forbidden extra fields; users therefore cannot construct a valid file without reading Python source or tests.

Useful? React with 👍 / 👎.


Environment variable: `DYN_BENCHMARK_PREFILL_GRANULARITY`
Environment variable: `DYN_BENCHMARK_POINTS_FILE`
</ParamField>
Comment on lines +194 to 198

@coderabbitai coderabbitai Bot Aug 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Document the --benchmark-mode prerequisite.

--benchmark-points-file fails at startup when --benchmark-mode is absent. State this requirement explicitly. The loader enforces it in components/src/dynamo/vllm/backend_args.py lines 497-504.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/fern/pages/reference/backends/vllm-configuration.mdx` around lines 194 -
198, Update the documentation for the --benchmark-points-file parameter to
explicitly state that --benchmark-mode must also be provided; retain the
existing description of how the file affects benchmark sampling and rank
forwarding.


<ParamField path="--benchmark-decode-length-granularity" type="integer" default="6">
Number of context length sample points for the decode sweep.
<ParamField path="--prefill-max-new-token-samples" type="integer" default="64">
Maximum number of iteration-total prefill new-token samples. If the CUDA-graph-aware axis has more points, points are selected uniformly across the sorted axis while always retaining its minimum and maximum. Must be at least 2.

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Document the protected eager-tail sampling

When a CUDA-graph-aware axis has an eager tail comprising at most 20% of its candidates, _limit_cudagraph_axis preserves every eager-tail point and uniformly samples only the graph-covered prefix; it does not sample uniformly across the full sorted axis as stated here. This affects both this prefill flag and the analogous decode batch-size description, and can mislead users predicting which operating points a constrained benchmark sweep will retain.

Useful? React with 👍 / 👎.


Environment variable: `DYN_BENCHMARK_DECODE_LENGTH_GRANULARITY`
Environment variable: `DYN_PREFILL_MAX_NEW_TOKEN_SAMPLES`
</ParamField>

<ParamField path="--benchmark-decode-batch-granularity" type="integer" default="6">
Number of batch size sample points per context length for the decode sweep.
<ParamField path="--prefill-max-kv-read-token-samples" type="integer" default="16">
Maximum number of iteration-total prefill KV-read-token samples for each (new tokens, batch size) pair. If the block-aligned KV ladder has more points, points are selected uniformly while always retaining zero and the feasible maximum. Must be at least 2.

Environment variable: `DYN_BENCHMARK_DECODE_BATCH_GRANULARITY`
Environment variable: `DYN_PREFILL_MAX_KV_READ_TOKEN_SAMPLES`
</ParamField>

<ParamField path="--prefix-max-batch-size-samples" type="integer" default="3">
Maximum number of prefill request-batch-size samples for each new-token point. Keeps the first N values from the sorted power-of-two-plus-legal-maximum axis, so the default of 3 selects `[1, 2, 4]` when all three are legal. Must be positive.

Environment variable: `DYN_PREFIX_MAX_BATCH_SIZE_SAMPLES`
</ParamField>

<ParamField path="--decode-max-kv-read-token-samples" type="integer" default="128">
Maximum number of iteration-total decode KV-read-token samples for each batch size. If the KV ladder has more points, points are selected uniformly while always retaining its minimum and feasible maximum. Must be at least 2.

Environment variable: `DYN_DECODE_MAX_KV_READ_TOKEN_SAMPLES`
</ParamField>

<ParamField path="--decode-max-batch-size-samples" type="integer" default="128">
Maximum number of decode batch-size samples. If the CUDA-graph-aware axis has more points, points are selected uniformly while always retaining the minimum and feasible maximum. Must be at least 2.

Environment variable: `DYN_DECODE_MAX_BATCH_SIZE_SAMPLES`
</ParamField>

<ParamField path="--benchmark-warmup-iterations" type="integer" default="5">
Expand Down Expand Up @@ -237,6 +255,36 @@ These flags are retained for backward compatibility and will be removed in a fut
Environment variable: `MODEL_EXPRESS_URL`
</ParamField>

<ParamField path="--benchmark-prefill-granularity" type="integer" default="null" deprecated={true}>
**Deprecated** — use `--prefill-max-new-token-samples`. Legacy values are translated to the new sampling limit.

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Qualify when legacy sampling values are translated

Legacy values are translated only when a benchmark mode is active and no explicit points file was loaded: _resolve_legacy_benchmark_sampling returns before processing these mappings when benchmark_mode is None or _benchmark_points is set. In particular, combining this alias with --benchmark-points-file silently ignores it without the stated translation or deprecation warning, so qualify this behavior as applying only to generated-grid benchmarks; the same issue affects the other four aliases added below.

Useful? React with 👍 / 👎.


Environment variable: `DYN_BENCHMARK_PREFILL_GRANULARITY`
</ParamField>

<ParamField path="--benchmark-prefill-kv-read-granularity" type="integer" default="null" deprecated={true}>
**Deprecated** — use `--prefill-max-kv-read-token-samples`. Legacy values are translated to the new sampling limit.

Environment variable: `DYN_BENCHMARK_PREFILL_KV_READ_GRANULARITY`
</ParamField>

<ParamField path="--benchmark-prefill-batch-granularity" type="integer" default="null" deprecated={true}>
**Deprecated** — use `--prefix-max-batch-size-samples`. Legacy values are translated to the new sampling limit.

Environment variable: `DYN_BENCHMARK_PREFILL_BATCH_GRANULARITY`
</ParamField>

<ParamField path="--benchmark-decode-length-granularity" type="integer" default="null" deprecated={true}>
**Deprecated** — use `--decode-max-kv-read-token-samples`. Legacy values are translated to the new sampling limit.

Environment variable: `DYN_BENCHMARK_DECODE_LENGTH_GRANULARITY`
</ParamField>

<ParamField path="--benchmark-decode-batch-granularity" type="integer" default="null" deprecated={true}>
**Deprecated** — use `--decode-max-batch-size-samples`. Legacy values are translated to the new sampling limit.

Environment variable: `DYN_BENCHMARK_DECODE_BATCH_GRANULARITY`
</ParamField>

## Validation rules

- `--embedding-worker` is only valid with `--disaggregation-mode=agg` (or the default aggregated mode) and cannot be combined with `--enable-multimodal` or `--benchmark-mode`.
Expand Down
Loading