Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 69 additions & 11 deletions docs/fern/pages/reference/backends/vllm-configuration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,8 @@ These flags control which role this worker plays in a disaggregated deployment.

These flags control the self-benchmark sweep that runs on startup before the worker begins accepting production requests.

The sweep does not walk a fixed grid of sample counts. Each axis is derived from the engine's own limits — CUDA-graph axes include every `{capture size, capture size + 1}` boundary and then continue geometrically to the engine limit, and KV-read axes use complete power-of-two block ladders plus their exact feasible maxima. The flags below are per-axis *sample limits* applied to those derived axes: if an axis has more points than its limit allows, points are selected uniformly across the sorted axis while the endpoints are retained. Raising a limit therefore measures more of the same axis; it does not change the axis itself.

<ParamField path="--benchmark-mode" type="string" default="null">
Run a self-benchmark on startup before accepting requests. Sweeps prefill input sequence lengths (ISLs) and/or decode `(context_length × batch_size)` operating points, collecting `ForwardPassMetrics` at each point.

Expand All @@ -191,22 +193,42 @@ These flags control the self-benchmark sweep that runs on startup before the wor
Environment variable: `DYN_BENCHMARK_MODE`
</ParamField>

<ParamField path="--benchmark-prefill-granularity" type="integer" default="16">
Number of ISL sample points for the prefill sweep.
<ParamField path="--benchmark-points-file" type="string" default="null">
JSON file of explicit pure prefill and decode benchmark points, applied uniformly to every data-parallel rank. The file is read and normalized once before the vLLM workers start, then the same contents are forwarded to every rank.

Environment variable: `DYN_BENCHMARK_PREFILL_GRANULARITY`
The file completely replaces generated grid sampling for the phases selected by `--benchmark-mode`, so all of the sample-limit flags below — and the deprecated granularity flags — are ignored when it is set. Setting it without `--benchmark-mode` raises a `ValueError`.

Environment variable: `DYN_BENCHMARK_POINTS_FILE`
</ParamField>

<ParamField path="--benchmark-decode-length-granularity" type="integer" default="6">
Number of context length sample points for the decode sweep.
<ParamField path="--prefill-max-new-token-samples" type="integer" default="64">
Maximum number of iteration-total prefill new-token samples. Must be at least 2, so that both endpoints of the axis are always retained.

Environment variable: `DYN_BENCHMARK_DECODE_LENGTH_GRANULARITY`
Environment variable: `DYN_PREFILL_MAX_NEW_TOKEN_SAMPLES`
</ParamField>

<ParamField path="--benchmark-decode-batch-granularity" type="integer" default="6">
Number of batch size sample points per context length for the decode sweep.
<ParamField path="--prefill-max-kv-read-token-samples" type="integer" default="16">
Maximum number of iteration-total prefill KV-read-token samples, applied for each `(new tokens, batch size)` pair. Sampling always retains zero and the feasible maximum. Must be at least 2.

Environment variable: `DYN_BENCHMARK_DECODE_BATCH_GRANULARITY`
Environment variable: `DYN_PREFILL_MAX_KV_READ_TOKEN_SAMPLES`
</ParamField>

<ParamField path="--decode-max-kv-read-token-samples" type="integer" default="128">
Maximum number of iteration-total decode KV-read-token samples, applied for each batch size. Sampling always retains the minimum and the feasible maximum. Must be at least 2.

Environment variable: `DYN_DECODE_MAX_KV_READ_TOKEN_SAMPLES`
</ParamField>

<ParamField path="--decode-max-batch-size-samples" type="integer" default="128">
Maximum number of decode batch-size samples over the CUDA-graph-aware axis. Sampling always retains the minimum and the feasible maximum. Must be at least 2.

Environment variable: `DYN_DECODE_MAX_BATCH_SIZE_SAMPLES`
</ParamField>

<ParamField path="--prefix-max-batch-size-samples" type="integer" default="3">
Maximum number of prefill request-batch-size samples for each new-token point. Unlike the limits above, this one keeps the first N values of the sorted power-of-two-plus-legal-maximum axis rather than sampling uniformly, so the default of 3 selects `[1, 2, 4]` when all three are legal. Must be positive.

Environment variable: `DYN_PREFIX_MAX_BATCH_SIZE_SAMPLES`
</ParamField>

<ParamField path="--benchmark-warmup-iterations" type="integer" default="5">
Expand All @@ -221,8 +243,8 @@ These flags control the self-benchmark sweep that runs on startup before the wor
Environment variable: `DYN_BENCHMARK_OUTPUT_PATH`
</ParamField>

<ParamField path="--benchmark-timeout" type="integer" default="300">
Maximum seconds to wait for the benchmark to complete. Worker startup fails if this limit is exceeded.
<ParamField path="--benchmark-timeout" type="integer" default="900">
Soft limit, in seconds, for the self-benchmark. Reaching it does not fail worker startup: the iteration being measured finishes, the partial results collected so far are returned, and engine startup continues. A bounded cleanup grace still fails closed if no result is written at all. Must be positive.

Environment variable: `DYN_BENCHMARK_TIMEOUT`
</ParamField>
Expand All @@ -237,6 +259,42 @@ These flags are retained for backward compatibility and will be removed in a fut
Environment variable: `MODEL_EXPRESS_URL`
</ParamField>

The five `--benchmark-*-granularity` flags below are the previous names for the benchmark sampling limits. They are read only when `--benchmark-mode` is set and `--benchmark-points-file` is not; otherwise they are ignored entirely, and none of the translation, range checking, or deprecation warnings described here takes place. Under those conditions each is translated to its replacement at startup, with a deprecation warning, and is accepted in the range 1 to 1024; a value outside it raises a `ValueError`. Because the four uniform-sampling limits need both endpoints of their axis, a legacy value of `1` maps to `2` for those flags.

<Warning>
With `--benchmark-mode` set and no points file, passing a legacy flag together with its replacement raises a `ValueError` at startup — `cannot combine --benchmark-decode-length-granularity with --decode-max-kv-read-token-samples`, and likewise for each of the other pairs. Migrate to the replacement rather than setting both. All five are ignored when `--benchmark-points-file` is set.
</Warning>
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

<ParamField path="--benchmark-prefill-granularity" type="integer" default="null" deprecated={true}>
**Deprecated** — use `--prefill-max-new-token-samples` instead.

Environment variable: `DYN_BENCHMARK_PREFILL_GRANULARITY`
</ParamField>

<ParamField path="--benchmark-prefill-kv-read-granularity" type="integer" default="null" deprecated={true}>
**Deprecated** — use `--prefill-max-kv-read-token-samples` instead.

Environment variable: `DYN_BENCHMARK_PREFILL_KV_READ_GRANULARITY`
</ParamField>

<ParamField path="--benchmark-prefill-batch-granularity" type="integer" default="null" deprecated={true}>
**Deprecated** — use `--prefix-max-batch-size-samples` instead. This is the one legacy flag whose replacement does not sample uniformly, so a legacy value of `1` is carried across unchanged.

Environment variable: `DYN_BENCHMARK_PREFILL_BATCH_GRANULARITY`
</ParamField>

<ParamField path="--benchmark-decode-length-granularity" type="integer" default="null" deprecated={true}>
**Deprecated** — use `--decode-max-kv-read-token-samples` instead.

Environment variable: `DYN_BENCHMARK_DECODE_LENGTH_GRANULARITY`
</ParamField>

<ParamField path="--benchmark-decode-batch-granularity" type="integer" default="null" deprecated={true}>
**Deprecated** — use `--decode-max-batch-size-samples` instead.

Environment variable: `DYN_BENCHMARK_DECODE_BATCH_GRANULARITY`
</ParamField>

## Validation rules

- `--embedding-worker` is only valid with `--disaggregation-mode=agg` (or the default aggregated mode) and cannot be combined with `--enable-multimodal` or `--benchmark-mode`.
Expand Down
Loading