diff --git a/docs/fern/pages/reference/backends/vllm-configuration.mdx b/docs/fern/pages/reference/backends/vllm-configuration.mdx index a5c18400879d..fc6b6eb85cfd 100644 --- a/docs/fern/pages/reference/backends/vllm-configuration.mdx +++ b/docs/fern/pages/reference/backends/vllm-configuration.mdx @@ -191,22 +191,40 @@ These flags control the self-benchmark sweep that runs on startup before the wor Environment variable: `DYN_BENCHMARK_MODE` - - Number of ISL sample points for the prefill sweep. + + JSON file of explicit pure prefill/decode benchmark points, applied uniformly to every data-parallel rank. The file completely replaces generated grid sampling for the phases selected by `--benchmark-mode`, so the sampling limits below are ignored when it is set. It is read and normalized once before vLLM workers start, then the same contents are forwarded to every rank. - Environment variable: `DYN_BENCHMARK_PREFILL_GRANULARITY` + Environment variable: `DYN_BENCHMARK_POINTS_FILE` - - Number of context length sample points for the decode sweep. + + Maximum number of iteration-total prefill new-token samples. If the CUDA-graph-aware axis has more points, points are selected uniformly across the sorted axis while always retaining its minimum and maximum. Must be at least 2. - Environment variable: `DYN_BENCHMARK_DECODE_LENGTH_GRANULARITY` + Environment variable: `DYN_PREFILL_MAX_NEW_TOKEN_SAMPLES` - - Number of batch size sample points per context length for the decode sweep. + + Maximum number of iteration-total prefill KV-read-token samples for each (new tokens, batch size) pair. If the block-aligned KV ladder has more points, points are selected uniformly while always retaining zero and the feasible maximum. Must be at least 2. - Environment variable: `DYN_BENCHMARK_DECODE_BATCH_GRANULARITY` + Environment variable: `DYN_PREFILL_MAX_KV_READ_TOKEN_SAMPLES` + + + + Maximum number of prefill request-batch-size samples for each new-token point. Keeps the first N values from the sorted power-of-two-plus-legal-maximum axis, so the default of 3 selects `[1, 2, 4]` when all three are legal. Must be positive. + + Environment variable: `DYN_PREFIX_MAX_BATCH_SIZE_SAMPLES` + + + + Maximum number of iteration-total decode KV-read-token samples for each batch size. If the KV ladder has more points, points are selected uniformly while always retaining its minimum and feasible maximum. Must be at least 2. + + Environment variable: `DYN_DECODE_MAX_KV_READ_TOKEN_SAMPLES` + + + + Maximum number of decode batch-size samples. If the CUDA-graph-aware axis has more points, points are selected uniformly while always retaining the minimum and feasible maximum. Must be at least 2. + + Environment variable: `DYN_DECODE_MAX_BATCH_SIZE_SAMPLES` @@ -237,6 +255,36 @@ These flags are retained for backward compatibility and will be removed in a fut Environment variable: `MODEL_EXPRESS_URL` + + **Deprecated** — use `--prefill-max-new-token-samples`. Legacy values are translated to the new sampling limit. + + Environment variable: `DYN_BENCHMARK_PREFILL_GRANULARITY` + + + + **Deprecated** — use `--prefill-max-kv-read-token-samples`. Legacy values are translated to the new sampling limit. + + Environment variable: `DYN_BENCHMARK_PREFILL_KV_READ_GRANULARITY` + + + + **Deprecated** — use `--prefix-max-batch-size-samples`. Legacy values are translated to the new sampling limit. + + Environment variable: `DYN_BENCHMARK_PREFILL_BATCH_GRANULARITY` + + + + **Deprecated** — use `--decode-max-kv-read-token-samples`. Legacy values are translated to the new sampling limit. + + Environment variable: `DYN_BENCHMARK_DECODE_LENGTH_GRANULARITY` + + + + **Deprecated** — use `--decode-max-batch-size-samples`. Legacy values are translated to the new sampling limit. + + Environment variable: `DYN_BENCHMARK_DECODE_BATCH_GRANULARITY` + + ## Validation rules - `--embedding-worker` is only valid with `--disaggregation-mode=agg` (or the default aggregated mode) and cannot be combined with `--enable-multimodal` or `--benchmark-mode`.