docs: sync router and planner reference pages to the shipped 1.4.0 behavior - #12983
Conversation
…havior Adapted port of #12975 and #12979 onto release/1.4.0. The docs tree was restructured on main after the branch cut, so paths differ and a straight cherry-pick does not apply. Three of the planner items are release-branch-only: they are already correct on main with no commit behind them, so they are applied by hand here. 6571617 (router docs): - nvext: DYN_ENABLE_FRONTEND_NVEXT does not exist; the switch is DYN_DISABLE_FRONTEND_NVEXT. Following the old text was a silent no-op - router-examples: document the kv_state_endpoint WorkerConfig parameter - metrics-catalog: add dynamo_kvrouter_ckf_mutation_total - metric-labels: add allocation_failed and ownership_degree_overflow, plus the CKF outcome label - router-guide: cross-link Multi-DC KV Routing 6572846 (planner and vLLM docs): - planner README: engine capacity metrics are engine_{prefill,decode}_capacity_requests_per_second - planner-config-reference and dgdr-reference: the Prometheus CA bundle applies regardless of ssl_verify, the opposite of what both pages said - planner-examples: the Planner main process takes a config file, not the eight CLI flags the examples used - vllm-config-reference: document the six current benchmark sampling parameters and demote the five deprecated granularity aliases Signed-off-by: Dan Gil <dagil@nvidia.com>
github-actions
Bot
added
docs
documentation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| The `nvext` protocol is enabled by default. To turn it off, set `DYN_DISABLE_FRONTEND_NVEXT` to a | ||
| truthy value (`1`, `true`, `yes`, or `on`, case-insensitive). The frontend then drops | ||
| `request.nvext` at handler entry, ignores the routing-override headers | ||
| (`x-dynamo-worker-instance-id`, `x-dynamo-prefill-instance-id`, `x-dynamo-dp-rank`, | ||
| `x-dynamo-prefill-dp-rank`), and ignores the response-side `extra_fields` opt-in. The top-level | ||
| backend-compatibility field is not part of the NvExt protocol. Cache salt is an isolation key, | ||
| not an authentication or authorization mechanism; |
There was a problem hiding this comment.
🟡 Tenant-isolation header is wrongly implied to keep working when the request-extension protocol is turned off
The rewritten paragraph lists only the four worker/rank routing headers as ignored when the request-extension protocol is disabled (docs/fern/components/frontend/nvext.md:109-115), dropping the previous statement that the tenant header is ignored too, so operators can believe tenant-based cache isolation still applies when it does not.
Impact: Someone who disables the extension protocol may assume per-tenant cache separation via the tenant header is still honored, while in reality the salt is silently dropped.
Why the tenant header is also gated by the disable switch
When nvext_enabled() is false, the handlers never call apply_header_routing_overrides and set request.nvext = None (lib/llm/src/http/service/openai.rs:676-681, lib/llm/src/http/service/openai.rs:1387, lib/llm/src/http/service/openai.rs:2361). That function is the only place x-tenant-id is mapped to cache_salt (lib/llm/src/protocols/common/extensions.rs:344-349), so with the disable flag set the tenant header has no effect — exactly what the removed sentence said. The new text enumerates only x-dynamo-worker-instance-id, x-dynamo-prefill-instance-id, x-dynamo-dp-rank, and x-dynamo-prefill-dp-rank.
| The `nvext` protocol is enabled by default. To turn it off, set `DYN_DISABLE_FRONTEND_NVEXT` to a | |
| truthy value (`1`, `true`, `yes`, or `on`, case-insensitive). The frontend then drops | |
| `request.nvext` at handler entry, ignores the routing-override headers | |
| (`x-dynamo-worker-instance-id`, `x-dynamo-prefill-instance-id`, `x-dynamo-dp-rank`, | |
| `x-dynamo-prefill-dp-rank`), and ignores the response-side `extra_fields` opt-in. The top-level | |
| backend-compatibility field is not part of the NvExt protocol. Cache salt is an isolation key, | |
| not an authentication or authorization mechanism; | |
| The `nvext` protocol is enabled by default. To turn it off, set `DYN_DISABLE_FRONTEND_NVEXT` to a | |
| truthy value (`1`, `true`, `yes`, or `on`, case-insensitive). The frontend then drops | |
| `request.nvext` at handler entry, ignores the routing-override headers | |
| (`x-dynamo-worker-instance-id`, `x-dynamo-prefill-instance-id`, `x-dynamo-dp-rank`, | |
| `x-dynamo-prefill-dp-rank`, and `x-tenant-id`), and ignores the response-side `extra_fields` | |
| opt-in. The top-level backend-compatibility field is not part of the NvExt protocol. Cache salt | |
| is an isolation key, not an authentication or authorization mechanism; |
Was this helpful? React with 👍 or 👎 to provide feedback.
| <ParamField path="dynamo_kvrouter_ckf_mutation_total" type="counter"> | ||
| CKF block-level mutation outcomes recorded by the router indexer, finer-grained than the event `status` on the counter above. Labeled by `outcome` — see [Metric-specific labels](metric-labels.mdx#metric-specific-labels). | ||
| </ParamField> |
There was a problem hiding this comment.
🟡 Newly documented router counter uses a name that the frontend and router never emit
The new counter is documented under the frontend/router KV indexer metrics as dynamo_kvrouter_ckf_mutation_total (docs/fern/reference/observability/metrics-catalog.mdx:335), but that name is emitted only by the standalone indexer service, so dashboards and queries built from this page will find nothing on a frontend or router.
Impact: Users copying the documented metric name into dashboards or alerts get no data from frontend/router deployments.
Component registration prefixes the suffix with `dynamo_component`
KvIndexerMetrics::from_component registers the counter with the suffix ckf_mutation_total (lib/kv-router/src/indexer/metrics.rs:197, lib/kv-router/src/indexer/metrics.rs:271-276), and component metric names are built as dynamo_component_<name> (lib/runtime/src/metrics/prometheus_names.rs:921-925) — hence dynamo_component_ckf_mutation_total. The literal dynamo_kvrouter_ckf_mutation_total (lib/kv-router/src/indexer/metrics.rs:199) is used only by new_registered, which is called from the standalone indexer server (lib/kv-router/src/services/indexer/server.rs:60). This mirrors the existing split the page already documents for dynamo_component_kv_cache_events_applied vs dynamo_kvrouter_kv_cache_events_applied. The same wrong name is repeated in docs/fern/reference/observability/metric-labels.mdx:115.
Prompt for agents
The newly added counter documentation names the metric `dynamo_kvrouter_ckf_mutation_total`, but that exact name is only produced by the standalone indexer service path (KvIndexerMetrics::new_registered, used by lib/kv-router/src/services/indexer/server.rs). In frontend and router deployments the counter is created through KvIndexerMetrics::from_component with suffix `ckf_mutation_total`, which the runtime prefixes as `dynamo_component_...`. Update docs/fern/reference/observability/metrics-catalog.mdx and the `outcome` label entry in docs/fern/reference/observability/metric-labels.mdx to use the component-prefixed name in the router metrics section, and mention the `dynamo_kvrouter_*` variant alongside the existing standalone-indexer note, matching how `kv_cache_events_applied` is already documented on both pages.
Was this helpful? React with 👍 or 👎 to provide feedback.
| <ParamField path="--decode-max-batch-size-samples" type="integer" default="128"> | ||
| Maximum number of decode batch-size samples. If the CUDA-graph-aware axis has more points, points are selected uniformly while always retaining the minimum and feasible maximum. Must be at least 2. | ||
|
|
||
| Environment variable: `DYN_DECODE_MAX_BATCH_SIZE_SAMPLES` |
There was a problem hiding this comment.
🔍 Neighbouring --benchmark-timeout default is stale (300 vs 900 in code)
While verifying the new benchmark sampling flags against components/src/dynamo/vllm/backend_args.py, the surrounding (unchanged) --benchmark-timeout entry still documents default="300", but VllmConfig.benchmark_timeout is 900 (components/src/dynamo/vllm/backend_args.py:455). Since this PR is specifically a benchmark-section accuracy sync, it would be worth correcting in the same pass.
Was this helpful? React with 👍 or 👎 to provide feedback.
| For workloads with daily/weekly patterns: | ||
|
|
||
| ```yaml | ||
| args: | ||
| - --load-predictor prophet | ||
| - --prophet-window-size 100 # Larger window for seasonal detection | ||
| - --load-predictor-log1p | ||
| load_predictor: prophet | ||
| prophet_window_size: 100 # Larger window for seasonal detection | ||
| load_predictor_log1p: true |
There was a problem hiding this comment.
🔍 Planner examples set optimization_target in only one of three YAML blocks
The new note declares each block a standalone PlannerConfig, but only the first block sets optimization_target: sla; the Kalman and Prophet blocks fall back to the default throughput. Given PlannerConfig gates throughput scaling on optimization_target == "sla" in the scale-interval derivation (components/src/dynamo/planner/config/planner_config.py:707-708) and warns that ttft_ms/itl_ms are ignored unless the target is sla (components/src/dynamo/planner/config/planner_config.py:797-812), it is worth confirming the predictor-only blocks behave as the examples imply, or making all three blocks consistent.
(Refers to lines 39-54)
Was this helpful? React with 👍 or 👎 to provide feedback.
![dynamo-review-agent[bot]](https://avatars.githubusercontent.com/u/201626793?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e614d09775
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| `request.nvext` at handler entry, ignores the routing-override headers | ||
| (`x-dynamo-worker-instance-id`, `x-dynamo-prefill-instance-id`, `x-dynamo-dp-rank`, | ||
| `x-dynamo-prefill-dp-rank`), and ignores the response-side `extra_fields` opt-in. The top-level |
There was a problem hiding this comment.
When an operator disables NvExt but continues sending x-tenant-id, the frontend skips apply_header_routing_overrides, so the header no longer populates cache_salt; the request-priority headers and compatibility aliases are likewise ignored. This exhaustive-looking list omits those headers and can lead readers to believe gateway-controlled tenant salting remains active, silently removing the intended cross-tenant KV-cache isolation. Include x-tenant-id, both request-priority headers, and the aliases, or state that all routing overrides are ignored.
AGENTS.md reference: docs/fern/AGENTS.md:L8-L9
Useful? React with 👍 / 👎.
| - --kalman-r 5.0 # Lower = trusts new measurements more | ||
| - --kalman-min-points 3 # Fewer points before forecasting starts | ||
| - --load-predictor-log1p # Often helps with request-rate series | ||
| load_predictor: kalman |
There was a problem hiding this comment.
When a reader runs either this Kalman block or the Prophet block as the standalone PlannerConfig promised above, optimization_target defaults to throughput; PlannerConfig._validate_config then disables enable_throughput_scaling, and BuiltinLoadPredict.Predict returns before using the configured predictor. Consequently both examples run successfully while all of their predictor tuning is inert. Add optimization_target: sla to both blocks, as the warm-start example already does.
AGENTS.md reference: docs/fern/AGENTS.md:L8-L9
Useful? React with 👍 / 👎.
| KV cache events applied to the router's radix-tree index. Includes events applied to the device tier and lower tiers such as host-pinned memory and disk. Appears only when `--router-kv-overlap-score-credit` is greater than 0 and workers publish KV events. Labeled by `status` and `event_type` — see [Metric-specific labels](metric-labels.mdx#metric-specific-labels). | ||
| </ParamField> | ||
|
|
||
| <ParamField path="dynamo_kvrouter_ckf_mutation_total" type="counter"> |
There was a problem hiding this comment.
In a normal Frontend+KV deployment, KvIndexerMetrics::from_component registers the ckf_mutation_total suffix through the runtime component registry, which exposes it as dynamo_component_ckf_mutation_total; dynamo_kvrouter_ckf_mutation_total is produced only by the standalone indexer's new_registered path. As written, PromQL copied from this main router catalog returns no series for embedded routers. Document the component-prefixed name here and note the standalone-prefixed variant separately, as the catalog already does for kv_cache_events_applied.
AGENTS.md reference: docs/fern/AGENTS.md:L8-L9
Useful? React with 👍 / 👎.
Brings this cherry-pick in line with the main-side PR that is being taken for
6572846, so the release branch does not ship without the parts only that PR
carried.
--benchmark-timeout was documented as default 300 with hard-limit semantics
("worker startup fails if this limit is exceeded"). The parser declares
benchmark_timeout: int = 900, and the limit is soft: the measured iteration
finishes, partial results are returned, and engine startup continues, with only
a bounded cleanup grace failing closed. The release branch carried the same
wrong text as main.
Also qualifies the deprecated granularity flags. They are read only when
--benchmark-mode is set and --benchmark-points-file is not, so the translation,
range check, and deprecation warning described in that section do not happen at
all in the common case. Adds the startup ValueError raised when a legacy flag is
passed together with its replacement.
Signed-off-by: Dan Gil <dagil@nvidia.com>
Three defects this PR introduced, each flagged independently by two review bots and confirmed against the code on release/1.4.0. The nvext disable paragraph listed four routing-override headers and dropped the tenant header the previous text had named. apply_header_routing_overrides sets cache_salt from x-tenant-id (extensions.rs:344, documented at :303), and the handler skips that whole function when the protocol is off, so per-tenant cache isolation stops applying. An operator reading the list as exhaustive would assume gateway-controlled tenant salting still held. Reworded to "every routing-override header" with the tenant header and its consequence named first, so an incomplete enumeration cannot mislead again. The CKF counter was documented as dynamo_kvrouter_ckf_mutation_total under the frontend and router metrics. That name comes only from new_registered, the standalone indexer path. An embedded router registers through from_component, which passes the bare suffix and yields dynamo_component_ckf_mutation_total, so PromQL copied from this page returned no series on a normal Frontend+KV deployment. Now documents the component name with the standalone variant noted alongside, matching how this same file already handles kv_cache_events_applied four lines above. The Kalman and Prophet predictor blocks omitted optimization_target while this PR newly declared each block a standalone PlannerConfig. planner_config.py:708 gates throughput scaling on optimization_target == "sla", so both examples ran successfully with all their tuning inert. Signed-off-by: Dan Gil <dagil@nvidia.com>
Re-snapshot docs/ from main (46 further commits since the first snapshot), re-apply the release ledger at the new paths - the 1.4.0 image-tag pins, the shipped-behavior reference sync (#12983), the CKF metric caveats (#13149), and the UCX/EFA facts (#13049) - and sync docs CI plus both READMEs from main so no link targets the deleted old layout. Promote v1.4.0 to current release across the docs values: releases.data.ts (CURRENT_* consts, v1.4.0 release entry with shipped pins SGLang 0.5.16 / TRT-LLM 1.3.0rc22 / vLLM 0.26.0 / NIXL 1.3.0-1.3.2 / UCX 1.21.x, CUDA rows, artifact tags) with regenerated reference pages, and bump copy-paste container tags in install guides (EN/zh-CN), READMEs, profiler and guide examples. Recipe-mirroring pages keep their deliberate validation pins. Signed-off-by: Dan Gil <dagil@nvidia.com>
2 participants
Summary
Adapted port of #12975 (6571617) and #12979 (6572846) onto
release/1.4.0. Closes both docs P0s.Why this is not a straight cherry-pick
Two reasons, both worth review attention:
mainafter the branch cut (docs: restructure docs site into tab-based navigation #10855), so every path differs.components/frontend/nvext.mdhere ispages/developer-guide/additional-resources/nvidia-request-extensions-nvext.mdthere.mainwith no commit behind them, so there is nothing to pick — they are applied by hand here: the planner engine-capacity metric names, the Prometheus CA-bundle description in two files, and the planner CLI examples.6571617 — router docs
nvext.mdDYN_ENABLE_FRONTEND_NVEXTdoes not exist. The switch isDYN_DISABLE_FRONTEND_NVEXT(lib/runtime/src/config/environment_names.rs), so following the old text was a silent no-op:nvextand the routing-override headers stayed activerouter-examples.mdkv_state_endpointWorkerConfigparameter and its serving-endpoint defaultmetrics-catalog.mdxdynamo_kvrouter_ckf_mutation_totalmetric-labels.mdxallocation_failedandownership_degree_overflow(the list carried 6 of 8), plus the CKFoutcomelabelrouter-guide.md6572846 — planner and vLLM docs
planner/README.mdengine_prefill_capacity_requests_per_secondandengine_decode_capacity_requests_per_second; the documented names never existedplanner-config-reference.mdx,dgdr-reference.mdxssl_verify, the opposite of what both pages said (planner_config.py)planner-examples.md--config planner.yaml, not the eight CLI flags the examples used. Converted toPlannerConfigYAML, matching mainvllm-config-reference.mdx--benchmark-*-granularityaliases, whichbackend_args.pyregisters as deprecated withdefault=NoneVerification
Every item was checked against this branch's source before editing, and re-checked after. Documentation only, no behavior change.
🤖 Generated with Claude Code