Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/backends/trtllm/trtllm-observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ Metric name constants are defined in `lib/runtime/src/metrics/prometheus_names.r

These metrics are only recorded in disaggregated (prefill + decode) deployments when a KV cache transfer actually occurs. They are sourced from TensorRT-LLM's `RequestPerfMetrics.timing_metrics`.

- `trtllm_kv_transfer_success_total` (Counter) — Total number of successful KV cache transfers (recorded on prefill side)
- `trtllm_kv_transfer_success_total` (Counter) — Total number of successful KV cache transfers (recorded on the decode worker, when it observes non-zero KV-transfer timing in `RequestPerfMetrics.timing_metrics`). Grows in lock-step with the `_count` of the sibling `trtllm_kv_transfer_latency_seconds` / `trtllm_kv_transfer_bytes` / `trtllm_kv_transfer_speed_gb_s` histograms for the same transfer events.
- Labels: `model_name`, `disaggregation_mode`, `engine_type`
- `trtllm_kv_transfer_latency_seconds` (Histogram) — KV cache transfer latency per request in seconds
- Labels: `model_name`, `disaggregation_mode`, `engine_type`
Expand Down
6 changes: 5 additions & 1 deletion docs/components/planner/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,11 @@ See [Planner Guide](planner-guide.md) for the full workflow.

Load-based scaling has the following known limitations. Throughput-based scaling is not affected by any of these.

**Requires ForwardPassMetrics (FPM).** Load-based scaling uses per-engine per-iteration metrics delivered via the Dynamo event plane (ForwardPassMetrics). FPM is currently only available for vllm and is automatically enabled when the engine uses `InstrumentedScheduler` and `DYN_FORWARDPASS_METRIC_PORT` is set. The KV Router is **not** required for load-based scaling.
**Requires ForwardPassMetrics (FPM).** Load-based scaling uses per-engine per-iteration metrics delivered via the Dynamo event plane (ForwardPassMetrics). The KV Router is **not** required for load-based scaling. FPM availability by backend:

- **vLLM** — supported. Automatically enabled when the engine uses `InstrumentedScheduler` and `DYN_FORWARDPASS_METRIC_PORT` is set.
- **TensorRT-LLM** — supported for non-attention-DP workers (`attention_dp_size == 1`); gated off when `attention_dp_size > 1` pending per-rank FPM emission.
- **SGLang** — pipeline wired in Dynamo, but the upstream SGLang FPM module is not included in the current 1.2.0 SGLang runtime image. See the [SGLang FPM section](../../backends/sglang/sglang-observability.md#forward-pass-metrics-fpm) for the runtime-image prerequisite.

### General

Expand Down
2 changes: 1 addition & 1 deletion docs/kubernetes/model-deployment-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -313,7 +313,7 @@ backend explicitly in these cases:
| MoE models (DeepSeek-R1, Qwen3-MoE) | `sglang` (full MoE support) |
| Using `searchStrategy: thorough` | Any except `auto` (required) |
| TensorRT-LLM compilation caching | `trtllm` (add a compilation cache PVC) |
| Need load-based planner scaling (FPM) | `vllm` (only backend with ForwardPassMetrics) |
| Need load-based planner scaling (FPM) | `vllm` (any config) or `trtllm` (non-attention-DP only). SGLang FPM is wired in Dynamo but the upstream module is not in the 1.2.0 runtime image. |

> [!WARNING]
> TensorRT-LLM does not support Python 3.11. If your environment uses
Expand Down
Loading