Add speculative-decoding (MTP) recording rules and Grafana dashboard

[Kaushalya]

Summary

Extends the existing Prometheus/Grafana metrics stack (added in #2) with speculative-decoding / MTP observability: new Prometheus recording rules plus a provisioned Grafana dashboard that shows draft acceptance and per-position MTP stats.

Changes

prometheus-rules.yaml

New vllm-spec-decode rule group (5s interval, 1m rate window, {job="vllm"} filter — same conventions as the existing vllm-throughput group):

Recorded metric	Meaning
vllm:spec_decode_acceptance_rate:rate1m	Fraction of draft tokens accepted (0..1)
vllm:spec_decode_mean_accepted_length:rate1m	Tokens emitted per verification step, including the bonus token
vllm:spec_decode_draft_tokens_per_second:rate1m	Draft tokens proposed per second
vllm:spec_decode_accepted_tokens_per_second:rate1m	Draft tokens accepted per second
vllm:spec_decode_acceptance_rate_by_pos:rate1m	Per-position acceptance rate (position label preserved)

grafana/dashboards/vllm-spec-decode.json (new)

Provisioned vLLM Speculative Decoding dashboard (uid vllm-spec-decode, tag spec-decode), loaded into the vLLM folder by the existing file provider. Five panels, styled consistently with vllm-throughput.json (datasource uid: prometheus, 5s refresh, 15m window):

1. Draft acceptance rate (1-minute average) — Stat, percent
2. Mean accepted length (incl. bonus token) — Stat
3. Draft tokens / second — Stat
4. Draft vs accepted tokens / second (1-minute average) — Time series; the gap is wasted draft compute
5. Acceptance rate by draft position (1-minute average) — Bar chart with Position {{position}} legend

docs/METRICS.md

• Stack overview now lists the spec-decode recording rules and the new dashboard file.
• "Speculative decoding and MTP panels" section rewritten to lead with the provisioned dashboard and rule names, documents the idle "No value" behavior and the Grafana restart needed when dashboard files change on disk.
• The raw-counter PromQL examples are kept as the equivalent direct queries for custom panels (e.g. longer rate windows).

Usage

1. Start (or re-run) the stack: docker compose -f docker-compose.metrics.yaml up -d — Prometheus hot-reloads the changed rules file; verify the new group at :9090/rules.
2. Restart Grafana so the file provider picks up the new dashboard: docker compose -f docker-compose.metrics.yaml restart grafana.
3. Open vLLM Speculative Decoding in the vLLM folder.

Behavior notes

• Panels show No value while vLLM is idle — the rules evaluate to NaN with no traffic, which avoids a misleading 100% acceptance rate at idle.
• With N speculative tokens (e.g. 3 for an MTP head), Mean accepted length ranges from 1.0 to N+1; the by-position chart shows which of the N positions actually contribute.
• Acceptance rate is not speedup; to size MTP's benefit, benchmark the same workload with and without --num-speculative-tokens and compare the generation tokens/s rule.
• Counter names follow docs/METRICS.md (..._num_draft_tokens_total, ..._num_drafts_total, ..._num_accepted_tokens_per_pos_total). Builds exposing different vLLM spec-decode counter names would need the rule metrics adjusted.

Testing

• prometheus-rules.yaml validated (YAML parse); all five rules confirmed loaded without errors at :9090/rules against the running stack.
• Dashboard JSON validated; every panel query was checked to resolve to one of the loaded recorded metrics.
• End-to-end panel data requires a speculative-decoding-enabled server producing traffic (two 5s scrapes after generation starts).

[Copilot]

Pull request overview

Adds speculative-decoding observability to the Prometheus/Grafana stack.

Changes:

• Adds five speculative-decoding recording rules.
• Adds a provisioned Grafana dashboard.
• Documents metrics, panels, and reload procedures.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
prometheus-rules.yaml	Defines speculative-decoding metrics.
grafana/dashboards/vllm-spec-decode.json	Adds the visualization dashboard.
docs/METRICS.md	Documents setup and metric usage.

Suppressed comments (2)

grafana/dashboards/vllm-spec-decode.json:104

• Like the acceptance-rate stat, lastNotNull over a range keeps displaying the previous mean length after the recording rule stops emitting samples. Use an instant query so this panel shows No value once the one-minute draft rate reaches zero.

          "range": true,

docs/METRICS.md:116

• This update also changes a bind-mounted Prometheus rule file, but Prometheus does not watch rule files and the Compose service does not enable the lifecycle reload endpoint. On an already-running stack, compose up -d leaves the container unchanged, so restarting only Grafana produces a dashboard whose recording metrics do not exist. Document a Prometheus restart as part of applying rule changes.

If you add or replace dashboard files on disk, restart the Grafana container
for the file provider to reload them:

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Kaushalya]

Confirmed — the blanket wording was still incorrect after the instant-query fix: only the guarded ratio rules (acceptance rate, mean accepted length, per-position) evaluate to no value at idle; the two unguarded token-rate rules keep recording zero, so those panels show 0 rather than No value. Fixed in b66a553, which splits the documented behavior by rule type. Verified against live Prometheus data: during a 46-minute true-idle window (target up, counters flat) the draft/accepted token-rate rules recorded 0.0 while the ratio rules produced no samples.

[Kaushalya]

Summary of the fixes in this round (4b62d71, cd5814c, 1238aaa):

Recipe: MTP speculative decoding enabled (4b62d71)

• recipes/qwen3.8-27b-nvfp4.yaml now launches with --speculative-config '{"method":"mtp","num_speculative_tokens":3}', so the panels in this dashboard have data when running the recipe. Requires the vLLM build with the Qwen GDN speculative-decoding fixes ([Bugfix] Align Qwen GDN gates with speculative tokens vllm-project/vllm#51812, #51674).

Review comment: stale stat values at idle (cd5814c) — confirmed valid, fixed

• The stat panels used range queries reduced with lastNotNull over the default 15-minute window. The rules' idle guard (and on() (rate > 0)) stops samples once traffic stops, but the last samples written while active stayed inside the window, so the stats kept showing the old acceptance rate / mean accepted length for up to ~15 minutes, contradicting the "No value" behavior documented in docs/METRICS.md.
• The stat panels are now instant queries (the unguarded draft-tokens/s stat was converted too, for consistency), so the stats go No value once the last recorded sample leaves Prometheus' five-minute lookback window. The docs/METRICS.md paragraph now describes this behavior.
• Verified against live Prometheus data: during a 17-minute idle gap, the 15-minute range query still returned the stale 0.5733 while the instant query at the same time was empty.

Review comment: per-position panel (1238aaa) — confirmed valid, fixed

• xTickLabel is not a Bar chart option (the schema only defines xTickLabelMaxLength/xTickLabelRotation/xTickLabelSpacing), so it was ignored.
• The bar chart resolves its category axis as: X-field option → first string field → time field. Prometheus frames carry no string field, so the x-axis stayed time-based and the bars remained unlabeled with the legend hidden — still true after the query was made instant, so the earlier fix was incomplete here.
• The panel is now the dedicated Bar gauge panel (the comment's second suggestion), keeping the instant query and Position {{position}} legend: one labeled bar per draft position on a 0–100% track.
• The categorical-transform alternative was not used because the fieldToLabels transform referenced in older Grafana docs does not exist in the Grafana version this stack runs (verified against the app bundles); the Bar gauge is the version-safe option.
• docs/METRICS.md's per-position section now describes the Bar gauge rendering.

Both dashboards were validated end-to-end against the live Prometheus + Grafana stack: all vllm:spec_decode_* recording metrics return live values and every panel renders with data.