fix(trtllm): decouple KV events from performance metrics

[alec-flowers]

Summary

• stop --publish-kv-events from implicitly setting TensorRT-LLM's return_perf_metrics and enable_iter_perf_stats
• initialize TensorRT-LLM performance collectors from the final explicit engine configuration instead of the KV-event flag
• keep KV router cache-load telemetry with the event publisher
• document independent YAML and JSON configuration for TRT-LLM performance metrics
• add focused regression coverage for the engine-argument boundary

Why

The legacy internal publish_events_and_metrics boolean controlled both KV-event publication and TensorRT-LLM performance statistics. This made a KV-event A/B also enable unrelated iteration and request metric work, contaminating performance measurements—particularly the DeepSeek-V4 KV-event benchmark that motivated this change.

Users can still enable performance collection explicitly through --extra-engine-args or --override-engine-args with:

return_perf_metrics: true
enable_iter_perf_stats: true

Validation

• git diff --check: passed
• Python compilation for the changed modules: passed
• pre-commit isort, black, flake8, codespell, ruff, and repository file checks: passed
• focused TRT-LLM unit test collection: skipped because the local host lacks CUDA/TRT-LLM
• repository-wide pytest-marker-report hook: skipped for the commit after it failed during unrelated test collection because /tmp/pytest_port_allocations.lock is owned by nobody and not writable

Summary by CodeRabbit

• Improvements

  • Separated KV-event publishing from TensorRT-LLM performance metrics.
  • KV-event publishing no longer automatically enables performance collectors or metrics.
  • Performance metrics can now be enabled explicitly through TensorRT-LLM engine arguments.

• Documentation

  • Updated setup and observability guides with the new configuration options and behavior.
  • Clarified configuration examples for cache-aware routing and Prometheus performance data.

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-01a06452-6cb8-73a8-9ae6-1b8ee930ae99.docs.buildwithfern.com/dynamo/dev

[devin-ai-integration]

Devin Review found 1 potential issue.

[coderabbitai]

Walkthrough

Changes

TensorRT-LLM metrics separation

Layer / File(s)	Summary
Worker configuration and metric gating components/src/dynamo/trtllm/backend_args.py, components/src/dynamo/trtllm/workers/llm_worker.py	KV-event publishing no longer implicitly adds TensorRT-LLM performance flags; metrics collection is enabled only when those flags are explicitly configured.
Engine argument regression coverage components/src/dynamo/trtllm/tests/test_trtllm_unit.py	The unit test isolates engine-argument overrides and verifies performance flags are absent when no overrides are provided.
CLI and operational documentation docs/backends/trtllm/trtllm-dp-rank-routing.md, docs/backends/trtllm/trtllm-observability.md	Examples and guidance distinguish KV-event publishing from explicit performance-metrics engine configuration.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description covers summary, rationale, and validation, but it omits the required template sections and issue linkage details.	Add Overview, Details, Where should reviewer start, and the required Related Issues section with either a real Closes/Relates reference or the no-issue confirmation.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly states the main change: separating KV event publishing from TensorRT-LLM performance metrics.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

components/src/dynamo/trtllm/tests/test_trtllm_unit.py (1)

427-453: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Cover the explicit-enablement path too.

This test only proves that KV-event publishing does not add performance flags. Add cases using return_perf_metrics and enable_iter_perf_stats overrides that assert those flags remain in engine_args; otherwise an implementation that disables metrics entirely would still pass.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@components/src/dynamo/trtllm/tests/test_trtllm_unit.py` around lines 427 -
453, Add explicit-enablement cases to the test around init_llm_worker, using
return_perf_metrics and enable_iter_perf_stats overrides. Assert each explicitly
enabled flag remains present with its configured value in
EngineArgsCaptured.engine_args, while preserving the existing publish-kv-events
assertions that these flags are absent by default.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/backends/trtllm/trtllm-observability.md`:
- Line 145: Update the TensorRT-LLM observability description to state that
request-level metrics are enabled by either return_perf_metrics or
enable_iter_perf_stats, while only KV-transfer metrics additionally require
RequestPerfMetrics.timing_metrics. Replace the current “and” wording without
changing the surrounding metric or configuration guidance.

---

Nitpick comments:
In `@components/src/dynamo/trtllm/tests/test_trtllm_unit.py`:
- Around line 427-453: Add explicit-enablement cases to the test around
init_llm_worker, using return_perf_metrics and enable_iter_perf_stats overrides.
Assert each explicitly enabled flag remains present with its configured value in
EngineArgsCaptured.engine_args, while preserving the existing publish-kv-events
assertions that these flags are absent by default.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 473fcbf2-05bf-41c7-8120-386b98d95190

📥 Commits

Reviewing files that changed from the base of the PR and between daf44c6 and 83dbc71.

📒 Files selected for processing (5)

• components/src/dynamo/trtllm/backend_args.py
• components/src/dynamo/trtllm/tests/test_trtllm_unit.py
• components/src/dynamo/trtllm/workers/llm_worker.py
• docs/backends/trtllm/trtllm-dp-rank-routing.md
• docs/backends/trtllm/trtllm-observability.md

[datadog-official]

⚠️ Warnings

🚦 6 Pipeline jobs failed

Docs link check | lychee     

PR | backend-status-check     

PR | deploy-status-check     

View all 6 failed jobs.

ℹ️ Info

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 38.49% (-4.42%)

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 157e828 | Docs | Datadog PR Page | Give us feedback!}