feat(trtllm): keep enable_iter_perf_stats off by default in every engine config

[nv-yna]

Summary

Bake enable_iter_perf_stats: false into every TRT-LLM engine section a recipe uses (prefill and decode, or aggregated; created when the recipe has none), under both frontend.type: dynamo and trtllm_serve, unless the recipe or observability sets the key.

Why an explicit false when TensorRT-LLM's own default is already false: srtctl passes --publish-metrics to Dynamo TRT-LLM workers by default (#421), and dynamo.trtllm turns that flag into enable_iter_perf_stats: true in the engine arguments (components/src/dynamo/trtllm/workers/llm_worker.py), so every worker collected TensorRT-LLM's per-iteration statistics (KV-cache stats and CUDA-event step timing on every executor loop) for gauges no benchmark client reads. The engine YAML is merged over the derived arguments and wins on conflicts (TensorRT-LLM update_llm_args_with_extra_dict), so the explicit key is what turns the statistics off. The request-level trtllm_* Prometheus series (request latency, TTFT, TPOT, queue/prefill/decode time, token counters) come from the per-request perf metrics, which --publish-metrics sets on the Dynamo path and return_perf_metrics: true sets for trtllm-serve, and are unaffected.

• New expand_trtllm_engine_defaults() applies TRTLLM_ENGINE_DEFAULTS with setdefault semantics: an explicit enable_iter_perf_stats: true in the recipe wins.
• expand_observability() now creates the engine sections for the modes in use instead of only touching existing ones, so its true still wins for a recipe with no engine yaml. It warns when it meets an explicit false, which is also what a saved or locked recipe carries after a load/dump round trip.
• New expand_engine_config_defaults() runs observability, trtllm-serve and engine defaults in order. load_config, SrtConfig.from_yaml and the override submit path all use it; the override path previously loaded the schema with no expansion step, so srtctl dry-run and the in-job loader disagreed.
• expand_trtllm_serve_defaults() shares the new _setdefault_trtllm_engine_keys helper. A non-mapping trtllm_config (or role section) is left alone so schema validation reports it instead of being replaced.
• Sections whose backend is the legacy tensorrt / trt engine are skipped: enable_iter_perf_stats is a TorchLlmArgs-only field, so containers older than TensorRT-LLM v1.3.0rc21 reject it, and that backend always collected the statistics anyway.
• srtctl dry-run shows a "TRT-LLM Engine Statistics" panel with the per-role enable_iter_perf_stats / return_perf_metrics values for every TRT-LLM backend.
• Docs: "Iteration statistics default" in config-reference.md, the trtllm-serve worker-metrics note, and the regenerated schema-reference.md row for publish_metrics.

Default behavior

Recipe	enable_iter_perf_stats in the engine YAML
Key omitted; observability absent or disabled	false (section created when missing)
Key omitted; observability.enabled: true	true (observability expands first)
Explicit true on a role	true for that role
Explicit false on a role with observability.enabled: true	false, with a load-time warning
Role backend: tensorrt / trt	untouched
Non-TRT-LLM backend	untouched

What changes on a default run

The iteration-level trtllm_* gauges (trtllm_kv_cache_*, running/waiting requests, iteration latency) are no longer produced, and on Dynamo the dynamo_component_kvstats_* gauges, the router worker-load sample and the Planner's forward-pass metrics stay at their seeded zero. No benchmark client reads them. The component dashboard's engine-tab KV-cache utilisation and hit-rate panels show no data, and the Dynamo bench dashboard's KV-utilisation series sits at the seeded 0 %; a follow-up could teach the dashboard to render an all-zero component gauge as "not captured". Opt back in per role with roles.<role>.args.enable_iter_perf_stats: true, or with observability.enabled: true.

Compatibility

Recipes that already set the key are unchanged. Observability runs are unchanged where the recipe has engine sections, and gain the iteration-level gauges where it has none. The trtllm-serve /prometheus/metrics route depends on return_perf_metrics only and is unchanged. The override submit path now applies the same expansions as load_config, which also means observability flags reach its dry-run output for the first time.

Validation

• make check: ruff passed, schema-docs-check up to date, 2431 passed, 2 skipped, 6 deselected (30 new tests in test_observability.py, test_dry_run.py, test_override.py, covering both frontends, section creation, explicit true/false precedence, observability ordering, load/dump/reload, the v2 roles: layout, the override dry-run path, non-mapping values and the legacy-backend skip).
• ty check src/srtctl/: 34 diagnostics, down from 36 on main; none in changed lines.
• The Dynamo and TensorRT-LLM merge behavior this relies on was verified by reading ai-dynamo/dynamo main (llm_worker.py: enable_iter_perf_stats derived from publish_metrics, then update_llm_args_with_extra_options(arg_map, config.extra_engine_args) with no explicit CLI keys) and TensorRT-LLM llm_args.py (llm_args = llm_args | llm_args_dict).
• Not run: a live SLURM job. No GPU jobs were launched and no existing run artifacts were modified.

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

❌ Patch coverage is 98.46154% with 1 line in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@00ee7f3). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
src/srtctl/core/config.py	98.00%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #446   +/-   ##
=======================================
  Coverage        ?   79.85%           
=======================================
  Files           ?      117           
  Lines           ?    17395           
  Branches        ?        0           
=======================================
  Hits            ?    13890           
  Misses          ?     3505           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.