feat(trtllm): publish Dynamo engine metrics by default

docs/config-reference.md

@@ -639,9 +639,43 @@ backend:
 | `prefill_environment` | dict   | {}      | Environment variables for prefill       |
 | `decode_environment`  | dict   | {}      | Environment variables for decode        |
 | `trtllm_config`       | object | null    | TRTLLM CLI configuration per mode       |
+| `publish_metrics` | bool | true | Pass `--publish-metrics` to Dynamo TRT-LLM workers; does not enable KV events |
+| `publish_events_and_metrics` | bool or null | unset | `false`: disable both publication flags; `true`: enable the combined flag; unset/null: inherit defaults |
+
+With `frontend.type: dynamo`, prefill, decode, and aggregated workers publish engine metrics
+by default using `--publish-metrics`, regardless of whether observability is enabled. Without
+observability, this does not enable KV events. `observability.enabled: true` retains its existing
+superset behavior: it additionally enables `--publish-events-and-metrics` when the combined
+setting is omitted or null. Explicitly requesting the combined flag also works without
+observability.
+
+**An explicit `backend.publish_events_and_metrics: false` is a master opt-out:** neither
+publication flag is passed, even if `publish_metrics` is true or observability is enabled.
+This differs from omitting the combined setting, which keeps metrics on by default. Unset
+values remain null through config serialization so a saved config does not acquire an opt-out.
+
+| `publish_events_and_metrics` | Observability | Default publication flags |
+| --- | --- | --- |
+| omitted / null | disabled / omitted | `--publish-metrics` |
+| omitted / null | enabled | `--publish-metrics --publish-events-and-metrics` |
+| `false` | either | none |
+| `true` | either | `--publish-metrics --publish-events-and-metrics` |
+
+**Compatibility:** the metrics-only flag requires a Dynamo build containing
+[ai-dynamo/dynamo#12162](https://github.com/ai-dynamo/dynamo/pull/12162) or equivalent support.
+Older builds (including Dynamo v1.4.2) reject the flag. Set `backend.publish_metrics: false`
+to omit only the new flag, including when observability is enabled; this does not disable a
+combined flag enabled by observability or the recipe. Set `backend.publish_events_and_metrics: false`
+to omit **both** flags. srt-slurm does not substitute the combined flag as an automatic
+compatibility fallback, because that would enable KV events. Omitting the flag does
+not override metrics-related environment variables supplied by the user. Metrics collection
+adds engine telemetry work; metrics-only does not mean zero overhead.
+
+These options do not change native `trtllm_serve` or sidecar worker commands. `srtctl dry-run`
+shows the publication flag selected for Dynamo TRT-LLM workers.
 
 **Key differences from SGLang backend**:
-- No aggregated mode support (prefill/decode only)
+- Supports prefill, decode, and aggregated workers
 - Uses MPI-style launching (one srun per endpoint with all nodes)
 - Uses `trtllm-llmapi-launch` for distributed launching
 - Automatically sets `TRTLLM_EPLB_SHM_NAME` with unique UUID per endpoint
@@ -739,9 +773,14 @@ their URLs are appended to `AIPERF_SERVER_METRICS_URLS` after the logical worker
 
 Two caveats for `AIPERF_SERVER_METRICS_URLS`:
 
-- **TRT-LLM worker URLs are omitted when the workers publish no metrics.** A Dynamo TRT-LLM worker
-  launched without `--publish-events-and-metrics` (the default; `observability.enabled` turns it
-  on) serves nothing on its sys-port `/metrics`, so those URLs are not advertised. With
+- **Dynamo TRT-LLM worker URLs are advertised when engine metrics are enabled.** This is the
+  default via `backend.publish_metrics: true` (`--publish-metrics`) when the combined setting
+  is omitted; `backend.publish_events_and_metrics: true` also enables them. Explicit
+  `backend.publish_events_and_metrics: false` suppresses both publication flags and worker
+  URLs, regardless of the metrics-only setting. URLs are also omitted when no flag is enabled
+  (for example, metrics-only false and the combined setting omitted without observability).
+  This applies to built-in AIPerf and custom benchmarks, excluding sidecars, whose behavior is
+  unchanged. Runtime-only metrics may still exist but do not constitute an engine-metrics capture. With
   `frontend.type: trtllm_serve` the gate is the worker's own engine config instead: its
   `/prometheus/metrics` URL is advertised when that mode's `return_perf_metrics` is true (the
   srtctl default for trtllm_serve recipes; an explicit `false` drops the URL). KVBM URLs are

src/srtctl/backends/trtllm.py

@@ -105,12 +105,16 @@ class TRTLLMProtocol:
     # engine does not recognise.
     served_model_name: str | None = None
 
-    # Whether dynamo.trtllm workers pass `--publish-events-and-metrics`.
-    # Enables the worker to publish KV-cache events (add/evict) + metrics, which
-    # the dynamo frontend consumes for KV-cache-aware routing (router-mode: kv).
-    # This may impact performance so should be disabled if exact KV aware routing
-    # is not needed.
-    publish_events_and_metrics: bool = False
+    # Publish TRT-LLM engine metrics without enabling KV-cache events.
+    # Requires a Dynamo build supporting --publish-metrics; set False to omit
+    # the flag for older builds. Native trtllm-serve and sidecars are unaffected.
+    publish_metrics: bool = True
+
+    # None means unspecified: metrics default on, events off (observability
+    # promotes this to True). Explicit False is a master opt-out of BOTH
+    # publication flags, even when publish_metrics is True. Preserve None in
+    # schema round-trips so an omitted value never becomes an explicit opt-out.
+    publish_events_and_metrics: bool | None = None
 
     # Controls batched startup of workers that share the same node.
     # 0 = start all workers in parallel (no constraint).
@@ -145,6 +149,18 @@ class TRTLLMProtocol:
 
     Schema: ClassVar[builtins.type[Schema]] = Schema
 
+    @property
+    def dynamo_metrics_flags(self) -> tuple[str, ...]:
+        """Effective publication flags, preserving the explicit legacy opt-out."""
+        if self.publish_events_and_metrics is False:
+            return ()
+        flags = []
+        if self.publish_metrics:
+            flags.append("--publish-metrics")
+        if self.publish_events_and_metrics:
+            flags.append("--publish-events-and-metrics")
+        return tuple(flags)
+
     # =========================================================================
     # BackendProtocol Implementation
     # =========================================================================
@@ -373,8 +389,7 @@ def build_worker_command(
             ]
         )
 
-        if self.publish_events_and_metrics:
-            cmd.append("--publish-events-and-metrics")
+        cmd.extend(self.dynamo_metrics_flags)
 
         return self._wrap_with_numa_cpu_bind(cmd)
 

src/srtctl/cli/mixins/benchmark_stage.py

@@ -638,14 +638,25 @@ def _get_aiperf_server_metrics_env(
         ranks are not advertised as separate engines.
         """
         urls: list[str] = []
+        dynamo_trtllm_metrics_disabled = (
+            self.config.frontend.type == "dynamo"
+            and self.config.backend_type == "trtllm"
+            and not (
+                (not self.config.dynamo.sidecar and getattr(self.config.backend, "dynamo_metrics_flags", ()))
+                or getattr(self.config.backend, "publish_events_and_metrics", False)
+            )
+        )
         # trtllm-serve serves Prometheus at /prometheus/metrics on the worker
         # OpenAI port (GET /metrics there is JSON iteration stats, not
         # exposition text); every other frontend serves it at /metrics.
         metrics_path = "/prometheus/metrics" if self.config.frontend.type == "trtllm_serve" else "/metrics"
         if logical_workers_only:
             if logical_endpoints is None:
                 logical_endpoints = self._logical_worker_endpoints()
-            urls = [f"http://{host}:{port}{metrics_path}" for _, host, port in logical_endpoints]
+            # Sidecars use native worker commands, so publish_metrics does not
+            # control their existing logical-worker URL discovery.
+            if self.config.dynamo.sidecar or not dynamo_trtllm_metrics_disabled:
+                urls = [f"http://{host}:{port}{metrics_path}" for _, host, port in logical_endpoints]
         else:
             if self.config.frontend.type in {"vllm", "vllm-router"}:
                 for process in self.backend_processes:
@@ -676,15 +687,13 @@ def _get_aiperf_server_metrics_env(
                         continue
                     host = get_hostname_ip(process.node, self.runtime.network_interface)
                     urls.append(f"http://{host}:{process.http_port}{metrics_path}")
-            # TRT-LLM workers only publish engine metrics when launched with
-            # --publish-events-and-metrics (pre-v1.3.0 Dynamo gates the whole
-            # worker /metrics surface on it; observability.enabled sets it at
-            # config load). Without the flag the sys-port endpoints serve
-            # nothing, so advertising them would only create the impression
-            # that worker metrics are being captured.
-            elif self.config.backend_type != "trtllm" or getattr(
-                self.config.backend, "publish_events_and_metrics", False
-            ):
+            # Dynamo TRT-LLM engine metrics require either the metrics-only
+            # flag (the default) or the legacy combined flag (also enabled by
+            # observability). Retain the existing sidecar gate because sidecars
+            # do not receive --publish-metrics. An explicit legacy False disables
+            # both flags. Runtime-only metrics may still exist with publication disabled,
+            # but must not be advertised as an engine-metrics capture.
+            elif not dynamo_trtllm_metrics_disabled:
                 for process in self.backend_processes:
                     if process.sys_port > 0:
                         host = get_hostname_ip(process.node, self.runtime.network_interface)

src/srtctl/cli/submit.py

@@ -218,6 +218,28 @@ def show_config_details(config: SrtConfig) -> None:
     environment variables (global and backend per-mode) so users can verify their
     config is correct before submitting.
     """
+    if config.frontend.type == "dynamo" and not config.dynamo.sidecar:
+        from srtctl.backends.trtllm import TRTLLMProtocol
+
+        if isinstance(config.backend, TRTLLMProtocol):
+            descriptions = {
+                "--publish-metrics": "metrics only",
+                "--publish-events-and-metrics": "metrics and KV events",
+            }
+            publication = [f"{flag} ({descriptions[flag]})" for flag in config.backend.dynamo_metrics_flags]
+            disabled_by = (
+                "publish_events_and_metrics"
+                if config.backend.publish_events_and_metrics is False
+                else "publish_metrics"
+            )
+            console.print(
+                Panel(
+                    "\n".join(publication) or f"No publication flag (backend.{disabled_by}: false)",
+                    title="Dynamo TRT-LLM Metrics",
+                    border_style="cyan",
+                )
+            )
+
     if config.frontend.type == "vllm":
         from srtctl.backends.vllm import VLLMProtocol, find_vllm_orchestration_recipe_flags
 

src/srtctl/core/config.py

@@ -624,9 +624,10 @@ def expand_observability(cfg: dict) -> dict:
     One knob, six effects -- see :class:`~srtctl.core.schema.ObservabilityConfig`
     for the rationale and the full list. Mutates ``cfg`` in place and returns it.
 
-    Every write is a ``setdefault``: an explicit value in the recipe always
-    wins. That makes it safe to flip ``observability.enabled`` on globally
-    without silently overriding a recipe that deliberately disabled something.
+    Defaults preserve explicit recipe values. The tri-state combined publishing
+    setting treats null as unset, while explicit False remains a master opt-out.
+    Enabling observability never overrides a recipe that deliberately disables
+    publication.
 
     No-op unless ``observability.enabled`` is truthy.
     """
@@ -662,23 +663,20 @@ def expand_observability(cfg: dict) -> dict:
     # keyed by x_request_id so all three legs join on one id.
     _setdefault_nested(frontend, "env", ANALYTICS_REQUEST_TRACE_ENV)
 
-    # --- metrics leg: the /metrics surface and what appears on it ------------
-    # publish_events_and_metrics is what creates the endpoint; without it the
-    # engine-config keys below have nowhere to publish to.
+    # --- metrics leg: engine metrics on the worker /metrics surface ----------
+    # Metrics-only publication defaults on independently of observability.
+    # Keep observability as the existing superset that also enables KV events.
     if backend.get("type", "sglang") == "trtllm":
-        # An explicit False here defeats the whole metrics leg -- no /metrics
-        # surface means no KV-cache gauges for anyone, including the in-job
-        # scraper. setdefault still lets the recipe win (that contract matters),
-        # but say so loudly: a recipe written before this knob existed will
-        # otherwise silently produce a run with half the data missing.
-        if backend.get("publish_events_and_metrics") is False:
+        # None preserves an omitted setting through schema dumps; treat it as
+        # unset here too. An explicit False must remain the master opt-out.
+        if backend.get("publish_events_and_metrics") is None:
+            backend["publish_events_and_metrics"] = True
+        if frontend.get("type", "dynamo") == "dynamo" and backend["publish_events_and_metrics"] is False:
             logger.warning(
-                "observability.enabled but backend.publish_events_and_metrics is "
-                "explicitly false — workers will NOT expose /metrics, so KV-cache "
-                "gauges and worker scrapes will be missing. Remove that line or "
-                "set it true to get the full analytics capture."
+                "observability.enabled but backend.publish_events_and_metrics is explicitly false "
+                "— srt-slurm will enable neither metrics nor KV-event publication. "
+                "This opt-out takes precedence over backend.publish_metrics."
             )
-        backend.setdefault("publish_events_and_metrics", True)
 
         trtllm_config = backend.get("trtllm_config")
         if not isinstance(trtllm_config, dict):

src/srtctl/core/schema.py

@@ -1177,8 +1177,10 @@ class ObservabilityConfig:
     having to remember six independent flags. It expands (at config-load time,
     via :func:`srtctl.core.config.expand_observability`) into:
 
-    * ``backend.publish_events_and_metrics: true`` -- the worker/frontend
-      Prometheus ``/metrics`` surface exists at all.
+    * ``backend.publish_events_and_metrics: true`` -- enable KV-cache events
+      and TRT-LLM engine metrics. Metrics-only publication already defaults on
+      independently via ``backend.publish_metrics``. An explicit
+      ``publish_events_and_metrics: false`` disables both publication flags.
     * ``enable_iter_perf_stats`` + ``return_perf_metrics`` on every engine
       config -- the ``trtllm_kv_cache_*`` occupancy gauges and per-request
       histograms appear on that surface.
@@ -1191,16 +1193,17 @@ class ObservabilityConfig:
       client does not already poll (see ``TelemetryStageMixin.start_tachometer``
       and ``tachometer`` below).
 
-    Every expansion uses setdefault semantics: an explicit value in the recipe
-    always wins, so ``observability.enabled`` is safe to switch on globally.
+    Expansion preserves explicit recipe values; the tri-state combined
+    publishing setting treats null as unset. Explicit False is never replaced.
 
     Scope is deliberately server-side. The knob configures what the workers and
     frontend *emit*, and captures that surface by scraping the endpoints
     directly. It never asks the benchmark client to re-export what the servers
     already publish. (One indirect exception: on TRT-LLM the client's
     ``AIPERF_SERVER_METRICS_URLS`` worker list exists only when
-    ``publish_events_and_metrics`` gives those endpoints content, and this knob
-    is one way that flag gets set — see ``BenchmarkStageMixin``.)
+    the effective publication flags give those endpoints engine metrics,
+    respecting the explicit combined-setting opt-out — see
+    ``BenchmarkStageMixin``.)
 
     It does **not** decide whether the component perf dashboard is built. That
     happens on every run (see :mod:`srtctl.analysis.perf_dashboard`); ``enabled``