feat(profiling): adopt the Dynamo Benchmark Playbook nsys recipe for TRT-LLM workers

.gitignore

@@ -51,6 +51,7 @@ bin/
 configs/nats-server
 configs/etcd
 configs/etcdctl
+configs/process-exporter
 configs/*.whl
 configs/*.deb
 configs/*.tar.gz

Makefile

@@ -2,6 +2,7 @@
 
 NATS_VERSION ?= v2.10.28
 ETCD_VERSION ?= v3.5.21
+PROCESS_EXPORTER_VERSION ?= 0.8.7
 LOGS_DIR ?= logs
 ARCH ?= $(shell uname -m)
 TACHOMETER_RELEASE ?= latest
@@ -128,6 +129,26 @@ setup: tachometer-scraper-download
 		echo "✅ ETCD installed to configs/etcd"; \
 	fi; \
 	echo ""; \
+	echo "--- process-exporter $(PROCESS_EXPORTER_VERSION) (Tachometer per-process/thread telemetry) ---"; \
+	if [ -f configs/process-exporter ] && file configs/process-exporter | grep -q "$$ARCH_FILE_PATTERN"; then \
+		echo "✅ process-exporter already installed at configs/process-exporter ($(ARCH))"; \
+	else \
+		echo "⬇️  Downloading process-exporter ($(PROCESS_EXPORTER_VERSION)) for $$ARCH_SHORT..."; \
+		PE_NAME="process-exporter-$(PROCESS_EXPORTER_VERSION).linux-$$ARCH_SHORT"; \
+		PE_TAR="$$PE_NAME.tar.gz"; \
+		PE_URL="https://github.com/ncabatoff/process-exporter/releases/download/v$(PROCESS_EXPORTER_VERSION)/$$PE_TAR"; \
+		if ! wget -q --show-progress --tries=3 --waitretry=5 "$$PE_URL" -O "configs/$$PE_TAR"; then \
+			rm -f "configs/$$PE_TAR"; \
+			echo "❌ Failed to download process-exporter from $$PE_URL"; \
+			exit 1; \
+		fi; \
+		echo "📁 Extracting process-exporter binary..."; \
+		tar -xzf "configs/$$PE_TAR" --strip-components=1 -C configs "$$PE_NAME/process-exporter"; \
+		chmod +x configs/process-exporter; \
+		rm "configs/$$PE_TAR"; \
+		echo "✅ process-exporter installed to configs/process-exporter"; \
+	fi; \
+	echo ""; \
 	echo "--- uv (compute node arch: $(ARCH)) ---"; \
 	if [ -f bin/uv ] && file bin/uv | grep -q "$$ARCH_FILE_PATTERN"; then \
 		echo "✅ uv already installed at bin/uv ($(ARCH))"; \

docs/config-reference.md

@@ -1277,7 +1277,7 @@ The legacy in-job Python RAW scraper is retired: a recipe still carrying `scrape
 
 The component perf dashboard is **not** configured here. It is built in post-processing on every run; `enabled` decides which capture legs exist and therefore which tabs the page carries. See [Component Performance Dashboard](component-dashboard.md).
 
-Tachometer collects every worker rank, frontend, DCGM, and node metrics by default (minus the client-polled complement described above) — the exporters launch from pinned multi-arch registry images with no configuration. Air-gapped clusters override the images via the `containers:` alias map in `srtslurm.yaml`; `default_exporters: false` disables the built-ins:
+Tachometer collects every worker rank, frontend, DCGM, node, and process metrics by default (minus the client-polled complement described above) — the exporters launch from pinned multi-arch registry images with no configuration. Air-gapped clusters override the images via the `containers:` alias map in `srtslurm.yaml`; `default_exporters: false` disables the built-ins:
 
 ```yaml
 observability:
@@ -1296,6 +1296,10 @@ observability:
     node_exporter:
       container_image: /containers/node-exporter.sqsh
       port: 9100
+    process_exporter:
+      binary: /opt/srt/configs/process-exporter   # host-native (default mode); or set container_image instead
+      container_image: ""
+      port: 9256
 ```
 
 | Tachometer field | Type | Default | Description |
@@ -1307,11 +1311,14 @@ observability:
 | `compaction_threads` | int | `4` | Value passed as `POLARS_MAX_THREADS` |
 | `storage_subdir` | string | `tachometer` | Output directory below the run log directory |
 | `extra_metadata` | dict | `{}` | Static string metadata added to every endpoint |
-| `default_exporters` | bool | `true` | Launch the built-in DCGM + node exporters when no explicit blocks are set (sweep path only) |
+| `default_exporters` | bool | `true` | Launch the built-in DCGM + node + process exporters when no explicit blocks are set (sweep path only) |
 | `dcgm_exporter` | object/null | built-in | Defaults to `nvcr.io#nvidia/k8s/dcgm-exporter:3.3.9-3.6.1-ubuntu22.04` on port 9401; an explicit block overrides |
-| `node_exporter` | object/null | built-in | Defaults to `quay.io#prometheus/node-exporter:v1.8.2` on port 9101; an explicit block overrides |
+| `node_exporter` | object/null | built-in | Defaults to `quay.io#prometheus/node-exporter:v1.8.2` on port 9101 with `--collector.disable-defaults` plus the `cpu`, `infiniband`, `meminfo`, `stat` (`node_procs_running/blocked`, context switches), `vmstat` (with `--collector.vmstat.fields` widened to include `pgsteal_*` next to `pgmajfault`), `pressure` (PSI; absent on kernels without `CONFIG_PSI`), `meminfo_numa` (`node_memory_numa_*`, kept per NUMA node by the scraper as `numa_node=N`) and `processes` (`node_processes_threads`, per-state `node_processes_state`) collectors; an explicit block overrides |
+| `process_exporter` | object/null | built-in | Defaults to the **host-native** `configs/process-exporter` binary (ncabatoff/process-exporter 0.8.7, installed by `make setup` for the compute arch, like `configs/nats-server` and `configs/etcd`) on port 9256, launched with plain `srun` (no container) on every node that hosts a backend rank or a frontend replica. Reads the host `/proc` and publishes per-process-group CPU seconds by mode, thread count, per-thread-name CPU and count (`-threads=true`), context switches, RSS and open fds. Groups (frontend, `dynamo_trtllm` / `dynamo_sglang` / `dynamo_vllm` handlers + engine ranks, launcher, client, infra daemons) come from `<log_dir>/process-exporter.yml`, written at launch. If the binary is missing the leg is skipped with a warning (submit warns too). An explicit block may set `binary` (absolute, or relative to the srtctl checkout) or instead a `container_image` with `binary` unset to run it containerized; the upstream `FROM scratch` image is not used by default because pyxis/enroot on some clusters cannot start shell-less images |
 
-`make setup ARCH=<compute_arch>` downloads and checksum-verifies the matching Tachometer binary from the latest srt-slurm release. The scraper runs as a native `srun` process on the head node; configured exporters remain containerized on worker nodes. Run `make tachometer-scraper` to build from source instead.
+Every exporter block accepts `container_image`, `port`, `command` and `binary`. `binary` selects host-native launch (the executable runs directly under `srun`, `container_image` is ignored and may be `""`); without it the exporter runs from `container_image`. One of the two must be set.
+
+`make setup ARCH=<compute_arch>` downloads and checksum-verifies the matching Tachometer binary from the latest srt-slurm release and installs the process-exporter binary for the same arch. The scraper and the process exporter run as native `srun` processes; the DCGM and node exporters remain containerized on worker nodes. Run `make tachometer-scraper` to build the scraper from source instead.
 
 Tachometer writes its Parquet stream under `<log_dir>/<storage_subdir>/raw/scrape/` (the leaf is created by the scraper itself — srtctl pre-creates only the parent, because the scraper refuses a pre-existing storage directory), compacting to `final.parquet` there on shutdown. Intermediate files remain in `<log_dir>/<storage_subdir>/local` until shutdown compaction completes. Rows carry an epoch `timestamp_ns` column, so they join directly with AIPerf records and Dynamo spans; the post-processing ingest converts the Parquet into the dashboard's `server_metrics_export.jsonl`.
 

docs/host-attribution-metrics.md

@@ -0,0 +1,80 @@
+# Host attribution metrics: pinning vs frontend placement
+
+srt-slurm's host sampler collects the telemetry needed to answer, **from a
+baseline run alone**, two questions that end-to-end serving metrics cannot:
+
+1. Are worker ranks losing CPU time to scheduler contention/migration?
+   (remedy: `backend.numa_cpu_bind: true`)
+2. Is the frontend/etcd interfering with the workers sharing its node?
+   (remedy: `frontend.dedicated_node` / `infra.etcd_nats_dedicated_node`)
+
+Both effects are real and independently worth ~1% output throughput each at
+high concurrency on GB300 disaggregated serving — but they are invisible in
+throughput/TTFT alone, which is why the collectors below exist.
+
+## Collection
+
+With `observability.enabled: true`, the `/proc` host sampler runs on the
+orchestrator node (in-process) **and on every other allocated node**
+(`observability.host_sampler_all_nodes`, default true: one persistent
+`srun --overlap` per node group running `host_sampler.py` standalone). Each
+node writes `host_samples_<node>.jsonl` into the run's log dir; the ingest
+merges them into `host_series.json` with a per-node `hosts` map. This closes
+the previous gaps: worker nodes had no per-process host telemetry, and a
+dedicated frontend node had none at all.
+
+## Metric set
+
+Per sampled process (workers, frontend, benchmark client — matched by cmdline):
+
+| Field (raw JSONL) | Ingest series | Diagnoses | Points at |
+|---|---|---|---|
+| `run_delay_ns` (`/proc/pid/schedstat`) | `run_delay_ms_per_s` | Task runnable but not running: scheduler contention on its cores | pinning |
+| `nr_migrations` (`/proc/pid/sched`) | `migrations_rate` | Cross-core churn; near-zero when pinned | pinning |
+| `affinity_ncpus` (`sched_getaffinity`) | `affinity_ncpus` | Direct pinning-state observable (144 = floating, 36 = pinned rank on GB200/GB300) | pinning (config state) |
+| `ctx_invol` (`/proc/pid/status`) | `ctx_invol_rate` | Involuntary descheduling (lock convoys, neighbor pressure) | pinning / placement |
+| `cpu_jiffies` | `cpu_pct` | Per-process CPU use — splits a shared node's load into frontend vs etcd vs ranks | placement |
+| host `procs_running/blocked` (`/proc/stat`) | `procs_runnable` | Whole-node run-queue pressure vs core count | either (localizes with the per-process rows) |
+| `t` + `t_mono` | — | Per-node clock-offset estimation; cross-node wall clocks have been observed seconds apart | metric hygiene |
+
+## Decision rubric (thresholds from the c1010 validation matrix, GB300/oci-aga)
+
+You are looking at `host_series.json` from ONE run. You do not need to know what
+"taskset" or "frontend placement" are — the rubric names the config change.
+
+**Step 1 — is the bottleneck host-CPU-side at all?**
+Look at the per-node `procs` map for the busiest process per worker node
+(highest `cpu_pct`). If every worker node shows `run_delay_ms_per_s` p50
+< 0.1 and `migrations_rate` ≈ 0, host-CPU scheduling is NOT the problem —
+stop here. (Validated: clean nodes sit at 0.00–0.01 ms/s.)
+
+**Step 2 — check the pinning state directly.**
+`affinity_ncpus` of the worker ranks equals the node's full logical-CPU count
+(e.g. 144 or 288) → the ranks are NOT pinned. Remedy:
+`backend.numa_cpu_bind: true`. Validated effect at c1010: +4.0% output
+throughput on 288-CPU GB300 nodes (+1.1% on 144-CPU nodes in the reference
+campaign — the gain grows with core count). If `affinity_ncpus` equals
+(CPUs ÷ GPUs per node), the ranks are already pinned.
+
+**Step 3 — look for the single-node asymmetry.**
+Compare each worker rank's `run_delay_ms_per_s` p50 against the median of its
+peers on other nodes. Threshold: **>10× the peer median AND >0.5 ms/s absolute,
+on exactly the node(s) that also host a non-worker process with
+`affinity_ncpus` = full width and `cpu_pct` > 1000** (the frontend: measured
+~4,100–5,000% of one core at c1010). That is co-location interference.
+Remedy: `frontend.dedicated_node: true`. Validated effect: +0.5% throughput
+(+0.85% in the reference campaign) — and the asymmetry itself is huge even
+when the throughput cost is small: measured 140–350× on the shared node,
+collapsing to 1× in all three dedicated-frontend runs.
+Note the dissociation, confirmed both ways across 7 runs: this asymmetry is
+UNCHANGED by pinning (190× with ranks pinned), and `affinity_ncpus` is
+UNCHANGED by moving the frontend. Each signal names exactly one remedy.
+
+**Expected-gain estimate**: single-node asymmetry affecting 1 of N prefill
+groups → small-percent gain (≈ its share of prefill capacity); full-width
+affinity on all ranks → the pinning gain for your node's core count.
+
+Cross-node timing comparisons must estimate per-node clock offsets first
+(pair `t` with `t_mono`, or use a constant frontend→worker dispatch offset);
+raw cross-node wall-clock deltas are unreliable at millisecond scale —
+observed inter-node skew up to 2.1 s.

docs/profiling.md

@@ -109,17 +109,103 @@ Profiling has specific requirements:
 
 ### nsys-specific behavior
 
-When using `nsys`, workers are wrapped with:
+**TRT-LLM workers (`backend.type: trtllm`)** are wrapped with the capture recipe from the Dynamo
+Benchmark Playbook §9.5.1.1 "Dynamo + TRTLLM", the flag set that produced every usable multi-node
+capture on nsys 2026.3.x / VR200 disaggregated serving:
 
 ```bash
-nsys profile -t cuda,nvtx --cuda-graph-trace=node \
-  -c cudaProfilerApi --capture-range-end stop \
-  [extra_nsys_args...] \
-  -o /logs/profiles/{mode}/{name} \
-  python3 -m sglang.launch_server ...
+nsys profile --force-overwrite=true \
+  -t cuda-sw,nvtx,python-gil --cuda-graph-trace=graph \
+  --sample=none --cpuctxsw=none --python-sampling=false --python-sampling-frequency=1000 \
+  --gpu-metrics-devices=none --flush-on-cudaprofilerstop=false --cuda-flush-interval=0 \
+  -c cudaProfilerApi --capture-range-end=stop \
+  [extra_nsys_args...] --kill none --wait all \
+  -o /logs/profiles/{mode}/{leader}_{mode}_w{index}_profile_rank%q{SLURM_PROCID} \
+  trtllm-llmapi-launch python3 -m dynamo.trtllm ...
 ```
 
-You can pass extra arguments via `profiling.extra_nsys_args` (e.g. `["--stats=true", "--trace=osrt"]`).
+One `nsys` runs per srun task, i.e. per engine rank; `%q{SLURM_PROCID}` is nsys's own env-var
+substitution, expanded inside the launched process, so each rank gets its own `.nsys-rep`.
+The capture window is the TRT-LLM iteration range from `prefill`/`decode` `start_step`/`stop_step`
+(`TLLM_PROFILE_START_STOP`): the PyExecutor calls `cudaProfilerStart`/`cudaProfilerStop` on every rank
+at those iterations and `--capture-range-end=stop` finalises the report the moment the range closes.
+Reports appear under `<log_dir>/profiles/<mode>/` while the benchmark is still running.
+
+Why each flag (from the playbook):
+
+| Flag | Reason |
+| ---- | ------ |
+| `-t cuda-sw`, not `cuda` | on nsys 2026.x plain `cuda` selects the HES hardware trace, which SIGSEGVs the KV transceiver's device-to-device `cudaMemcpyAsync`; `cuda-sw` forces the software tracer |
+| `--cuda-graph-trace=graph` | decode runs CUDA graphs; with `node` the launches are invisible (~0.3 % GPU busy reads as idle), with `graph` each iteration is one entry and utilisation is real |
+| `--sample=none --cpuctxsw=none` | `--sample=process-tree` wedged workers at `cudaProfilerStop` across 8 runs. Cost: no scheduler/CPU-time data, every duration is wall-clock. `nsys_cpuctxsw: process-tree` alone (context switches, no IP sampling) is the safer experiment; both need `kernel.perf_event_paranoid <= 2` on the compute node |
+| `-c cudaProfilerApi --capture-range-end=stop` | finalise-at-process-exit has never produced a report on this stack |
+| `--flush-on-cudaprofilerstop=false --cuda-flush-interval=0` | matches the known-good captures; nsys 2026.4 flips the flush default, so it is set explicitly |
+| `--gpu-metrics-devices=none` | a second collection path with its own failure modes |
+
+Worker environment set alongside: `TLLM_PROFILE_START_STOP=<start>-<stop>`, `TLLM_LLMAPI_ENABLE_NVTX=1`,
+`TLLM_PROFILE_LOG_RANKS=<log_ranks>` (default `all`), `DYN_ENABLE_RUST_NVTX=1` (Dynamo's Rust NVTX
+ranges; effective only on a wheel built with the `nvtx` cargo feature) and, when
+`nvtx_injection_path` is set, `NVTX_INJECTION64_PATH` pointing at nsys's injection library inside the
+container (TRT-LLM containers ship it next to nsys, e.g.
+`/usr/local/cuda-0.gpgpu/NsightSystems-cli-2026.3.0/target-linux-sbsa-armv8/libToolsInjection64.so`).
+
+Knobs (all optional, defaults = the recipe above): `nsys_trace`, `nsys_cuda_graph_trace` (`graph`|`node`),
+`nsys_sample` and `nsys_cpuctxsw` (`none`|`process-tree`|`system-wide`), `nsys_python_sampling`,
+`nsys_python_sampling_frequency`, `nsys_gpu_metrics_devices`, `nvtx_injection_path`, `log_ranks`,
+plus `extra_nsys_args` (appended before `-o`). `nsys-time` uses the same trace flags with
+`--delay`/`--duration` instead of the cudaProfilerApi window. The `profiling:` block is independent of
+`observability.enabled`, so the capture cost can be measured against a default-visibility run.
+
+Sizing the window: `start_step` counts engine iterations, which begin with traffic, not with worker
+start. On the 8-node AgentX recipe the decode worker ran ~20 iterations/s and each prefill worker
+~2.7/s in steady state, so `decode: 6000-6600` and `prefill: 1200-1300` both capture about 30-40 s
+roughly 12 minutes into the benchmark.
+
+**SGLang / vLLM workers** keep the time-based or `--trace-fork-before-exec` prefixes described in the
+examples below; `extra_nsys_args` applies to them as well.
+
+### Profiling the Dynamo frontend too (`profiling.frontend`)
+
+The worker windows are iteration based (cudaProfilerApi). The Dynamo frontend does no CUDA
+work and never calls `cudaProfilerStart`, so it gets a time window instead: `profiling.frontend`
+wraps every `python3 -m dynamo.frontend` process in
+`nsys profile -t <trace> --delay <delay_secs> --duration <duration_secs> --kill none --wait all`
+and exports `DYN_ENABLE_RUST_NVTX=1` (plus `NVTX_INJECTION64_PATH` when `nvtx_injection_path` is
+set) so Dynamo's Rust NVTX ranges are recorded. Reports land in
+`/logs/profiles/frontend/<node>_frontend_<idx>.nsys-rep`; nsys stays attached for the whole run
+and the frontend keeps serving after the window closes.
+
+```yaml
+profiling:
+  type: nsys
+  frontend:
+    delay_secs: 1500     # seconds after the frontend starts (workers need ~10-15 min to load first)
+    duration_secs: 120
+    trace: nvtx          # or "nvtx,osrt"; CUDA tracing is pointless on this process
+  prefill: {start_step: 1200, stop_step: 1300}
+  decode:  {start_step: 6000, stop_step: 6600}
+```
+
+Requires `frontend.type: dynamo` and an nsys profiling type; other frontends are not wrapped.
+
+### Time-windowed sessions and the srun task (`nsys-time`, `profiling.frontend`)
+
+`nsys profile --delay D --duration T --kill none <app>` writes its report when the window closes and then
+**exits**, leaving the application running as an orphan. Under Slurm the exiting nsys is the srun task, so the
+step ends and slurmstepd kills the orphan — on hecate job 595056 the frontend and the decode workers died three
+seconds after their reports were written. srtctl therefore wraps every time-windowed launch (the profiled
+frontend, and workers in `nsys-time` mode) so the task waits for the profiled process after nsys exits
+(`srtctl.core.nsys_keepalive`). Iteration-based captures need no wrapper: nsys stays attached until the engine exits.
+
+### Teardown grace for open capture ranges (`profiling.teardown_grace_secs`)
+
+nsys writes a report when its capture range closes. If the range is still open when the run ends —
+`stop_step` was never reached, or an `nsys-time` duration outlasts the benchmark — the report is written
+only after the engine exits, and the default 10-second SIGTERM→SIGKILL grace that srtctl gives every
+process loses it. When `profiling.type` is `nsys`/`nsys-time`, srtctl therefore waits
+`profiling.teardown_grace_secs` (default 180) after SIGTERM before SIGKILL for the nsys-wrapped worker
+sruns and the profiled frontend. The waits overlap (every process is signalled first, then each is
+reaped), so a run pays the grace once, not once per worker. Unprofiled processes keep the 10 s grace.
 
 ## Example Configurations