feat(observability): host sampler on every node + scheduler-contention metrics

[nv-yna]

What

Extends the /proc host sampler from orchestrator-node-only to every allocated node, and adds the scheduler-level metrics needed to attribute host-CPU interference to a remedy from a single baseline run — the collection half of the pinning-vs-frontend-placement attribution methodology documented in the new docs/host-attribution-metrics.md.

Why

Two independent host-CPU effects are each worth ~1% output throughput at high concurrency on GB300 disaggregated serving (measured on the DSV4 c1010 A/B campaign): worker-rank CPU pinning, and moving frontend+etcd off the prefill node. Neither is visible in throughput/TTFT, and until now srt-slurm had zero host-CPU telemetry on worker nodes (and none at all on a dedicated frontend node). Retro-analysis of the c1010 runs showed the frontend-sharing node's ranks run measurably slower (cache-matched +1.3–2.0% median prefill time) and its GPUs 2–3 pp less utilized — signals these collectors now capture live.

New metrics (per sampled process)

Field	Diagnoses	Remedy it points at
run_delay_ns (schedstat)	runnable-but-not-running time — scheduler contention	backend.numa_cpu_bind
nr_migrations (/proc/pid/sched)	cross-core churn; ~0 when pinned	backend.numa_cpu_bind
affinity_ncpus (sched_getaffinity)	direct pinning-state observable (144 floating / 36 pinned)	config state
ctx_invol (existing) + per-proc cpu_pct on the shared node	frontend/etcd stealing CPU from co-located ranks	frontend.dedicated_node
host procs_running/blocked, t_mono	node run-queue pressure; clock-offset hygiene	—

How

• host_sampler.py: new fields; stdlib-only standalone CLI mode writing host_samples_<node>.jsonl (SIGTERM-clean, exits nonzero if its sampler thread dies, never samples itself)
• benchmark_stage: launches the standalone sampler on all non-orchestrator nodes, one srun --overlap per het group, no container; best-effort contract (a node without python3 logs and is skipped). Gated by new observability.host_sampler_all_nodes (default true, follows observability.enabled)
• ingest: host_series.json gains a per-node hosts map + the new rate series; rate denominators prefer the monotonic clock (NTP-step-proof)
• teardown hardening from a 12-agent adversarial review (8 confirmed findings, all fixed): benchmark proc creation moved inside the try so sampler sruns can't leak on placement/launch errors; terminate_and_reap escalation; early-exit logging

make check: 1492 passed. 12 new tests (scheduler fields, standalone mode e2e, launch planning, remote-launch mocking, ingest rates incl. NTP-step case, multi-file merge).

Status: first production evidence in

Round 1 of the validation matrix (baseline vs pinned at c1010) completed with these collectors enabled: all-node capture worked (13 files/run), and the baseline-only placement fingerprint is real — the frontend-sharing node's worker rank shows a ~100× run-queue-delay asymmetry vs clean peers, while affinity_ncpus directly dissociates pinned (72) from unpinned (144) ranks. Full perf-diff and attribution tables: see the round-1 evidence comment below. Dedicated-frontend variants (C/D incl. the etcd-deconfounded cell) + repeats 2–3 are running to complete the double dissociation and set the rubric thresholds in docs/host-attribution-metrics.md.

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

❌ Patch coverage is 67.27273% with 36 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@00fc543). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
src/srtctl/analysis/host_sampler.py	61.19%	26 Missing ⚠️
src/srtctl/cli/mixins/benchmark_stage.py	76.19%	10 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #356   +/-   ##
=======================================
  Coverage        ?   72.23%           
=======================================
  Files           ?       96           
  Lines           ?    13169           
  Branches        ?        0           
=======================================
  Hits            ?     9513           
  Misses          ?     3656           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[nv-yna]

First production evidence: round 1 of the c1010 validation matrix

Both round-1 runs (DSV4 c1010, 13× GB300 on oci-aga, collectors from this PR enabled) completed cleanly, and the samplers delivered on all nodes: 12 per-node host_samples_<node>.jsonl + 1 orchestrator file per run, all new fields populated.

Perf diff (single run per variant — repeats in progress)

Metric	A: baseline (unpinned, FE co-located) — job 593609	B: pinned (numa_cpu_bind: true) — job 593610	Δ B vs A
Output tok/s (server-reported)	59,443.5	61,934.9	+4.2%
Output tok/s (client-estimated)	44,715.2	46,554.3	+4.1%
TTFT p50 / p95 / p99	1.426 / 2.587 / 3.833 s	1.323 / 2.537 / 3.859 s	−7.2% / −1.9% / +0.7%
Per-request output speed p50	75.6 tok/s	78.6 tok/s	+4.0%
Measured requests (errors)	202,421 (1)	198,993 (0)	—

Caveats: single runs on different NVL72 domains (d021 vs d195); the reference campaign's own single-run spread was 1–2%, so treat +4.2% as directionally right, not a settled effect size — rounds 2–3 bound it.

Attribution evidence (what this PR exists for)

Steady-window scheduler rates for the busiest worker rank per node, from the baseline run alone:

Node (role)	run_delay ms/s p50 / p90	migrations/s	affinity
CTX_0 + frontend + etcd (shared)	1.36 / 3.03	21.6	144
CTX_1 (clean)	0.01 / 0.02	0.0	144
CTX_2 (clean)	0.01 / 0.02	0.0	144
CTX_3 (clean)	0.01 / 0.02	0.0	144
CTX_4 (clean)	0.00 / 0.02	0.5	144
GEN (per node)	0.03 / 0.32	1.0	144

• Placement fingerprint, baseline-only: the frontend-sharing node's rank shows a ~100× run-queue-delay asymmetry vs its clean peers, co-resident with an unpinned ~49-core process (the dynamo frontend — it stays at affinity 144 in the pinned run while every worker drops to 72). No A/B needed to see it.
• Pinning fingerprint: affinity_ncpus flips 144 → 72 on every worker rank between variants — the direct config-state observable. (oci-aga GB300 nodes expose 288 logical CPUs → 72 per pinned rank.)
• Discriminating prediction: CTX_0's elevated delay persists under pinning (1.86 / 4.12 ms/s in variant B) — pinning does not remove co-location interference. If these metrics are causally right, the dedicated-frontend variants (C, and the etcd-deconfounded D — running now) must collapse CTX_0's run-delay to peer levels. That completes the double dissociation.

🤖 Generated with Claude Code

[nv-yna]

Validation campaign complete: 9 runs, 3 per variant — double dissociation confirmed

Final results of the c1010 validation matrix (DSV4-Pro, 13–14× GB300, all runs with this PR's collectors enabled; orchestration parity-proven in #353):

Throughput hierarchy (server-reported output tok/s)

Variant	runs	mean ± spread
A baseline (FE+etcd co-located, unpinned)	59,443 · 59,278 · 60,202	59,641 ± 0.8%
B pinned (numa_cpu_bind: true)	61,935 · 61,486 · 61,953	61,791 ± 0.4%
C dedicated FE (+ pinned)	61,928 · 62,032 · 62,074	62,011 ± 0.1%

C > B > A, A/B ranges disjoint; pinning +3.6% (cluster-dependent: these nodes expose 288 logical CPUs; the 144-CPU reference cluster measured +1.1%), placement +0.36% on top (reference +0.85%). TTFT p50 monotone: 1.43 → 1.31 → 1.32 s.

The attribution metrics dissociate, replicated 3×

Placement metric — shared-node worker rank's run-queue delay + migrations vs clean peers:

Cell	shared-node rank (delay ms/s p50/p90 · migr/s)	peers	ratio
A1 / A2 / A3	1.36/3.03·21.6 — 3.53/6.20·22.2 — 3.92/6.70·24.1	0.00–0.01 / 0.02–0.03 · ~0	140–390×
B1 / B3 (pinned, FE co-located)	1.86/4.12·21.6 — 3.40/6.18·21.7	same	190–340× — unchanged by pinning
C1 / C2 / C3 (FE moved)	0.00–0.01 / 0.03 · 0	identical	1× — collapsed

Pinning metric — affinity_ncpus on every worker rank: 144 in A×3, 72 in B×3 + C×3 — unchanged by placement. The co-resident interferer is directly visible in the same table: the frontend at 4,100–5,000% of one core, affinity 144, on exactly the asymmetric node.

Each metric responds to exactly its own remedy — the double dissociation these collectors were built to enable. Rubric thresholds derived from these numbers are in docs/host-attribution-metrics.md (previous commit).

Known limitation, filed as follow-up: with frontend.dedicated_node: true, head=infra=the reserved node, so etcd always follows the frontend — the "etcd stays with the prefill node" deconfound cell needs a small infra-placement option. etcd's contribution is bounded small regardless (<3% of one core in all 9 runs).

🤖 Generated with Claude Code