Skip to content

feat(observability): collect host scheduler-pressure metrics (PSI, procs, reclaim, per-NUMA) - #415

Merged
nv-yna merged 3 commits into
NVIDIA:mainfrom
nv-yna:yna/tachometer-host-scheduler-metrics
Sep 12, 2026
Merged

nv-yna merged 3 commits into
NVIDIA:mainfrom
nv-yna:yna/tachometer-host-scheduler-metrics

Conversation

@nv-yna

@nv-yna nv-yna commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

Default Tachometer host capture lacked runnable/blocked process counts, host context-switch counters, major faults/reclaim, PSI, and NUMA memory breakdowns. This change enables those node-exporter collectors on worker nodes and keeps each NUMA node distinct in the raw capture.

Behavior

  • Add stat, vmstat, pressure, and meminfo_numa alongside the existing cpu, infiniband, meminfo, and processes collectors. The vmstat field filter includes pgsteal_* while retaining major faults. Explicit recipe commands continue to take precedence.
  • Preserve the exporter’s NUMA node label as numa_node in raw metric names, separately from host metadata. This applies to NUMA memory gauges and allocation counters, and retains feat(observability): scrape a process exporter with tachometer by default #413’s process-state and passthrough fixes.
  • Add cumulative PSI stall totals, in microseconds, to the existing optional local host sampler. It remains gated by observability.enabled, covers only the sweep/orchestrator host, and has no dependency on the closed all-node sampler PR feat(observability): host sampler on every node + scheduler-contention metrics #356.
  • Missing or unreadable PSI files produce unavailable data, not zero pressure. This does not extend node-exporter placement to dedicated frontend/client nodes. Collector overhead has not been measured for this change.

Deployment

Use make tachometer-scraper to build a scraper containing this change, or use a release that includes it. Older scraper binaries can collapse NUMA labels even when the exporter collects them.

Validation

  • Rebased onto 0ecc0bdb8f8fc507d769cda0449aec70c52286de after feat(observability): scrape a process exporter with tachometer by default #413 merged; preserved the combined collector set.
  • make check: 2,152 passed, 2 skipped, 6 deselected. Final source/test lint and formatting passed.
  • cargo test --locked -p tachometer-scraper: all 34 tests passed; Rust formatting passed.
  • A regression reproduces NUMA label loss before the fix. Tests cover separate NUMA nodes for MemFree and allocation counters, unchanged host metadata and ordinary memory names, actual launched collector arguments, reclaim matching, custom-command precedence, and PSI present/absent cases.
  • No new cluster workload was launched. Hosted CI passed all seven checks, including the full Rust workspace tests and release build, on 55bb240d2bb379ac9667db08ae7633d046dadece before merge.

@codecov-commenter

codecov-commenter commented Sep 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.00000% with 1 line in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@0ecc0bd). Learn more about missing BASE report.

Files with missing lines Patch % Lines
src/srtctl/analysis/host_sampler.py 93.33% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #415   +/-   ##
=======================================
  Coverage        ?   75.25%           
=======================================
  Files           ?      109           
  Lines           ?    16115           
  Branches        ?        0           
=======================================
  Hits            ?    12127           
  Misses          ?     3988           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

nv-yna added a commit to nv-yna/srt-slurm that referenced this pull request Sep 10, 2026
…xporterFilter

Two node_exporter families enabled by the combined observability branch lost
their only distinguishing label in the vendored scraper filter, so every series
of the family collapsed into one metric name in the parquet:

- node_memory_numa_*{node="N"} (--collector.meminfo_numa, NVIDIA#415): the generic
  `memory_` arm emitted the bare metric name, folding all NUMA nodes together.
  Now `memory_numa_<x>{numa_node=N}`; host-wide meminfo stays label-free.
- node_processes_state{state="R"|"S"|"D"|...} and
  node_processes_threads_state{thread_state=...} (--collector.processes, NVIDIA#413):
  the default arm keeps only a fixed label whitelist that had neither key.
  `state` and `thread_state` are added to the whitelist.

Label-free stat/vmstat/pressure families were already passed through unchanged;
a test pins that too.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com>
Enable stat, vmstat, pressure, and NUMA collectors alongside the existing node-exporter collectors. Include page-reclaim counters and retain explicit command overrides. Add optional PSI totals to the existing local host sampler.

Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com>
Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com>
Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com>
@nv-yna
nv-yna force-pushed the yna/tachometer-host-scheduler-metrics branch from 68c05db to 55bb240 Compare September 12, 2026 07:23
@nv-yna
nv-yna marked this pull request as ready for review September 12, 2026 07:24
@nv-yna
nv-yna merged commit 4d3e8bd into NVIDIA:main Sep 12, 2026
7 checks passed
ishandhanani added a commit that referenced this pull request Sep 13, 2026
Nine conflicting files, resolved by hand:

- The process exporter (#413) and the wider node-exporter collector set (#415)
  arrive as tachometer-stage exporter launches; on this branch the exporters
  are services. process-exporter is now a service kind (`type:
  process-exporter`, implied while tachometer runs, `placement.node: all`),
  host-native from the `configs/process-exporter` binary `make setup`
  installs (no container, no mounts), skipped with a warning when the binary
  is missing, container launch when a recipe declares a `container`. The
  group file is written by the kind's `prepare` hook; `ServiceKind` gains
  `host_native`, `prepare`, and `skip_reason`. node-exporter's built command
  carries the stat/vmstat/pressure/meminfo_numa/processes collectors and the
  widened vmstat field filter. The templates, the group YAML, and the host
  binary resolver move to `srtctl.services.exporters`; their tests follow.
- Load-window tachometer (#359): start/stop inside run_benchmark is kept;
  `stop_tachometer` now terminates through `ManagedProcess.terminate`, which
  signals the Slurm step (SIGTERM to the srun client would abort the step and
  SIGKILL the scraper). The scraper's `terminate_timeout` is the recipe's
  `shutdown_grace_secs`; the 90 s module constant is gone.
- Ingest timestamp fallback (#414) is taken from main wholesale (it also
  offers `--start-ns`); this branch's own fallback is dropped, its dedup
  module and docstring edits re-applied.
- CPU power telemetry (#410, #422): taken as is (power is frozen); the
  energy report runs before the S3 upload, which returns the URL only.
- Makefile: the `examples`/`golden-check` targets plus the cpu-power targets;
  the `recipes/`-based runner targets stay deleted. SUMMARY: both new pages,
  `analyzing.md` stays deleted. submit.py: the direct-host renderer import
  stays deleted; the arch helpers the cpu-power preflight uses are kept.
  telemetry.py: IPv6-safe host and the SGLang gateway metrics port together.

2355 tests on Python 3.10 and 3.13, lint, schema docs, 21 examples validated,
golden 574 identical / 0 mismatched.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants