feat(observability): collect host scheduler-pressure metrics (PSI, procs, reclaim, per-NUMA) - #415
Merged
nv-yna merged 3 commits intoSep 12, 2026
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #415 +/- ##
=======================================
Coverage ? 75.25%
=======================================
Files ? 109
Lines ? 16115
Branches ? 0
=======================================
Hits ? 12127
Misses ? 3988
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
nv-yna
added a commit
to nv-yna/srt-slurm
that referenced
this pull request
Sep 10, 2026
…xporterFilter
Two node_exporter families enabled by the combined observability branch lost
their only distinguishing label in the vendored scraper filter, so every series
of the family collapsed into one metric name in the parquet:
- node_memory_numa_*{node="N"} (--collector.meminfo_numa, NVIDIA#415): the generic
`memory_` arm emitted the bare metric name, folding all NUMA nodes together.
Now `memory_numa_<x>{numa_node=N}`; host-wide meminfo stays label-free.
- node_processes_state{state="R"|"S"|"D"|...} and
node_processes_threads_state{thread_state=...} (--collector.processes, NVIDIA#413):
the default arm keeps only a fixed label whitelist that had neither key.
`state` and `thread_state` are added to the whitelist.
Label-free stat/vmstat/pressure families were already passed through unchanged;
a test pins that too.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com>
Enable stat, vmstat, pressure, and NUMA collectors alongside the existing node-exporter collectors. Include page-reclaim counters and retain explicit command overrides. Add optional PSI totals to the existing local host sampler. Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com>
Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com>
Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com>
nv-yna
force-pushed
the
yna/tachometer-host-scheduler-metrics
branch
from
September 12, 2026 07:23
68c05db to
55bb240
Compare
nv-yna
marked this pull request as ready for review
September 12, 2026 07:24
nv-yna
requested review from
alec-flowers,
csahithi,
ishandhanani and
nlevin-ui
as code owners
September 12, 2026 07:24
ishandhanani
added a commit
that referenced
this pull request
Sep 13, 2026
Nine conflicting files, resolved by hand: - The process exporter (#413) and the wider node-exporter collector set (#415) arrive as tachometer-stage exporter launches; on this branch the exporters are services. process-exporter is now a service kind (`type: process-exporter`, implied while tachometer runs, `placement.node: all`), host-native from the `configs/process-exporter` binary `make setup` installs (no container, no mounts), skipped with a warning when the binary is missing, container launch when a recipe declares a `container`. The group file is written by the kind's `prepare` hook; `ServiceKind` gains `host_native`, `prepare`, and `skip_reason`. node-exporter's built command carries the stat/vmstat/pressure/meminfo_numa/processes collectors and the widened vmstat field filter. The templates, the group YAML, and the host binary resolver move to `srtctl.services.exporters`; their tests follow. - Load-window tachometer (#359): start/stop inside run_benchmark is kept; `stop_tachometer` now terminates through `ManagedProcess.terminate`, which signals the Slurm step (SIGTERM to the srun client would abort the step and SIGKILL the scraper). The scraper's `terminate_timeout` is the recipe's `shutdown_grace_secs`; the 90 s module constant is gone. - Ingest timestamp fallback (#414) is taken from main wholesale (it also offers `--start-ns`); this branch's own fallback is dropped, its dedup module and docstring edits re-applied. - CPU power telemetry (#410, #422): taken as is (power is frozen); the energy report runs before the S3 upload, which returns the URL only. - Makefile: the `examples`/`golden-check` targets plus the cpu-power targets; the `recipes/`-based runner targets stay deleted. SUMMARY: both new pages, `analyzing.md` stays deleted. submit.py: the direct-host renderer import stays deleted; the arch helpers the cpu-power preflight uses are kept. telemetry.py: IPv6-safe host and the SGLang gateway metrics port together. 2355 tests on Python 3.10 and 3.13, lint, schema docs, 21 examples validated, golden 574 identical / 0 mismatched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Default Tachometer host capture lacked runnable/blocked process counts, host context-switch counters, major faults/reclaim, PSI, and NUMA memory breakdowns. This change enables those node-exporter collectors on worker nodes and keeps each NUMA node distinct in the raw capture.
Behavior
stat,vmstat,pressure, andmeminfo_numaalongside the existingcpu,infiniband,meminfo, andprocessescollectors. The vmstat field filter includespgsteal_*while retaining major faults. Explicit recipe commands continue to take precedence.nodelabel asnuma_nodein raw metric names, separately from host metadata. This applies to NUMA memory gauges and allocation counters, and retains feat(observability): scrape a process exporter with tachometer by default #413’s process-state and passthrough fixes.observability.enabled, covers only the sweep/orchestrator host, and has no dependency on the closed all-node sampler PR feat(observability): host sampler on every node + scheduler-contention metrics #356.Deployment
Use
make tachometer-scraperto build a scraper containing this change, or use a release that includes it. Older scraper binaries can collapse NUMA labels even when the exporter collects them.Validation
0ecc0bdb8f8fc507d769cda0449aec70c52286deafter feat(observability): scrape a process exporter with tachometer by default #413 merged; preserved the combined collector set.make check: 2,152 passed, 2 skipped, 6 deselected. Final source/test lint and formatting passed.cargo test --locked -p tachometer-scraper: all 34 tests passed; Rust formatting passed.55bb240d2bb379ac9667db08ae7633d046dadecebefore merge.