test(benchmark): unit + golden-file tests for the agentperf porter - #355
Merged
Merged
Conversation
Unit tests cover the porter's utility layer: path-prefix rewriting (single value + nested trees), absolute-path collection, worker-env translation (drop/rename/DYN_LOG pinning), quoted env-token-list extraction from srun lines, frontend-arg reconstruction from env, and settle-time recovery from output filename stems. The golden e2e test materializes a real harness run directory as a stable fixture (tests/fixtures/agentperf_porter/c1010 — the DSV4 c1010 GB300 baseline, usernames anonymized, content otherwise verbatim: real engine configs, the real 35-token worker env, the real client banner, real srun line shapes) plus checked-in expected recipe/workload YAMLs. The test runs the porter on the fixture and byte-compares normalized output against the goldens, so any behavior drift over time fails loudly; regeneration instructions are in the module docstring. A second test loads the golden recipe through SrtConfig's schema. Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com>
nv-yna
requested review from
alec-flowers,
csahithi,
ishandhanani and
nlevin-ui
as code owners
August 27, 2026 17:00
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #355 +/- ##
=======================================
Coverage ? 72.09%
=======================================
Files ? 96
Lines ? 13064
Branches ? 0
=======================================
Hits ? 9419
Misses ? 3645
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Closed
ishandhanani
added a commit
that referenced
this pull request
Sep 13, 2026
Nine conflicting files, resolved by hand: - The process exporter (#413) and the wider node-exporter collector set (#415) arrive as tachometer-stage exporter launches; on this branch the exporters are services. process-exporter is now a service kind (`type: process-exporter`, implied while tachometer runs, `placement.node: all`), host-native from the `configs/process-exporter` binary `make setup` installs (no container, no mounts), skipped with a warning when the binary is missing, container launch when a recipe declares a `container`. The group file is written by the kind's `prepare` hook; `ServiceKind` gains `host_native`, `prepare`, and `skip_reason`. node-exporter's built command carries the stat/vmstat/pressure/meminfo_numa/processes collectors and the widened vmstat field filter. The templates, the group YAML, and the host binary resolver move to `srtctl.services.exporters`; their tests follow. - Load-window tachometer (#359): start/stop inside run_benchmark is kept; `stop_tachometer` now terminates through `ManagedProcess.terminate`, which signals the Slurm step (SIGTERM to the srun client would abort the step and SIGKILL the scraper). The scraper's `terminate_timeout` is the recipe's `shutdown_grace_secs`; the 90 s module constant is gone. - Ingest timestamp fallback (#414) is taken from main wholesale (it also offers `--start-ns`); this branch's own fallback is dropped, its dedup module and docstring edits re-applied. - CPU power telemetry (#410, #422): taken as is (power is frozen); the energy report runs before the S3 upload, which returns the URL only. - Makefile: the `examples`/`golden-check` targets plus the cpu-power targets; the `recipes/`-based runner targets stay deleted. SUMMARY: both new pages, `analyzing.md` stays deleted. submit.py: the direct-host renderer import stays deleted; the arch helpers the cpu-power preflight uses are kept. telemetry.py: IPv6-safe host and the SGLang gateway metrics port together. 2355 tests on Python 3.10 and 3.13, lint, schema docs, 21 examples validated, golden 574 identical / 0 mismatched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Follow-up to #353: locks the behavior of
scripts/agentperf/port_harness_run.py(the Step-0 harness-run porter) with two layers of tests.Unit tests (utility layer)
rewrite/rewrite_tree: path-prefix rewriting on single values and nested YAML trees (first-matching-rule semantics, non-prefix strings untouched)absolute_paths: nested absolute-path collection (drives the reachability TODOs)translate_env: env drops (CUDA_VISIBLE_DEVICES,PATH, …), renames (DYN_KV_BLOCK_SIZE→DYN_TRTLLM_KV_BLOCK_SIZE,DYN_UCX_TLS→UCX_TLS), and the pinned-but-overridableDYN_LOG_quoted_env_tokens: picks the env token list out of an srun line and ignores scalar quoted args like'1010'/'0-0'build_frontend_args: env-driven flag reconstruction, KV-events on/off, and the guarantee that the worker-only--dyn-*-parserflags are never emittedSETTLE_RE: integer and decimal settle stemsGolden e2e test (stability over time)
tests/fixtures/agentperf_porter/c1010/materializes a real harness run directory as a stable test input — the DSV4 c1010 GB300 baseline with usernames anonymized and content otherwise verbatim: the real prefill/decode engine configs, the real 35-token worker environment, the real client resolved-config banner, and real srun launch-line shapes (5 CTX + 1 GEN over 8 nodes + frontend + etcd).expected-recipe.yaml/expected-workload.yamlare the checked-in expected outputs (machine-dependent paths normalized to<FIXTURE>/<WORKLOAD_OUT>placeholders). The test runs the porter on the fixture and byte-compares the normalized output against the goldens, so any behavior drift fails loudly with regeneration instructions in the module docstring. A second test round-trips the golden recipe throughSrtConfig's schema.make check: 1494 passed.🤖 Generated with Claude Code