feat(benchmark): add agentperf benchmark type for trajectory replay

[nv-yna]

What

Adds an agentperf benchmark type that lets srt-slurm orchestrate the agentperf-client trajectory-replay load generator (deterministic agentic workload, Rust streaming core, per-phase *__traj*.{jsonl,txt,json} outputs + per-request requests.jsonl).

This is a different client from the InferenceX AgentX harness in #342 — the two are complementary benchmark types.

Launching an AgentPerf load: client-side steps

Step 0 (optional) — port an existing script-harness run automatically

If you already have a run directory from the standalone disagg-harness (a bm_agentperf_*/.../c<N>/ dir), you don't need to reverse-engineer it by hand — this PR ships a porter:

python3 src/srtctl/benchmarks/scripts/agentperf/port_harness_run.py \
    /path/to/old/.../c1010 \
    --out my-recipe.yaml --workload-out my-workload.yaml \
    --dataset-root /scratch/.../datasets      # where the original trajectory/assignments live

You don't need to know how the old harness worked. The script reads the artifacts every harness run leaves behind — the job parameter file (container image, model, concurrency), the engine configs (copied verbatim into the recipe), the client launch script (which pinned agentperf-client checkout was used), the client's own startup banner in client.log (all workload knobs: phase timeout, settling time, worker/token limits, routing flags, seed), and the srun launch lines in job.log (worker/frontend environment and the node topology) — and emits a ready-to-edit srt-slurm recipe plus the client workload YAML.

It also applies the translations that are easy to get wrong when porting by hand: environment variables that only the old harness understood are renamed to the ones dynamo actually reads (e.g. the KV block size, which would otherwise silently fall back to a wrong default), variables that srt-slurm or the container manage are dropped, /lustre paths are rewritten for clusters where containers only mount /scratch, CPU pinning is auto-detected from the old worker logs, and frontend flags that would crash the frontend are excluded.

Everything it generates is traceable: it prints a provenance report (each field → the artifact it came from) and a TODO list of the few things it cannot infer — chiefly where the original datasets live (the logs only show staged copies; --dataset-root lets it search), your cluster's gpu_type, and whether paths owned by other users are readable from your compute containers. Review the TODOs, then continue at Step 4.

Verified: ported the DSV4 c1010 GB300 baseline run and compared against a hand-validated parity recipe — 176/184 recipe fields and 19/19 workload fields identical, every remaining diff benign and TODO-flagged.

Step 1 — mount a pinned agentperf-client checkout

The client is not vendored. Put a checkout at a pinned commit somewhere container-visible and mount it:

extra_mount:
  - "/scratch/.../agentperf-client:/agentperf-client"

Step 2 — write the workload YAML

This is the client's own config (benchmark.agentperf_config). It carries what the client CLI can't set: settling_time_seconds, user_spawn_rate, stop criteria, dataset paths. Three rules:

• It must contain syntactically valid placeholder base_url / model / concurrencies — the client validates the YAML before merging CLI overrides; srtctl then overrides all three at run time.
• phase_timeout_seconds ≥ (max concurrency − 1)/user_spawn_rate + settling_time_seconds + min_measurement_seconds, computed for the concurrency the recipe injects.
• The user-assignments file must cover the highest concurrency level (the client fails loudly otherwise).

Step 3 — recipe benchmark section

benchmark:
  type: agentperf
  agentperf_client_dir: /agentperf-client
  agentperf_config: /workloads/my-workload.yaml
  concurrency: 1010              # or concurrencies: [64, 1010] — one client phase per level
  client_placement: last_decode  # optional: put the client on the decode leader node

Step 4 — validate and submit

srtctl dry-run -f recipe.yaml   # shows mounts/env so you can verify before burning nodes
srtctl apply -f recipe.yaml

Step 5 — what happens and where results land

On first run the bench step preflights an isolated client runtime under /tmp/agentperf-<jobid> (pinned Rust + uv, uv sync --frozen, maturin rustcore build, tokenizer cache) and stages the trajectory/assignments datasets to node-local /tmp — needs network egress from the benchmark node, adds ~5–10 min before the first phase. Results land in the run's log dir: agentperf/*__traj*.{txt,jsonl,json} (per-phase summary + per-request metrics), agentperf/requests.jsonl, agentperf/phase_manifest.jsonl, and a normalized benchmark-rollup.json.

Implementation notes

• src/srtctl/benchmarks/agentperf.py — runner: validates agentperf_client_dir / agentperf_config / concurrencies, injects endpoint + served model + concurrency via the client's CLI overrides.
• scripts/agentperf/bench.sh — a faithful translation of the standalone disagg-harness client invocation (same isolated /tmp runtime layout, pinned rust/uv toolchain preflight, uv sync --frozen + maturin rustcore build, node-local dataset staging). READY marker is self-validating (arch + client commit) with an atomic build lock for relocated runtimes.
• scripts/agentperf/port_harness_run.py — the Step-0 porter (host-side tool; provenance report + TODOs).
• scripts/agentperf/rollup.py — normalizes per-phase stats into benchmark-rollup.json.
• Schema: two new BenchmarkConfig fields (agentperf_client_dir, agentperf_config).
• Docs: config-reference.md types table + full section. Tests: runner + porter suites; make check: 1480 passed.

Draft until parity is proven

This PR is the first half of a two-step deliverable: the same DSV4 c1010 configuration (5p1d, dep4/dep32, GB300) is being run under srt-slurm orchestration and compared against the script-harness reference numbers (server-reported output throughput, TTFT percentiles). Marking ready once the parity evidence is attached.

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

❌ Patch coverage is 93.61702% with 3 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@575997f). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
src/srtctl/benchmarks/agentperf.py	93.33%	3 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #353   +/-   ##
=======================================
  Coverage        ?   72.09%           
=======================================
  Files           ?       96           
  Lines           ?    13064           
  Branches        ?        0           
=======================================
  Hits            ?     9418           
  Misses          ?     3646           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[nv-yna]

Parity evidence: script-harness vs srt-slurm orchestration (DSV4 c1010, GB300)

The parity gate this PR was drafted against is now closed. The same c1010 configuration (DeepSeek-V4-Pro, 5 prefill workers dep4 + 1 decode worker dep32 across 13 GB300 nodes, dynamo frontend co-located with etcd and prefill_0, agentperf-client pinned at the same tree) was run under srt-slurm orchestration (oci-aga job 592210, recipe dsv4-5p1d-c1010-parity-baseline) and compared against the script-harness reference run:

Metric	script harness	srt-slurm	Δ
Output tok/s (server-reported)	60,822.2	60,573.2	−0.41%
Output tok/s (client-estimated)	45,776.2	45,697.7	−0.17%
Total tok/s (server-reported)	8,145,082.8	8,014,563.4	−1.60%
TTFT p50	1.348 s	1.376 s	+2.1%
TTFT p95	2.562 s	2.495 s	−2.6%
TTFT p99	4.158 s	3.759 s	−9.6%
Cache hit rate	96.7%	96.7%	=
Avg input tokens/request	40,544.8	40,448.1	−0.24%
Avg output tokens/request	305.0	308.0	+1.0%
Per-request output speed p50 (local / SSE)	77.5 / 138.3 tok/s	77.4 / 136.7 tok/s	−0.1% / −1.2%
Measured requests (errors)	230,509 (0)	216,835 (0)	—

Verdict: no systematic difference from the orchestration translation. The primary metric (server-reported output throughput) lands −0.41% from the reference, well inside the ~1–2% run-to-run band observed between the reference campaign's own single-run variants. TTFT medians agree within ~2%; the p99 tail is noisy across runs and domains in the reference data and the srt-slurm run's tail falls at the favorable edge of the observed band.

Notes for reproducers:

• Recipe knobs that mattered: DYN_TRTLLM_KV_BLOCK_SIZE=128 in worker env (dynamo's env fallback; the harness's DYN_KV_BLOCK_SIZE is inert under srt-slurm), explicit UCX_TLS, dynamo.install: false, no --dyn-*-parser on the frontend, and all /lustre paths rewritten to /scratch (a root symlink on this cluster — containers only mount /scratch). All of these are handled automatically by scripts/agentperf/port_harness_run.py.
• Bring-up for this model/topology takes ~20–30 min; the recipe raises health_check.max_attempts to 300 to cover the observed variance.

🤖 Generated with Claude Code