Skip to content
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
874a95c
feat(agent-skills): add skills and updated docs for perf analyzer sub…
ar-gupte Jul 21, 2026
cf07222
refactor(agent-skills): rename deploy and troubleshoot skills
ar-gupte Jul 21, 2026
9e955f6
feat(agents): add recipe optimization workflow
ar-gupte Aug 3, 2026
08ad3e0
feat(agent-skills): preserve latest optimization work
ar-gupte Aug 4, 2026
bbf5e7a
Ben's edits: clean-room UAT hardening of the optimization skills (con…
BenHamm Aug 14, 2026
a780ac3
fix(skills): greenfield engagements are a first-class iteration-0 pat…
BenHamm Aug 18, 2026
fad4553
Merge remote-tracking branch 'origin/main' into optimization-pack-v2
BenHamm Aug 18, 2026
c079b41
docs: surface the agent skills where users already look
BenHamm Aug 18, 2026
39086c6
docs(fern): add Agent Skills tab with overview page
BenHamm Aug 19, 2026
f35d390
docs(fern): give Agent Skills tab an explicit slug
BenHamm Aug 19, 2026
d92d423
docs(fern): correct deployment-skill names and refresh example model
BenHamm Aug 19, 2026
7865343
ci: cover .codex/ in CI filters
BenHamm Aug 19, 2026
eb33e64
fix: address review findings 1-7 and 10-14 from PR review
BenHamm Aug 20, 2026
91eb11b
fix: make the search-calibration ledger authoritative for stop-requests
BenHamm Aug 20, 2026
801992d
fix: declare .codex/, agents/, agent-docs/ ownership in areas.yaml
BenHamm Aug 20, 2026
3e522d8
fix: resolve both review passes plus full contract-consistency sweep
BenHamm Aug 20, 2026
3fac9f1
docs(fern): pin agent-docs links to a commit SHA, drop lycheeignore s…
BenHamm Aug 20, 2026
6b3478a
docs(fern): correct pinned commit SHA in agent-docs links
BenHamm Aug 20, 2026
7e14ea4
fix(skills): harden deploy-dynamo-recipe from live-cluster validation
BenHamm Aug 20, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
142 changes: 142 additions & 0 deletions .agents/skills/analyze-aiperf-results/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
---
name: analyze-aiperf-results
description: >-
Validates and normalizes raw AIPerf outputs, then evaluates valid results against target SLOs and comparable prior
candidates. Use after an AIPerf Job completes to produce benchmark audit, summary, and performance analysis artifacts.
license: Apache-2.0
metadata:
author: NVIDIA
tags:
- dynamo
- aiperf
- benchmarking
---

# Analyze AIPerf Results

Audit benchmark evidence before interpreting performance. Preserve raw files unchanged and do not claim an unmeasured
server-side cause.

Read:

- `agent-docs/rules/benchmarking/benchmark-isolation.md`;
- `agent-docs/rules/benchmarking/comparison-uncertainty.md`;
- `agent-docs/rules/benchmarking/evidence-eligibility.md`;
- `agent-docs/rules/benchmarking/result-storage.md`;
- `agent-docs/rules/benchmarking/series-boundaries.md`;
- `agent-docs/rules/optimization/evidence-before-spend.md`;
- `agent-docs/rules/optimization/one-variable.md`;
- `agent-docs/rules/verification/config-engagement.md`;
- `agent-docs/rules/verification/implausible-speedup.md`;
- `agent-docs/rules/verification/overlap.md`; and
- `agent-docs/rules/verification/stack-verdict.md`.

Also read the user workload, active benchmark plan, execution ledger, AIPerf config, raw outputs, all prior
candidate audits and summaries, and the profile-export documentation matching the pinned AIPerf source or runtime.

## Audit And Normalize

- Parse per-request `profile_export.jsonl` with AIPerf's native Pydantic models when available. Record the parser and
runtime version used.
- Parse `profile_export_aiperf.json` and multi-run aggregate/search artifacts when configured.
- Confirm files are readable, non-empty, and internally consistent.
- Confirm the executed config matches the active plan and candidate endpoint.
- Confirm trace hash or static-shape identity, schedule mode, endpoint type, model, tokenizer, warmup, seed, request
count/duration, repetitions, and load controls.
- Separate warmup from profiling records and exclude only data the active plan says to exclude.
- Compare attempted, successful, failed, cancelled, and timed-out request counts.
- Check actual ISL/OSL distributions against the input workload and report any output-length shortfall.
- Check timestamps, benchmark duration, fixed-schedule coverage, duplicate/missing request ids, malformed metrics,
NaN/inf values, units, and impossible negative latencies.
- Recompute user-requested percentiles from raw profiling records when AIPerf does not export them directly.

If the aggregate export is missing but complete raw records exist, reconstruct it once using the pinned AIPerf
models and metric definitions. Record `valid_with_recovery`, the missing file, method, and generated summary. Never
modify or replace the raw directory.

## Write Audit Artifacts And Gate Analysis

Write `benchmark_audit.json` with:

- `status`: `valid`, `valid_with_recovery`, or `invalid`;
- benchmark-series ID, plan path and SHA256, performance question, and workload identity;
- expected versus actual requests and phases;
- error/cancellation breakdown;
- integrity checks and recovery actions;
- parser/AIPerf versions;
- blockers and `next_action`: `continue_analysis`, `rerun_benchmark`, or `stop`.

Write `benchmark_summary.json` with normalized benchmark metadata and every numerical metric reported by AIPerf,
including units and available statistics. Include requested custom percentiles and per-GPU derived throughput with the
GPU-count source. Do not include gain/loss interpretation in the summary.

For `valid` or `valid_with_recovery`, set `next_action` to `continue_analysis` and continue below in the same
invocation.

For repairable `invalid` evidence, set `next_action` to `rerun_benchmark`. Return `benchmark_audit.json` and
`benchmark_execution.json` to `run-aiperf-benchmark`, preserve the invalid run as failed evidence, and rerun the active
series unchanged without overwriting its raw artifacts. Invoke `analyze-aiperf-results` again after the rerun.

If repair would change workload semantics or a bounded rerun repeats the same invalid result, set `next_action` to
`stop`. Do not write performance analysis or promote the candidate until a valid rerun exists. Never discard an
invalid run.

## Select Comparable History

Use only valid runs whose benchmark-series ID matches the active plan. From that set identify:

- `series_baseline`: earliest valid result in the series;
- `previous_valid`: most recent valid iteration before the current one;
- `best_prior`: best prior run for each objective, respecting metric direction and SLO feasibility;
- `history`: every valid same-series iteration.

Verify that every reference required by the plan is present before making a direct comparison. If no prior valid
same-series result exists, treat the current result as the series baseline and report absolute performance only.
Cross-series results may provide context but never a gain, loss, or Pareto calculation.

## Analyze

1. Evaluate every target SLO and report the observed statistic, threshold, pass/fail, and missing evidence.
2. Report goodput and good-request fraction when configured, including the attainment target.
3. Report throughput, output throughput per GPU, output throughput per user, TTFT, ITL, request latency, errors, and
workload-shape metrics available for the run.
4. For trace workloads, include time-sliced behavior when it reveals warmup leakage, bursts, collapse, or instability.
5. When comparable references exist, compare current versus each highlighted prior and produce a compact same-series
history table.
6. Calculate signed percent change as `(current - prior) / prior * 100`. Also state whether the value is higher or
lower and whether that direction is an improvement or regression.
7. Classify an absolute performance change at or below the measured noise floor of the active benchmark series (see
`comparison-uncertainty.md`) as noise and report it without recommending a repeat
solely because the delta is small.
8. Analyze one valid run by default. A clear, substantial, plausible gain or loss may support a conclusion without a
repeat; state that it is single-run evidence.
9. Use multi-run confidence intervals and coefficient of variation only when deliberate comparable repetitions exist
and those statistics help resolve the decision. Do not treat degraded single-run intervals as confidence evidence.
10. Recommend another valid run only when the existing evidence cannot support a consequential decision, another run
is likely to resolve the uncertainty, and the information value justifies the GPU cost. Record that rationale. If
uncertainty remains but a repeat is not justified, report `inconclusive` and stop.
11. State whether the current candidate is characterized, SLO-feasible, Pareto-improving, mixed, regressed, or
inconclusive, as supported by the available evidence.
12. Identify client-visible symptoms and missing measurements. Do not convert them into kernel, communication,
scheduler, router, or backend root-cause claims.
13. Mark recipe-proxy results as proxy-scoped and carry the workload mismatch into limitations.

## Write Analysis Artifacts

Write `performance_analysis.json` containing:

- current candidate, performance question, benchmark-plan identity, and benchmark-series identity;
- target-SLO evaluation;
- absolute current metrics;
- comparisons to `series_baseline`, `previous_valid`, and per-objective `best_prior` when available;
- full valid same-series history and any required reference still missing;
- run count and confidence/uncertainty;
- `repeat_decision`: `not_needed`, `necessary`, or `not_justified`, with the GPU-cost rationale and the decision the
repeat is expected to resolve;
- verdict, client-visible symptoms, missing evidence, and limitations.

Write `performance_analysis.md` with a concise executive verdict, SLO table, current metrics, applicable comparisons,
same-series history, insights, and limitations.

Append one compact record to `EXP_ROOT/analysis/performance_findings.jsonl` containing the iteration, verdict, primary
absolute metrics, applicable deltas, SLO status, and paths to the full artifacts. Preserve prior records.
109 changes: 109 additions & 0 deletions .agents/skills/configure-aiperf-benchmark/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
---
name: configure-aiperf-benchmark
description: >-
Selects and freezes a question-driven AIPerf workload, objective, load policy, and Kubernetes execution manifest for
a successfully deployed Dynamo candidate. Use when a candidate needs performance characterization or a comparable
measurement against a reference.
license: Apache-2.0
metadata:
author: NVIDIA
tags:
- dynamo
- aiperf
- benchmarking
---

# Configure AIPerf Benchmark

Create a reproducible benchmark that answers the current performance question without changing the deployed candidate.
Freeze semantics only for runs used in the same direct comparison.

Read `agent-docs/rules/execution/user-workload.md`,
`agent-docs/rules/benchmarking/benchmark-isolation.md`,
`agent-docs/rules/benchmarking/comparison-uncertainty.md`,
`agent-docs/rules/benchmarking/concurrency-grid.md`,
`agent-docs/rules/benchmarking/evidence-eligibility.md`,
`agent-docs/rules/benchmarking/proxy-workload-selection.md`, and
`agent-docs/rules/benchmarking/series-boundaries.md` before selecting flags. Also read the user workload, deployment
ledger, and the AIPerf documentation matching the pinned source or runtime. Inspect matching Dynamo recipe `perf.yaml`
files when available.

## Inputs

Require:

- exact user-workload path and SHA256;
- current `DEPLOY_ROOT`, applied DGD, deployment ledger, and successful smoke test;
- the baseline-characterization goal or approved candidate's performance question and target operating region; and
- an exact existing benchmark-plan path, SHA256, and series ID only when considering series reuse.

## Workflow

1. Require a successful smoke test and resolve the in-cluster frontend endpoint, served model, tokenizer, Kubernetes
context, namespace, artifact collection path, and GPU count.
2. Decide whether the run is an absolute characterization or a direct comparison. For a comparison, identify the
candidate and reference measurements needed to answer the question.
3. Reuse an existing series only when its workload and measurement semantics still answer the question. Otherwise
assign a new series ID and require any comparison reference to be measured under the new plan.
4. Select workload input in this order:
- user-provided Mooncake trace;
- exact user-provided ISL/OSL and traffic controls;
- closest Dynamo recipe trace as a `recipe_proxy`.
5. Validate a selected trace as JSONL. Record its path, SHA256, row count, timestamp range, ISL/OSL distribution,
prefix/hash information, and any rows outside the served context limit. Do not silently filter or clip rows.
6. Choose the experiment that best exposes the behavior under investigation:
- `trace_fidelity`: preserve timestamps and fixed-schedule behavior;
- `static_shape`: preserve the exact synthetic ISL/OSL target;
- `capacity`: vary concurrency or request rate over a bounded range.
7. Select the objective and metrics needed for the question:
- with target SLOs, use goodput/good-request fraction and the stated attainment constraints;
- without sufficient SLOs, preserve a throughput/latency/error Pareto view.
8. Configure one measured run by default. Set the measurement duration to 30 minutes or less. For request-count or
fixed-schedule workloads, choose or validate a count or schedule that is expected to complete within that limit and
set a bounded execution timeout. Stop rather than silently truncate or weaken a workload that cannot fit.
9. Do not enable repetitions only to obtain confidence intervals. Configure more than one run only when prior analysis
documents why another run is necessary to resolve a consequential decision and why that value justifies the GPU
cost.
10. Pin the AIPerf runtime version or source commit. Write the immutable series plan and the run-scoped
`<DEPLOY_ROOT>/benchmark/aiperf-config.yaml` and `<DEPLOY_ROOT>/benchmark/perf.yaml`.

## Benchmark Plan

For a new series, write:

```text
<EXP_ROOT>/inputs/benchmark-plans/<series-id>.json
```

Use a filesystem-safe series ID. Never overwrite a plan. When reusing a series, use its exact existing path and SHA256
and change only endpoint address, Job identity, namespace, or artifact wiring in the run-scoped files.

The plan must identify:

- performance question, target operating region, and characterization or comparison intent;
- plan ID, benchmark-series ID, and any required reference candidates;
- user-workload path and SHA256;
- workload source: `user_trace`, `user_static`, or `recipe_proxy`;
- trace path/hash or exact synthetic distribution;
- fixed-schedule, concurrency, request-rate, request-count/duration, warmup, a per-run measurement limit of no more
than 30 minutes, repetitions with a default of one, any approved repeat rationale, confidence policy and level when
needed, and seed;
- endpoint type, streaming behavior, model, and tokenizer;
- target metrics, SLOs, goodput thresholds, and optimization direction;
- AIPerf source commit when available and required runtime version;
- proxy rationale and limitations when applicable.


## Manifest Rules

- `perf.yaml` is a Kubernetes Job, not the AIPerf-native config.
- Run AIPerf inside the cluster and use the target context and namespace.
- Prefer an existing compatible recipe Job as a starting point. Modify only the run-scoped copy.
- Enforce the 30-minute measurement limit with AIPerf workload controls. Do not rely only on a Kubernetes Job timeout.
- Ensure raw artifacts remain available after Job completion, either on a PVC or in the completed pod until copied.
- Read referenced secret names from existing manifests; never copy secret values into benchmark files.
- Do not embed a local host path that is unavailable to the benchmark pod.

Stop with a concrete configuration blocker when the performance question is missing, no defensible workload can be
selected, a required comparison cannot be made, or the planned run cannot fit within 30 minutes without changing its
semantics.
Loading
Loading