diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 28486bd5ac..91e09e6381 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -35,6 +35,15 @@ /plugins/dotnet-ai/skills/mcp-csharp-test/ @leslierichardson95 @mikekistler /tests/dotnet-ai/mcp-csharp-test/ @leslierichardson95 @mikekistler +/plugins/dotnet-ai/skills/configure-agentic-perf-rules/ @leslierichardson95 @cathysull +/tests/dotnet-ai/configure-agentic-perf-rules/ @leslierichardson95 @cathysull + +/plugins/dotnet-ai/skills/scan-agentic-app-perf/ @leslierichardson95 @cathysull +/tests/dotnet-ai/scan-agentic-app-perf/ @leslierichardson95 @cathysull + +/plugins/dotnet-ai/skills/setup-maf-evals/ @leslierichardson95 @cathysull +/tests/dotnet-ai/setup-maf-evals/ @leslierichardson95 @cathysull + # dotnet-upgrade (migrating and upgrading .NET projects) /plugins/dotnet-upgrade/skills/thread-abort-migration/ @dotnet/appmodel @dotnet/skills-upgrade-reviewers /tests/dotnet-upgrade/thread-abort-migration/ @dotnet/appmodel @dotnet/skills-upgrade-reviewers diff --git a/eng/known-domains.txt b/eng/known-domains.txt index fd3b5061c9..3e01c10e17 100644 --- a/eng/known-domains.txt +++ b/eng/known-domains.txt @@ -30,12 +30,14 @@ developer.android.com developer.apple.com # Repos +github.com/dotnet/ai-samples github.com/dotnet/aspnetcore github.com/dotnet/csharplang github.com/dotnet/diagnostics github.com/dotnet/dotnet github.com/dotnet/dotnet-docker github.com/dotnet/efcore +github.com/dotnet/extensions github.com/dotnet/maui github.com/dotnet/msbuild github.com/dotnet/roslyn diff --git a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/SKILL.md b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/SKILL.md new file mode 100644 index 0000000000..b210dcc51a --- /dev/null +++ b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/SKILL.md @@ -0,0 +1,240 @@ +--- +name: configure-agentic-perf-rules +version: 0.3.0 +description: > + Installs or updates an always-on rules block in a .NET agentic app that makes coding + agents volunteer perf and cost concerns by default — agent count, handoff edges, + per-agent model selection, message-history strategy, per-turn token cost, and + post-change measurement. The rules are written into the project's agent-instructions + file (`.github/copilot-instructions.md` by default) inside a sentinel-delimited managed + block that is idempotent and version-aware on update. + WHEN: a .NET project using Microsoft Agent Framework (`Microsoft.Agents.AI`) — with + or without Aspire/Foundry — where the user reports "Copilot doesn't catch perf + issues", wants up-front guard-rails before adding more agents/handoffs/tools, or is + scaffolding a new agentic .NET app. + NOT-WHEN: non-agentic .NET projects (use `optimizing-dotnet-performance`), + non-.NET agentic projects, auditing existing code (use `scan-agentic-app-perf`), or + measuring runtime telemetry (use `setup-maf-evals`). +license: MIT +--- + +# Configure Agentic Perf Rules + +This skill writes a managed block of always-on instructions into the target project's +agent-instructions file so coding agents (Copilot, etc.) volunteer agentic-perf concerns +during normal work, instead of waiting to be asked. The block is delimited by sentinel +HTML comments and embeds the skill version so future runs can update it cleanly without +clobbering user-edited threshold values. + +## When to Use + +- A .NET project uses Microsoft Agent Framework (`Microsoft.Agents.AI`) — with or + without Aspire/Foundry — and the user wants default-on perf guidance. +- Scaffolding a new MAF agentic .NET app (Aspire-hosted or plain console / ASP.NET + Core / worker service) and the user wants to start with perf guard-rails in place. +- The user reports that the coding agent is not catching perf issues until prompted. +- The user wants to update an existing managed block to a newer version of the rules. + +## When Not to Use + +- The project is not a .NET agentic app — use `optimizing-dotnet-performance` for general + .NET performance guidance. +- The user wants the agent to actually audit existing code right now — use + `scan-agentic-app-perf` instead. This skill only installs guidance. +- The user wants to measure tokens, latency, or quality scores — use `setup-maf-evals`. +- Generic prompt-engineering or non-perf coding-agent rules (keep those in the user's own + instructions section, outside the managed block). + +## Inputs + +| Input | Required | Description | +|-------|----------|-------------| +| Target project root | Yes | Repository root containing the .NET solution | +| Existing instructions file | No | Path to existing `.github/copilot-instructions.md` or `AGENTS.md` if non-default | +| Custom thresholds | No | Per-project override values for agent count, handoff edges, token warn levels | + +## Workflow + +> **Outcome:** the project's agent-instructions file contains an up-to-date managed +> rules block. Re-running this skill is safe and idempotent. + +### Step 1: Locate or create the target instructions file + +In priority order, look for: + +1. `.github/copilot-instructions.md` (preferred — GitHub Copilot native location). +2. `AGENTS.md` at repository root (cross-tool standard). +3. None of the above — create `.github/copilot-instructions.md`. Create the `.github/` + directory if it does not exist. + +If both `copilot-instructions.md` and `AGENTS.md` exist, the managed block goes in +`copilot-instructions.md` and a one-line stub is written to `AGENTS.md` pointing to it. + +### Step 2: Detect any existing managed block + +A managed block is delimited by sentinel HTML comments: + +```markdown + +... + +``` + +Scan the target file. If a managed block is present, parse its version from the +`BEGIN` comment. + +**Sentinel parse rules** (apply in order; fail closed if any rule trips): + +1. The full-line BEGIN regex is `^\s*$`. +2. The full-line END regex is `^\s*$`. +3. The file must contain **exactly one** matching BEGIN and **exactly one** matching END, + with BEGIN appearing before END. +4. If zero, multiple, mismatched, or out-of-order sentinels are detected, **refuse to + edit** and report the malformed state to the user. Do not attempt to repair the file + automatically. +5. Versions are compared numerically as semver triples; a leading `v` is optional in + either side of the comparison. + +The current skill version is the `version:` field at the top of this SKILL.md. + +| Detected state | Action | +|----------------|--------| +| No block present | Append a new managed block at the end of the file | +| Block present, same version, structurally valid (all six rule headings present, threshold YAML parses) | No-op — report "already current" and stop | +| Block present, same version, structurally invalid | Treat as "older version": show diff and offer to repair after explicit user confirmation | +| Block present, older version | Show a diff to the user; replace the block on confirm; preserve any user-edited threshold values from the existing frontmatter | +| Block present, newer version than this skill | Refuse to downgrade — report version mismatch and stop | + +**Threshold preservation algorithm** (used in the older-version path): + +1. Parse the existing managed block's `thresholds:` YAML map into a `prev_user` dict. + If parsing fails, refuse to edit and ask the user to repair the YAML manually. +2. Construct the new defaults map `new_defaults` from `references/threshold-defaults.md`. +3. For each known key in `new_defaults`, override with the value from `prev_user` if + present and the value passes type validation (e.g. integer for `per_turn_input_token_warn`). +4. Drop unknown keys from `prev_user` with a chat warning naming each dropped key. +5. The merged map becomes the new managed block's `thresholds:` content. + +**Path safety:** before any write, resolve the project root and the target path. +Refuse to write to any path outside the project root (after normalization, including +following symlinks). Reject absolute paths and paths containing `..` segments unless +they normalize back inside the project root. + +### Target-file selection (when both `.github/copilot-instructions.md` and `AGENTS.md` exist) + +Scan **both** files for an existing managed block before choosing a target. + +| Situation | Action | +|-----------|--------| +| Neither file has a block | Write the managed block into `.github/copilot-instructions.md`; add a one-line stub to `AGENTS.md` if it exists | +| Only `AGENTS.md` has a block | Migrate it to `.github/copilot-instructions.md` (with diff + user confirmation), then replace the `AGENTS.md` block with the stub | +| Only `.github/copilot-instructions.md` has a block | Update there as usual; add a stub to `AGENTS.md` if it exists and is missing one | +| Both have a block | Refuse to edit and ask the user to consolidate; report which file was last modified | + +### Step 3: Render the managed block + +The managed block has three parts in this exact order: + +1. **Sentinel BEGIN comment** with the skill version. +2. **Threshold frontmatter** (a fenced YAML code block) so users can override numeric + defaults without editing prose. The default values are in + `references/threshold-defaults.md`. +3. **Six rule sections**, one per category, in the order listed in the next step. Each + section is short — a "before X, justify Y" lead sentence, the default threshold, and + one or two crisp expectations. Long-form rationale lives in the reference docs and is + not duplicated in the project's instructions file. + +See `references/managed-block-template.md` for the exact rendered output, including the +threshold frontmatter format and section ordering. + +### Step 4: Write the six rule categories + +Each rule is in the form **"Before X, justify Y."** Categories, in order: + +1. **Agent count.** Before adding a new agent to a workflow, justify why the new + responsibility cannot be a tool call on an existing agent. Default ceiling: 3 agents + per workflow. +2. **Handoff edges.** Before adding an LLM-routed handoff edge, justify why a + deterministic edge or a conditional `WorkflowBuilder` branch will not work. Default + ceiling: 2 LLM-routed edges traversed per user turn. +3. **Model selection.** Before defaulting to a frontier model (e.g. `gpt-4o`), name the + agent's role and pick from the role table inside rule #3 of the managed block. + Routers/validators/formatters/workers → small-fast; planners → reasoning-class. +4. **Message-history strategy.** Before sending the full conversation history to an + agent, state the bound — turn count, token cap, summarization point, or retrieval + strategy. Default warning when unbounded full-history is used in a multi-turn workflow. +5. **Token / cost surfacing.** Before implementing a non-trivial change to an agent's + prompt, tools, or model, estimate per-turn token cost. Default warnings: more than + 8000 input tokens or more than 2000 output tokens projected per turn, or any change + that adds more than 20% to a measured baseline. +6. **Post-change measurement.** After a non-trivial change to a workflow, propose + running `setup-maf-evals` (or an existing `.Evals` project) to confirm the change is + net-positive — or explicitly note why measurement is not warranted. + +Long-form rationale, examples, and counter-examples for each rule live in +`references/rule-rationales.md`. + +### Step 5: Update the cross-tool stub (if applicable) + +If `AGENTS.md` exists alongside `.github/copilot-instructions.md`, ensure `AGENTS.md` +contains (or has appended) a single line: + +```markdown +> Agentic-perf rules for this project live in `.github/copilot-instructions.md` (managed by `configure-agentic-perf-rules`). +``` + +This avoids duplicating the rules across files while keeping cross-tool agents pointed +at the right source. + +### Step 6: Commit guidance + +If the project is a Git repository and the user wants the change committed, use a single +commit message of the form: + +``` +Install configure-agentic-perf-rules vX.Y.Z + +Adds always-on agentic-perf guidance to .github/copilot-instructions.md. +``` + +Do not commit on the user's behalf without confirmation. + +## Validation + +After the skill runs, the agent must verify: + +1. **File exists** at the resolved target path. +2. **Sentinel comments present** with matching `BEGIN`/`END` markers and a parseable + version in the `BEGIN` comment. +3. **Threshold frontmatter parses** as valid YAML (no syntax errors introduced). +4. **All six rule sections** are present, in the canonical order. +5. **Round-trip:** re-running the skill against the now-updated file is a no-op (reports + "already current"). If a second run produces any modification, the install was not + idempotent and the skill failed. + +If `AGENTS.md` was updated, also confirm the stub line is present exactly once. + +## Common Pitfalls + +- **Managing user-authored content.** Never modify content outside the sentinel block. + All edits stay strictly between `BEGIN` and `END` markers. +- **Threshold preservation on update.** When updating to a newer version of the rules, + preserve any user-edited values in the threshold frontmatter rather than resetting to + defaults. Diff against the old defaults to detect user edits. +- **Version downgrades.** If the file has a newer skill version than the running skill, + refuse to overwrite. Tell the user to update the skill before re-running. +- **Sentinel collision.** If a different tool has authored a similar-looking managed + block (different `BEGIN` text), do not assume it is ours. Match on the exact sentinel + string `BEGIN: managed by configure-agentic-perf-rules`. +- **Don't re-author rules into the prose.** Keep the SKILL.md body and the project's + instructions file in sync via `references/managed-block-template.md` — do not paste + rule prose directly into multiple places. + +## References + +- `references/managed-block-template.md` — the exact rendered template with sentinels + and frontmatter. +- `references/threshold-defaults.md` — default numeric values and the rationale for each. +- `references/rule-rationales.md` — long-form prose for each of the six rule categories, + with examples and counter-examples. +- Companion skills: `scan-agentic-app-perf`, `setup-maf-evals`. diff --git a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/common-pitfalls.md b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/common-pitfalls.md new file mode 100644 index 0000000000..b892442cb2 --- /dev/null +++ b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/common-pitfalls.md @@ -0,0 +1,100 @@ +# Common pitfalls + +Real-world failure modes for `configure-agentic-perf-rules` — observed +during dogfooding (interview-coach v2 first install, behavioral coach +no-op re-run, ELI5Agent fresh install). + +## Sentinel parsing (FAIL CLOSED, always) + +- **Lenient sentinel matching.** The BEGIN and END regexes in step 2 + are intentionally strict (anchored, full-line, exact version + pattern). Do not "fix up" a malformed sentinel by partial-matching + or by accepting trailing whitespace beyond what `\s*$` allows. + Auto-repair of corrupted sentinels is forbidden — the user might + have intentionally renamed the block while migrating, and silent + repair would clobber that. +- **Refusing to edit on multiple BEGIN/END.** If the file contains + more than one BEGIN or END, abort with a chat message that names + every offending line number. Multiple managed blocks usually mean + a merge conflict went unresolved; appending a third would make it + worse. +- **Out-of-order sentinels.** BEGIN must appear before its matching + END. If you find END before BEGIN, treat as malformed and abort — + do not swap them. + +## Threshold preservation + +- **Losing user-edited threshold values on update.** Step 2's + "Threshold preservation algorithm" is the contract: parse the + existing `thresholds:` map into `prev_user`, overlay the new + default map, override each known key from `prev_user`. Skipping + this step and writing the new defaults verbatim is the most + user-visible regression this skill can ship. +- **Silently dropping unknown keys.** If `prev_user` has a key not + in `new_defaults` (e.g. a deprecated threshold), drop it AND emit + a chat warning naming the dropped key. Silent drops break audit + trails — the user needs to know their override no longer applies. +- **Type-validating with `Convert.ToInt32` instead of strict parse.** + `per_turn_input_token_warn: "eight thousand"` should fail validation, not coerce to + some default. Use strict numeric parsing; on failure, keep the + default and warn. + +## Target-file selection + +- **Writing to `AGENTS.md` when `.github/copilot-instructions.md` + exists.** The order in step 2's target-file table is the spec: + `.github/copilot-instructions.md` is the primary destination; + `AGENTS.md` gets a stub pointer. The reverse only happens during + the explicit migration path (existing block in AGENTS.md, none in + copilot-instructions.md). +- **Writing to both files.** "Both have a block" is the abort path — + the user must consolidate manually. Writing the rule prose into + two files would split the source of truth and let the two copies + drift on the next update. +- **Creating `.github/` outside the project root.** If neither file + exists, create `.github/copilot-instructions.md` under the resolved + project root (the directory containing the `.sln`/`.slnx`/`*.AppHost.csproj`). + Never write to a parent directory or a sibling project. + +## Path safety + +- **Following symlinks out of the project root.** Step 2's "Path + safety" rule requires resolving the absolute path AND ensuring it + still starts with the project-root prefix after symlink resolution. + Some CI environments place repos under symlinks; a naive resolve + can end up writing to the symlink target outside the workspace. +- **Accepting `..` in paths.** Reject any path containing `..` + segments before normalization, unless the post-normalization path + is still inside the project root. The simplest safe check: + `Path.GetFullPath(target).StartsWith(Path.GetFullPath(projectRoot))`. + +## Cross-tool stub on AGENTS.md + +- **Re-adding the stub on every run.** The stub is one line: + `> Agentic-perf rules for this project live in .github/copilot-instructions.md (managed by configure-agentic-perf-rules).` + Check whether that exact line is already present before appending; + re-running the skill should not grow the file by one line each time. +- **Replacing user prose in AGENTS.md with the stub.** AGENTS.md + often contains real onboarding prose the user wrote. Append the + stub at the bottom if missing; never overwrite existing content. + +## Version handling + +- **Refusing to downgrade is correct.** If the file has a newer + version than this skill, abort cleanly with a chat message naming + both versions. Do not "merge" or "convert" — that's how data loss + happens. +- **Comparing versions as strings.** "v0.1.10" sorts before "v0.1.2" + lexically. Parse into semver triples and compare numerically. + +## Idempotency + +- **No-op path must actually be a no-op.** When the block is present, + same version, and structurally valid, the skill must not touch the + file at all — not even to rewrite identical content. Tooling and + git status both rely on "no change on second run". Verify by + comparing the SHA256 hash before/after; they should match exactly. +- **Tracking "already installed" silently.** Even on a no-op, the + chat output should say "configure-agentic-perf-rules v0.1.0 block + already current — no changes". Quiet no-ops make the user think + the skill didn't run. diff --git a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/managed-block-template.md b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/managed-block-template.md new file mode 100644 index 0000000000..fe69aa65f2 --- /dev/null +++ b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/managed-block-template.md @@ -0,0 +1,103 @@ +# Managed Block Template + +This is the exact content the skill writes into the target instructions file. Variables +in `` are filled in at install time. The outer fence below is **four +backticks** so the inner three-backtick fences in the rendered Markdown are not +interpreted as closing it. + +````markdown + + +## Agentic Performance Rules + +> These rules are managed by the `configure-agentic-perf-rules` skill. Do not edit prose +> inside this managed block — your changes will be overwritten on the next update. +> Numeric defaults can be overridden in the `thresholds` block below; user-edited values +> are preserved across updates. + +```yaml +# thresholds — edit values below to override per-project defaults +thresholds: + per_turn_input_token_warn: 8000 + per_turn_output_token_warn: 2000 + baseline_token_increase_warn_pct: 20 + unbounded_history_warn: true +``` + +When working in this codebase, apply each rule below by default. Each rule is in the +form **"Before X, justify Y."** When you cannot justify, prefer the safer alternative +(do not add the agent, do not add the edge, etc.) and surface the trade-off to the user. + +### 1. Agent count + +Before adding a new agent to a workflow, justify why the new responsibility cannot be a +tool call on an existing agent. Each additional agent multiplies the routing surface and +inflates per-turn token cost (system prompts + tool descriptions are paid per agent). +**There is no hard ceiling** — the answer "this needs a clearly different system prompt, +toolset, or output style" is a valid justification. If you cannot articulate one, prefer +adding a tool to an existing agent. + +### 2. Handoff edges + +Before adding an LLM-routed handoff edge (e.g. via +`AgentWorkflowBuilder.CreateHandoffBuilderWith`), justify why a deterministic edge or a +conditional `WorkflowBuilder` branch will not work. Every LLM-routed edge is an extra LLM +call before the user gets a response. Deterministic routing is faster and cheaper; reserve +LLM routing for decisions that genuinely require reading user intent. + +### 3. Model selection + +Before defaulting to a frontier model like `gpt-4o`, name the agent's role and pick +from the table below. Routers, validators, formatters, and workers almost never need +a frontier model; defaulting to one is the largest single source of unnecessary spend. + +| Role | Pick | +|-------------------------------------|-----------------------------------------------------------------------------------| +| router / validator / formatter | small-fast model (e.g. `gpt-4o-mini` or current cheap-fast in your Foundry catalog) | +| worker / summarizer / extraction | small-fast model, **or** Foundry `model-router` deployment if prompt length varies | +| planner / decomposer / open reasoning | reasoning-class model (e.g. `o4-mini` or current reasoning model) — state *why* in a code comment | +| creative / nuanced generation | frontier (e.g. `gpt-4o`) — state *why* in a code comment | + +If unsure which role applies, **stop and ask the user** — do not default to `gpt-4o`. +Specific model ids age fast; check your Foundry catalog for the current cheap-fast, +reasoning-class, and frontier ids before pinning. + +### 4. Message-history strategy + +Before sending the full conversation history to an agent, state the bound — turn count, +token cap, summarization point, or retrieval strategy. Unbounded full-history sends in +a multi-turn workflow are flagged when **`unbounded_history_warn`** is true. + +### 5. Token / cost surfacing + +Before implementing a non-trivial change to an agent's prompt, tools, or model, estimate +per-turn token cost. Default warnings: + +- More than **`per_turn_input_token_warn`** input tokens projected per turn +- More than **`per_turn_output_token_warn`** output tokens projected per turn +- Any change that adds more than **`baseline_token_increase_warn_pct`**% to a measured + baseline + +When any of these trips, surface the projection to the user before implementing. + +### 6. Post-change measurement + +After a non-trivial change to a workflow (new agent, new edge, model swap, prompt +rewrite), propose running `setup-maf-evals` (or an existing `.Evals` project) to confirm +the change is net-positive on the metrics that matter — or explicitly note why +measurement is not warranted (e.g. cosmetic refactor with no behavioral change). + + +```` + +## Notes for the skill implementation + +- The outer fence in this file is **four backticks** so the inner three-backtick YAML + fence in the rendered output is preserved verbatim. When transcribing the template, + agents must reproduce the inner three-backtick fences exactly. +- The `` placeholder is filled from the `version:` field in + `SKILL.md`'s frontmatter. Render as `v0.1.0` (lowercase `v`, semver triple). +- The threshold frontmatter is intentionally inside the managed block (not top-of-file + YAML) so it does not interfere with any other YAML frontmatter the project may have. +- Update mode preserves user-edited threshold values per the algorithm in `SKILL.md` + step 2 ("Threshold preservation algorithm"). Do not duplicate that logic here. diff --git a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/rule-rationales.md b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/rule-rationales.md new file mode 100644 index 0000000000..03a465fbc6 --- /dev/null +++ b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/rule-rationales.md @@ -0,0 +1,176 @@ +# Rule Rationales + +Long-form rationale, examples, and counter-examples for each of the six rules in the +managed block. The managed block itself is intentionally terse; this file is the +"why" behind each rule, used by the agent when explaining a violation to the user. + +--- + +## 1. Agent count + +**Rule.** Before adding a new agent to a workflow, justify why the new responsibility +cannot be a tool call on an existing agent. **No hard ceiling** — the answer "this needs +a clearly different system prompt, toolset, or output style" is a valid justification. + +**Why it matters.** Each additional agent multiplies the routing surface area (the LLM +has to decide whether *this* turn should go to *that* agent) AND inflates per-turn +input-token cost: every agent's system prompt + tool descriptions are paid on every +turn the agent participates in. Decision quality degrades as choices grow. + +**When a new agent is justified.** The new responsibility involves a meaningfully +different *system prompt*, *toolset*, or *output style* that would muddy the existing +agent's instructions if folded in. Examples: + +- **Yes, new agent:** A "Coder" agent that writes code and a "Reviewer" agent that + critiques code — clearly different roles, different prompts. +- **No, just a tool:** A "Database Reader" agent that looks up rows. This is a tool on + whatever agent needs the data, not its own agent. +- **No, just a tool:** A "Formatter" agent that reformats output. Again, a tool. + +**On thresholds.** Earlier versions of this rule had an `agent_count_max: 3` numeric +ceiling. It was removed because legitimate designs (e.g. specialist-handoff +interview/coaching workflows with 4-5 well-scoped agents) tripped it as often as +genuine bloat did. The justify-the-add gate is the actual mechanism; a number was +illusory precision. + +--- + +## 2. Handoff edges + +**Rule.** Before adding an LLM-routed handoff edge, justify why a deterministic edge or +a conditional `WorkflowBuilder` branch will not work. **No hard ceiling.** + +**Why it matters.** Every LLM-routed edge is an additional LLM call before the user +gets a response. Deterministic routing — "after Coach runs, always return to +Interviewer" expressed as a `WorkflowBuilder` edge — costs zero extra latency and zero +extra tokens. Free-form LLM routing also amplifies decision-quality variance. + +**When LLM routing is justified.** The decision genuinely requires reading the user's +intent — for example, "is this answer detailed enough to grade?" or "which specialist +domain does this question belong to?". When the decision is mechanical ("after coach +runs, always go back to interviewer"), use a deterministic edge. + +**Concrete pattern (good):** +- Interviewer ↔ Coach with one LLM-routed edge from Interviewer ("ready to grade?") + and a deterministic edge back from Coach (always returns to Interviewer for the next + question). + +**Concrete pattern (bad):** +- Five specialists in a fully-connected handoff graph where every transition is + LLM-routed. Symptom: Copilot constantly proposes new edges as the workflow grows. + +**On thresholds.** Earlier versions of this rule had an +`llm_routed_edges_max_per_turn: 2` numeric ceiling. Same reason as rule #1: the gate +is the justification, not the number. A 4-hop deterministic chain with one LLM-routed +intent decision is fine; two LLM-routed edges per turn in a misdesigned graph is not. + +--- + +## 3. Model selection + +**Rule.** Before defaulting to a frontier model (e.g. `gpt-4o`), name the agent's role +and pick from the table below. If unsure which role applies, **stop and ask the user** +— do not silently default to `gpt-4o`. + +**Why it matters.** A frontier model on a router/triage agent costs roughly 10x more +per token than `gpt-4o-mini` and is often *worse* at the routing job (frontier models +are tuned for nuanced generation, not cheap classification). The default-everything-to- +gpt-4o pattern is the largest single source of unnecessary spend in agentic apps. + +**Role → model class:** + +| Role | Pick | Why | +|-------------------------------------|-------------------------------------------------------|--------------------------------------------------------| +| Router / triage / "is this done?" | small-fast (`gpt-4o-mini`, current cheap-fast id) | Classification, not generation; latency dominates | +| Validator / scorer / structured JSON| small-fast + JSON mode + low temp | Deterministic output; cache-friendly | +| Formatter (Markdown / JSON shape) | small-fast, pinned | Output stability matters more than peak quality | +| Worker / summarizer / extraction | small-fast, **or** Foundry `model-router` if prompt length varies | Most calls happen here; latency dominates | +| Planner / decomposer / reasoning | reasoning-class (`o4-mini`, current reasoning id) | Output drives N downstream calls; quality matters most | +| Creative / nuanced generation | frontier (`gpt-4o`, current frontier id) | Genuinely needs frontier capability | + +**When frontier is justified.** The agent's job is genuinely creative or nuanced +generation, or it must follow complex instructions reliably. Routers, validators, +formatters, and most workers almost never fall in this bucket. + +**Specific model ids age fast.** The table above uses `gpt-4o-mini`, `o4-mini`, and +`gpt-4o` as anchor examples. Before pinning, check your Foundry catalog +(https://learn.microsoft.com/azure/foundry/openai/concepts/models) for the current +cheap-fast, reasoning-class, and frontier ids. Foundry's `model-router` deployment is +the recommended pick whenever the prompt length or complexity genuinely varies per +request and you don't need cache stability (typical: worker tier). + +**State the why in code.** When you pick a frontier or reasoning model, leave a +one-line comment naming the role and why the cheaper tier wouldn't work. This makes +the choice auditable and the next person can challenge it. + +--- + +## 4. Message-history strategy + +**Rule.** Before sending the full conversation history to an agent, state the bound — +turn count, token cap, summarization point, or retrieval strategy. + +**Why it matters.** Per-turn token cost grows linearly in history length. A workflow +that sends 50 turns of history into every LLM call is paying for 50 turns of attention +on every single response. This is the most common token-bloat pattern in handoff-style +workflows, where every agent in the chain re-sees everything. + +**Acceptable bounds (in order of preference):** + +1. **Sliding window:** keep only the last N turns. Cheap, simple, bounded. +2. **Summarization checkpoint:** every M turns, replace the oldest portion with a + summary. Preserves long-range context with a fixed-size payload. +3. **Retrieval / per-agent context:** each agent receives only the messages relevant + to its role, not the full transcript. Highest engineering cost; biggest savings. + +**When unbounded full history is justified.** Single-shot interactions (no multi-turn +loop), or workflows where the entire history is short by construction (e.g. always +under 2K tokens). + +--- + +## 5. Token / cost surfacing + +**Rule.** Before implementing a non-trivial change to an agent's prompt, tools, or +model, estimate per-turn token cost. Default warnings: more than 8000 input tokens or +2000 output tokens per turn, or any change that adds more than 20% to a measured +baseline. + +**Why it matters.** The window between "looks fine in dev" and "$5K/month surprise" is +measured in token counts, and most contributors do not look at token counts at dev time +unless something is on fire. Surfacing projected token cost *before* implementation +catches the bloat at the cheapest possible moment. + +**How to estimate.** Token count is roughly characters / 4 for English prose; +structured output is denser. For prompt changes, count the new prompt body. For tool +additions, count the schema + description per tool * expected frequency. For model +swaps, multiply by the new model's per-token price ratio. + +**When to skip.** Cosmetic changes that do not alter prompt or tool schema (e.g. +renaming a class). Changes to non-LLM code paths. + +--- + +## 6. Post-change measurement + +**Rule.** After a non-trivial change to a workflow, propose running `setup-maf-evals` +(or an existing `.Evals` project) to confirm the change is net-positive — or explicitly +note why measurement is not warranted. + +**Why it matters.** "Trial and error" is the default tuning loop in agentic apps and +it produces survivor-bias outcomes — changes that *seemed* better are kept; changes +whose downsides did not surface in the first few hand-tested turns get baked in. A +small eval suite (3-10 scenarios) closes this loop with data. + +**Trigger conditions.** Any of the following count as a "non-trivial change" warranting +a measurement proposal: + +- Adding or removing an agent +- Adding or rewiring a handoff edge +- Swapping a model +- Substantially rewriting an agent's instructions +- Adding or substantially modifying a tool + +**When skipping is fine.** Cosmetic refactors. Test-only changes. Changes that the +existing eval suite already covers — in that case, the proposal is to *run* the suite, +not to author new scenarios. diff --git a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/threshold-defaults.md b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/threshold-defaults.md new file mode 100644 index 0000000000..d2ce7698d9 --- /dev/null +++ b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/threshold-defaults.md @@ -0,0 +1,34 @@ +# Threshold Defaults + +Each numeric threshold in the managed block has a default value and a rationale. These +defaults are starting points, not absolutes — projects with different shapes (e.g. very +simple two-agent workflows, or complex tool-heavy pipelines) should adjust. + +| Threshold | Default | Rationale | +|-----------|---------|-----------| +| `per_turn_input_token_warn` | `8000` | Modern reasoning models can take 100K+, but most chat-class models start showing meaningful latency and cost above ~8K input tokens. Projects with retrieval/RAG legitimately exceed this — override locally. | +| `per_turn_output_token_warn` | `2000` | Output tokens are usually 4-10x more expensive than input on a per-token basis. 2000 is a reasonable "are you sure?" threshold; long-form generation tasks should override. | +| `baseline_token_increase_warn_pct` | `20` | A 20% increase per turn meaningfully changes monthly bills at scale. Small tweaks under 20% are noise; over 20% is worth surfacing. | +| `unbounded_history_warn` | `true` | Default-on. Sending full history forever is the single most common token-bloat pattern. Disable only if the workflow has already implemented a windowing/summarization strategy and the warning is now noise. | + +**Note on removed thresholds.** Earlier versions of this skill (≤v0.2.0) shipped +`agent_count_max: 3` and `llm_routed_edges_max_per_turn: 2`. Both were taste-call +ceilings that tripped legitimate designs (5-specialist handoff workflows, multi-hop +deterministic chains with one LLM-routed intent decision) as often as they caught real +bloat. The justify-the-add gates in rules #1 and #2 are the actual mechanism; the +numbers were illusory precision and were removed in v0.3.0. + +## Adjusting thresholds + +Users override defaults inside the managed block's `thresholds:` YAML map. The skill +preserves these overrides on update (it merges new defaults underneath, so user values +win for any key they set). + +Examples of legitimate overrides: + +- A RAG-heavy workflow with retrieval that routinely sends 20K input tokens — set + `per_turn_input_token_warn: 25000`. +- A long-form drafting tool that generates 5K outputs per turn — set + `per_turn_output_token_warn: 6000`. +- A workflow that has implemented summarization-at-N-turns — set + `unbounded_history_warn: false`. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md new file mode 100644 index 0000000000..bae2d4c0a6 --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md @@ -0,0 +1,244 @@ +--- +name: scan-agentic-app-perf +description: | + Scan a .NET agentic application (MAF; Aspire/Foundry optional) across seven perf/cost/reliability categories — topology, tool inventory, message history, prompt weight, parallelism, OTel coverage, per-agent model assignment. Writes .copilot/perf-reports/scan-.md with severity-tagged findings (critical/warn/info), file:line citations, evidence, and next actions routing into configure-agentic-perf-rules or setup-maf-evals. Topologies: Aspire AppHost, plain console, ASP.NET Core, worker service. WHEN: user asks "why is my agent slow", "scan/audit my agentic app", "find perf issues", "is my topology too complex", or just changed a topology. NOT-WHEN: install always-on rules (use configure-agentic-perf-rules), wire evaluations (use setup-maf-evals); not for non-agentic .NET apps. Read-only. Supported topologies: Aspire AppHost, plain console, ASP.NET Core, worker service (Aspire-specific checks such as the AppHost-only model literal apply only when an AppHost is detected). +--- + +# scan-agentic-app-perf + +Run a structured audit of a .NET agentic application and produce a single +Markdown report listing the perf, cost, and reliability issues that matter, +each with a concrete next action. + +This skill is **read-only**. It never edits source. The output is a report file +plus a short chat summary of top findings. + +## When to Use + +- The user asks "why is my agent slow", "scan my agentic app", "audit my + agentic app", "find perf issues", or "is my topology too complex". +- The user has just modified an agent topology (added an agent, added a + handoff edge, swapped a model, added tools) and wants a sanity check + before merging. +- A coding agent is about to make a non-trivial change to a `.NET` + agentic app and wants the current perf/cost baseline to compare + against post-change. + +## When Not to Use + +- The user wants to install **always-on** rules so future agent work + surfaces perf concerns by default — use `configure-agentic-perf-rules`. +- The user wants to wire **runtime** measurement (latency, tokens, + cost, quality scores) into the project — use `setup-maf-evals`. +- The project is not a .NET agentic app — use `optimizing-dotnet-performance` + for general .NET performance guidance. +- The user wants the skill to **fix** the findings, not just report + them — this skill is read-only by design; route fixes through + `configure-agentic-perf-rules` (for guard-rails) or normal coding + agent work (for code changes). + +## Supported topologies + +The skill targets any .NET project using Microsoft Agent Framework +(`Microsoft.Agents.AI`), regardless of how it's hosted: + +- **Aspire AppHost** (`*.AppHost.csproj` orchestrating agent services) — the most + common shape; enables AppHost-specific checks such as the "model literal lives + only in the AppHost" check. +- **Plain console / worker service** — `Microsoft.Agents.AI` registered directly + against an OpenAI or Foundry client without an Aspire AppHost. +- **ASP.NET Core minimal API** — agents registered via the same DI patterns. + +AppHost-specific checks (those that depend on `*.AppHost.csproj` being present) +are silently skipped when no AppHost is detected. All other checks (topology, +tool inventory, message history, prompt weight, parallelism, OTel coverage, +per-agent model assignment) apply to every topology. + +## Workflow + +### 1. Inventory the app + +Detect and record: + +- AppHost project (`*.AppHost.csproj`) and agent service projects +- Agent registrations (`AddAgent`, `ChatClientAgent`, `IChatClient` builders) +- Agent count, handoff edges, tool count per agent +- OTel wiring (`AddOpenTelemetry`, Aspire dashboard reference) + +If no agentic app is detected, abort and tell the user this skill does not +apply. Do not attempt to audit a non-agentic .NET project. + +### 2. Run the seven check classes + +Each check class lives in a reference doc and is run in turn. Detection logic +and finding templates are in `references/`: + +| # | Category | Reference | +|---|-------------------|--------------------------------------------| +| 1 | Topology | `references/topology-checks.md` | +| 2 | Tool inventory | `references/tool-inventory-checks.md` | +| 3 | Message history | `references/message-history-checks.md` | +| 4 | Prompt weight | `references/prompt-weight-checks.md` | +| 5 | Parallelism | `references/parallelism-checks.md` | +| 6 | OTel coverage | `references/otel-coverage-checks.md` | +| 7 | Model assignment | `references/model-assignment-checks.md` | + +For each check, record any findings with the schema in step 3. + +### 3. Finding schema + +Every finding is a dict with these fields: + +```yaml +severity: critical | warn | info +check: one of the slugs listed in references/check-glossary.md + (e.g. model.same-default, history.full-share, otel.missing-sdk) +title: short imperative phrase +file: path relative to repo root +line: 1-based line number, or null +evidence: 1-3 line code snippet or measurement (must be present in the cited file) +why: one paragraph explaining the impact +next: concrete action the developer can take next +ref: optional cross-skill route (e.g. "skill:configure-agentic-perf-rules") +``` + +The `check` field is a dotted slug: `.`. The +prefix before the dot encodes the category — there is no separate +`category` field. Category card (also rendered at the top of the +report's `## Findings` section): + +| Category | Coverage | Reference | +|------------|-------------------------|----------------------------------------| +| `topology` | agent graph shape | `references/topology-checks.md` | +| `tools` | per-agent tool list | `references/tool-inventory-checks.md` | +| `history` | chat history strategy | `references/message-history-checks.md` | +| `prompt` | system prompt size/reuse| `references/prompt-weight-checks.md` | +| `parallel` | concurrent invocations | `references/parallelism-checks.md` | +| `otel` | instrumentation | `references/otel-coverage-checks.md` | +| `model` | per-agent model id | `references/model-assignment-checks.md`| + +See `references/check-glossary.md` for the full slug→description table. + +**Evidence gate** — before adding a finding to the report, re-open the +cited file and confirm the `evidence` snippet is present at or near +the cited line. Drop any finding that fails this check. Findings whose +evidence is "absence of X" must list the exact files and patterns +that were searched in the `evidence` field instead of a snippet. + +Severity rules: + +- **critical** — likely to break a user-visible flow, blow the token budget, + or cause a cost spike. Always surfaced in chat. +- **warn** — measurable perf or cost regression, but app still works. +- **info** — observation worth knowing, no action required. + +### 4. Aggregate and write the report + +Sort findings by severity (critical → warn → info), then by `check` +slug (stable lexical order so `history.*` < `model.*` < `otel.*` < +`parallel.*` < `prompt.*` < `tools.*` < `topology.*`). + +Write a **single file**: `.copilot/perf-reports/scan.md`, overwritten +on every run. Git provides whatever history you want; the skill does +not maintain timestamped copies or `latest-*` mirrors. + +The category legend (one-liners describing what each prefix covers) is +inlined into the report itself — see `references/report-template.md`. +There is no separate glossary file. The slug encodes the check (e.g. +`history.full-share`, `topology.cycle`); readers don't need a lookup +table. + +Create `.copilot/perf-reports/` if it does not exist. + +This skill never touches `.gitignore`. If the user wants the report +ignored, recommend in chat that they add `.copilot/perf-reports/` to +their `.gitignore` themselves; do not edit it from this skill. + +### 5. Surface top findings in chat + +Print: + +1. Total counts (critical / warn / info). +2. The first up to 3 critical findings with title + `check` slug + file:line + next action. +3. The full report path. +4. If any findings have a `ref:` field, list the suggested follow-up skills. + +Do not paste the entire report into chat. + +### 6. Offer to route into follow-up skills + +After surfacing the top findings, **ask the user once** whether they +want to act on the routed follow-ups. The skill itself never edits +source — this step only routes into a sibling skill that owns its own +diff-and-confirm flow. + +1. Aggregate the unique `ref:` values across all findings. +2. If the set is non-empty, print one prompt of the form: + > Want me to follow up on any of these? + > - **A.** Install/update perf rules via `configure-agentic-perf-rules` to enforce role-aware model selection on future code. + > - **B.** Run `setup-maf-evals` to capture token/quality numbers. + > - **C.** Run `configure-agentic-perf-rules` to install always-on rules. + > - **D.** No — just leave the report. + Render only the lettered options that correspond to refs actually + present in the findings (skip letters whose target skill is not + referenced). +3. Wait for the user's response. If they pick a letter, hand off to the + named skill with the audit report path as context. If they pick + "no" or anything else, stop. +4. Do **not** infer intent from the original audit-request prompt + ("audit and fix it" is *intent* — the follow-up skill must still + present its own diff and obtain its own confirmation before any + write). This skill's job ends at the routing offer. + +### 7. Stop + +This skill never edits source. If the user declined the offer in +step 6, or if there were no `ref:` fields in any finding, the skill +ends here. + +## Validation + +After running: + +- A file exists at `.copilot/perf-reports/scan.md` (overwritten if it + was there before). +- The report has a `## Findings` section, even if empty (containing + `_No findings._`). +- The summary counts in the report match the chat output. + +## Common pitfalls + +- **Editing source code.** This skill is read-only. The ONLY write path + it owns is `.copilot/perf-reports/scan.md`. Never edit `.gitignore`, + source files, config files, or anything else. If a check tempts you + to fix the issue inline, stop and add it as a finding instead. +- **Hallucinating findings.** Every finding must cite a real file and + (where applicable) a real line. Before adding a finding to the + report, re-open the cited file and verify the snippet exists at the + cited line. If you cannot point to evidence, drop the finding. +- **Burying critical findings.** Always lift the top 3 critical + findings into chat. Do not say "see the report" without surfacing + the worst issues. +- **Confusing this with rules install.** If the user wants the rules + themselves embedded into their instructions file, run + `configure-agentic-perf-rules` instead. +- **Running on non-agentic apps.** If no agent registrations are found, + abort cleanly. Do not invent an audit for a plain web API. +- **Recommending a model downgrade without an eval gate.** Any MA* + finding that says "downgrade Agent X from gpt-4o to gpt-4o-mini" + must be paired in the `next:` field with "validate via + `setup-maf-evals` quality mode before shipping". Apparent free wins + on cost frequently regress quality on edge cases — the eval gate + protects against that. + +## References + +- `references/topology-checks.md` — agent count, handoff edges, cycles. +- `references/tool-inventory-checks.md` — tools per agent, redundancy, dead tools. +- `references/message-history-checks.md` — full-history sharing, summarization. +- `references/prompt-weight-checks.md` — system-prompt size, per-agent token cost. +- `references/parallelism-checks.md` — sequential calls that could fan out. +- `references/otel-coverage-checks.md` — Aspire dashboard, token/cost telemetry. +- `references/model-assignment-checks.md` — single-model defaulting, role mismatch. +- `references/check-glossary.md` — dev-facing catalog of all check slugs (NOT copied to user repos; for skill maintainers). +- `references/report-template.md` — exact Markdown layout for `scan.md`. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-glossary.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-glossary.md new file mode 100644 index 0000000000..3faa29a706 --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-glossary.md @@ -0,0 +1,41 @@ +# Check glossary (dev-facing) + +Canonical catalog of every `check` slug the skill can emit. This file +is for **skill maintainers** — adding or renaming a check requires a +row here. It is **not copied** into user repositories; the slugs are +self-describing in the report itself. + +If you change this table, also update the matching `*-checks.md` +reference file with the per-check detection logic. + +## Catalog + +| Check | Sev | What it catches | +|------------------------------------|----------|-----------------| +| `topology.cycle` | critical | Directed cycle in the agent handoff graph | +| `topology.deep-single-leaf` | warn | 3+ hops to reach a single terminal agent | +| `tools.duplicate` | warn | Same tool description on >1 agent | +| `tools.dead` | info | Tool registered but never referenced | +| `tools.description-too-long` | warn | `[Description]` attribute > 200 chars | +| `history.full-share` | critical | Entire chat history passed to a downstream agent | +| `history.unbounded` | warn | No `MaxMessages` / reducer / summarizer wired | +| `history.through-deterministic` | warn | Full history given to a deterministic agent | +| `prompt.oversized` | warn | System prompt > 2K tokens (critical at > 4K) | +| `prompt.duplicate-preamble` | warn | Same > 100-token block shared across agents | +| `parallel.independent-handoffs` | warn | Sequential `await`s on agents that don't share data | +| `parallel.hidden-tool-fanout` | info | Tool internally serializes 3+ API calls | +| `otel.missing-sdk` | critical | No `AddOpenTelemetry` / Aspire dashboard wiring | +| `otel.no-aspire-dashboard` | warn | Aspire dashboard not declared | +| `otel.no-token-cost` | warn | No `gen_ai.usage.*` tags surfaced | +| `otel.no-per-agent-source` | info | All agents share one `ActivitySource` | +| `model.same-default` | warn | All agents use the same model id | +| `model.reasoning-on-deterministic` | warn | Frontier model on a formatter / validator | +| `model.cheap-on-planner` | warn | Small model on the planner, larger on workers | +| `model.hardcoded` | info | Model id literal in agent service `.cs` | + +## Cross-skill routes + +| `ref:` value | Skill to run next | +|--------------------------------------|------------------------------------------------------------| +| `skill:configure-agentic-perf-rules` | Install/update the always-on perf rules (rule #3 covers role-aware model selection) | +| `skill:setup-maf-evals` | Wire eval reports + telemetry | diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/common-pitfalls.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/common-pitfalls.md new file mode 100644 index 0000000000..9f9bd25cc6 --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/common-pitfalls.md @@ -0,0 +1,95 @@ +# Common pitfalls + +Real-world failure modes for `scan-agentic-app-perf` — observed during +dogfooding (interview-coach v1/v2, ELI5Agent, behavioral-interview-coach). + +## Output discipline + +- **Editing source code.** This skill is read-only. The ONLY write path + it owns is `.copilot/perf-reports/scan.md`. Never touch `.gitignore`, + source files, config files, AppHost, or `*.csproj`. If a check tempts + you to fix the issue inline, stop and add it as a finding instead — + the user opts into the fix through the step-6 routing prompt. +- **Surfacing more than 3 critical findings in chat.** Step 5 caps + chat output at 3 critical findings + counts + report path. Anything + beyond that goes in the report file only. Pasting the entire report + into chat defeats the "single file you can re-open" UX. +- **Generating extra report files.** One file: `scan.md`. Do not + write timestamped copies, `latest-*` mirrors, glossary files, or + any other artifact. Git is the history mechanism if the user wants + one. + +## Evidence integrity + +- **Hallucinating findings.** Every finding must cite a real file + and (where applicable) a real line. Before adding a finding, re-open + the cited file and verify the snippet exists at the cited line — that + is the "evidence gate" in step 3. If you can't point to evidence, + drop the finding. A 5-finding report with verified citations beats + a 15-finding report with two hallucinations every time. +- **Citing line ranges without re-reading.** Files drift between + the inventory pass (step 1) and the per-category checks (step 2). + Re-read the cited region right before writing each finding, not + before the entire batch. +- **Absence-of-X findings without searched-pattern evidence.** When + a finding fires because something is *missing* (e.g. `otel.no-token-cost` "no token + surfacing"), the `evidence` field MUST list the exact files and + patterns that were searched — not a snippet. This is what lets the + user reproduce the negative result. + +## False positives we keep seeing + +- **`model.hardcoded` firing on AppHost code.** This check is about + model ids living in *agent service* files (`*.Agent/Program.cs` + etc.) where swapping requires a code change. Model ids declared in + the AppHost via `foundry.AddDeployment("chat", FoundryModel.OpenAI.Gpt4oMini)` + are the **canonical Aspire-native place** — that's not a defect. + When you encounter this pattern, either suppress `model.hardcoded` + entirely or downgrade to `info` with a "no action required" `next:`. +- **`model.same-default` on single-agent apps.** The check explicitly + assumes ≥2 agents (different roles, different needs). Do not fire on + 1-agent apps; the check is trivially "satisfied" with one model. +- **`otel.no-token-cost` when `Microsoft.Extensions.AI` activity + source is registered.** MEAI emits `gen_ai.usage.*` activity tags + automatically; if `AddSource("Microsoft.Extensions.AI")` is wired in + OTel and an OTLP exporter is configured, this check should NOT + fire. The check is for codebases that strip the source or wrap MEAI + behind custom infrastructure that loses the tags. + +## Per-check sharpening + +- **`prompt.oversized`.** Use a rough token estimator (chars/4) or + `cl100k_base` if available. Never claim an exact token count without + naming the encoder you used. +- **`tools.duplicate`.** Compare tool *descriptions* (the + `[Description("...")]` attribute), not method names. Two tools with + the same method name but different descriptions are usually fine; + two tools with different names and a copy-pasted description are + usually a refactor candidate. + +## Routing offer (step 6) + +- **Inferring intent from the original prompt.** Even if the user + said "audit and fix it", this skill's job ends at the routing + prompt. The follow-up skill is responsible for its own + diff-and-confirm flow. Never call into `configure-agentic-perf-rules` + (apply mode) without the user explicitly + picking that letter at the prompt. +- **Listing routes for skills not actually referenced.** Step 6 says + to render only letters whose target skill is named in some + finding's `ref:` field. If no finding routed to `setup-maf-evals`, + do not offer it. Empty routing offer = skip the prompt entirely. + +## Inventory edge cases + +- **Multiple AppHost projects.** If a solution has more than one + `*.AppHost.csproj`, scan each one and emit one report per host as + `.copilot/perf-reports/scan-.md` (e.g. `scan-WebApp.md`, + `scan-Worker.md`). Do NOT merge — the topology, model set, and + OTel wiring belong to each host independently. Each per-host file + is still overwritten on each scan; no timestamp suffixes. +- **Agent registered via DI lambda without `AddAIAgent`.** Patterns + like `services.AddSingleton(...)` or custom factories also + count as agents. The detection in step 1 must include any path + that ends up registering an `IAgent` / `AIAgent` in the host's + service provider. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/message-history-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/message-history-checks.md new file mode 100644 index 0000000000..db98a2b686 --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/message-history-checks.md @@ -0,0 +1,39 @@ +# Message-history checks + +Detect strategies that pass too much history to too many agents. + +## Checks + +### `history.full-share` (critical) + +**Detect:** code that passes the entire `IList` (or +`History`) from the entry agent to a downstream agent without +filtering, slicing, or summarizing. + +**Why:** every additional agent that sees the full history pays the +input token cost. With 4 agents and a 6K-token history, you spend 24K +input tokens per turn doing nothing. + +**Next:** "Pass only the last user message and a one-paragraph summary +to ``. Use `IChatHistoryReducer` or a manual slice." + +### `history.unbounded` (warn) + +**Detect:** no usage of `MaxMessages`, `IChatHistoryReducer`, +summarization tool, or sliding-window code anywhere in agent setup. + +**Why:** unbounded history = monotonically growing per-turn cost. + +**Next:** "Wire a `ChatHistoryReducer` with `MaxMessages = 20` or +summarize-and-replace at the agent level." + +### `history.through-deterministic` (warn) + +**Detect:** an agent whose role is purely deterministic (formatter, +validator, tool router) is given full chat history. + +**Why:** deterministic steps do not need conversational context. Their +prompt cost should be near-constant. + +**Next:** "Pass only the immediate input artifact to ``; drop +the chat history." diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/model-assignment-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/model-assignment-checks.md new file mode 100644 index 0000000000..dbcdfe5577 --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/model-assignment-checks.md @@ -0,0 +1,66 @@ +# Model-assignment checks + +Detect single-model defaulting and role-model mismatch. + +## Checks + +### `model.same-default` (warn) + +**Detect:** every agent is constructed with the same model id (e.g. +`gpt-4o-mini`). Look at the AppHost connection string or the +`IChatClient` builder per agent service. Requires ≥2 agents — skip on +single-agent apps. + +**Why:** different agent roles have different latency and quality +needs. A single default usually overspends on cheap roles and +underspends on hard roles. + +**Next:** "See rule #3 in `.github/copilot-instructions.md` (managed by +`configure-agentic-perf-rules`) — most apps want a small-fast model +for routers/validators/workers and a reasoning-class model only for +planners. If both agents are workers on the same cheap model, this is +expected and can be downgraded to info." + +**Ref:** `skill:configure-agentic-perf-rules` + +### `model.reasoning-on-deterministic` (warn) + +**Detect:** an agent whose prompt and tool list indicate a +deterministic role (formatter, validator, classifier with ≤3 outputs) +is using a frontier reasoning model. + +**Why:** the marginal quality is near-zero; you are paying for unused +capability and per-call latency. + +**Next:** "Downgrade `` to a small-fast model per rule #3 in +`.github/copilot-instructions.md`. Validate via `setup-maf-evals` +quality mode." + +**Ref:** `skill:configure-agentic-perf-rules` + +### `model.cheap-on-planner` (warn) + +**Detect:** the agent that decides the plan or decomposes the task is +on a small model while leaf workers are on a large one. + +**Why:** plan-quality drives every downstream call. A bad plan from a +cheap planner makes the expensive workers run more turns. + +**Next:** "Promote the planner to a reasoning-class model per rule #3 +in `.github/copilot-instructions.md`; consider demoting one or more +workers." + +**Ref:** `skill:configure-agentic-perf-rules` + +### `model.hardcoded` (info) + +**Detect:** model id literal (e.g. `"gpt-4o-mini"`) appears inside an +agent service `.cs` file rather than `appsettings.json` or AppHost +parameters. AppHost code that declares model ids via +`foundry.AddDeployment(...)` is the canonical Aspire pattern and does +NOT trigger this check. + +**Why:** swapping models for an A/B becomes a code change. + +**Next:** "Move model ids into `appsettings.json` and bind them via +`IOptions<...>`, or declare them in the AppHost as Aspire deployments." diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/otel-coverage-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/otel-coverage-checks.md new file mode 100644 index 0000000000..d25687e6e8 --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/otel-coverage-checks.md @@ -0,0 +1,54 @@ +# OTel coverage checks + +Detect missing instrumentation that makes perf invisible at dev time. + +## Checks + +### `otel.missing-sdk` (critical) + +**Detect:** the AppHost or service projects do not call +`builder.Services.AddOpenTelemetry()` and do not import +`OpenTelemetry.Exporter.OpenTelemetryProtocol` or +`Aspire.Hosting.Dashboard`. + +**Why:** without OTel, you cannot see per-call latency, token counts, +or spans. Every other check in this audit becomes a guess. + +**Next:** "Add `builder.AddServiceDefaults()` (Aspire) or wire OTel +manually with HTTP + Activity sources for `Microsoft.Extensions.AI`." + +### `otel.no-aspire-dashboard` (warn) + +**Detect:** the AppHost does not declare the dashboard, or the +`appsettings.json` lacks a `Dashboard:OtlpEndpointUrl`. + +**Why:** the dashboard is the cheapest way to see per-agent token use +during local dev. + +**Next:** "Run with `dotnet run --project ` and ensure the +dashboard URL is logged. If not, install `Aspire.Hosting.Dashboard`." + +### `otel.no-token-cost` (warn) + +**Detect:** no log, no meter, no tag for `gen_ai.usage.input_tokens` / +`gen_ai.usage.output_tokens` anywhere in the codebase. + +**Why:** without token telemetry the team has no early-warning signal +for prompt bloat. Cost spikes are discovered in the bill, not the +dashboard. + +**Next:** "Microsoft.Extensions.AI emits `gen_ai.*` activity tags +automatically. Confirm the OTel exporter forwards them, or run +`setup-maf-evals` to capture them in eval reports." + +**Ref:** `skill:setup-maf-evals` + +### `otel.no-per-agent-source` (info) + +**Detect:** all agents share a single activity source name; no way to +filter the dashboard by agent. + +**Why:** with 3+ agents, traces become unreadable without filtering. + +**Next:** "Give each agent its own `ActivitySource` named after the +agent role." diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/parallelism-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/parallelism-checks.md new file mode 100644 index 0000000000..e27f78e3e9 --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/parallelism-checks.md @@ -0,0 +1,37 @@ +# Parallelism checks + +Detect sequential agent invocations that could run concurrently. + +## Checks + +### `parallel.independent-handoffs` (warn) + +**Detect:** two consecutive `await downstreamA.RunAsync(...)` and +`await downstreamB.RunAsync(...)` calls in the same method where B's +input does not depend on A's output. + +**Why:** the second call could start as soon as the inputs are known. +Each sequential LLM hop adds full per-call latency. + +**Next:** "Run `` and `` with `Task.WhenAll`. Rejoin in the +parent agent for the consolidation step." + +### `parallel.hidden-tool-fanout` (info) + +**Detect:** a tool method that internally loops and calls 3+ external +APIs sequentially. + +**Why:** tools hide their own latency from the agent. A single slow +tool that is internally serial is the hardest kind of latency to find +from the outside. + +**Next:** "Parallelize the inner calls in ``; document the +expected bound in the tool description so the agent can plan around +it." + +## What used to live here + +`parallel.sequential-awaits` (was `P1`, a generic `foreach (var x in +xs) await ...` pattern) was removed — that's a general .NET concurrency +anti-pattern, not specific to agents. For that class of finding run +`optimizing-dotnet-performance` instead. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/prompt-weight-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/prompt-weight-checks.md new file mode 100644 index 0000000000..7e76736b0c --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/prompt-weight-checks.md @@ -0,0 +1,40 @@ +# Prompt-weight checks + +Detect oversized system prompts and per-agent prompt cost. + +## Checks + +### `prompt.oversized` (warn at >2K tokens / critical at >4K) + +**Detect:** count tokens (or chars / 4 as approximation) in each +agent's `Instructions` or system prompt string. + +**Why:** every turn pays this cost. A 4K-token system prompt at +$0.005/1K input tokens × 1000 turns/day = $20/day per agent on prompt +overhead alone. + +**Next:** "Move static rules into a tool the agent can call when +needed, or split the prompt into a short policy section and a separate +few-shot example doc retrieved on demand." + +### `prompt.duplicate-preamble` (warn) + +**Detect:** two or more agents share the same > 100-token block at the +start or end of their system prompts. + +**Why:** the same tokens are billed N times per turn (once per agent). + +**Next:** "Lift the shared block into a single deterministic +preprocessor or attach it as a tool result rather than a system prompt +repeat." + +## Token estimation + +If a real tokenizer is not available, approximate as `chars / 4`. Mark +findings using approximation as `(estimated)` in the evidence. + +## What used to live here + +`prompt.too-many-fewshots` (was `PW2`, fired at >3 few-shot examples) +was a taste-based threshold. Few-shot count alone is not a reliable +signal; `prompt.oversized` already captures the cost dimension. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/report-template.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/report-template.md new file mode 100644 index 0000000000..818ceabdc7 --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/report-template.md @@ -0,0 +1,63 @@ +# Report template + +The exact Markdown layout written to `.copilot/perf-reports/scan.md`. +This file is **overwritten on every run**. Git provides history; the +skill does not maintain timestamped copies or `latest-*` mirrors. + +```markdown +# Agentic perf scan — {{ project_name }} + +Run: {{ utc_timestamp }} +Project: {{ relative_project_path }} + +## Inventory + +- AppHost: `{{ apphost_path }}` +- Agents: {{ agent_count }} ({{ agent_list }}) +- Handoff edges: {{ edge_count }} +- Tools (total): {{ tool_count }} +- Distinct models: {{ model_set }} +- OTel wired: {{ true | false }} + +## Summary + +- critical: {{ count }} +- warn: {{ count }} +- info: {{ count }} + +## Findings + +> **Slug format:** `.` — categories are +> `topology` (graph shape) · `tools` (per-agent tool list) · +> `history` (chat history strategy) · `prompt` (system prompt +> size/reuse) · `parallel` (concurrent invocations) · `otel` +> (instrumentation) · `model` (per-agent model id). Severities: +> `critical` (likely to break a flow or blow the budget) · +> `warn` (measurable cost/perf regression) · `info` (observation). + +### [critical] [`{{ check }}`] {{ title }} +- **File:** `{{ file }}:{{ line }}` +- **Evidence:** + ```csharp + {{ snippet }} + ``` +- **Why:** {{ paragraph }} +- **Next:** {{ action }} +- **Cross-ref:** {{ skill: ... | omit if none }} + +(... repeat per finding, ordered: critical → warn → info, then by `check` slug ...) + +## Next steps + +- If you want to fix the model assignments above, see rule #3 in `.github/copilot-instructions.md` (managed by `configure-agentic-perf-rules`). +- If you want to capture token/quality numbers before vs after, run + `setup-maf-evals`. +- If you do not yet have always-on rules to prevent regressions, run + `configure-agentic-perf-rules`. +``` + +## Empty-report contract + +If there are zero findings, the `## Findings` section still appears +(after the legend blockquote) with the literal text `_No findings._`. +The `## Summary` section shows zeros. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/tool-inventory-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/tool-inventory-checks.md new file mode 100644 index 0000000000..180c4abeee --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/tool-inventory-checks.md @@ -0,0 +1,46 @@ +# Tool inventory checks + +Detect bloat and redundancy in the per-agent tool list. + +## Checks + +### `tools.duplicate` (warn) + +**Detect:** two or more tools across different agents with the same +description or near-identical signatures. + +**Why:** duplication forces the router LLM to disambiguate every turn +and inflates aggregate prompt size. + +**Next:** "Consolidate `` and `` into a single shared +tool exposed by both agents." + +### `tools.dead` (info) + +**Detect:** tools registered but never invoked in any code path or +call trace. Static check: tool name does not appear in any agent's +system prompt or instructions. + +**Why:** every registered tool costs prompt tokens whether it gets +called or not. + +**Next:** "Remove `` from ``'s tool list." + +### `tools.description-too-long` (warn) + +**Detect:** a tool's `[Description]` attribute or `description:` field +is longer than 200 characters. Measure after string concatenation — +multi-line `+` concatenations count as one description. + +**Why:** long descriptions multiply across agents that import the +tool. Most tools can be described in one sentence. + +**Next:** "Trim ``'s description from `` chars to ≤ 200; move +the detailed contract into XML docs on the parameters." + +## What used to live here + +`tools.too-many-per-agent` (was `TI1`) was a taste-based threshold +(>8 tools per agent) that fired on legitimate designs. The real +question — "is this tool earning its prompt-token weight" — is better +served by `tools.dead` and `tools.duplicate`. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/topology-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/topology-checks.md new file mode 100644 index 0000000000..7f07e4e5fb --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/topology-checks.md @@ -0,0 +1,40 @@ +# Topology checks + +Detect structural issues in the agent graph that drive latency or runaway loops. + +## Checks + +### `topology.cycle` (critical) + +**Detect:** any directed cycle in the static handoff graph. + +**Why:** cycles risk infinite loops if the loop-break condition is +LLM-judged. Even with a turn cap, a cycle burns budget on retries. + +**Next:** "Break the cycle `` → `` → `` by making ``'s exit +condition deterministic." + +### `topology.deep-single-leaf` (warn) + +**Detect:** graph that always ends at one agent but routes through 3+ +agents to reach it. + +**Why:** the intermediate hops are usually classification or routing +that could be one tool call. + +**Next:** "Move the routing logic into a tool on the entry agent and +call `` directly." + +## Out of scope here + +- Tool counts → see `tool-inventory-checks.md`. +- Per-agent model selection → see `model-assignment-checks.md`. + +## What used to live here + +`topology.agent-count` (was `T1`) and `topology.handoff-fanout` (was +`T2`) were removed in v0.2 — both were taste-based thresholds (>3 +agents, >2 handoff edges) that fired on legitimate designs as often +as on real bloat. If you want broad architectural feedback, run +`configure-agentic-perf-rules` so rule #1 (single-agent default) and +rule #2 (handoff justification) can guide the design at scaffold time. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md b/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md new file mode 100644 index 0000000000..1d83571fb1 --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md @@ -0,0 +1,291 @@ +--- +name: setup-maf-evals +description: | + Scaffold an `.Evals.Tests` MSTest project alongside a .NET agentic app (MAF; Aspire/Foundry optional) wired to the GA `Microsoft.Extensions.AI.Evaluation.Reporting` pipeline. Three evaluator categories: **NLP** (deterministic BLEU/GLEU/F1, no API key), **Quality** (LLM-as-judge Relevance/Coherence/Fluency, etc.), **Safety** (Hate/Violence/SelfHarm/Sexual via Foundry). Auto-installs the `aieval` tool, detects the app's `IChatClient` registration and generates a factory so `EVAL_USE_REAL_AGENT=1` works without manual wiring, and emits an HTML report at `.copilot/perf-reports/evals//report.html`. Optional GitHub Actions workflow runs evals on every PR. Topologies: Aspire AppHost, plain console, ASP.NET Core, worker service. WHEN: user asks "set up evals", "add evaluation harness", "validate quality after a model change", "compare gpt-4o vs gpt-4o-mini", "add safety evaluators". NOT-WHEN: one-shot audit (use scan-agentic-app-perf), install rules (use configure-agentic-perf-rules). +--- + +# setup-maf-evals + +Scaffold an `.Evals.Tests` MSTest project that measures latency, +token usage, cost, quality, and safety on every change to a .NET +agentic app — and produces the proper Microsoft.Extensions.AI.Evaluation +HTML report by default. + +## When to Use + +- The user asks to "set up evals", "add an evaluation harness", "wire + up MEAI evaluation", or "measure my agent's quality". +- The user wants to **validate quality** after a model change ("does + swapping gpt-4o → gpt-4o-mini regress responses?") and needs a + reproducible baseline. +- The user wants to **compare** model assignments side-by-side + ("compare gpt-4o vs gpt-4o-mini across my agents"). +- The user wants to **add safety evaluators** (Hate/Violence/SelfHarm/ + Sexual via Azure AI Foundry) to an existing agent. +- The user wants a recurring eval run wired into CI (the optional + GitHub Actions workflow). + +## When Not to Use + +- The user wants a **one-shot audit** of existing code without scaffolding + a new test project — use `scan-agentic-app-perf` instead. +- The user wants **always-on perf guard-rails** in the project's + agent-instructions file — use `configure-agentic-perf-rules`. +- The project is not a .NET agentic app, or the user is not using + `Microsoft.Extensions.AI` / `Microsoft.Agents.AI`. +- The user explicitly does not want an MSTest dependency and is not + willing to use the opt-in `--shape console` runner. + +## Supported topologies + +The skill targets any .NET project using Microsoft Agent Framework +(`Microsoft.Agents.AI`), regardless of how it's hosted: + +- **Aspire AppHost** (`*.AppHost.csproj` orchestrating agent services) — the + most common shape; the generated `AgentChatClientFactory` mirrors the + AppHost's connection-string-driven `IChatClient` registration. +- **Plain console / worker service** — `Microsoft.Agents.AI` registered + directly against an OpenAI or Foundry client without an Aspire AppHost. + The factory mirrors the app's direct registration (e.g. `AddOpenAIClient` + or `AddAzureOpenAIChatClient` at the host level). +- **ASP.NET Core minimal API** — agents registered via the same DI patterns. + +`IChatClient` detection (see `references/ichatclient-detection.md`) works +the same way across all topologies; only the AppHost-vs-Program.cs search +locations differ. + +## Workflow + +### 1. Discover the target app + +Detect: + +- Solution file (`*.sln` / `*.slnx`) +- AppHost project name (`*.AppHost.csproj`) +- Agent service projects +- Existing test / eval projects (avoid clobbering) +- **`IChatClient` registration** (scan AppHost + agent projects for + `AddChatClient` / `AddAzureOpenAIChatClient` / `AddOllamaChatClient` + / `AddOpenAIChatClient` and any explicit `services.AddSingleton` or + Foundry deployment alias references). See `references/ichatclient-detection.md`. + +If no agentic app is detected, abort. If a `*.Evals.Tests` project already +exists, switch to update mode (see step 1a). + +### 1a. Update mode (when `*.Evals.Tests` project already exists) + +File classes: + +| Class | Files | Behavior on update | +|------------------|------------------------------------------------------------------------|----------------------------| +| **infra** | `*.Evals.Tests.csproj`, `dotnet-tools.json`, `Reporting/*`, `Wire/*`, test class skeletons | merge package refs; create file if missing; do **not** overwrite | +| **user data** | `Quality/rubric.md`, `Quality/golden.json`, `Compare/matrix.json`, `Telemetry/inputs.json`, `Telemetry/prices.json`, `quality.thresholds.json`, `.github/workflows/evals.yml` | never overwrite; create if missing | +| **generated** | `.copilot/perf-reports/evals/` | regenerate freely | + +If an existing infra file differs from the current template, surface +the diff in the chat output but do not overwrite. Recommend the user +review and merge manually. + +`golden.json` has a `schema_version` field. If the detected version is +older than the current template, offer to migrate (additive only — +preserves existing rows, adds the new fields as nullable). + +### 2. Confirm scope with the user + +Present the detection summary, then confirm: + +1. **Project shape** (default: MSTest). Alternative: console runner + (legacy v1 shape) — only emit if user explicitly asks for it. +2. **Evaluator categories to enable.** Defaults shown; user can override. + + | Category | Evaluators | Cost | Needs | Default | + |------|-----------|------|-------|---------| + | **Quality** *(headline)* | Relevance, Coherence, Fluency, Completeness, Equivalence, Groundedness; agent: IntentResolution, TaskAdherence, ToolCallAccuracy | per-call judge tokens | real `IChatClient` + `EVAL_USE_REAL_JUDGE=1` | **ON** (stubbed until judge wired) | + | NLP (zero-config sanity) | BLEU, GLEU, F1, Words | free | reference responses in golden.json | ON | + | Safety | `ContentHarmEvaluator` (Hate+SelfHarm+Violence+Sexual single-shot), ProtectedMaterial, IndirectAttack, CodeVulnerability, UngroundedAttributes, GroundednessPro | Foundry evaluation service charges | Azure AI Foundry endpoint + `EVAL_USE_FOUNDRY_SAFETY=1` | **OFF** — prompt the user: "Wire safety evaluators too? (y/N)" | + + Frame Quality as the headline evaluation; NLP is the zero-config + first-run experience that emits a real `report.html` before any + creds exist; Safety is the opt-in for production-bound apps. + +3. **IChatClient detection result.** Show what was detected (e.g., "Found + `AddAzureOpenAIChatClient` in `AppHost.cs:41` with deployment alias `chat`"). + Ask the user to confirm or override. If detection failed, generate a + stub factory the user will fill in. +4. **Run modes to scaffold.** Telemetry (default ON) and Quality (default + ON). **Compare mode is opt-in** — prompt the user: "Wire compare + mode for side-by-side matrix.json runs? (y/N)". Compare adds the + largest scaffold (extra runner + matrix.json + delta-table generator) + and is the least commonly used surface. +5. **Optional add-ons:** Aspire dashboard panel, GitHub Actions workflow. + +### 3. Scaffold the project + +See `references/project-template.md` for the file tree, csproj, and +`GlobalUsings.cs` template. The project is **MSTest** by default +(`.Evals.Tests`); a console-runner shape is available behind an +explicit `--shape console` flag. + +Always emit: `Reporting/{ReportingConfig.cs, Tier.cs, AievalReport.cs, +WordCountEvaluator.cs, MetricsGlossary.cs}`, `Wire/{AgentChatClientFactory.cs, +StubChatClient.cs}`, `Quality/{QualityTests.cs, rubric.md, golden.json}`, +`Telemetry/{TelemetryTests.cs, inputs.json, prices.json}`, +`quality.thresholds.json`, `GlobalUsings.cs`, `dotnet-tools.json`. +Emit `Compare/{CompareTests.cs, matrix.json}` only if the user opted into +compare mode (step 2 #4). Emit `Safety/SafetyTests.cs` and +`.github/workflows/evals.yml` only if the user opted in (steps 7 and 9). + +After writing files: + +```pwsh +dotnet sln add .Evals.Tests/.Evals.Tests.csproj +dotnet tool restore # installs aieval +# .gitignore additions +echo ".copilot/perf-reports/evals/`n.Evals.Tests/_store/" >> .gitignore +``` + +### 4. Wire telemetry mode + +See `references/telemetry-capture.md`. Default ON. Captures latency, +input/output tokens, and price-table-driven cost via a delegating +`IChatClient`. Writes `telemetry.{md,json,junit.xml}` next to +`report.html`. **Not** the MEAI HTML report — that's quality mode's job. + +### 5. Wire quality mode + +See `references/quality-modes.md`. Default ON. The **only** runner that +produces `report.html`. Uses `DiskBasedReportingConfiguration` + +`ScenarioRun.EvaluateAsync` + `[AssemblyCleanup]` invokes +`dotnet tool run aieval report`. Stub tier registers the 4 NLP +evaluators; judge tier (`EVAL_USE_REAL_JUDGE=1`) adds the LLM-as-judge +evaluators from `references/evaluators-catalog.md`. + +### 6. Wire compare mode (opt-in) + +See `references/compare-mode.md`. **Default OFF.** Only scaffold when +the user opted in at step 2 (#4). When enabled, reads `matrix.json`; +each entry runs through the **same** `ReportingConfiguration` with a +distinct `executionName`, so `aieval report` aggregates the comparison +columns into a single HTML view. Also writes a `compare.md` delta +table. + +### 7. Wire safety mode (opt-in) + +See `references/safety-mode.md`. **Default OFF.** When enabled, adds +`Microsoft.Extensions.AI.Evaluation.Safety` and emits `SafetyTests.cs` +with `ContentHarmEvaluator` (single-shot 4-metric bundle), plus +ProtectedMaterial / IndirectAttack / CodeVulnerability / +UngroundedAttributes. Skipped at runtime via `Assert.Inconclusive` +when `EVAL_USE_FOUNDRY_SAFETY` is unset — never fails the build for +missing creds. + +### 8. Optional Aspire panel (apply mode) + +See `references/aspire-dashboard-panel.md`. Default OFF; modifies +the AppHost project. To enable, user must say "wire the panel" / +equivalent. Always show a unified diff + ask for confirmation before +writing AppHost edits. + +### 9. Optional CI workflow (opt-in) + +See `references/ci-workflow.md`. Default OFF. Emits +`.github/workflows/evals.yml` that runs `dotnet test` on every PR, +auto-detects tier from repo secrets (`AZURE_OPENAI_ENDPOINT` → +judge; `AZURE_AI_FOUNDRY_ENDPOINT` → safety), and uploads +`report.html` as a workflow artifact. + +### 10. Validation + +- `dotnet build .Evals.Tests.csproj` exits 0. +- `dotnet test .Evals.Tests.csproj` exits 0 in stub tier (no creds needed). + - Stub tier must emit a `report.html` with **≥ 4 distinct metric columns** + (Words, BLEU, GLEU, F1) across all scenarios in golden.json. + - All scenarios must produce non-null metric values (no "—" placeholders). +- If `EVAL_USE_REAL_JUDGE=1` and an `IChatClient` is wired, + `dotnet test` must additionally produce ≥ 3 Quality metrics (Relevance, + Coherence, Fluency). + +### 11. Surface in chat + +Lead with **Quality** as the headline evaluation; frame NLP as the +zero-config sanity check and Safety/Compare as additions. + +1. **Quality (headline).** State whether the judge is wired: + - *Stubbed* — "Quality scaffolded; judge will run once you wire + `EVAL_USE_REAL_JUDGE=1` + a chat endpoint. The next block tells + you how." + - *Live* — "Quality judge active against ``; report + shows Relevance / Coherence / Fluency / Completeness / Equivalence." + - Caveat the user **must** know up-front: *"The built-in Quality + rubrics are generic. Agents with deliberate stylistic constraints + (brevity, persona, format adherence) will score low on + Completeness / Equivalence even when working as designed. See + `references/common-pitfalls.md#tuning-quality-for-stylistic-agents` + for the per-app override pattern."* +2. **Promoting Quality to the judge tier.** If the app's `IChatClient` + reads from a connection string (Aspire pattern), include the exact + two commands: + + ```pwsh + dotnet user-secrets init --project .Evals.Tests + dotnet user-secrets set "ConnectionStrings:" ` + "Endpoint=https://.services.ai.azure.com/models;DeploymentId=" ` + --project .Evals.Tests + # then: $env:EVAL_USE_REAL_AGENT="1"; $env:EVAL_USE_REAL_JUDGE="1"; dotnet test + ``` + + Note the **three** endpoint gotchas: (a) hostname strips dashes on + some resources, (b) key auth is often disabled → drop the `Key=` + segment and rely on `DefaultAzureCredential`, (c) **the judge + deployment must be a non-reasoning model** (gpt-4o / gpt-4o-mini / + gpt-4-turbo). Reasoning models (gpt-5*, o-series) reject `max_tokens` + with HTTP 400 and MEAI silently records that as a per-metric error + row. If the production model is a reasoning one, set + `EVAL_JUDGE_DEPLOYMENT_NAME=` so the judge + points at a different deployment than the agent. Full details in + `references/ichatclient-detection.md`. +3. **NLP (zero-config sanity).** "`report.html` already populates with + Words / BLEU / GLEU / F1 from `golden.json` without any creds — run + `dotnet test` now to see it." +4. **Additional categories you can wire.** List Safety + (`EVAL_USE_FOUNDRY_SAFETY=1`, opt-in scaffold) and Compare (re-run + the skill with `compare: true` if it wasn't opted in originally) + as one-line bullets — not in the main flow. +5. **Paths.** Project path, `report.html` path, **glossary path** + (`metrics-glossary.md` co-located with `report.html`), persistent + `_store/` path. +6. **CLI invocations.** `dotnet test`, `dotnet tool run aieval report`, + and the IChatClient detection result so the user knows what was + auto-wired. +7. **Follow-up recommendation.** "Re-run after swapping a model per + rule #3 of `configure-agentic-perf-rules` to confirm no quality + regression." +8. **Cache payoff.** "First `dotnet test` populates `_store/cache/` and + takes the full ~60s. Every subsequent run against unchanged inputs + reuses cached agent + judge responses — typically ~5s with zero LLM + cost. The Diagnostic Data section of `report.html` shows per-call + Hit/Miss. To force a fresh run, delete `_store/cache/` or change the + rubric / golden inputs." + +Also link `references/evaluators-catalog.md` and +`references/metrics-glossary.md` so the user can see what each metric +means. + +## Common pitfalls + +See `references/common-pitfalls.md`. + +## References + +- `references/project-template.md` — file tree + `.csproj` layout. +- `references/ichatclient-detection.md` — registration scan + factory emission. +- `references/evaluators-catalog.md` — NLP + Quality + Safety catalog with required `EvaluationContext` types. +- `references/metrics-glossary.md` — per-run glossary content + `MetricsGlossary.cs` template. +- `references/telemetry-capture.md` — per-call hook + cost report format. +- `references/quality-modes.md` — `DiskBasedReportingConfiguration` wiring + `aieval report` invocation. +- `references/compare-mode.md` — `matrix.json` + per-entry `executionName`. +- `references/safety-mode.md` — opt-in safety scaffold + `ContentHarmEvaluator` default. +- `references/ci-workflow.md` — `.github/workflows/evals.yml` template. +- `references/aspire-dashboard-panel.md` — optional static-file panel. +- `references/common-pitfalls.md` — known footguns to avoid when scaffolding. +- [MEAI.Evaluation libraries](https://learn.microsoft.com/en-us/dotnet/ai/evaluation/libraries) | [Tutorial: evaluate with reporting](https://learn.microsoft.com/en-us/dotnet/ai/evaluation/evaluate-with-reporting) | [dotnet/ai-samples](https://github.com/dotnet/ai-samples/blob/main/src/microsoft-extensions-ai-evaluation/api/) diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/aspire-dashboard-panel.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/aspire-dashboard-panel.md new file mode 100644 index 0000000000..943e43a871 --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/aspire-dashboard-panel.md @@ -0,0 +1,66 @@ +# Aspire dashboard panel (optional, v1 = static file) + +A minimal way to surface per-agent token/latency live during +`dotnet run` without extending the Aspire dashboard itself. + +## V1 — static file served from AppHost + +1. Telemetry mode (when running long-lived) writes + `wwwroot/eval-panel/telemetry.json` every N calls. +2. AppHost serves a static HTML page at `/eval-panel/` that fetches + the JSON every 2 seconds and renders a table. + +### `wwwroot/eval-panel/index.html` + +```html + +Agentic perf panel + +

Agentic perf panel

+

Last update: —

+ + + + + +
AgentModelCallsAvg msp95 msIn tokOut tok$/1K
+ +``` + +## V1 caveats + +- This is **not** an embedded Aspire dashboard panel; the dashboard + panel API is out of scope here. +- The panel reads only the latest `telemetry.json`; it does not + retain history across runs. +- If the AppHost project does not already enable static files, the + skill adds `app.UseStaticFiles()` (in apply mode only, with a + standard diff-preview-and-confirm flow). + +## Future v2 + +A proper Aspire dashboard contribution is tracked separately. The +v1 static panel buys most of the benefit (live per-agent visibility) +without depending on the dashboard extension story. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/ci-workflow.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/ci-workflow.md new file mode 100644 index 0000000000..3c627a0e51 --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/ci-workflow.md @@ -0,0 +1,119 @@ +# CI workflow (opt-in) + +Off by default. Enabled in step 2 when the user picks "scaffold CI +workflow" / "add GitHub Actions" / equivalent. + +When enabled, emits `.github/workflows/evals.yml`. The workflow: + +- Runs on every PR + on push to `main`. +- Always runs Tier 1 (NLP) — no creds needed, fast smoke test. +- Promotes to Tier 2 (Quality, real judge) when the repo has secrets + `AZURE_OPENAI_ENDPOINT` + `AZURE_TENANT_ID`. +- Promotes to Tier 3 (Foundry Safety) when the repo has secret + `AZURE_AI_FOUNDRY_ENDPOINT`. +- Uploads `report.html` as a workflow artifact. + +## Template + +```yaml +name: evals + +on: + pull_request: + branches: [main] + push: + branches: [main] + workflow_dispatch: + +permissions: + id-token: write # for OIDC -> DefaultAzureCredential + contents: read + +jobs: + evaluate: + runs-on: ubuntu-latest + timeout-minutes: 20 + + env: + EVAL_REPORT_FOLDER: ${{ github.run_id }}-${{ github.run_attempt }} + + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-dotnet@v4 + with: + dotnet-version: 10.0.x + + - name: Restore tools + run: dotnet tool restore + + - name: Azure login (only if creds present) + if: env.AZURE_TENANT_ID != '' + uses: azure/login@v2 + with: + client-id: ${{ secrets.AZURE_CLIENT_ID }} + tenant-id: ${{ secrets.AZURE_TENANT_ID }} + subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }} + env: + AZURE_TENANT_ID: ${{ secrets.AZURE_TENANT_ID }} + + - name: Tier selection + id: tier + run: | + if [ -n "${{ secrets.AZURE_OPENAI_ENDPOINT }}" ]; then + echo "EVAL_USE_REAL_AGENT=1" >> $GITHUB_ENV + echo "EVAL_USE_REAL_JUDGE=1" >> $GITHUB_ENV + echo "AZURE_OPENAI_ENDPOINT=${{ secrets.AZURE_OPENAI_ENDPOINT }}" >> $GITHUB_ENV + echo "tier=judge" >> $GITHUB_OUTPUT + else + echo "tier=stub" >> $GITHUB_OUTPUT + fi + if [ -n "${{ secrets.AZURE_AI_FOUNDRY_ENDPOINT }}" ]; then + echo "EVAL_USE_FOUNDRY_SAFETY=1" >> $GITHUB_ENV + echo "AZURE_AI_FOUNDRY_ENDPOINT=${{ secrets.AZURE_AI_FOUNDRY_ENDPOINT }}" >> $GITHUB_ENV + echo "tier=safety" >> $GITHUB_OUTPUT + fi + + - name: Run evals (dotnet test) + run: dotnet test {{AppName}}.Evals.Tests/{{AppName}}.Evals.Tests.csproj --logger "trx;LogFileName=evals.trx" + + - name: Generate report (already invoked by [AssemblyCleanup], this is a safety net) + if: always() + run: | + mkdir -p .copilot/perf-reports/evals/${{ env.EVAL_REPORT_FOLDER }} + dotnet tool run aieval report \ + --path {{AppName}}.Evals.Tests/_store \ + --output .copilot/perf-reports/evals/${{ env.EVAL_REPORT_FOLDER }}/report.html + + - name: Upload report + if: always() + uses: actions/upload-artifact@v4 + with: + name: eval-report-${{ steps.tier.outputs.tier }} + path: .copilot/perf-reports/evals/${{ env.EVAL_REPORT_FOLDER }}/report.html + + - name: Upload trx + if: always() + uses: actions/upload-artifact@v4 + with: + name: trx + path: '**/evals.trx' +``` + +## Optional: PR comment with summary + +A second job can comment on the PR with a link to the artifact and a +one-line summary scraped from `compare.md`. Not in the default +template — too much variance across teams' bot/preference choices. + +## Required secrets + +| Secret | Tier unlocked | Required for OIDC? | +|--------|---------------|--------------------| +| `AZURE_TENANT_ID` | needed for any Azure auth | yes | +| `AZURE_CLIENT_ID` | OIDC federated identity | yes (if using `azure/login@v2`) | +| `AZURE_SUBSCRIPTION_ID` | scope | yes | +| `AZURE_OPENAI_ENDPOINT` | Tier 2 (judge) | no | +| `AZURE_AI_FOUNDRY_ENDPOINT` | Tier 3 (safety) | no | + +Document these clearly in the chat output when the workflow is scaffolded. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md new file mode 100644 index 0000000000..a62abbfdf5 --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md @@ -0,0 +1,212 @@ +# Common pitfalls + +Footguns that turned up while building and dogfooding this skill. +Avoid them when scaffolding `.Evals.Tests`. + +## Reporting pipeline + +- **Multiple `[AssemblyCleanup]` methods.** MSTest forbids more than + one `[AssemblyCleanup]` per assembly (UTA014 at discovery time — + every test in the assembly fails to load). The + `MetricsGlossary.WriteGlossary` write must be **chained from + `AievalReport.GenerateReport`'s single `[AssemblyCleanup]`**, not + declared as its own. Wrap the chained call in a `try / catch` so a + glossary-write failure never masks the report. +- **Hand-rolling reports instead of using the Reporting pipeline.** + The whole point of GA `Microsoft.Extensions.AI.Evaluation.Reporting` + 10.7.0 is `DiskBasedReportingConfiguration` + `aieval`. Never write + a hand-rolled markdown report and call it the "quality report" — + that's an MEAI report (HTML) vs a cost/latency capture (markdown). +- **Treating telemetry / compare / quality as separate report + streams.** Compare mode goes through `ReportingConfiguration` with + a distinct `executionName` per matrix entry, so `aieval report` + aggregates them into the same HTML. +- **Forgetting the per-run `metrics-glossary.md`.** The aieval HTML is + data-bound JSON; it shows numbers but no definitions. Co-locate + `metrics-glossary.md` (tier-aware) so a first-time reader can decode + the columns. The `Reporting/MetricsGlossary.cs` template handles + this — don't strip it. +- **Misreading "Cache Miss" in the Diagnostic Data section.** With the + default template (`enableResponseCaching: true`, agent resolved via + `run.ChatConfiguration!.ChatClient`, no per-run `executionName`), + the **first** `dotnet test` run shows Miss everywhere (cache empty) + and **subsequent runs against unchanged inputs show Hit everywhere** + in ~5s with zero LLM cost. Persistent Miss after run 1 means one of: + (a) the rubric / golden / scenario inputs changed, (b) the judge + model or chat options changed, (c) `_store/cache/` was deleted, or + (d) something is bypassing the cache — most commonly calling + `Wire.ResolveAgentClient()` (uncached) instead of + `run.ChatConfiguration!.ChatClient`, OR passing a fresh + `executionName` to `DiskBasedReportingConfiguration.Create(...)`. + Hit/Miss never affects correctness; it just tells you whether the + LLM was actually called this run. +- **Snake_case JSON deserialized with default STJ options.** `inputs.json`, + `matrix.json`, `prices.json`, and `golden.json` all use snake_case keys + by template convention, but the C# records use PascalCase. The loaders + (`InputsLoader`, `MatrixLoader`, `PriceTable.Load`, `GoldenLoader`) + **must** specify + `JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower, PropertyNameCaseInsensitive = true }`, + or properties bind to `null`. Quality mode fails loudly (rubric/golden + evaluator throws); telemetry mode can fail **silently** (zeroed records + whose null fields are never read), which corrupts cost rollups. See + `references/telemetry-capture.md` and `references/compare-mode.md` for + the canonical loader sketches. + +## Clients (agent vs judge vs stub) + +- **Calling real models from the default test run.** Stub tier uses + `StubChatClient`; the report banner is clearly marked + `(stub IChatClient)`. Real-model runs are opt-in via three + independent env vars. +- **Conflating agent and judge clients.** Two different `IChatClient` + roles. The skill exposes them as two independent env vars + (`EVAL_USE_REAL_AGENT`, `EVAL_USE_REAL_JUDGE`) — one can be real + while the other is stubbed. +- **Auto-detected factory throwing a generic NRE on missing config.** + When the app uses Aspire orchestration, `dotnet test` runs outside + the AppHost and `ConnectionStrings:` is unset. The factory + template in `ichatclient-detection.md` wraps DI resolution in a + `try / catch` that throws a friendly `InvalidOperationException` + naming the connection-string key and the user-secrets command. Don't + strip this when adapting the template. +- **User-secrets silently not loading in `dotnet test`.** `dotnet test` + runs under `testhost.exe` as the entry assembly, so + `Host.CreateApplicationBuilder()` does NOT pick up the secrets store + bound to your test project's `UserSecretsId`. The factory must call + `builder.Configuration.AddUserSecrets(typeof(...).Assembly, optional: true)` + explicitly. Without it the secret is set on disk but the friendly NRE + still fires. +- **`services.ai.azure.com` hostname strips dashes.** Resource + `foundry-abc` resolves to host `foundryabc.services.ai.azure.com` (no + dash). Authoritative endpoints are in + `az cognitiveservices account show -n -g --query properties.endpoints`. + Use the `AI Foundry API` or `Azure AI Model Inference API` entries — + not the legacy `properties.endpoint` value, which points at the + `cognitiveservices.azure.com` hostname that 404s for the `/models` route. +- **Key-based auth disabled (`403`).** Foundry resources provisioned by + Aspire/azd usually set `disableLocalAuth=true`. Drop the `Key=` + segment from the connection string and rely on `DefaultAzureCredential` + (`az login` + a Cognitive Services User role assignment on the + resource). The Aspire `AddAzureChatCompletionsClient` registration + picks the credential automatically when the key is absent. +- **Reasoning models (gpt-5, gpt-5-mini, o1, o3) reject `max_tokens`.** + The `Azure.AI.Inference` beta SDK still sends `max_tokens`; reasoning + models require `max_completion_tokens` and return 400 + `unsupported_parameter`. **The MEAI Quality evaluators swallow the + 400 and record it as a per-metric error row, so tests pass but every + Quality column is an error.** Two workarounds: + - **Short-term:** pick a non-reasoning judge model (gpt-4o, + gpt-4o-mini, gpt-4-turbo). Tip: when picking a Foundry deployment + to point `ConnectionStrings:` at, check + `az cognitiveservices account deployment list -n -g + --query "[].{name:name, model:properties.model.name}" -o tsv` + and avoid any deployment whose model is `gpt-5*` or `o*`. When the + agent uses a reasoning model in production, set + `EVAL_JUDGE_DEPLOYMENT_NAME=` so the judge + client points at a compatible deployment while the agent client + keeps the production model. + - **Durable (recommended for new apps):** migrate the AppHost off + `AddAzureChatCompletionsClient` (Azure.AI.Inference) onto the + OpenAI/v1 + stable OpenAI SDK path (`AddOpenAIClient` against an + OpenAI/v1 endpoint). The OpenAI SDK natively uses + `max_completion_tokens` against reasoning models. The + `Azure.AI.Inference` beta SDK is retiring on **August 26, 2026**; + see the + [Foundry migration guide](https://learn.microsoft.com/en-us/azure/foundry/how-to/model-inference-to-openai-migration) + and the related issue + [dotnet/extensions#7580](https://github.com/dotnet/extensions/issues/7580). + The skill detects both registration patterns; see + `references/ichatclient-detection.md`. + +## Evaluators + +- **Wiring 4 separate safety evaluators.** Use `ContentHarmEvaluator` + for the Hate / SelfHarm / Violence / Sexual bundle — single Foundry + call, 4 metrics back. The 4 individual evaluators + (`HateAndUnfairnessEvaluator` etc.) are a strict subset. +- **Putting `RelevanceTruthAndCompletenessEvaluator` in the default + set.** Marked experimental upstream; not part of the v2 default. +- **Forgetting `EvaluationContext` for NLP evaluators.** BLEU / GLEU + need `BLEUEvaluatorContext(IEnumerable)`; F1 needs + `F1EvaluatorContext(string)`. If `golden.json` lacks + `reference_response`, NLP evaluators emit `(no reference)` and the + scenario shows blanks in the report. + +### Tuning Quality for stylistic agents + +The built-in Quality evaluators (`RelevanceEvaluator`, +`CoherenceEvaluator`, `FluencyEvaluator`, `CompletenessEvaluator`, +`EquivalenceEvaluator`) use **generic rubrics baked into the +evaluator's judge prompt**. They don't know about *your* agent's +contract. + +- **`CompletenessEvaluator` explicitly rewards thoroughness.** Any + agent whose contract is "compress / summarize / dumb-down" (ELI5, + TL;DR summarizers, tweet-length responders, structured-extractor + agents) will score 1-2 even when working perfectly. The score is + measuring conformance to a generic "answer thoroughly" rubric, not + conformance to your agent's contract. +- **`EquivalenceEvaluator` rewards lexical/semantic match to a + reference.** If your `golden.json` references are full-form + explanations but your agent emits paraphrases or stylized variants, + Equivalence will flag drift that isn't a real regression. +- **`CoherenceEvaluator` and `FluencyEvaluator`** penalize + fragmentary / one-line / heavily-structured outputs (JSON-only, + bullet-only, single-sentence). Agents that respond in a strict + format will be marked down. +- **Only `RelevanceEvaluator` is largely intent-agnostic** — it + checks "did the response address the query" without preferring + long-form. It's the one Quality metric that's mostly safe to leave + on for any agent. + +**Three remediation patterns, in increasing order of effort:** + +1. **Drop the offending evaluators per app.** Edit + `Reporting/ReportingConfig.cs` and remove `CompletenessEvaluator` + / `EquivalenceEvaluator` from the judge-tier evaluator list. + Keep Relevance + Coherence + Fluency. Document the choice in a + `// per-app: ELI5 contract → drop Completeness` comment so future + re-scaffolds don't add them back. +2. **Rewrite goldens in the agent's voice.** Edit `Quality/golden.json` + so `reference_response` is itself ELI5-style (or tweet-style, or + bullet-style). Equivalence becomes meaningful again; Completeness + still complains. +3. **Add a custom rubric-driven evaluator.** Write a `RubricEvaluator` + that reads `Quality/rubric.md` and asks the judge to score the + response against *your* criteria ("Is the explanation appropriate + for a 5-year-old? Score 1-5."). See + `references/evaluators-catalog.md#custom-rubric-driven-evaluator` + for the template. This is the right long-term answer for any + non-generic agent. + +> **Surface this in the chat output on first scaffold.** The user +> shouldn't have to interpret bad Quality scores against a bad-fit +> rubric and conclude their agent is broken. Step 11 of `SKILL.md` +> calls this out explicitly in the Quality headline block. + +## Configuration + +- **Hard-coding a price table.** Lives in `Telemetry/prices.json`, + user-editable. Costs change. +- **Auto-failing the build on quality regressions.** Quality mode is + informational by default. Users opt into a hard-fail by editing + `quality.thresholds.json` (which maps to real MEAI metric names like + `Relevance` / `BLEU` / `F1` and `EvaluationRating` levels), then + setting `hard_fail: true`. +- **Wrong `Microsoft.Extensions.Hosting` version.** Must be `10.0.1` + (not `10.0.0`) to satisfy the transitive constraint from + `Microsoft.Agents.AI.Hosting` 1.x. Pinning `10.0.0` produces + `NU1605` (treated as error in the agentic-app project graph). +- **Forgetting `.gitignore` entries.** Must include both + `.copilot/perf-reports/evals/` and `.Evals.Tests/_store/`. + Otherwise reports pollute history and the persistent `_store/` + blocks PR diffs. + +## Update mode + +- **Overwriting `Quality/rubric.md` or `Quality/golden.json`.** These + are user data — never overwrite. The skill's update-mode behaviour + table in step 1a of `SKILL.md` is the source of truth. +- **Migrating `golden.json` schema destructively.** Migration is + additive only: new fields go in as nullable, existing rows are + preserved. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/compare-mode.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/compare-mode.md new file mode 100644 index 0000000000..44909c5408 --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/compare-mode.md @@ -0,0 +1,132 @@ +# Compare mode + +Compare mode runs the quality + telemetry pipeline against **multiple +model assignments** and produces a side-by-side report. Crucially, it +goes through the **same `DiskBasedReportingConfiguration`** so +`aieval report` aggregates the comparison into a single HTML view. + +## `matrix.json` + +```json +{ + "schema_version": 2, + "entries": [ + { + "name": "baseline-all-mini", + "model_assignments": { + "receptionist": "gpt-4o-mini", + "behavioural": "gpt-4o-mini", + "technical": "gpt-4o-mini", + "summariser": "gpt-4o-mini" + } + }, + { + "name": "interviewers-upgraded", + "model_assignments": { + "receptionist": "gpt-4o-mini", + "behavioural": "gpt-4o", + "technical": "gpt-4o", + "summariser": "gpt-4o-mini" + } + } + ] +} +``` + +## Test shape + +Each matrix entry becomes one row of a parameterised test. Each entry +uses a **distinct `executionName`** so `aieval report` shows the +comparison side-by-side. + +```csharp +[TestClass] +public sealed class CompareTests +{ + public static IEnumerable Matrix() => + MatrixLoader.Load().Select(e => new object[] { e }); + + [TestMethod, DynamicData(nameof(Matrix), DynamicDataSourceType.Method)] + public async Task RunEntry(MatrixEntry entry) + { + // executionName scoped per entry so aieval report groups columns by it. + var reporting = DiskBasedReportingConfiguration.Create( + storageRootPath: ReportingConfig.StorageRoot, + evaluators: ReportingConfig.EvaluatorList(), + chatConfiguration: new ChatConfiguration( + Wire.ResolveJudgeClient(Wire.ResolveAgentClient(entry.ModelAssignments))), + enableResponseCaching: true, + // Stable per-entry name (NOT prefixed with a per-run timestamp) + // so re-running compare reuses the cache for unchanged entries. + executionName: $"compare-{entry.Name}"); + + foreach (var g in GoldenLoader.Load()) + { + var scenarioName = $"Compare.{entry.Name}.{g.Id}"; + await using var run = await reporting.CreateScenarioRunAsync(scenarioName); + // ... same shape as QualityTests + } + } +} +``` + +## Override the per-agent model id + +`Wire.ResolveAgentClient(IDictionary overrides)` is the +extension point. The generated factory (`AgentChatClientFactory`) +exposes an overload accepting per-agent model assignments — useful when +the app uses multiple deployment aliases or supports model swapping +via `ChatOptions.ModelId`. + +## `MatrixLoader` (snake_case JSON, PascalCase records) + +`matrix.json` uses snake_case keys (`schema_version`, `model_assignments`) +that bind to PascalCase C# properties (`SchemaVersion`, `ModelAssignments`). +The loader **must** opt into snake_case-aware deserialization or the +properties come back `null` and `CompareTests` crashes on enumeration: + +```csharp +internal static class MatrixLoader +{ + private static readonly JsonSerializerOptions s_options = new() + { + PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower, + PropertyNameCaseInsensitive = true, + }; + + public static List Load() + { + var path = Path.Combine(AppContext.BaseDirectory, "Compare", "matrix.json"); + var root = JsonSerializer.Deserialize( + File.ReadAllText(path), s_options); + return root?.Entries ?? new(); + } +} + +internal sealed record MatrixRoot(int SchemaVersion, List Entries); +internal sealed record MatrixEntry(string Name, Dictionary ModelAssignments); +``` + +Reuse the same `s_options` for `InputsLoader`, `PriceTable.Load()`, and +`GoldenLoader.Load()` — STJ defaults will silently null-out PascalCase +properties bound to snake_case JSON. + +## Compare-specific report + +`compare.md` (still emitted, in addition to the aggregated +`report.html`): + +| name | avg ms | in tok | out tok | $ | mean quality | mean BLEU | +|------|--------|--------|---------|---|--------------|-----------| +| baseline-all-mini | 432 | 380 | 210 | 0.0021 | 3.8 | 0.31 | +| interviewers-upgraded | 891 | 380 | 245 | 0.0210 | 4.4 | 0.42 | + +| Recommendation | +|----------------| +| `interviewers-upgraded`: +0.6 quality at +9.4× cost. Promote only if quality bar requires it. | + +The recommendation row is rule-based: + +- If cost increases > 3× and quality delta < 0.3 → **do not promote**. +- If cost increases ≤ 1.5× and quality delta ≥ 0.5 → **promote**. +- Otherwise → **manual review**. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/dotnet-tools-manifest.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/dotnet-tools-manifest.md new file mode 100644 index 0000000000..c8e50335d4 --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/dotnet-tools-manifest.md @@ -0,0 +1,50 @@ +# `dotnet-tools.json` manifest + +The skill scaffolds a **local** tool manifest. Global install is +explicitly avoided — pinning the tool version to the project makes +evals reproducible across machines and CI runs. + +## Why local + +- Reproducible: every clone gets the exact same `aieval` version. +- CI-friendly: `dotnet tool restore` in the workflow is one line. +- No PATH conflicts with developers who may have other versions installed. + +## Template + +```json +{ + "version": 1, + "isRoot": true, + "tools": { + "microsoft.extensions.ai.evaluation.console": { + "version": "10.7.0", + "commands": ["aieval"], + "rollForward": false + } + } +} +``` + +Place at `.Evals.Tests/dotnet-tools.json` (not the repo root) so +the manifest follows the project. The skill emits `dotnet tool restore` +as the next step in the chat output after scaffolding. + +## Why `rollForward: false` + +The output of `aieval report` is consumed by humans visually and by +artifact comparison in CI. A silent rollForward of the tool to a +newer version can change report layout, charts, or metric grouping — +breaking trend comparisons and visual diffs. + +If a user wants to upgrade the tool, the skill should emit a +`dotnet tool update microsoft.extensions.ai.evaluation.console` +command in the chat output (with a note: "this may change report +layout"). + +## Conflict with existing manifest + +If `dotnet-tools.json` already exists at the project (or any parent +directory), the skill **merges** the `aieval` entry rather than +overwriting. If a different version of `aieval` is already pinned, +surface the diff in chat output and require user confirmation. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md new file mode 100644 index 0000000000..127968f217 --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md @@ -0,0 +1,271 @@ +# Evaluators catalog + +Full catalog of evaluators wired by `setup-maf-evals`, grouped by tier. +The skill defaults to Tier 1 (NLP) always on, Tier 2 (Quality) on but +gated by `EVAL_USE_REAL_JUDGE`, Tier 3 (Safety) off unless opted in. + +Source: [learn.microsoft.com/en-us/dotnet/ai/evaluation/libraries](https://learn.microsoft.com/en-us/dotnet/ai/evaluation/libraries) + +## Tier 1 — NLP (deterministic, no LLM) + +Package: `Microsoft.Extensions.AI.Evaluation.NLP` (preview 10.7.0). + +| Evaluator | Metric | Context type needed | Notes | +|-----------|--------|---------------------|-------| +| `BLEUEvaluator` | `BLEU` | `BLEUEvaluatorContext(IEnumerable references)` | n-gram overlap with one or more reference strings | +| `GLEUEvaluator` | `GLEU` | `GLEUEvaluatorContext(IEnumerable references)` | sentence-level BLEU variant | +| `F1Evaluator` | `F1` | `F1EvaluatorContext(string groundTruth)` | unigram-level F1 | + +Plus a built-in custom evaluator the skill always scaffolds: + +| Evaluator | Metric | Why | +|-----------|--------|-----| +| `WordCountEvaluator` (custom) | `Words` | Sanity check: response is non-empty and reasonable length. Same pattern as the Learn doc tutorial. | + +### `WordCountEvaluator` reference implementation + +Scaffold this file verbatim (the Learn-doc canonical pattern) into +`Reporting/WordCountEvaluator.cs`. It runs in stub tier with no API key. + +```csharp +public sealed class WordCountEvaluator : IEvaluator +{ + public const string MetricName = "Words"; + public IReadOnlyCollection EvaluationMetricNames { get; } = [MetricName]; + + public ValueTask EvaluateAsync( + IEnumerable messages, + ChatResponse modelResponse, + ChatConfiguration? chatConfiguration = null, + IEnumerable? additionalContext = null, + CancellationToken cancellationToken = default) + { + var text = modelResponse?.Text ?? string.Empty; + var count = text.Split( + new[] { ' ', '\t', '\r', '\n' }, + StringSplitOptions.RemoveEmptyEntries).Length; + + var metric = new NumericMetric(MetricName, value: count) + { + Interpretation = count switch + { + < 5 => new EvaluationMetricInterpretation(EvaluationRating.Poor, reason: "Response too short"), + > 500 => new EvaluationMetricInterpretation(EvaluationRating.Average, reason: "Response very long"), + _ => new EvaluationMetricInterpretation(EvaluationRating.Good), + } + }; + return new ValueTask(new EvaluationResult(metric)); + } +} +``` + +## Tier 2 — Quality (LLM-as-judge) + +Package: `Microsoft.Extensions.AI.Evaluation.Quality` (GA 10.7.0). + +Requires `EVAL_USE_REAL_JUDGE=1` and a real `IChatClient`. The skill +wires the following by default: + +| Evaluator | Metric | Context type | Notes | +|-----------|--------|--------------|-------| +| `RelevanceEvaluator` | `Relevance` | none | how relevant is the response to the query | +| `CoherenceEvaluator` | `Coherence` | none | logical, orderly presentation | +| `FluencyEvaluator` | `Fluency` | none | grammar, readability | +| `CompletenessEvaluator` | `Completeness` | `CompletenessEvaluatorContext(string groundTruth)` | comprehensive and accurate | +| `EquivalenceEvaluator` | `Equivalence` | `EquivalenceEvaluatorContext(string groundTruth)` | similarity vs ground truth wrt query | +| `GroundednessEvaluator` | `Groundedness` | `GroundednessEvaluatorContext(string context)` | alignment with given context | + +Agent-focused (added when `*.AppHost.csproj` is detected, indicating +this is an agentic app and not just a chat completion app): + +| Evaluator | Metric | Context type | Notes | +|-----------|--------|--------------|-------| +| `IntentResolutionEvaluator` | `Intent Resolution` | none | identifies + resolves user intent | +| `TaskAdherenceEvaluator` | `Task Adherence` | none | sticks to assigned task | +| `ToolCallAccuracyEvaluator` | `Tool Call Accuracy` | `ToolCallAccuracyEvaluatorContext(...)` | uses tools correctly | + +**Not wired by default:** `RelevanceTruthAndCompletenessEvaluator` +(marked experimental in upstream docs), `RetrievalEvaluator` (specific +to RAG pipelines — separate skill territory). + +## Tier 3 — Safety (Foundry Evaluation service) + +Package: `Microsoft.Extensions.AI.Evaluation.Safety` (preview 10.7.0). + +Off by default. Enabled when user opts in during step 2 of the +workflow. Requires `EVAL_USE_FOUNDRY_SAFETY=1` and an Azure AI Foundry +endpoint (the skill prompts for `AZURE_AI_FOUNDRY_ENDPOINT` + Entra credentials). + +**Always wire the bundle, not the 4 separate evaluators:** + +| Evaluator | Metrics produced | Notes | +|-----------|------------------|-------| +| `ContentHarmEvaluator` | `Hate And Unfairness`, `Self Harm`, `Violence`, `Sexual` | **Single-shot — one Foundry call returns all 4 metrics.** Always prefer this over the 4 separate evaluators below. | + +Additional safety evaluators (each wired as separate `[TestMethod]`): + +| Evaluator | Metric | Notes | +|-----------|--------|-------| +| `ProtectedMaterialEvaluator` | `Protected Material` | copyrighted material in output | +| `IndirectAttackEvaluator` | `Indirect Attack` | prompt-injection-style indirect attacks | +| `CodeVulnerabilityEvaluator` | `Code Vulnerability` | vulnerable code in output | +| `UngroundedAttributesEvaluator` | `Ungrounded Attributes` | inferred human attributes | +| `GroundednessProEvaluator` | `Groundedness Pro` | fine-tuned Foundry-hosted groundedness check | + +The 4 separate evaluators (`HateAndUnfairnessEvaluator`, +`SelfHarmEvaluator`, `ViolenceEvaluator`, `SexualEvaluator`) are +**not** scaffolded — they're a strict subset of `ContentHarmEvaluator` +and cost 4× more Foundry calls for the same metrics. + +## Threshold mapping + +`quality.thresholds.json` maps **real MEAI metric names** to minimum +`EvaluationRating` enum values: + +```json +{ + "schema_version": 2, + "hard_fail": false, + "thresholds": { + "Relevance": { "min_rating": "Good" }, + "Coherence": { "min_rating": "Good" }, + "Fluency": { "min_rating": "Average" }, + "Groundedness":{ "min_rating": "Good" }, + "BLEU": { "min_value": 0.20 }, + "F1": { "min_value": 0.30 }, + "Words": { "min_value": 5, "max_value": 500 } + } +} +``` + +`hard_fail: true` makes any below-threshold metric fail the test. +Default `false` makes the test pass-through informational — failures +show in the report only. + +## Custom rubric-driven evaluator + +The built-in Quality evaluators (Relevance, Coherence, Fluency, +Completeness, Equivalence) judge against generic rubrics baked into +the evaluator's judge prompt. They cannot read `Quality/rubric.md`. +That makes them ill-fitting for agents with deliberate stylistic +constraints (ELI5 / summarizer / strict-format / persona-bound +agents) — see `common-pitfalls.md#tuning-quality-for-stylistic-agents`. + +For those agents, scaffold a `RubricEvaluator` that reads +`Quality/rubric.md` and asks the judge to score against *your* +criteria. The pattern is shape-identical to `WordCountEvaluator` — +it just delegates to the judge chat client. + +```csharp +// Reporting/RubricEvaluator.cs +// Custom rubric-driven evaluator. Reads Quality/rubric.md and asks +// the judge to score the response against per-app criteria. Emits a +// single "RubricFit" metric (numeric 1-5) plus a free-text rationale. +public sealed class RubricEvaluator : IEvaluator +{ + public const string MetricName = "RubricFit"; + public IReadOnlyCollection EvaluationMetricNames { get; } = [MetricName]; + + private readonly string _rubric; + + public RubricEvaluator(string rubricMarkdown) => _rubric = rubricMarkdown; + + public static RubricEvaluator FromFile(string path) => + new(File.ReadAllText(path)); + + public async ValueTask EvaluateAsync( + IEnumerable messages, + ChatResponse modelResponse, + ChatConfiguration? chatConfiguration = null, + IEnumerable? additionalContext = null, + CancellationToken cancellationToken = default) + { + if (chatConfiguration?.ChatClient is null) + { + // Judge tier not active — return a stubbed Inconclusive metric. + return new EvaluationResult(new NumericMetric(MetricName) + { + Interpretation = new EvaluationMetricInterpretation( + EvaluationRating.Inconclusive, reason: "Judge not wired.") + }); + } + + var userQuery = string.Join("\n", messages.Select(m => m.Text)); + var responseText = modelResponse.Text ?? string.Empty; + + // NOTE: $$"""...""" (double-$) raw string. Inside, single { } is literal + // (matters for the JSON braces below), and {{var}} is interpolation. + // Plain $"""...""" would reject `{{ ... }}` as literal with CS9006. + var prompt = $$""" + You are scoring an assistant response against the rubric below. + + ## Rubric + {{_rubric}} + + ## User query + {{userQuery}} + + ## Assistant response + {{responseText}} + + Respond with strict JSON: { "score": <1-5 int>, "rationale": "<1-2 sentences>" }. + 5 = perfectly satisfies every rubric clause. 1 = ignores the rubric. + """; + + var judge = await chatConfiguration.ChatClient.GetResponseAsync( + prompt, cancellationToken: cancellationToken).ConfigureAwait(false); + + // Tolerant parse: fall back to Inconclusive on bad JSON. + var (score, rationale) = TryParse(judge.Text ?? ""); + + var metric = new NumericMetric(MetricName, value: score) + { + Interpretation = new EvaluationMetricInterpretation( + rating: score switch { >= 4 => EvaluationRating.Good, + 3 => EvaluationRating.Average, + _ => EvaluationRating.Poor }, + reason: rationale) + }; + return new EvaluationResult(metric); + } + + private static (int score, string rationale) TryParse(string raw) + { + try + { + using var doc = JsonDocument.Parse(raw.Trim().Trim('`', ' ', '\n', '\r')); + return (doc.RootElement.GetProperty("score").GetInt32(), + doc.RootElement.GetProperty("rationale").GetString() ?? ""); + } + catch { return (3, "Judge response was unparseable; defaulted to Average."); } + } +} +``` + +Wire it from `Reporting/ReportingConfig.cs`'s judge-tier evaluator +list **alongside** Relevance/Coherence/Fluency (drop Completeness + +Equivalence per the pitfall guidance): + +```csharp +// In ReportingConfig.ForQuality(), when EvalEnv.UseRealJudge: +evaluators.Add(new RelevanceEvaluator()); +evaluators.Add(new CoherenceEvaluator()); +evaluators.Add(new FluencyEvaluator()); +evaluators.Add(RubricEvaluator.FromFile( + Path.Combine(AppContext.BaseDirectory, "Quality", "rubric.md"))); +// CompletenessEvaluator + EquivalenceEvaluator deliberately dropped +// for this app — see common-pitfalls.md tuning section. +``` + +Ensure `Quality/rubric.md` is copied to output by adding to the csproj: + +```xml + + + +``` + +This `RubricFit` metric will appear in `report.html` alongside the +built-ins, and the rationale shows in the per-scenario detail drawer. +Update `Reporting/MetricsGlossary.cs`'s `QualityEntries` constant to +add a one-line `RubricFit` definition pointing at `Quality/rubric.md`. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md new file mode 100644 index 0000000000..ad7ac0b3ec --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md @@ -0,0 +1,226 @@ +# IChatClient detection + +The skill auto-detects how the target app registers its `IChatClient` +and emits `Wire/AgentChatClientFactory.cs` so `EVAL_USE_REAL_AGENT=1` +works without further code. Detection result is surfaced in step 2 +(scope confirmation) and can be overridden by the user. + +## Patterns to scan for + +Scan **`*.AppHost.csproj` directory** and **all `*.Agent*.csproj` +directories**. Match (case-insensitive, multi-line): + +| Pattern (regex-ish) | Inferred client | Example file:line | +|---------------------|-----------------|--------------------| +| `AddAzureOpenAIChatClient\s*\(` or `AddAzureOpenAIClient\s*\(` | Azure OpenAI | `AppHost.cs`, `Program.cs` | +| `AddOpenAIChatClient\s*\(` | OpenAI direct | `Program.cs` | +| `AddOllamaChatClient\s*\(` | Ollama | `Program.cs` | +| `AddAIInference\s*\(` (Foundry deployment alias) | Azure AI Foundry | `AppHost.cs` | +| `AddAzureChatCompletionsClient\s*\([^)]*\)\s*\.AddChatClient\s*\(` | Aspire `Aspire.Azure.AI.Inference` (Foundry-routed, **legacy — see deprecation note below**) | `Program.cs` | +| `AddOpenAIClient\s*\([^)]*\)\s*\.AddChatClient\s*\(` | Aspire OpenAI/v1 (Foundry-routed via the OpenAI SDK) | `Program.cs` | +| `services\.AddSingleton` (any explicit registration) | custom | varies | +| `\.AsIChatClient\(\)` (after an SDK client) | manual wrap | varies | + +> The `AddAzureChatCompletionsClient(...).AddChatClient(...)` chain is the +> Aspire 13.2 way of wiring an `IChatClient` against a Foundry chat +> deployment via the `Azure.AI.Inference` beta SDK. The argument is the +> **connection-string name**, which Aspire's AppHost populates +> automatically (`AddDeployment("chat", ...)` -> connection string `chat`). +> The factory mirrors both calls verbatim. +> +> ⚠️ **Deprecation note (as of 2026):** the `Azure.AI.Inference` beta SDK +> is being retired on **August 26, 2026** in favor of the GA OpenAI/v1 +> API + stable OpenAI SDK. New apps should prefer the +> `AddOpenAIClient(...).AddChatClient(...)` pattern against an OpenAI/v1 +> endpoint, which avoids several quirks (notably the reasoning-model +> `max_tokens` rejection documented in +> `references/common-pitfalls.md`). The skill still detects and supports +> the legacy pattern unchanged. See the +> [Foundry migration guide](https://learn.microsoft.com/en-us/azure/foundry/how-to/model-inference-to-openai-migration). + +Capture the deployment alias / model id literal if present (e.g., +`builder.AddAIInference("chat", "gpt-4o-mini")` → alias `chat`). + +## What to emit + +### Case A — exactly one registration found + +Emit a factory that resolves from the host: + +```csharp +// Wire/AgentChatClientFactory.cs +namespace {{AppName}}.Evals.Tests; + +internal static class AgentChatClientFactory +{ + /// + /// Resolves the same IChatClient the app uses, by building a minimal + /// host that mirrors the app's DI registration. + /// Detected: {{DetectionSummary}} at {{File}}:{{Line}} + /// + public static IChatClient Create() + { + var builder = Host.CreateApplicationBuilder(); + + // In test hosts (`dotnet test`), the entry assembly is testhost.exe so + // user-secrets are NOT auto-loaded by CreateApplicationBuilder. + // Add them explicitly from THIS assembly's UserSecretsId. + builder.Configuration.AddUserSecrets(typeof(AgentChatClientFactory).Assembly, optional: true); + + // {{InsertDetectedRegistrationCallVerbatim}} + var host = builder.Build(); + try + { + return host.Services.GetRequiredService(); + } + catch (InvalidOperationException ex) + { + throw new InvalidOperationException( + "EVAL_USE_REAL_AGENT=1 but IChatClient could not be resolved. " + + "The detected registration ({{DetectionSummary}}) reads connection " + + "string \"{{ConnStrName}}\" from configuration. Aspire's AppHost " + + "populates this at runtime, but `dotnet test` runs standalone. " + + "Wire one of:\n" + + " # Key-based auth (only if the resource has it enabled):\n" + + " dotnet user-secrets set \"ConnectionStrings:{{ConnStrName}}\" " + + "\"Endpoint=https://.services.ai.azure.com/models;Key=;DeploymentId={{ConnStrName}}\" --project {{AppName}}.Evals.Tests\n" + + " # Entra-ID auth (DefaultAzureCredential — works when key auth is disabled):\n" + + " dotnet user-secrets set \"ConnectionStrings:{{ConnStrName}}\" " + + "\"Endpoint=https://.services.ai.azure.com/models;DeploymentId={{ConnStrName}}\" --project {{AppName}}.Evals.Tests\n" + + "Note: the hostname strips dashes from the resource name " + + "(resource `foundry-abc` -> host `foundryabc.services.ai.azure.com`).\n" + + "Get the exact endpoint from `azd env get-values` or " + + "`az cognitiveservices account show -n -g --query properties.endpoints`.", + ex); + } + } +} +``` + +Where `{{InsertDetectedRegistrationCallVerbatim}}` is the literal call +copied from the detection source (with any required `using`s in scope +via `GlobalUsings.cs`), and `{{ConnStrName}}` is the connection-string +literal extracted from the call (e.g., the `"chat"` argument). + +### Case B — multiple registrations found + +Emit the same factory but with a comment listing all candidates, and +have the chat output ask the user to pick one before write. Do **not** +write the file until confirmation. + +### Case C — no registration found + +Emit a stub factory the user fills in: + +```csharp +// Wire/AgentChatClientFactory.cs +namespace {{AppName}}.Evals.Tests; + +internal static class AgentChatClientFactory +{ + /// + /// Auto-detection failed: no IChatClient registration found in + /// AppHost or agent projects. Wire your client manually. + /// + public static IChatClient Create() => + throw new NotImplementedException( + "Wire your IChatClient here. See https://learn.microsoft.com/en-us/dotnet/ai/microsoft-extensions-ai"); +} +``` + +## Runtime selection + +`ReportingConfig` picks agent client based on tier: + +```csharp +internal static IChatClient ResolveAgentClient() => + EvalEnv.UseRealAgent ? AgentChatClientFactory.Create() : new StubChatClient(); +``` + +And by default the judge client is the **same** instance as the agent +client. This saves a duplicate Azure credential setup AND lets the +MEAI response cache serve both the agent call and the judge call from +one cache (because `QualityTests` uses `run.ChatConfiguration!.ChatClient` +for the agent call — that's the cached wrapper around the shared +instance). The user can override by setting `EVAL_JUDGE_DEPLOYMENT_NAME` +to a different deployment alias when (e.g.) the production model is a +reasoning model that can't be used as a judge. Trade-off: with the +override active, `run.ChatConfiguration!.ChatClient` becomes the +**judge** client, so the agent call would silently use the judge +model. To avoid that, `QualityTests` falls back to +`Wire.ResolveAgentClient()` (uncached) for agent calls whenever the +override is set; cache hits then apply only to judge calls. + +## Connection-string setup for standalone test runs + +When the target app uses Aspire orchestration, the AppHost populates +`ConnectionStrings:` automatically. **`dotnet test` runs outside +the AppHost and does not get this for free.** Surface this in the chat +output any time the detected pattern reads from a connection string +(`AddAzureChatCompletionsClient`, `AddAIInference`, `AddOllamaChatClient`, +or `AddAzureOpenAIChatClient` without an explicit endpoint). + +> **Important — the factory must opt into user-secrets explicitly.** In a +> `dotnet test` process the entry assembly is `testhost.exe`, so the +> `dotnet user-secrets` payload tied to *your* `UserSecretsId` is NOT auto +> loaded by `Host.CreateApplicationBuilder()`. The Case A template above +> calls `builder.Configuration.AddUserSecrets(typeof(...).Assembly)` for +> this reason. Without that line, the secret is set on disk but never read. + +Recommend the user wire one of: + +```pwsh +# 0 — one-time: bind the test project to a secrets store +dotnet user-secrets init --project .Evals.Tests + +# Option A — Key-based auth (only if the resource has key auth enabled) +dotnet user-secrets set "ConnectionStrings:" ` + "Endpoint=https://.services.ai.azure.com/models;Key=;DeploymentId=" ` + --project .Evals.Tests + +# Option B — Entra-ID auth (DefaultAzureCredential; works when key auth disabled) +dotnet user-secrets set "ConnectionStrings:" ` + "Endpoint=https://.services.ai.azure.com/models;DeploymentId=" ` + --project .Evals.Tests +# requires `az login` and a Cognitive Services User / Azure AI User role +# on the resource for the signed-in identity. + +# Option C — env var (works in CI without a secrets file) +$env:ConnectionStrings__ = "Endpoint=;DeploymentId=" +``` + +**Two endpoint gotchas to call out in chat:** + +1. The `services.ai.azure.com/models` hostname **strips dashes** from the + resource name. Resource `foundry-abc` -> host `foundryabc.services.ai.azure.com`. + Use `az cognitiveservices account show -n -g --query properties.endpoints` + to see all valid endpoint hostnames (`AI Foundry API` / `Azure AI Model Inference API`). +2. If the resource has `disableLocalAuth=true` (common on Foundry resources + provisioned by Aspire/azd), key-based auth returns `403 Key based authentication + is disabled for this resource`. Drop the `Key=` segment and use Entra (Option B). + +For Foundry-routed clients the connection string is what `azd env get-values` +prints for `connectionString` against the deployment resource. Document this +in the chat output along with the tier banner so the user doesn't see a +silent NRE on first real-agent run. + +## Chat output (step 2) + +When detection succeeds, surface as: + +``` +IChatClient detection: + - AddAzureOpenAIChatClient at AppHost.cs:41 + - Deployment alias: "chat" → gpt-4o-mini + - Will generate Wire/AgentChatClientFactory.cs that resolves it from the host. + +Override [y/N]? +``` + +When detection fails: + +``` +IChatClient detection: no registration found. + - Will generate a stub factory you'll need to fill in. + - Tier 2 (Quality) and Tier 3 (Safety) will be skipped until wired. +``` diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/metrics-glossary.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/metrics-glossary.md new file mode 100644 index 0000000000..476c995158 --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/metrics-glossary.md @@ -0,0 +1,247 @@ +# Metrics glossary + +Source of truth for the per-run `metrics-glossary.md` artifact that the +scaffolded `.Evals.Tests/Reporting/MetricsGlossary.cs` writes next +to `report.html`. The aieval HTML report is data-bound JSON — it shows +numbers but no definitions — so we co-locate this glossary with the +report so a first-time reader has a one-page cheat-sheet. + +The skill emits **only the entries for evaluators that actually ran** in +the active tier, so a stub-tier user sees Words/BLEU/GLEU/F1 and +nothing else. + +## NLP tier (deterministic, no LLM) + +### `Words` +- **Custom evaluator** scaffolded by this skill (see `evaluators-catalog.md`). +- **What it measures:** raw token count of the response text. +- **Scale:** integer ≥ 0. +- **Interpretation:** `< 5` → Poor (response too short / empty). `5–500` → Good. `> 500` → Average (response unusually long). +- **Trust for:** sanity-checking that the model produced *anything* and isn't running away with a 10-page essay. +- **Don't trust for:** quality. A 50-word wrong answer scores the same as a 50-word correct answer. + +### `BLEU` — Bilingual Evaluation Understudy +- **What it measures:** n-gram (1-4) overlap between the response and one or more reference strings, with a brevity penalty. +- **Scale:** 0.0 – 1.0. +- **Interpretation:** ~0.0–0.1 weak overlap, often paraphrased; ~0.1–0.3 normal for free-form generation; ~0.3–0.5 strong; > 0.5 near-quotation. +- **Trust for:** "is the response in the same lexical neighbourhood as the reference?" — useful as a regression signal when the reference is canonical. +- **Don't trust for:** semantic correctness. A correct paraphrase scores low; an incorrect copy-paste of reference fragments scores high. +- **Reference:** [`BLEUEvaluator` in MEAI.Evaluation.NLP](https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.ai.evaluation.nlp.bleuevaluator). + +### `GLEU` — Google-BLEU +- **What it measures:** sentence-level BLEU variant. Symmetric — penalises both missing reference n-grams and extra invented ones. +- **Scale:** 0.0 – 1.0. +- **Interpretation:** same buckets as BLEU; GLEU usually tracks BLEU but is less brittle on short outputs. +- **Trust for:** the same use cases as BLEU when individual scenarios are short (1-2 sentences) and BLEU's brevity penalty would be misleading. +- **Don't trust for:** any semantic claim. Same caveats as BLEU. +- **Reference:** [`GLEUEvaluator`](https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.ai.evaluation.nlp.gleuevaluator). + +### `F1` — Token-level F1 +- **What it measures:** harmonic mean of unigram precision and recall against the ground-truth string. Order-insensitive. +- **Scale:** 0.0 – 1.0. +- **Interpretation:** ~0.0–0.2 mostly-disjoint vocab; 0.3–0.5 typical for free-form generation; > 0.6 strong word-level match. +- **Trust for:** QA / extraction-style scenarios where the *set of words* matters more than phrasing (SQuAD-style benchmarks use this). +- **Don't trust for:** word-order-sensitive tasks (e.g., "yes" vs "no" answers buried in long output) — the F1 score will be high even if the polarity is wrong. +- **Reference:** [`F1Evaluator`](https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.ai.evaluation.nlp.f1evaluator). + +> **NLP-tier headline:** all three of BLEU/GLEU/F1 are *lexical* metrics. +> They're cheap, deterministic, and free, but they cannot tell you the +> response is *correct* — only that it's *similar in wording*. Treat them +> as a regression early-warning, not a quality verdict. + +## Quality tier (LLM-as-judge — `EVAL_USE_REAL_JUDGE=1`) + +All Quality metrics produce a 1-5 `EvaluationRating` (Poor → Excellent) +plus a free-text rationale from the judge model. + +### `Relevance` +- **What it measures:** does the response actually address the user's query? +- **Trust for:** catching off-topic regressions (model started monologuing about a different topic). +- **Don't trust for:** factual correctness — a *relevantly wrong* answer can score high. + +### `Coherence` +- **What it measures:** is the response logically structured / orderly / easy to follow? +- **Trust for:** detecting rambling, contradictory, or logically broken outputs. +- **Don't trust for:** depth or correctness — coherent nonsense scores high. + +### `Fluency` +- **What it measures:** grammar, readability, naturalness. +- **Trust for:** detecting broken-English / token-soup outputs from undertrained or quantised models. +- **Don't trust for:** anything beyond surface text quality. A fluent lie scores high. + +### `Completeness` +- **What it measures:** how comprehensive and accurate the response is, given a reference (`CompletenessEvaluatorContext(groundTruth)`). +- **Trust for:** catching responses that are correct but partial (covered 2 of 4 required points). +- **Don't trust for:** brevity-as-a-feature scenarios — a short-but-correct answer can score lower than a long-and-padded one. + +### `Equivalence` +- **What it measures:** semantic similarity between response and ground truth in the context of the original query. +- **Trust for:** distinguishing "right answer, different words" (high) from "wrong answer" (low). Better than BLEU/F1 when paraphrasing is acceptable. +- **Don't trust for:** any case where the ground truth is itself ambiguous or one of multiple valid answers. + +### `Groundedness` +- **What it measures:** alignment between the response and a supplied source-of-truth context (`GroundednessEvaluatorContext(context)`). +- **Trust for:** RAG pipelines — flags when the model invented facts not in the retrieved context. +- **Don't trust for:** open-ended chat without a context document. + +### Agentic-only + +The skill wires these only when an `*.AppHost.csproj` is detected. + +- **`Intent Resolution`** — did the model identify and resolve the user's actual intent (vs answering a related-but-different question)? +- **`Task Adherence`** — did the model stick to the assigned task or wander into other agent territory? +- **`Tool Call Accuracy`** — did the model invoke the right tools with the right arguments? Requires `ToolCallAccuracyEvaluatorContext`. + +> **Quality-tier headline:** these are LLM-judge subjective scores. They +> drift across judge model versions. Pin the judge model in +> `quality.thresholds.json` if you want comparable scores across runs. + +## Safety tier (Foundry — `EVAL_USE_FOUNDRY_SAFETY=1`) + +All Safety evaluators return a 1-5 severity (1 = safe, 5 = severe harm) +plus a confidence score. Each metric is an *output classifier* — it +inspects the model's response, not the input prompt. + +### `Hate And Unfairness`, `Self Harm`, `Violence`, `Sexual` +- Wired together as the single-shot `ContentHarmEvaluator` (one Foundry call → all 4 metrics). +- **Trust for:** detecting harmful content in agent outputs. +- **Don't trust for:** input-side filtering — these never see the user's prompt. Pair with input-side Azure AI Content Safety for full coverage. + +### `Protected Material` +- Detects copyrighted text / song lyrics / book passages reproduced in the output. + +### `Indirect Attack` +- Detects prompt-injection-style content that would indicate the model picked up an indirect attack from retrieved content or tool output. Closest thing to an input-side check available in this tier. + +### `Code Vulnerability` +- Flags vulnerable patterns in code the model emitted (SQL injection, hardcoded credentials, weak crypto, etc.). + +### `Ungrounded Attributes` +- Detects inferred human attributes (race, gender, age, religion, etc.) in the response that weren't in the input. + +### `Groundedness Pro` +- Foundry-hosted fine-tuned groundedness evaluator. More accurate than the open-source `GroundednessEvaluator` but costs a Foundry call per scenario. + +> **Safety-tier headline:** all safety metrics are *output classifiers*. +> They protect downstream consumers from the agent's outputs; they do not +> protect the agent from its inputs. + +## Quick reference card + +| Metric | Tier | Scale | Needs ground truth? | Needs LLM? | +|--------|------|-------|---------------------|-----------| +| Words | NLP | int | no | no | +| BLEU | NLP | 0-1 | yes (references) | no | +| GLEU | NLP | 0-1 | yes (references) | no | +| F1 | NLP | 0-1 | yes (one string) | no | +| Relevance / Coherence / Fluency | Quality | 1-5 | no | yes (judge) | +| Completeness / Equivalence | Quality | 1-5 | yes (one string) | yes (judge) | +| Groundedness | Quality | 1-5 | yes (context) | yes (judge) | +| Intent Resolution / Task Adherence | Quality (agentic) | 1-5 | no | yes (judge) | +| Tool Call Accuracy | Quality (agentic) | 1-5 | yes (expected_tool_calls) | yes (judge) | +| Content Harm bundle (4 metrics) | Safety | 1-5 severity | no | yes (Foundry) | +| Protected / Indirect / Code Vuln. / Ungrounded | Safety | 1-5 severity | varies | yes (Foundry) | +| Groundedness Pro | Safety | 1-5 | yes (context) | yes (Foundry) | + +## `MetricsGlossary.cs` template + +The scaffolded `.Evals.Tests/Reporting/MetricsGlossary.cs` writes +the tier-relevant slice of this glossary next to `report.html` after +each `dotnet test` run. + +> **MSTest constraint:** an assembly may declare **only one** +> `[AssemblyCleanup]` method. The skill emits `MetricsGlossary` as a +> plain static class (no `[TestClass]`, no `[AssemblyCleanup]`) and +> chains `MetricsGlossary.WriteGlossary()` from +> `AievalReport.GenerateReport`'s single `[AssemblyCleanup]`. +> Wrap the call in a `try/catch` so a glossary-write failure never +> masks the report. + +```csharp +internal static class MetricsGlossary +{ + public static void WriteGlossary() + { + var outDir = Path.Combine( + RepoRoot.Find(), ".copilot", "perf-reports", "evals", EvalEnv.ReportFolder); + Directory.CreateDirectory(outDir); + var path = Path.Combine(outDir, "metrics-glossary.md"); + + var sb = new StringBuilder(); + sb.AppendLine($"# Metrics glossary — {EvalEnv.Tier} tier"); + sb.AppendLine(); + sb.AppendLine($"Generated: {DateTime.UtcNow:O}"); + sb.AppendLine(); + sb.AppendLine($"Companion to `report.html` in this folder. The aieval HTML report shows numbers; this file explains them."); + sb.AppendLine(); + + sb.AppendLine(NlpEntries); + if (EvalEnv.UseRealJudge) sb.AppendLine(QualityEntries); + if (EvalEnv.UseFoundrySafety) sb.AppendLine(SafetyEntries); + + sb.AppendLine(); + sb.AppendLine("> Source: setup-maf-evals references/metrics-glossary.md"); + + File.WriteAllText(path, sb.ToString()); + Console.WriteLine($"[MetricsGlossary] {path}"); + } + + private const string NlpEntries = """ + ## NLP tier (deterministic, no LLM) + - **Words** (int): response length sanity check. <5 too short, 5-500 ok, >500 long. + - **BLEU** (0-1): n-gram overlap with reference(s). 0.1-0.3 normal, >0.3 strong, >0.5 near-quotation. *Lexical, not semantic.* + - **GLEU** (0-1): sentence-level BLEU; better for short outputs. Same buckets as BLEU. + - **F1** (0-1): unigram token F1 vs ground-truth. 0.3-0.5 typical, >0.6 strong word-level match. Order-insensitive. + + > Headline: NLP metrics measure wording similarity, not correctness. Use for regression early-warning, not as quality verdicts. + """; + + private const string QualityEntries = """ + ## Quality tier (LLM-as-judge) + Each rated 1-5 (Poor → Excellent) with a free-text rationale. + - **Relevance**: addresses the user's query. Catches off-topic regressions. + - **Coherence**: logically structured. Catches rambling/contradictory outputs. + - **Fluency**: grammar/readability. Catches broken-English outputs. + - **Completeness** (needs reference): comprehensive and accurate. + - **Equivalence** (needs reference): semantic similarity in context of the query. + - **Groundedness** (needs context): aligned with supplied source-of-truth. + - **Intent Resolution / Task Adherence / Tool Call Accuracy** (agentic only). + + > Headline: judge scores drift across model versions. Pin the judge model for comparable runs. + """; + + private const string SafetyEntries = """ + ## Safety tier (Foundry) + Each rated 1-5 severity (1 safe → 5 severe). + - **ContentHarm bundle** (single-shot, 4 metrics): Hate-And-Unfairness, Self-Harm, Violence, Sexual. + - **Protected Material**: copyrighted text reproduced. + - **Indirect Attack**: prompt-injection content from retrieved/tool data. + - **Code Vulnerability**: vulnerable code patterns (SQLi, weak crypto, etc.). + - **Ungrounded Attributes**: inferred human attributes not in input. + - **Groundedness Pro**: Foundry-hosted fine-tuned groundedness check. + + > Headline: all safety metrics inspect *outputs*, not inputs. Pair with Azure AI Content Safety on the request side for full coverage. + """; +} +``` + +Then in `Reporting/AievalReport.cs` (the assembly's single +`[AssemblyCleanup]` host): + +```csharp +[TestClass] +public static class AievalReport +{ + [AssemblyCleanup] + public static void GenerateReport() + { + // ... aieval invocation ... + + try { MetricsGlossary.WriteGlossary(); } + catch (Exception ex) { Console.Error.WriteLine($"[MetricsGlossary] Failed: {ex.Message}"); } + } +} +``` + +The skill should emit both files verbatim — change the `private const` +strings only if upstream MEAI changes a metric definition. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/project-template.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/project-template.md new file mode 100644 index 0000000000..9bcbb88ac7 --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/project-template.md @@ -0,0 +1,137 @@ +# Project template (MSTest shape) + +The scaffold creates `.Evals.Tests` as a **MSTest** project. This +matches the +[upstream Learn doc tutorial](https://learn.microsoft.com/en-us/dotnet/ai/evaluation/evaluate-with-reporting) +and the [dotnet/ai-samples evaluation unit tests](https://github.com/dotnet/ai-samples/blob/main/src/microsoft-extensions-ai-evaluation/api/), +which is the canonical pattern for `Microsoft.Extensions.AI.Evaluation`. + +A console-runner shape is available behind an explicit +`--shape console` flag for users who can't take an MSTest dependency, +but is no longer the default — every CI system understands `dotnet test` +out of the box, and Test Explorer integration is automatic. + +## File tree + +``` +.Evals.Tests/ + .Evals.Tests.csproj + dotnet-tools.json + GlobalUsings.cs + Reporting/ + ReportingConfig.cs # DiskBasedReportingConfiguration factory; tier-aware evaluator list + Tier.cs # EvalTier enum + EvalEnv reader + AievalReport.cs # [AssemblyCleanup] that invokes the dotnet tool + Wire/ + AgentChatClientFactory.cs # auto-generated from IChatClient detection + StubChatClient.cs # used when EVAL_USE_REAL_AGENT is unset + Telemetry/ + TelemetryTests.cs + inputs.json + prices.json + Quality/ + QualityTests.cs + rubric.md + golden.json + Compare/ + CompareTests.cs + matrix.json + Safety/ # only if user opted in + SafetyTests.cs + quality.thresholds.json +.github/ + workflows/ + evals.yml # optional, opt-in +``` + +`.gitignore` additions (idempotent): + +``` +# setup-maf-evals +.copilot/perf-reports/evals/ +.Evals.Tests/_store/ +``` + +## `.csproj` template + +```xml + + + net10.0 + enable + enable + false + {{AppName}}.Evals.Tests + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +``` + +**Why these versions:** + +- `Microsoft.Extensions.AI.Evaluation.{Reporting,Quality,Console}` are GA at `10.7.0`. +- `Microsoft.Extensions.AI.Evaluation.{NLP,Safety}` are still preview at `10.7.0-preview.1.26309.5`. NLP is opt-in-on; Safety is opt-in-off. +- `Microsoft.Extensions.Hosting` and `Microsoft.Extensions.Configuration.*` must be `10.0.1` (not `10.0.0`) to satisfy the transitive constraint from `Microsoft.Agents.AI.Hosting`. Pinning `10.0.0` produces `NU1605`. + +## `dotnet-tools.json` + +```json +{ + "version": 1, + "isRoot": true, + "tools": { + "microsoft.extensions.ai.evaluation.console": { + "version": "10.7.0", + "commands": ["aieval"], + "rollForward": false + } + } +} +``` + +After scaffold: `dotnet tool restore` (the skill runs this automatically). + +## `GlobalUsings.cs` + +```csharp +global using Microsoft.Extensions.AI; +global using Microsoft.Extensions.AI.Evaluation; +global using Microsoft.Extensions.AI.Evaluation.Reporting; +global using Microsoft.Extensions.AI.Evaluation.Reporting.Storage; +global using Microsoft.Extensions.AI.Evaluation.NLP; +global using Microsoft.Extensions.AI.Evaluation.Quality; +global using Microsoft.Extensions.Configuration; // AddUserSecrets in AgentChatClientFactory +global using Microsoft.Extensions.DependencyInjection; // GetRequiredService in AgentChatClientFactory +global using Microsoft.Extensions.Hosting; // Host.CreateApplicationBuilder +global using Microsoft.VisualStudio.TestTools.UnitTesting; +``` diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/quality-modes.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/quality-modes.md new file mode 100644 index 0000000000..e9df0de7b9 --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/quality-modes.md @@ -0,0 +1,253 @@ +# Quality mode + +`Quality/QualityTests.cs` is the MSTest class that actually drives +`Microsoft.Extensions.AI.Evaluation.Reporting`. It's the **only** +runner that produces `report.html`. + +## Response cache (read first) + +`DiskBasedReportingConfiguration.Create(..., enableResponseCaching: true)` +wraps the supplied `IChatClient` with a content-addressable cache stored +under `_store/cache/`. The first `dotnet test` run populates it; every +subsequent run against the same scenarios is **near-instant with zero +LLM cost** because both the agent call and the judge call are served +from disk. + +Two rules MUST be followed to make the cache work: + +1. **The agent call must go through `run.ChatConfiguration!.ChatClient`** + (NOT `Wire.ResolveAgentClient()` directly). The run-scoped client is + the cached wrapper. Calling the factory directly bypasses the cache. +2. **Do not pass a per-run `executionName`** to `Create(...)`. The + `executionName` is part of the cache scope; a fresh timestamp per run + guarantees misses. Use a separate `EvalEnv.ReportFolder` value for + the report-output directory if you want per-run history. + +If both rules are followed, the only legitimate reasons to see Miss in +the report's Diagnostic Data section are: (a) the cache directory was +deleted, (b) the rubric / golden / scenario input changed, (c) the +judge model or chat options changed, or (d) it's the first run. + +## Pipeline + +``` +[ClassInitialize] → build ReportingConfig (tier-aware evaluator list) +[TestMethod] → one per golden.json entry + ├─ CreateScenarioRunAsync(scenarioName) + ├─ get cached IChatClient from run.ChatConfiguration + ├─ get agent response (cached) + ├─ build per-evaluator EvaluationContext (BLEU refs, F1 ground truth, ...) + └─ scenarioRun.EvaluateAsync(messages, response, contexts) // judge calls cached +[AssemblyCleanup] → dotnet tool run aieval report --path _store --output /report.html +``` + +## Reporting config (sketch) + +```csharp +// Reporting/ReportingConfig.cs +internal static class ReportingConfig +{ + public static readonly string StorageRoot = + Path.Combine(RepoRoot.Find(), "_store"); + + public static ReportingConfiguration ForQuality() + { + // ONE client serves both the agent call and the judge call. MEAI wraps + // it with the response cache when handed to ChatConfiguration, so the + // *agent* call gets cached too when QualityTests calls it through + // `run.ChatConfiguration!.ChatClient` (see "Test class" below). + // On re-runs against unchanged inputs, the entire run is a cache hit + // and finishes in seconds with zero LLM cost. Override the judge with + // a separate model via EVAL_JUDGE_DEPLOYMENT_NAME (advanced). + var agent = Wire.ResolveAgentClient(); + var judge = Wire.ResolveJudgeClient(agent); + + var evaluators = new List + { + // Tier 1 — always on, deterministic + new WordCountEvaluator(), + new BLEUEvaluator(), + new GLEUEvaluator(), + new F1Evaluator(), + }; + + if (EvalEnv.UseRealJudge) + { + // Tier 2 — needs real judge + evaluators.Add(new RelevanceEvaluator()); + evaluators.Add(new CoherenceEvaluator()); + evaluators.Add(new FluencyEvaluator()); + evaluators.Add(new CompletenessEvaluator()); + evaluators.Add(new EquivalenceEvaluator()); + evaluators.Add(new GroundednessEvaluator()); + + if (AgenticAppDetected) + { + evaluators.Add(new IntentResolutionEvaluator()); + evaluators.Add(new TaskAdherenceEvaluator()); + evaluators.Add(new ToolCallAccuracyEvaluator()); + } + } + + // executionName: deliberately omitted. MEAI's default keeps the cache + // scope stable across runs so re-runs hit. The report folder + // timestamp lives separately in EvalEnv.ReportFolder. + return DiskBasedReportingConfiguration.Create( + storageRootPath: StorageRoot, + evaluators: evaluators, + chatConfiguration: new ChatConfiguration(judge), + enableResponseCaching: true); + } +} +``` + +## Test class (sketch) + +```csharp +[TestClass] +public sealed class QualityTests +{ + private static ReportingConfiguration s_reporting = null!; + + [ClassInitialize] + public static void Init(TestContext _) => + s_reporting = ReportingConfig.ForQuality(); + + public static IEnumerable Golden() => + GoldenLoader.Load().Select(g => new object[] { g }); + + [TestMethod, DynamicData(nameof(Golden), DynamicDataSourceType.Method)] + public async Task Evaluate(GoldenItem g) + { + var scenarioName = $"{nameof(QualityTests)}.{g.Id}"; + await using var run = await s_reporting.CreateScenarioRunAsync(scenarioName); + + // IMPORTANT: use the run's ChatClient, NOT Wire.ResolveAgentClient(), + // for the agent call. The run-scoped client is wrapped with MEAI's + // response cache (set up by ReportingConfig.ForQuality), so identical + // inputs across runs return cached responses instead of paying for + // a fresh LLM call. Calling AgentChatClientFactory directly bypasses + // the cache and guarantees a cache miss on every judge call too + // (judge cache key includes the agent response, which then varies). + // + // EDGE CASE: when EVAL_JUDGE_DEPLOYMENT_NAME splits judge from agent, + // run.ChatConfiguration.ChatClient IS the judge client — using it as + // the agent would silently call the wrong model. Fall back to the + // uncached agent factory in that case (judge calls still cache). + var agent = string.IsNullOrEmpty( + Environment.GetEnvironmentVariable("EVAL_JUDGE_DEPLOYMENT_NAME")) + ? run.ChatConfiguration!.ChatClient + : Wire.ResolveAgentClient(); + var messages = new List + { + new(ChatRole.System, RubricLoader.SystemPrompt()), + new(ChatRole.User, g.UserMessage), + }; + var response = await agent.GetResponseAsync(messages); + + var contexts = new List(); + if (!string.IsNullOrEmpty(g.ReferenceResponse)) + { + contexts.Add(new BLEUEvaluatorContext(new[] { g.ReferenceResponse })); + contexts.Add(new GLEUEvaluatorContext(new[] { g.ReferenceResponse })); + contexts.Add(new F1EvaluatorContext(g.ReferenceResponse)); + contexts.Add(new EquivalenceEvaluatorContext(g.ReferenceResponse)); + contexts.Add(new CompletenessEvaluatorContext(g.ReferenceResponse)); + } + if (!string.IsNullOrEmpty(g.Context)) + contexts.Add(new GroundednessEvaluatorContext(g.Context)); + + var result = await run.EvaluateAsync(messages, response, contexts); + Thresholds.ApplyOrLog(result, g.Id); // hard_fail in JSON => Assert.Fail + } +} +``` + +## Report generation + +```csharp +// Reporting/AievalReport.cs +[TestClass] +public static class AievalReport +{ + [AssemblyCleanup] + public static void GenerateReport() + { + var outDir = Path.Combine( + RepoRoot.Find(), ".copilot", "perf-reports", "evals", + EvalEnv.ReportFolder); + Directory.CreateDirectory(outDir); + var html = Path.Combine(outDir, "report.html"); + + var psi = new ProcessStartInfo("dotnet", + $"tool run aieval report --path \"{ReportingConfig.StorageRoot}\" --output \"{html}\"") + { + WorkingDirectory = RepoRoot.Find(), + RedirectStandardOutput = true, + RedirectStandardError = true, + UseShellExecute = false, + CreateNoWindow = true, + }; + using var p = Process.Start(psi)!; + p.WaitForExit(); + TestContext.Out?.WriteLine($"Eval report: {html}"); + } +} +``` + +## EvalEnv (sketch, in `Reporting/Tier.cs`) + +```csharp +internal static class EvalEnv +{ + public static bool UseRealAgent => + Environment.GetEnvironmentVariable("EVAL_USE_REAL_AGENT") == "1"; + + public static bool UseRealJudge => + Environment.GetEnvironmentVariable("EVAL_USE_REAL_JUDGE") == "1"; + + public static bool UseFoundrySafety => + Environment.GetEnvironmentVariable("EVAL_USE_FOUNDRY_SAFETY") == "1"; + + public static string Tier => + UseFoundrySafety ? "Safety" : UseRealJudge ? "Judge" : "Stub"; + + // Per-run timestamp ONLY for the report output folder under + // .copilot/perf-reports/evals//. NOT passed to MEAI's + // ReportingConfiguration — that would scope the response cache per + // run and defeat caching. Override with EVAL_REPORT_FOLDER in CI to + // align with a build number. + public static readonly string ReportFolder = + Environment.GetEnvironmentVariable("EVAL_REPORT_FOLDER") + ?? DateTime.UtcNow.ToString("yyyyMMdd-HHmmss"); +} +``` + +## `golden.json` schema (v2) + +```json +{ + "schema_version": 2, + "scenarios": [ + { + "id": "g-receptionist-greeting", + "user_message": "Hi, I'd like to start an interview.", + "reference_response": "Hello! I'd be happy to start an interview with you. What role are you preparing for?", + "context": null, + "expected_traits": ["on_topic", "safe", "format_correct"], + "expected_tool_calls": null + } + ] +} +``` + +- `reference_response`: required for BLEU/GLEU/F1/Equivalence/Completeness. +- `context`: required for Groundedness. Free text providing the + source-of-truth context the response should be grounded in. +- `expected_traits`: free-form labels surfaced in the report rubric + view. Read by the LLM judge. +- `expected_tool_calls`: required for `ToolCallAccuracyEvaluator`. + +Migration from v1 (no `schema_version`, no `reference_response`): the +skill adds the fields as `null` so existing tests don't fail. NLP +evaluators emit `(no reference)` when null. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/safety-mode.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/safety-mode.md new file mode 100644 index 0000000000..2b38264d3b --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/safety-mode.md @@ -0,0 +1,89 @@ +# Safety mode (opt-in) + +Off by default. Enabled in step 2 of the workflow when the user picks +"include Safety tier" / says "wire safety" / equivalent. + +When enabled, the skill: + +1. Adds `Microsoft.Extensions.AI.Evaluation.Safety` (preview 10.7.0) to the csproj. +2. Generates `Safety/SafetyTests.cs` using `ContentHarmEvaluator` as the + default bundle (4 metrics in 1 Foundry call), plus + `ProtectedMaterialEvaluator`, `IndirectAttackEvaluator`, + `CodeVulnerabilityEvaluator`, `UngroundedAttributesEvaluator`, and + optionally `GroundednessProEvaluator`. +3. Adds the required Azure AI Foundry endpoint config keys to + `quality.thresholds.json` and surfaces them in the chat output. + +## Runtime gating + +Safety tests must **never** fail the build when Foundry creds are +missing — they're an opt-in capability. The pattern: + +```csharp +[TestClass] +public sealed class SafetyTests +{ + [ClassInitialize] + public static void Init(TestContext _) + { + if (!EvalEnv.UseFoundrySafety) + Assert.Inconclusive( + "Safety tier disabled. Set EVAL_USE_FOUNDRY_SAFETY=1 and " + + "AZURE_AI_FOUNDRY_ENDPOINT to enable."); + } + + public static IEnumerable Golden() => + GoldenLoader.Load().Select(g => new object[] { g }); + + [TestMethod, DynamicData(nameof(Golden), DynamicDataSourceType.Method)] + public async Task ContentHarm(GoldenItem g) + { + var reporting = ReportingConfig.ForSafety(); // separate config — Foundry chat client + await using var run = await reporting.CreateScenarioRunAsync($"Safety.ContentHarm.{g.Id}"); + var agent = Wire.ResolveAgentClient(); + var messages = new List { new(ChatRole.User, g.UserMessage) }; + var response = await agent.GetResponseAsync(messages); + await run.EvaluateAsync(messages, response); // ContentHarmEvaluator returns all 4 metrics + } + + // Repeat for ProtectedMaterial / IndirectAttack / CodeVulnerability / etc. +} +``` + +## Why `ContentHarmEvaluator` (not 4 separate) + +From the upstream docs: + +> ContentHarmEvaluator provides single-shot evaluation for the four +> metrics supported by HateAndUnfairnessEvaluator, SelfHarmEvaluator, +> ViolenceEvaluator, and SexualEvaluator. + +That's **1 Foundry call instead of 4** for the same metric set. Always +wire `ContentHarmEvaluator` unless the user has a strict reason to +isolate one harm category. + +## Config keys surfaced in chat + +When Safety tier is enabled, the skill output adds: + +``` +Safety tier enabled. To activate at runtime: + export EVAL_USE_FOUNDRY_SAFETY=1 + export AZURE_AI_FOUNDRY_ENDPOINT=https://.cognitiveservices.azure.com + az login --tenant # DefaultAzureCredential + +Safety tests are MARKED INCONCLUSIVE (not failed) when the env vars are unset, +so your default `dotnet test` run will not break. +``` + +## What Safety evaluators do **not** cover + +Document this explicitly in the rubric: safety evaluators are *output* +classifiers, not *input* classifiers. They do not protect the agent +from receiving harmful prompts — for that, use a separate input filter +(e.g., Azure AI Content Safety on the request side). + +`IndirectAttackEvaluator` is the closest to an input-side check; it +looks for prompt-injection-style content in the *response* that would +indicate the model picked up an indirect attack from retrieved content +or tool output. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/telemetry-capture.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/telemetry-capture.md new file mode 100644 index 0000000000..fab6e2eb7f --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/telemetry-capture.md @@ -0,0 +1,143 @@ +# Telemetry mode + +Telemetry mode captures **latency, input tokens, output tokens, and +cost** per agent call across a fixed input set. It is **not** the same +as the MEAI eval report — it produces a separate cost/latency capture. + +## Why separate from quality + +Quality mode answers "is the response any good?" via +`Microsoft.Extensions.AI.Evaluation.Reporting` and produces +`report.html`. + +Telemetry mode answers "how much does it cost and how slow is it?" +via a delegating `IChatClient` wrapper. Different question, different +artifact. Conflating them is the #1 most common scaffolding mistake. + +## Artifacts (each in `.copilot/perf-reports/evals//`) + +- `telemetry.md` — human-readable per-input table. +- `telemetry.json` — machine-readable for CI scraping. +- `telemetry.junit.xml` — for test-result dashboards. + +These are **distinct** from `report.html` (which only quality mode +writes). The skill output should never refer to `telemetry.md` as +"the eval report." + +## Test shape + +```csharp +[TestClass] +public sealed class TelemetryTests +{ + public static IEnumerable Inputs() => + InputsLoader.Load().Select(i => new object[] { i }); + + [TestMethod, DynamicData(nameof(Inputs), DynamicDataSourceType.Method)] + public async Task Capture(TelemetryInput input) + { + var inner = Wire.ResolveAgentClient(); + var wrapped = new TelemetryCapturingChatClient(inner, PriceTable.Load()); + + var messages = new List { new(ChatRole.User, input.Text) }; + var sw = Stopwatch.StartNew(); + var response = await wrapped.GetResponseAsync(messages); + sw.Stop(); + + TelemetryStore.Record(new TelemetryRecord( + AgentName: input.Agent, + Model: response.ModelId ?? "unknown", + InputTokens: response.Usage?.InputTokenCount ?? 0, + OutputTokens: response.Usage?.OutputTokenCount ?? 0, + LatencyMs: sw.ElapsedMilliseconds, + CostUsd: wrapped.LastCostUsd)); + } + + [ClassCleanup] + public static void WriteReports() => TelemetryStore.FlushTo( + Path.Combine(RepoRoot.Find(), ".copilot", "perf-reports", "evals", + ReportingConfig.ExecutionName)); +} +``` + +## Delegating client (sketch) + +```csharp +internal sealed class TelemetryCapturingChatClient(IChatClient inner, PriceTable prices) : IChatClient +{ + public decimal LastCostUsd { get; private set; } + + public async Task GetResponseAsync( + IEnumerable messages, ChatOptions? options = null, + CancellationToken cancellationToken = default) + { + var resp = await inner.GetResponseAsync(messages, options, cancellationToken); + var usage = resp.Usage; + LastCostUsd = prices.Cost(resp.ModelId, + usage?.InputTokenCount ?? 0, usage?.OutputTokenCount ?? 0); + return resp; + } + + // GetStreamingResponseAsync delegates similarly; usage parsed off the last update. + + public object? GetService(Type serviceType, object? serviceKey = null) => + inner.GetService(serviceType, serviceKey); + + public void Dispose() => inner.Dispose(); +} +``` + +## `inputs.json` + +```json +[ + { "agent": "receptionist", "text": "Hi" }, + { "agent": "behavioural", "text": "Tell me about a tough project" }, + { "agent": "technical", "text": "How would you throttle requests?" }, + { "agent": "summariser", "text": "Wrap up the interview" } +] +``` + +## `prices.json` + +```json +{ + "gpt-4o-mini": { "input_per_1k": 0.00015, "output_per_1k": 0.0006 }, + "gpt-4o": { "input_per_1k": 0.0025, "output_per_1k": 0.01 }, + "o4-mini": { "input_per_1k": 0.003, "output_per_1k": 0.012 } +} +``` + +Edit freely — costs change. The price table is **never** baked into source. + +## `InputsLoader` (the deserializer the test uses) + +`inputs.json` is snake_case; the C# records (`TelemetryInput`, etc.) are +PascalCase. The loader **must** specify a `JsonSerializerOptions` with +`PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower` and +`PropertyNameCaseInsensitive = true`, or properties deserialize to +`null` and tests fail (or, worse, silently produce zeroed records when +the property is never read — telemetry mode is especially prone to +this because it never accesses some fields). + +```csharp +internal static class InputsLoader +{ + private static readonly JsonSerializerOptions s_options = new() + { + PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower, + PropertyNameCaseInsensitive = true, + }; + + public static List Load() + { + var path = Path.Combine(AppContext.BaseDirectory, "Telemetry", "inputs.json"); + return JsonSerializer.Deserialize>( + File.ReadAllText(path), s_options) ?? new(); + } +} +``` + +Same applies to `PriceTable.Load()` (`prices.json`), `MatrixLoader.Load()` +(`matrix.json`), and `GoldenLoader.Load()` (`golden.json`). Use a shared +options instance — do NOT rely on STJ defaults. diff --git a/tests/dotnet-ai/configure-agentic-perf-rules/eval.yaml b/tests/dotnet-ai/configure-agentic-perf-rules/eval.yaml new file mode 100644 index 0000000000..2a23305c90 --- /dev/null +++ b/tests/dotnet-ai/configure-agentic-perf-rules/eval.yaml @@ -0,0 +1,240 @@ +config: + max_parallel_scenarios: 1 + max_parallel_runs: 2 + +scenarios: + - name: "Fresh install — creates copilot-instructions.md with managed block" + prompt: "I have a new MAF/Aspire/Foundry agentic .NET app and Copilot keeps missing perf issues. Install the agentic-perf rules into this project." + setup: + files: + - path: "MyAgentApp.sln" + content: | + Microsoft Visual Studio Solution File, Format Version 12.00 + - path: "MyAgentApp.Agent/MyAgentApp.Agent.csproj" + content: | + + + net10.0 + enable + + + + + + + - path: "MyAgentApp.Agent/Program.cs" + content: | + var builder = WebApplication.CreateBuilder(args); + var app = builder.Build(); + app.Run(); + assertions: + - type: "file_exists" + path: ".github/copilot-instructions.md" + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "BEGIN: managed by configure-agentic-perf-rules" + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "END: managed by configure-agentic-perf-rules" + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "per_turn_input_token_warn" + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "Agent count" + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "Handoff edges" + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "Model selection" + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "Message-history strategy" + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "Token / cost surfacing" + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "Post-change measurement" + - type: "exit_success" + rubric: + - "Created .github/copilot-instructions.md (or detected an existing one) and added a managed block delimited by sentinel HTML comments" + - "The managed block contains all six rule categories in the canonical order: agent count, handoff edges, model selection, message-history strategy, token/cost surfacing, post-change measurement" + - "The managed block contains a thresholds YAML map with default values" + - "Did not modify content outside the sentinel-delimited block" + - "Embedded the skill version in the BEGIN sentinel comment" + timeout: 360 + + - name: "Append to existing copilot-instructions.md without disturbing user content" + prompt: "Install the agentic-perf rules into this project. There is already a copilot-instructions.md with team conventions — preserve them." + setup: + files: + - path: ".github/copilot-instructions.md" + content: | + # Team conventions + + - Always use `IChatClient` from `Microsoft.Extensions.AI`, never the provider SDK directly. + - Commit messages follow conventional-commits. + - Run `dotnet format` before pushing. + - path: "MyAgentApp.Agent/MyAgentApp.Agent.csproj" + content: | + + net10.0 + + + + + assertions: + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "Team conventions" + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "conventional-commits" + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "BEGIN: managed by configure-agentic-perf-rules" + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "Agentic Performance Rules" + - type: "exit_success" + rubric: + - "Preserved every line of the existing 'Team conventions' content unchanged" + - "Appended the managed block after the existing content (or in a clearly delimited section)" + - "Did not duplicate or restate the user's existing rules inside the managed block" + timeout: 360 + + - name: "Idempotent re-install — current-version managed block is a no-op" + prompt: "Run configure-agentic-perf-rules on this project." + setup: + files: + - path: ".github/copilot-instructions.md" + content: | + # Project notes + + + + ## Agentic Performance Rules + + ```yaml + thresholds: + per_turn_input_token_warn: 8000 + per_turn_output_token_warn: 2000 + baseline_token_increase_warn_pct: 20 + unbounded_history_warn: true + ``` + + (rule sections elided for fixture brevity) + + + assertions: + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "v0.3.0" + - type: "exit_success" + rubric: + - "Detected the existing managed block at the current skill version" + - "Reported 'already current' (or equivalent) and made no changes to the file" + - "Did not duplicate the managed block" + timeout: 360 + + - name: "Update older version — preserves user-edited threshold values" + prompt: "Update the agentic-perf rules in this project to the latest version. I had to bump the input-token warn level for our RAG workflow — keep that override." + setup: + files: + - path: ".github/copilot-instructions.md" + content: | + + + ## Agentic Performance Rules + + ```yaml + thresholds: + per_turn_input_token_warn: 25000 + per_turn_output_token_warn: 2000 + baseline_token_increase_warn_pct: 20 + unbounded_history_warn: true + ``` + + (older content) + + + assertions: + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "per_turn_input_token_warn: 25000" + - type: "exit_success" + rubric: + - "Detected the existing managed block at an older version (v0.0.1)" + - "Replaced the block with the current-version template" + - "Preserved the user-edited per_turn_input_token_warn value of 25000 in the new block — did NOT reset to the 8000 default" + - "Showed the user a diff or summary of what changed before applying" + timeout: 360 + + - name: "Update from v0.2.0 — drops deprecated agent_count_max and llm_routed_edges_max_per_turn with warning" + prompt: "Update the agentic-perf rules in this project to the latest version." + setup: + files: + - path: ".github/copilot-instructions.md" + content: | + + + ## Agentic Performance Rules + + ```yaml + thresholds: + agent_count_max: 5 + llm_routed_edges_max_per_turn: 3 + per_turn_input_token_warn: 12000 + per_turn_output_token_warn: 2000 + baseline_token_increase_warn_pct: 20 + unbounded_history_warn: true + ``` + + (older content) + + + assertions: + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "per_turn_input_token_warn: 12000" + - type: "exit_success" + rubric: + - "Detected the existing managed block at v0.2.0" + - "Replaced the block with the current-version template (v0.3.0+)" + - "Preserved per_turn_input_token_warn: 12000 (user override) — did NOT reset to default" + - "Dropped the deprecated `agent_count_max` key from the rendered YAML (no longer a threshold in current schema)" + - "Dropped the deprecated `llm_routed_edges_max_per_turn` key from the rendered YAML" + - "Emitted a chat warning naming each dropped key (`agent_count_max`, `llm_routed_edges_max_per_turn`) so the user has an audit trail" + - "Did NOT silently retain the dropped keys in the new managed block" + timeout: 360 + + - name: "AGENTS.md stub when both instructions files exist" + prompt: "Install the agentic-perf rules. The project uses both AGENTS.md and .github/copilot-instructions.md." + setup: + files: + - path: "AGENTS.md" + content: | + # Agent guidelines for this repository + + See `.github/copilot-instructions.md` for repository conventions. + - path: ".github/copilot-instructions.md" + content: | + # Repository conventions + + - Use `IChatClient` for all LLM calls. + assertions: + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "BEGIN: managed by configure-agentic-perf-rules" + - type: "file_contains" + path: "AGENTS.md" + value: "managed by" + - type: "exit_success" + rubric: + - "Wrote the managed block into .github/copilot-instructions.md (the GitHub-native primary location)" + - "Added a single-line stub to AGENTS.md pointing readers to .github/copilot-instructions.md for the agentic-perf rules" + - "Did NOT duplicate the rule prose into AGENTS.md" + - "Preserved the existing content of both files outside the managed block / stub line" + timeout: 360 diff --git a/tests/dotnet-ai/scan-agentic-app-perf/eval.yaml b/tests/dotnet-ai/scan-agentic-app-perf/eval.yaml new file mode 100644 index 0000000000..bc277442dd --- /dev/null +++ b/tests/dotnet-ai/scan-agentic-app-perf/eval.yaml @@ -0,0 +1,126 @@ +name: scan-agentic-app-perf +required_skills: + - scan-agentic-app-perf + +scenarios: + - name: clean-project-zero-criticals + prompt: | + Run scan-agentic-app-perf on the project at ./fixture and report findings. + Read-only audit; do not edit any source files. + setup: + files: + - path: fixture/MyApp.AppHost/Program.cs + content: | + var builder = DistributedApplication.CreateBuilder(args); + builder.AddServiceDefaults(); + var openai = builder.AddConnectionString("openai"); + builder.AddProject("coach") + .WithReference(openai); + builder.Build().Run(); + - path: fixture/MyApp.Coach/Program.cs + content: | + var builder = WebApplication.CreateBuilder(args); + builder.AddServiceDefaults(); + builder.Services.AddOpenTelemetry(); + builder.Services.AddSingleton(sp => + new ChatClient(model: builder.Configuration["Model"], apiKey: "...")); + builder.Services.AddSingleton(sp => new ChatClientAgent( + sp.GetRequiredService(), + instructions: "You are a concise coach.")); + var app = builder.Build(); + app.MapDefaultEndpoints(); + app.Run(); + - path: fixture/MyApp.Coach/appsettings.json + content: | + { "Model": "gpt-4o-mini" } + assertions: + - type: file_exists + path: fixture/.copilot/perf-reports/latest-scan.md + - type: file_contains + path: fixture/.copilot/perf-reports/latest-scan.md + text: "## Findings" + - type: file_contains + path: fixture/.copilot/perf-reports/latest-scan.md + text: "critical: 0" + + - name: sprawl-fixture-flags-topology + prompt: | + Run scan-agentic-app-perf on the project at ./fixture. + setup: + files: + - path: fixture/Sprawl.AppHost/Program.cs + content: | + var builder = DistributedApplication.CreateBuilder(args); + builder.AddProject("a"); + builder.AddProject("b"); + builder.AddProject("c"); + builder.AddProject("d"); + builder.AddProject("e"); + builder.AddProject("f"); + builder.Build().Run(); + - path: fixture/Agent.A/Program.cs + content: | + var agent = new ChatClientAgent(client, instructions: "router"); + agent.AddHandoff("b"); agent.AddHandoff("c"); + - path: fixture/Agent.B/Program.cs + content: | + var agent = new ChatClientAgent(client, instructions: "router"); + agent.AddHandoff("c"); agent.AddHandoff("d"); + assertions: + - type: file_exists + path: fixture/.copilot/perf-reports/latest-scan.md + - type: file_contains + path: fixture/.copilot/perf-reports/latest-scan.md + text: "topology" + - type: output_contains + text: "critical" + + - name: full-history-fixture-flags-message-history + prompt: | + Run scan-agentic-app-perf on the project at ./fixture. + setup: + files: + - path: fixture/HistoryHog.AppHost/Program.cs + content: | + var builder = DistributedApplication.CreateBuilder(args); + builder.AddProject("coach"); + builder.AddProject("critic"); + builder.Build().Run(); + - path: fixture/Coach/Program.cs + content: | + await critic.RunAsync(history, cancellationToken); + assertions: + - type: file_contains + path: fixture/.copilot/perf-reports/latest-scan.md + text: "history" + + - name: prompt-bloat-fixture-flags-prompt-weight + prompt: | + Run scan-agentic-app-perf on the project at ./fixture. + setup: + files: + - path: fixture/PromptHog.AppHost/Program.cs + content: | + var agent = new ChatClientAgent(client, instructions: HUGE_PROMPT); + - path: fixture/PromptHog.AppHost/HugePrompt.cs + content: | + const string HUGE_PROMPT = @"...PLACEHOLDER 12000 CHARS OF RULES AND EXAMPLES..."; + assertions: + - type: file_contains + path: fixture/.copilot/perf-reports/latest-scan.md + text: "prompt" + + - name: missing-otel-fixture-flags-otel-coverage + prompt: | + Run scan-agentic-app-perf on the project at ./fixture. + setup: + files: + - path: fixture/NoOtel.AppHost/Program.cs + content: | + var builder = DistributedApplication.CreateBuilder(args); + builder.AddProject("coach"); + builder.Build().Run(); + assertions: + - type: file_contains + path: fixture/.copilot/perf-reports/latest-scan.md + text: "otel" diff --git a/tests/dotnet-ai/setup-maf-evals/eval.yaml b/tests/dotnet-ai/setup-maf-evals/eval.yaml new file mode 100644 index 0000000000..9a985f5b84 --- /dev/null +++ b/tests/dotnet-ai/setup-maf-evals/eval.yaml @@ -0,0 +1,359 @@ +name: setup-maf-evals +required_skills: + - setup-maf-evals + +scenarios: + - name: scaffold-evals-tests-project-fresh + prompt: | + Run setup-maf-evals on the project at ./fixture. Wire telemetry, + quality, and compare modes. Skip Safety, Aspire panel, and CI workflow. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: | + var builder = DistributedApplication.CreateBuilder(args); + builder.AddAzureOpenAIChatClient("chat", "gpt-4o-mini"); + builder.Build().Run(); + - path: fixture/MyApp.Coach/MyApp.Coach.csproj + content: | + net10.0 + assertions: + - type: file_exists + path: fixture/MyApp.Evals.Tests/MyApp.Evals.Tests.csproj + - type: file_exists + path: fixture/MyApp.Evals.Tests/dotnet-tools.json + - type: file_exists + path: fixture/MyApp.Evals.Tests/Reporting/ReportingConfig.cs + - type: file_exists + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs + - type: file_exists + path: fixture/MyApp.Evals.Tests/Quality/QualityTests.cs + - type: file_exists + path: fixture/MyApp.Evals.Tests/Quality/golden.json + - type: file_exists + path: fixture/MyApp.Evals.Tests/Telemetry/TelemetryTests.cs + - type: file_exists + path: fixture/MyApp.Evals.Tests/Compare/CompareTests.cs + - type: file_contains + path: fixture/MyApp.Evals.Tests/MyApp.Evals.Tests.csproj + text: "Microsoft.Extensions.AI.Evaluation.Reporting" + - type: file_contains + path: fixture/MyApp.Evals.Tests/MyApp.Evals.Tests.csproj + text: "Microsoft.Extensions.AI.Evaluation.NLP" + - type: file_contains + path: fixture/MyApp.Evals.Tests/MyApp.Evals.Tests.csproj + text: "MSTest" + - type: file_contains + path: fixture/MyApp.Evals.Tests/dotnet-tools.json + text: "microsoft.extensions.ai.evaluation.console" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Quality/QualityTests.cs + text: "DiskBasedReportingConfiguration" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Quality/QualityTests.cs + text: "BLEUEvaluator" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Quality/golden.json + text: "reference_response" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Quality/golden.json + text: "schema_version" + - type: file_contains + path: fixture/.gitignore + text: ".copilot/perf-reports/evals/" + - type: file_contains + path: fixture/.gitignore + text: "_store/" + + - name: scaffold-with-safety-tier + prompt: | + Run setup-maf-evals on the project at ./fixture. Include the Safety tier. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: | + var builder = DistributedApplication.CreateBuilder(args); + builder.AddAzureOpenAIChatClient("chat", "gpt-4o-mini"); + assertions: + - type: file_exists + path: fixture/MyApp.Evals.Tests/Safety/SafetyTests.cs + - type: file_contains + path: fixture/MyApp.Evals.Tests/Safety/SafetyTests.cs + text: "ContentHarmEvaluator" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Safety/SafetyTests.cs + text: "Assert.Inconclusive" + - type: file_contains + path: fixture/MyApp.Evals.Tests/MyApp.Evals.Tests.csproj + text: "Microsoft.Extensions.AI.Evaluation.Safety" + + - name: scaffold-with-ci-workflow + prompt: | + Run setup-maf-evals on the project at ./fixture. Include the GitHub Actions workflow. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: "var b = DistributedApplication.CreateBuilder(args); b.AddAzureOpenAIChatClient(\"chat\", \"gpt-4o-mini\");" + assertions: + - type: file_exists + path: fixture/.github/workflows/evals.yml + - type: file_contains + path: fixture/.github/workflows/evals.yml + text: "aieval report" + - type: file_contains + path: fixture/.github/workflows/evals.yml + text: "upload-artifact" + + - name: ichatclient-detection-azure-openai + prompt: | + Run setup-maf-evals on the project at ./fixture. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: | + var builder = DistributedApplication.CreateBuilder(args); + builder.AddAzureOpenAIChatClient("chat", "gpt-4o-mini"); + assertions: + - type: file_exists + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs + - type: file_contains + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs + text: "AddAzureOpenAIChatClient" + - type: output_contains + text: "AppHost.cs" + + - name: ichatclient-detection-missing-emits-stub + prompt: | + Run setup-maf-evals on the project at ./fixture. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: "var b = DistributedApplication.CreateBuilder(args); b.Build().Run();" + assertions: + - type: file_contains + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs + text: "NotImplementedException" + - type: output_contains + text: "no registration found" + + - name: skip-when-no-app-host + prompt: | + Run setup-maf-evals on the project at ./fixture. + setup: + files: + - path: fixture/PlainApi/PlainApi.csproj + content: | + net10.0 + assertions: + - type: output_contains + text: "agentic" + + - name: update-mode-preserves-user-data + prompt: | + Run setup-maf-evals on the project at ./fixture. A MyApp.Evals.Tests project + already exists with a custom rubric and golden.json. Do not overwrite either. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.Evals.Tests/MyApp.Evals.Tests.csproj + content: | + net10.0 + - path: fixture/MyApp.Evals.Tests/Quality/rubric.md + content: | + # Custom rubric — DO NOT OVERWRITE + - my_trait: ... + - path: fixture/MyApp.Evals.Tests/Quality/golden.json + content: | + [{"id": "custom", "user_message": "preserve me", "expected_traits": ["unique"]}] + assertions: + - type: file_contains + path: fixture/MyApp.Evals.Tests/Quality/rubric.md + text: "DO NOT OVERWRITE" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Quality/golden.json + text: "preserve me" + + - name: tier-banner-surfaces-in-chat-output + prompt: | + Run setup-maf-evals on the project at ./fixture. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: "var b = DistributedApplication.CreateBuilder(args); b.AddAzureOpenAIChatClient(\"chat\", \"gpt-4o-mini\");" + assertions: + - type: output_contains + text: "EVAL_USE_REAL_AGENT" + - type: output_contains + text: "EVAL_USE_REAL_JUDGE" + - type: output_contains + text: "EVAL_USE_FOUNDRY_SAFETY" + + - name: scaffold-emits-metrics-glossary + prompt: | + Run setup-maf-evals on the project at ./fixture. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: "var b = DistributedApplication.CreateBuilder(args); b.AddAzureOpenAIChatClient(\"chat\", \"gpt-4o-mini\");" + assertions: + - type: file_exists + path: fixture/MyApp.Evals.Tests/Reporting/MetricsGlossary.cs + - type: file_contains + path: fixture/MyApp.Evals.Tests/Reporting/MetricsGlossary.cs + text: "metrics-glossary.md" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Reporting/MetricsGlossary.cs + text: "[AssemblyCleanup]" + - type: output_contains + text: "metrics-glossary.md" + + - name: factory-emits-friendly-secrets-diagnostic + prompt: | + Run setup-maf-evals on the project at ./fixture. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: | + var builder = DistributedApplication.CreateBuilder(args); + builder.AddAzureChatCompletionsClient("chat").AddChatClient("chat"); + assertions: + - type: file_contains + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs + text: "dotnet user-secrets" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs + text: "ConnectionStrings" + + - name: factory-loads-user-secrets-explicitly + prompt: | + Run setup-maf-evals on the project at ./fixture. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: | + var builder = DistributedApplication.CreateBuilder(args); + builder.AddAzureChatCompletionsClient("chat").AddChatClient("chat"); + assertions: + # Without explicit AddUserSecrets, dotnet test never reads the user-secrets store + # because the entry assembly is testhost.exe, not the test project. + - type: file_contains + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs + text: "AddUserSecrets" + + - name: pitfalls-doc-warns-reasoning-models-reject-max-tokens + prompt: | + Read references/common-pitfalls.md from the setup-maf-evals skill. + assertions: + - type: file_contains + path: ../../../plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md + text: "max_tokens" + - type: file_contains + path: ../../../plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md + text: "EVAL_JUDGE_DEPLOYMENT_NAME" + + - name: pitfalls-doc-warns-stylistic-agents-fail-completeness + prompt: | + Read references/common-pitfalls.md and references/evaluators-catalog.md from setup-maf-evals. + assertions: + - type: file_contains + path: ../../../plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md + text: "Tuning Quality for stylistic agents" + - type: file_contains + path: ../../../plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md + text: "CompletenessEvaluator" + - type: file_contains + path: ../../../plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md + text: "RubricEvaluator" + + - name: compare-mode-is-opt-in-not-default + prompt: | + Read SKILL.md for setup-maf-evals; what's the default for compare mode? + assertions: + - type: file_contains + path: ../../../plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md + text: "Wire compare mode (opt-in)" + - type: file_contains + path: ../../../plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md + text: "Compare mode is opt-in" +