diff --git a/plugins/dotnet-ai/.codex-plugin/plugin.json b/plugins/dotnet-ai/.codex-plugin/plugin.json index 4981aba703..20fbc8bec8 100644 --- a/plugins/dotnet-ai/.codex-plugin/plugin.json +++ b/plugins/dotnet-ai/.codex-plugin/plugin.json @@ -2,6 +2,5 @@ "name": "dotnet-ai", "version": "0.1.0", "description": "AI and ML skills for .NET: technology selection, LLM integration, agentic workflows, RAG pipelines, MCP, and classic ML with ML.NET.", - "skills": ["./skills/"], - "agents": ["./agents/agentic-perf-reviewer.agent.md"] + "skills": ["./skills/"] } diff --git a/plugins/dotnet-ai/agents/agentic-perf-reviewer.agent.md b/plugins/dotnet-ai/agents/agentic-perf-reviewer.agent.md deleted file mode 100644 index 96b45cfd74..0000000000 --- a/plugins/dotnet-ai/agents/agentic-perf-reviewer.agent.md +++ /dev/null @@ -1,128 +0,0 @@ ---- -description: "Reviews .NET agentic applications (Microsoft Agent Framework + Aspire + Foundry) for performance, cost, and reliability issues across topology, tools, message history, prompts, parallelism, OTel coverage, and per-agent model selection. Orchestrates scan-agentic-app-perf, select-agent-models, setup-maf-evals, and configure-agentic-perf-rules to produce a single end-to-end review with actionable recommendations. Use when reviewing an MAF agentic app for perf or cost, when an agent app feels slow, or after non-trivial topology changes. Do NOT use for non-agentic .NET performance reviews (hot-path optimization, allocations, LINQ, async, serialization, general code perf) — use optimizing-dotnet-performance instead." -name: agentic-perf-reviewer -tools: ['read', 'search', 'task', 'skill', 'ask_user'] -license: MIT ---- - -# agentic-perf-reviewer - -You are an architect for .NET agentic applications. Help developers find -and fix the perf, cost, and reliability issues that Copilot routinely -overlooks: agent sprawl, single-model defaulting, full-history sharing, -prompt bloat, missing parallelism, and missing telemetry. - -## Three-Pass Review - -Every review uses three passes. All are mandatory unless the user has -explicitly asked for "quick triage only", in which case stop after -Pass 1 and recommend running the full audit. - -### Pass 1: Direct Read (No Skills) - -Analyze the project using your own knowledge. Do not load skills. - -1. Detect the agentic app: - - Look for `*.AppHost.csproj` files first. - - If none, look for project references to `Microsoft.Agents.AI`, - `Microsoft.Extensions.AI`, `ChatClientAgent`, `IChatClient` builders, - or Foundry agent config. - - If the user named a specific project path, use that even if no - AppHost is present. - - If neither AppHost nor agent signals are present and the user did - not name a target, ask one clarifying question, then stop if the - user cannot identify the agentic app. -2. Inventory: AppHost, agent service projects, per-agent models. -3. Identify the agent topology (count, handoff edges, cycles). -4. Identify the obvious performance smells (one-model defaulting, - full-history sharing, oversized system prompts, sequential awaits). -5. Provide a one-paragraph initial impression. Use **qualitative** - language only — do not produce numeric latency / cost / quality - estimates without telemetry, benchmark, or eval evidence. - -Label this section **"Pass 1: Initial Review"**. - -### Pass 2: Skill-Based Deep Audit - -**Always execute after Pass 1** unless the user asked for quick -triage. Do not ask whether to proceed. - -1. Load **scan-agentic-app-perf** and run it. Capture the report - path at `.copilot/perf-reports/scan-.md`. -2. Read the report file. For each finding, look at the `check_id`. The - prefix encodes the category — `T*` topology, `TI*` tool inventory, - `MH*` message history, `PW*` prompt weight, `P*` parallelism, `O*` - OTel coverage, `MA*` model assignment. Routing rules: - - Any `MA*` finding (model assignment) → suggest loading - `select-agent-models` in recommend mode. - - Any `O*` finding (OTel coverage) → suggest loading - `setup-maf-evals` so telemetry/cost are surfaced going forward. - - If the project has no `.github/copilot-instructions.md` managed - block from `configure-agentic-perf-rules`, suggest installing it - so future sessions volunteer perf concerns by default. -3. Cite findings by `check_id` and `file:line` from the report. Do - not summarize from memory. - -Label this section **"Pass 2: Deep Audit"**. - -### Pass 3: Synthesis - -After Pass 2, produce a single prioritized action list: - -1. The 3 highest-impact changes the user should make first. -2. For each, the skill that performs it (or "manual fix"). -3. The expected effect — qualitative only (e.g. "lower per-turn - token cost", "shorter critical-path latency"). Use numeric - estimates only if `setup-maf-evals` has already produced a report - you can cite. -4. The risk and how to validate (almost always: run setup-maf-evals). -5. **Offer the follow-ups.** Once the action list is on screen, ask - the user **once** whether they want to invoke any of the routed - skills now. Render only the lettered options whose target skill is - actually referenced in the synthesis, e.g.: - > Want me to run any of these now? - > - **A.** `select-agent-models` (recommend mode) for the model - > findings. - > - **B.** `setup-maf-evals` to capture token/quality numbers. - > - **C.** `configure-agentic-perf-rules` to install always-on rules. - > - **D.** No — leave the report and stop. - If the user picks a letter, hand off to that skill with the audit - report path as context. The invoked skill still owns its own - diff-and-confirm flow — do not pre-confirm on the user's behalf - (see Boundaries). - -## Boundaries - -- **Do not edit source.** This agent has no `edit` tool. If a fix - requires file modifications, route to a skill that owns the - diff-and-confirm flow. -- Do not pick models without running `select-agent-models`. -- Do not recommend a model downgrade without recommending a - `setup-maf-evals` quality follow-up. -- Cite findings by `check_id` and `file:line`; do not summarize the - audit report from memory. -- **Apply-mode chaining:** if the user says something like "apply the - fixes" in the same turn as invoking this agent, treat that as - *intent* but not as *confirmation*. The invoked skill (e.g. - `select-agent-models` apply mode) must still present its own diff - and obtain its own confirmation before any write. Do not pre-confirm - on the user's behalf. -- Do not apply this agent to non-agentic .NET apps. If detection in - Pass 1 fails, say so and stop. - -## Output Format - -Keep reports concise and actionable. - -1. **Pass 1: Initial Review** — paragraph + 3-5 bullets. -2. **Pass 2: Deep Audit** — top critical / warn findings cited by - `check_id` and `file:line` with the report path. -3. **Pass 3: Synthesis** — numbered action list with skill routes. -4. **Next steps** — exact commands or skill names to run. - -## Skills used - -- `scan-agentic-app-perf` — read-only audit, the workhorse of Pass 2. -- `select-agent-models` — per-agent model recommendations. -- `setup-maf-evals` — telemetry / quality / compare harness. -- `configure-agentic-perf-rules` — install always-on rules. diff --git a/plugins/dotnet-ai/plugin.json b/plugins/dotnet-ai/plugin.json index 4981aba703..20fbc8bec8 100644 --- a/plugins/dotnet-ai/plugin.json +++ b/plugins/dotnet-ai/plugin.json @@ -2,6 +2,5 @@ "name": "dotnet-ai", "version": "0.1.0", "description": "AI and ML skills for .NET: technology selection, LLM integration, agentic workflows, RAG pipelines, MCP, and classic ML with ML.NET.", - "skills": ["./skills/"], - "agents": ["./agents/agentic-perf-reviewer.agent.md"] + "skills": ["./skills/"] } diff --git a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/SKILL.md b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/SKILL.md index 2247e5f4a2..e1ec6c5bf2 100644 --- a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/SKILL.md +++ b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/SKILL.md @@ -1,6 +1,6 @@ --- name: configure-agentic-perf-rules -version: 0.1.0 +version: 0.3.0 description: > Installs or updates an always-on rules block in a .NET agentic app that makes coding agents volunteer perf and cost concerns by default — agent count, handoff edges, @@ -43,7 +43,6 @@ clobbering user-edited threshold values. - The user wants the agent to actually audit existing code right now — use `scan-agentic-app-perf` instead. This skill only installs guidance. - The user wants to measure tokens, latency, or quality scores — use `setup-maf-evals`. -- The user wants to pick or change per-agent model assignments — use `select-agent-models`. - Generic prompt-engineering or non-perf coding-agent rules (keep those in the user's own instructions section, outside the managed block). @@ -113,7 +112,7 @@ The current skill version is the `version:` field at the top of this SKILL.md. If parsing fails, refuse to edit and ask the user to repair the YAML manually. 2. Construct the new defaults map `new_defaults` from `references/threshold-defaults.md`. 3. For each known key in `new_defaults`, override with the value from `prev_user` if - present and the value passes type validation (e.g. integer for `agent_count_max`). + present and the value passes type validation (e.g. integer for `per_turn_input_token_warn`). 4. Drop unknown keys from `prev_user` with a chat warning naming each dropped key. 5. The merged map becomes the new managed block's `thresholds:` content. @@ -160,7 +159,8 @@ Each rule is in the form **"Before X, justify Y."** Categories, in order: deterministic edge or a conditional `WorkflowBuilder` branch will not work. Default ceiling: 2 LLM-routed edges traversed per user turn. 3. **Model selection.** Before defaulting to a frontier model (e.g. `gpt-4o`), name the - agent's role and pick from the role→model matrix in the `select-agent-models` skill. + agent's role and pick from the role table inside rule #3 of the managed block. + Routers/validators/formatters/workers → small-fast; planners → reasoning-class. 4. **Message-history strategy.** Before sending the full conversation history to an agent, state the bound — turn count, token cap, summarization point, or retrieval strategy. Default warning when unbounded full-history is used in a multi-turn workflow. @@ -238,4 +238,4 @@ If `AGENTS.md` was updated, also confirm the stub line is present exactly once. - `references/threshold-defaults.md` — default numeric values and the rationale for each. - `references/rule-rationales.md` — long-form prose for each of the six rule categories, with examples and counter-examples. -- Companion skills: `scan-agentic-app-perf`, `select-agent-models`, `setup-maf-evals`. +- Companion skills: `scan-agentic-app-perf`, `setup-maf-evals`. diff --git a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/common-pitfalls.md b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/common-pitfalls.md new file mode 100644 index 0000000000..b892442cb2 --- /dev/null +++ b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/common-pitfalls.md @@ -0,0 +1,100 @@ +# Common pitfalls + +Real-world failure modes for `configure-agentic-perf-rules` — observed +during dogfooding (interview-coach v2 first install, behavioral coach +no-op re-run, ELI5Agent fresh install). + +## Sentinel parsing (FAIL CLOSED, always) + +- **Lenient sentinel matching.** The BEGIN and END regexes in step 2 + are intentionally strict (anchored, full-line, exact version + pattern). Do not "fix up" a malformed sentinel by partial-matching + or by accepting trailing whitespace beyond what `\s*$` allows. + Auto-repair of corrupted sentinels is forbidden — the user might + have intentionally renamed the block while migrating, and silent + repair would clobber that. +- **Refusing to edit on multiple BEGIN/END.** If the file contains + more than one BEGIN or END, abort with a chat message that names + every offending line number. Multiple managed blocks usually mean + a merge conflict went unresolved; appending a third would make it + worse. +- **Out-of-order sentinels.** BEGIN must appear before its matching + END. If you find END before BEGIN, treat as malformed and abort — + do not swap them. + +## Threshold preservation + +- **Losing user-edited threshold values on update.** Step 2's + "Threshold preservation algorithm" is the contract: parse the + existing `thresholds:` map into `prev_user`, overlay the new + default map, override each known key from `prev_user`. Skipping + this step and writing the new defaults verbatim is the most + user-visible regression this skill can ship. +- **Silently dropping unknown keys.** If `prev_user` has a key not + in `new_defaults` (e.g. a deprecated threshold), drop it AND emit + a chat warning naming the dropped key. Silent drops break audit + trails — the user needs to know their override no longer applies. +- **Type-validating with `Convert.ToInt32` instead of strict parse.** + `per_turn_input_token_warn: "eight thousand"` should fail validation, not coerce to + some default. Use strict numeric parsing; on failure, keep the + default and warn. + +## Target-file selection + +- **Writing to `AGENTS.md` when `.github/copilot-instructions.md` + exists.** The order in step 2's target-file table is the spec: + `.github/copilot-instructions.md` is the primary destination; + `AGENTS.md` gets a stub pointer. The reverse only happens during + the explicit migration path (existing block in AGENTS.md, none in + copilot-instructions.md). +- **Writing to both files.** "Both have a block" is the abort path — + the user must consolidate manually. Writing the rule prose into + two files would split the source of truth and let the two copies + drift on the next update. +- **Creating `.github/` outside the project root.** If neither file + exists, create `.github/copilot-instructions.md` under the resolved + project root (the directory containing the `.sln`/`.slnx`/`*.AppHost.csproj`). + Never write to a parent directory or a sibling project. + +## Path safety + +- **Following symlinks out of the project root.** Step 2's "Path + safety" rule requires resolving the absolute path AND ensuring it + still starts with the project-root prefix after symlink resolution. + Some CI environments place repos under symlinks; a naive resolve + can end up writing to the symlink target outside the workspace. +- **Accepting `..` in paths.** Reject any path containing `..` + segments before normalization, unless the post-normalization path + is still inside the project root. The simplest safe check: + `Path.GetFullPath(target).StartsWith(Path.GetFullPath(projectRoot))`. + +## Cross-tool stub on AGENTS.md + +- **Re-adding the stub on every run.** The stub is one line: + `> Agentic-perf rules for this project live in .github/copilot-instructions.md (managed by configure-agentic-perf-rules).` + Check whether that exact line is already present before appending; + re-running the skill should not grow the file by one line each time. +- **Replacing user prose in AGENTS.md with the stub.** AGENTS.md + often contains real onboarding prose the user wrote. Append the + stub at the bottom if missing; never overwrite existing content. + +## Version handling + +- **Refusing to downgrade is correct.** If the file has a newer + version than this skill, abort cleanly with a chat message naming + both versions. Do not "merge" or "convert" — that's how data loss + happens. +- **Comparing versions as strings.** "v0.1.10" sorts before "v0.1.2" + lexically. Parse into semver triples and compare numerically. + +## Idempotency + +- **No-op path must actually be a no-op.** When the block is present, + same version, and structurally valid, the skill must not touch the + file at all — not even to rewrite identical content. Tooling and + git status both rely on "no change on second run". Verify by + comparing the SHA256 hash before/after; they should match exactly. +- **Tracking "already installed" silently.** Even on a no-op, the + chat output should say "configure-agentic-perf-rules v0.1.0 block + already current — no changes". Quiet no-ops make the user think + the skill didn't run. diff --git a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/managed-block-template.md b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/managed-block-template.md index 568d302d4e..fe69aa65f2 100644 --- a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/managed-block-template.md +++ b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/managed-block-template.md @@ -18,8 +18,6 @@ interpreted as closing it. ```yaml # thresholds — edit values below to override per-project defaults thresholds: - agent_count_max: 3 - llm_routed_edges_max_per_turn: 2 per_turn_input_token_warn: 8000 per_turn_output_token_warn: 2000 baseline_token_increase_warn_pct: 20 @@ -33,23 +31,36 @@ form **"Before X, justify Y."** When you cannot justify, prefer the safer altern ### 1. Agent count Before adding a new agent to a workflow, justify why the new responsibility cannot be a -tool call on an existing agent. Default ceiling: **`agent_count_max`** agents per -workflow. If the workflow already has that many agents, do not add another without -explicit user direction. +tool call on an existing agent. Each additional agent multiplies the routing surface and +inflates per-turn token cost (system prompts + tool descriptions are paid per agent). +**There is no hard ceiling** — the answer "this needs a clearly different system prompt, +toolset, or output style" is a valid justification. If you cannot articulate one, prefer +adding a tool to an existing agent. ### 2. Handoff edges Before adding an LLM-routed handoff edge (e.g. via `AgentWorkflowBuilder.CreateHandoffBuilderWith`), justify why a deterministic edge or a -conditional `WorkflowBuilder` branch will not work. Default ceiling: -**`llm_routed_edges_max_per_turn`** LLM-routed edges traversed per user turn. +conditional `WorkflowBuilder` branch will not work. Every LLM-routed edge is an extra LLM +call before the user gets a response. Deterministic routing is faster and cheaper; reserve +LLM routing for decisions that genuinely require reading user intent. ### 3. Model selection -Before defaulting to a frontier model (e.g. `gpt-4o`), name the agent's role and pick -from the role→model matrix in the `select-agent-models` skill. Routers, classifiers, -and summarizers usually want a smaller/faster model; reasoning steps may want a -reasoning-class model. +Before defaulting to a frontier model like `gpt-4o`, name the agent's role and pick +from the table below. Routers, validators, formatters, and workers almost never need +a frontier model; defaulting to one is the largest single source of unnecessary spend. + +| Role | Pick | +|-------------------------------------|-----------------------------------------------------------------------------------| +| router / validator / formatter | small-fast model (e.g. `gpt-4o-mini` or current cheap-fast in your Foundry catalog) | +| worker / summarizer / extraction | small-fast model, **or** Foundry `model-router` deployment if prompt length varies | +| planner / decomposer / open reasoning | reasoning-class model (e.g. `o4-mini` or current reasoning model) — state *why* in a code comment | +| creative / nuanced generation | frontier (e.g. `gpt-4o`) — state *why* in a code comment | + +If unsure which role applies, **stop and ask the user** — do not default to `gpt-4o`. +Specific model ids age fast; check your Foundry catalog for the current cheap-fast, +reasoning-class, and frontier ids before pinning. ### 4. Message-history strategy diff --git a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/rule-rationales.md b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/rule-rationales.md index 3832820c5f..03a465fbc6 100644 --- a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/rule-rationales.md +++ b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/rule-rationales.md @@ -9,12 +9,13 @@ managed block. The managed block itself is intentionally terse; this file is the ## 1. Agent count **Rule.** Before adding a new agent to a workflow, justify why the new responsibility -cannot be a tool call on an existing agent. Default ceiling: 3 agents per workflow. +cannot be a tool call on an existing agent. **No hard ceiling** — the answer "this needs +a clearly different system prompt, toolset, or output style" is a valid justification. -**Why it matters.** Each additional agent multiplies the routing surface area: the -LLM has to decide whether *this* turn should go to *that* agent, and decision quality -falls off as choices grow. Real-world failure mode: a 5-specialist workflow where the -router constantly mis-handoffs because too many specialists have overlapping domains. +**Why it matters.** Each additional agent multiplies the routing surface area (the LLM +has to decide whether *this* turn should go to *that* agent) AND inflates per-turn +input-token cost: every agent's system prompt + tool descriptions are paid on every +turn the agent participates in. Decision quality degrades as choices grow. **When a new agent is justified.** The new responsibility involves a meaningfully different *system prompt*, *toolset*, or *output style* that would muddy the existing @@ -26,22 +27,23 @@ agent's instructions if folded in. Examples: whatever agent needs the data, not its own agent. - **No, just a tool:** A "Formatter" agent that reformats output. Again, a tool. -**When you must add a 4th+ agent.** Surface the trade-off to the user. Mention the -default ceiling, why it exists, and what mitigations apply (e.g. tighter routing -prompts, explicit `WorkflowBuilder` branches instead of free-form handoffs). +**On thresholds.** Earlier versions of this rule had an `agent_count_max: 3` numeric +ceiling. It was removed because legitimate designs (e.g. specialist-handoff +interview/coaching workflows with 4-5 well-scoped agents) tripped it as often as +genuine bloat did. The justify-the-add gate is the actual mechanism; a number was +illusory precision. --- ## 2. Handoff edges **Rule.** Before adding an LLM-routed handoff edge, justify why a deterministic edge or -a conditional `WorkflowBuilder` branch will not work. Default ceiling: 2 LLM-routed -edges traversed per user turn. +a conditional `WorkflowBuilder` branch will not work. **No hard ceiling.** **Why it matters.** Every LLM-routed edge is an additional LLM call before the user -gets a response. Two routed decisions per turn (e.g. "router → specialist", "specialist -→ done-or-continue") is the practical latency ceiling before users notice. -Free-form-everywhere graphs also amplify decision-quality variance. +gets a response. Deterministic routing — "after Coach runs, always return to +Interviewer" expressed as a `WorkflowBuilder` edge — costs zero extra latency and zero +extra tokens. Free-form LLM routing also amplifies decision-quality variance. **When LLM routing is justified.** The decision genuinely requires reading the user's intent — for example, "is this answer detailed enough to grade?" or "which specialist @@ -57,32 +59,49 @@ runs, always go back to interviewer"), use a deterministic edge. - Five specialists in a fully-connected handoff graph where every transition is LLM-routed. Symptom: Copilot constantly proposes new edges as the workflow grows. +**On thresholds.** Earlier versions of this rule had an +`llm_routed_edges_max_per_turn: 2` numeric ceiling. Same reason as rule #1: the gate +is the justification, not the number. A 4-hop deterministic chain with one LLM-routed +intent decision is fine; two LLM-routed edges per turn in a misdesigned graph is not. + --- ## 3. Model selection **Rule.** Before defaulting to a frontier model (e.g. `gpt-4o`), name the agent's role -and pick from the role→model matrix in the `select-agent-models` skill. +and pick from the table below. If unsure which role applies, **stop and ask the user** +— do not silently default to `gpt-4o`. **Why it matters.** A frontier model on a router/triage agent costs roughly 10x more per token than `gpt-4o-mini` and is often *worse* at the routing job (frontier models are tuned for nuanced generation, not cheap classification). The default-everything-to- gpt-4o pattern is the largest single source of unnecessary spend in agentic apps. -**Quick role mapping (full matrix in `select-agent-models`):** +**Role → model class:** -| Role | Recommended class | -|------|-------------------| -| Router / triage / "is this done?" | small-fast (gpt-4o-mini, gpt-5-mini, phi-4) | -| Classifier / scorer with structured JSON | small + JSON mode + low temp | -| Summarizer / extraction | small with high context | -| Open-ended reasoning / planning | reasoning class (o1, o3-mini) | -| Tool-heavy specialist | mid-tier with strong function-calling fidelity | -| Creative generation / nuanced writing | frontier (gpt-4o) | +| Role | Pick | Why | +|-------------------------------------|-------------------------------------------------------|--------------------------------------------------------| +| Router / triage / "is this done?" | small-fast (`gpt-4o-mini`, current cheap-fast id) | Classification, not generation; latency dominates | +| Validator / scorer / structured JSON| small-fast + JSON mode + low temp | Deterministic output; cache-friendly | +| Formatter (Markdown / JSON shape) | small-fast, pinned | Output stability matters more than peak quality | +| Worker / summarizer / extraction | small-fast, **or** Foundry `model-router` if prompt length varies | Most calls happen here; latency dominates | +| Planner / decomposer / reasoning | reasoning-class (`o4-mini`, current reasoning id) | Output drives N downstream calls; quality matters most | +| Creative / nuanced generation | frontier (`gpt-4o`, current frontier id) | Genuinely needs frontier capability | **When frontier is justified.** The agent's job is genuinely creative or nuanced -generation, or it must follow complex instructions reliably. Routers and scorers -almost never fall in this bucket. +generation, or it must follow complex instructions reliably. Routers, validators, +formatters, and most workers almost never fall in this bucket. + +**Specific model ids age fast.** The table above uses `gpt-4o-mini`, `o4-mini`, and +`gpt-4o` as anchor examples. Before pinning, check your Foundry catalog +(https://learn.microsoft.com/azure/foundry/openai/concepts/models) for the current +cheap-fast, reasoning-class, and frontier ids. Foundry's `model-router` deployment is +the recommended pick whenever the prompt length or complexity genuinely varies per +request and you don't need cache stability (typical: worker tier). + +**State the why in code.** When you pick a frontier or reasoning model, leave a +one-line comment naming the role and why the cheaper tier wouldn't work. This makes +the choice auditable and the next person can challenge it. --- diff --git a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/threshold-defaults.md b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/threshold-defaults.md index 987d204188..d2ce7698d9 100644 --- a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/threshold-defaults.md +++ b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/threshold-defaults.md @@ -6,13 +6,18 @@ simple two-agent workflows, or complex tool-heavy pipelines) should adjust. | Threshold | Default | Rationale | |-----------|---------|-----------| -| `agent_count_max` | `3` | Most workflows that need more than 3 agents are better served by tools-on-fewer-agents. Real-world frustration: 5+ specialist workflows with free-form LLM routing dramatically slow per-turn latency and confuse handoff decisions. | -| `llm_routed_edges_max_per_turn` | `2` | Each LLM-routed edge is an extra LLM call. Two routed decisions per user turn (e.g. "router → specialist", "specialist → done") is the practical ceiling before latency becomes user-visible. | | `per_turn_input_token_warn` | `8000` | Modern reasoning models can take 100K+, but most chat-class models start showing meaningful latency and cost above ~8K input tokens. Projects with retrieval/RAG legitimately exceed this — override locally. | | `per_turn_output_token_warn` | `2000` | Output tokens are usually 4-10x more expensive than input on a per-token basis. 2000 is a reasonable "are you sure?" threshold; long-form generation tasks should override. | | `baseline_token_increase_warn_pct` | `20` | A 20% increase per turn meaningfully changes monthly bills at scale. Small tweaks under 20% are noise; over 20% is worth surfacing. | | `unbounded_history_warn` | `true` | Default-on. Sending full history forever is the single most common token-bloat pattern. Disable only if the workflow has already implemented a windowing/summarization strategy and the warning is now noise. | +**Note on removed thresholds.** Earlier versions of this skill (≤v0.2.0) shipped +`agent_count_max: 3` and `llm_routed_edges_max_per_turn: 2`. Both were taste-call +ceilings that tripped legitimate designs (5-specialist handoff workflows, multi-hop +deterministic chains with one LLM-routed intent decision) as often as they caught real +bloat. The justify-the-add gates in rules #1 and #2 are the actual mechanism; the +numbers were illusory precision and were removed in v0.3.0. + ## Adjusting thresholds Users override defaults inside the managed block's `thresholds:` YAML map. The skill diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md index cff5c4f788..28c30589bb 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md @@ -1,7 +1,7 @@ --- name: scan-agentic-app-perf description: | - Scan a .NET agentic application (Microsoft Agent Framework + Aspire + Foundry) for performance, cost, and reliability issues across seven check categories: topology, tool inventory, message-history strategy, prompt weight, parallelism, OTel coverage, and per-agent model assignment. Produces a Markdown report at .copilot/perf-reports/scan-.md (plus latest-scan.md) with severity-tagged findings (critical/warn/info), file:line citations, evidence, and concrete next actions that can route into select-agent-models, setup-maf-evals, or configure-agentic-perf-rules. WHEN: user asks "why is my agent slow", "scan my agentic app", "audit my agentic app", "find perf issues", "is my topology too complex", or has just modified an agent topology. NOT-WHEN: user wants to install always-on rules (use configure-agentic-perf-rules), pick models per role (use select-agent-models), or wire up evaluations (use setup-maf-evals); not for non-agentic .NET apps. Read-only — never edits source files. + Scan a .NET agentic application (Microsoft Agent Framework + Aspire + Foundry) for performance, cost, and reliability issues across seven check categories: topology, tool inventory, message-history strategy, prompt weight, parallelism, OTel coverage, and per-agent model assignment. Produces a Markdown report at .copilot/perf-reports/scan-.md (plus latest-scan.md) with severity-tagged findings (critical/warn/info), file:line citations, evidence, and concrete next actions that can route into configure-agentic-perf-rules or setup-maf-evals. WHEN: user asks "why is my agent slow", "scan my agentic app", "audit my agentic app", "find perf issues", "is my topology too complex", or has just modified an agent topology. NOT-WHEN: user wants to install always-on rules (use configure-agentic-perf-rules), or wire up evaluations (use setup-maf-evals); not for non-agentic .NET apps. Read-only — never edits source files. --- # scan-agentic-app-perf @@ -50,29 +50,33 @@ Every finding is a dict with these fields: ```yaml severity: critical | warn | info -check_id: T1 | T2 | T3 | T4 | TI1 | TI2 | TI3 | TI4 | MH1 | MH2 | MH3 | PW1 | PW2 | PW3 | P1 | P2 | P3 | O1 | O2 | O3 | O4 | MA1 | MA2 | MA3 | MA4 +check: one of the slugs listed in references/check-glossary.md + (e.g. model.same-default, history.full-share, otel.missing-sdk) title: short imperative phrase file: path relative to repo root line: 1-based line number, or null evidence: 1-3 line code snippet or measurement (must be present in the cited file) why: one paragraph explaining the impact next: concrete action the developer can take next -ref: optional cross-skill route (e.g. "skill:select-agent-models") +ref: optional cross-skill route (e.g. "skill:configure-agentic-perf-rules") ``` -The `check_id` prefix encodes the category — there is no separate -`category` field. Prefix glossary (also rendered at the top of the +The `check` field is a dotted slug: `.`. The +prefix before the dot encodes the category — there is no separate +`category` field. Category card (also rendered at the top of the report's `## Findings` section): -| Prefix | Category | Reference | -|--------|-------------------------|----------------------------------------| -| `T` | topology | `references/topology-checks.md` | -| `TI` | tool inventory | `references/tool-inventory-checks.md` | -| `MH` | message history | `references/message-history-checks.md` | -| `PW` | prompt weight | `references/prompt-weight-checks.md` | -| `P` | parallelism | `references/parallelism-checks.md` | -| `O` | OTel coverage | `references/otel-coverage-checks.md` | -| `MA` | model assignment | `references/model-assignment-checks.md`| +| Category | Coverage | Reference | +|------------|-------------------------|----------------------------------------| +| `topology` | agent graph shape | `references/topology-checks.md` | +| `tools` | per-agent tool list | `references/tool-inventory-checks.md` | +| `history` | chat history strategy | `references/message-history-checks.md` | +| `prompt` | system prompt size/reuse| `references/prompt-weight-checks.md` | +| `parallel` | concurrent invocations | `references/parallelism-checks.md` | +| `otel` | instrumentation | `references/otel-coverage-checks.md` | +| `model` | per-agent model id | `references/model-assignment-checks.md`| + +See `references/check-glossary.md` for the full slug→description table. **Evidence gate** — before adding a finding to the report, re-open the cited file and confirm the `evidence` snippet is present at or near @@ -89,27 +93,32 @@ Severity rules: ### 4. Aggregate and write the report -Sort findings by severity (critical → warn → info), then by `check_id` -(stable lexical order so `T1` < `T2` < `TI1` < `MH1` < ...). Write to: +Sort findings by severity (critical → warn → info), then by `check` +slug (stable lexical order so `history.*` < `model.*` < `otel.*` < +`parallel.*` < `prompt.*` < `tools.*` < `topology.*`). -- `.copilot/perf-reports/scan-.md` (timestamped, kept) -- `.copilot/perf-reports/latest-scan.md` (overwritten each run) +Write a **single file**: `.copilot/perf-reports/scan.md`, overwritten +on every run. Git provides whatever history you want; the skill does +not maintain timestamped copies or `latest-*` mirrors. -See `references/report-template.md` for the exact layout. +The category legend (one-liners describing what each prefix covers) is +inlined into the report itself — see `references/report-template.md`. +There is no separate glossary file. The slug encodes the check (e.g. +`history.full-share`, `topology.cycle`); readers don't need a lookup +table. Create `.copilot/perf-reports/` if it does not exist. This skill never touches `.gitignore`. If the user wants the report -folder ignored, recommend in chat that they add -`.copilot/perf-reports/` to their `.gitignore` themselves; do not edit -it from this skill. +ignored, recommend in chat that they add `.copilot/perf-reports/` to +their `.gitignore` themselves; do not edit it from this skill. ### 5. Surface top findings in chat Print: 1. Total counts (critical / warn / info). -2. The first up to 3 critical findings with title + check_id + file:line + next action. +2. The first up to 3 critical findings with title + `check` slug + file:line + next action. 3. The full report path. 4. If any findings have a `ref:` field, list the suggested follow-up skills. @@ -125,7 +134,7 @@ diff-and-confirm flow. 1. Aggregate the unique `ref:` values across all findings. 2. If the set is non-empty, print one prompt of the form: > Want me to follow up on any of these? - > - **A.** Run `select-agent-models` (recommend mode) for the `MA*` findings. + > - **A.** Install/update perf rules via `configure-agentic-perf-rules` to enforce role-aware model selection on future code. > - **B.** Run `setup-maf-evals` to capture token/quality numbers. > - **C.** Run `configure-agentic-perf-rules` to install always-on rules. > - **D.** No — just leave the report. @@ -150,19 +159,18 @@ ends here. After running: -- A new file exists at `.copilot/perf-reports/scan-.md`. -- `latest-scan.md` exists in the same folder and matches the timestamped - file byte-for-byte. +- A file exists at `.copilot/perf-reports/scan.md` (overwritten if it + was there before). - The report has a `## Findings` section, even if empty (containing `_No findings._`). - The summary counts in the report match the chat output. ## Common pitfalls -- **Editing source code.** This skill is read-only and only writes to - `.copilot/perf-reports/`. Never edit `.gitignore`, source files, - config files, or anything else. If a check tempts you to fix the - issue inline, stop and add it as a finding instead. +- **Editing source code.** This skill is read-only. The ONLY write path + it owns is `.copilot/perf-reports/scan.md`. Never edit `.gitignore`, + source files, config files, or anything else. If a check tempts you + to fix the issue inline, stop and add it as a finding instead. - **Hallucinating findings.** Every finding must cite a real file and (where applicable) a real line. Before adding a finding to the report, re-open the cited file and verify the snippet exists at the @@ -175,6 +183,12 @@ After running: `configure-agentic-perf-rules` instead. - **Running on non-agentic apps.** If no agent registrations are found, abort cleanly. Do not invent an audit for a plain web API. +- **Recommending a model downgrade without an eval gate.** Any MA* + finding that says "downgrade Agent X from gpt-4o to gpt-4o-mini" + must be paired in the `next:` field with "validate via + `setup-maf-evals` quality mode before shipping". Apparent free wins + on cost frequently regress quality on edge cases — the eval gate + protects against that. ## References @@ -185,4 +199,5 @@ After running: - `references/parallelism-checks.md` — sequential calls that could fan out. - `references/otel-coverage-checks.md` — Aspire dashboard, token/cost telemetry. - `references/model-assignment-checks.md` — single-model defaulting, role mismatch. -- `references/report-template.md` — exact Markdown layout for the report. +- `references/check-glossary.md` — dev-facing catalog of all check slugs (NOT copied to user repos; for skill maintainers). +- `references/report-template.md` — exact Markdown layout for `scan.md`. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-glossary.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-glossary.md new file mode 100644 index 0000000000..3faa29a706 --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-glossary.md @@ -0,0 +1,41 @@ +# Check glossary (dev-facing) + +Canonical catalog of every `check` slug the skill can emit. This file +is for **skill maintainers** — adding or renaming a check requires a +row here. It is **not copied** into user repositories; the slugs are +self-describing in the report itself. + +If you change this table, also update the matching `*-checks.md` +reference file with the per-check detection logic. + +## Catalog + +| Check | Sev | What it catches | +|------------------------------------|----------|-----------------| +| `topology.cycle` | critical | Directed cycle in the agent handoff graph | +| `topology.deep-single-leaf` | warn | 3+ hops to reach a single terminal agent | +| `tools.duplicate` | warn | Same tool description on >1 agent | +| `tools.dead` | info | Tool registered but never referenced | +| `tools.description-too-long` | warn | `[Description]` attribute > 200 chars | +| `history.full-share` | critical | Entire chat history passed to a downstream agent | +| `history.unbounded` | warn | No `MaxMessages` / reducer / summarizer wired | +| `history.through-deterministic` | warn | Full history given to a deterministic agent | +| `prompt.oversized` | warn | System prompt > 2K tokens (critical at > 4K) | +| `prompt.duplicate-preamble` | warn | Same > 100-token block shared across agents | +| `parallel.independent-handoffs` | warn | Sequential `await`s on agents that don't share data | +| `parallel.hidden-tool-fanout` | info | Tool internally serializes 3+ API calls | +| `otel.missing-sdk` | critical | No `AddOpenTelemetry` / Aspire dashboard wiring | +| `otel.no-aspire-dashboard` | warn | Aspire dashboard not declared | +| `otel.no-token-cost` | warn | No `gen_ai.usage.*` tags surfaced | +| `otel.no-per-agent-source` | info | All agents share one `ActivitySource` | +| `model.same-default` | warn | All agents use the same model id | +| `model.reasoning-on-deterministic` | warn | Frontier model on a formatter / validator | +| `model.cheap-on-planner` | warn | Small model on the planner, larger on workers | +| `model.hardcoded` | info | Model id literal in agent service `.cs` | + +## Cross-skill routes + +| `ref:` value | Skill to run next | +|--------------------------------------|------------------------------------------------------------| +| `skill:configure-agentic-perf-rules` | Install/update the always-on perf rules (rule #3 covers role-aware model selection) | +| `skill:setup-maf-evals` | Wire eval reports + telemetry | diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/common-pitfalls.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/common-pitfalls.md new file mode 100644 index 0000000000..9f9bd25cc6 --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/common-pitfalls.md @@ -0,0 +1,95 @@ +# Common pitfalls + +Real-world failure modes for `scan-agentic-app-perf` — observed during +dogfooding (interview-coach v1/v2, ELI5Agent, behavioral-interview-coach). + +## Output discipline + +- **Editing source code.** This skill is read-only. The ONLY write path + it owns is `.copilot/perf-reports/scan.md`. Never touch `.gitignore`, + source files, config files, AppHost, or `*.csproj`. If a check tempts + you to fix the issue inline, stop and add it as a finding instead — + the user opts into the fix through the step-6 routing prompt. +- **Surfacing more than 3 critical findings in chat.** Step 5 caps + chat output at 3 critical findings + counts + report path. Anything + beyond that goes in the report file only. Pasting the entire report + into chat defeats the "single file you can re-open" UX. +- **Generating extra report files.** One file: `scan.md`. Do not + write timestamped copies, `latest-*` mirrors, glossary files, or + any other artifact. Git is the history mechanism if the user wants + one. + +## Evidence integrity + +- **Hallucinating findings.** Every finding must cite a real file + and (where applicable) a real line. Before adding a finding, re-open + the cited file and verify the snippet exists at the cited line — that + is the "evidence gate" in step 3. If you can't point to evidence, + drop the finding. A 5-finding report with verified citations beats + a 15-finding report with two hallucinations every time. +- **Citing line ranges without re-reading.** Files drift between + the inventory pass (step 1) and the per-category checks (step 2). + Re-read the cited region right before writing each finding, not + before the entire batch. +- **Absence-of-X findings without searched-pattern evidence.** When + a finding fires because something is *missing* (e.g. `otel.no-token-cost` "no token + surfacing"), the `evidence` field MUST list the exact files and + patterns that were searched — not a snippet. This is what lets the + user reproduce the negative result. + +## False positives we keep seeing + +- **`model.hardcoded` firing on AppHost code.** This check is about + model ids living in *agent service* files (`*.Agent/Program.cs` + etc.) where swapping requires a code change. Model ids declared in + the AppHost via `foundry.AddDeployment("chat", FoundryModel.OpenAI.Gpt4oMini)` + are the **canonical Aspire-native place** — that's not a defect. + When you encounter this pattern, either suppress `model.hardcoded` + entirely or downgrade to `info` with a "no action required" `next:`. +- **`model.same-default` on single-agent apps.** The check explicitly + assumes ≥2 agents (different roles, different needs). Do not fire on + 1-agent apps; the check is trivially "satisfied" with one model. +- **`otel.no-token-cost` when `Microsoft.Extensions.AI` activity + source is registered.** MEAI emits `gen_ai.usage.*` activity tags + automatically; if `AddSource("Microsoft.Extensions.AI")` is wired in + OTel and an OTLP exporter is configured, this check should NOT + fire. The check is for codebases that strip the source or wrap MEAI + behind custom infrastructure that loses the tags. + +## Per-check sharpening + +- **`prompt.oversized`.** Use a rough token estimator (chars/4) or + `cl100k_base` if available. Never claim an exact token count without + naming the encoder you used. +- **`tools.duplicate`.** Compare tool *descriptions* (the + `[Description("...")]` attribute), not method names. Two tools with + the same method name but different descriptions are usually fine; + two tools with different names and a copy-pasted description are + usually a refactor candidate. + +## Routing offer (step 6) + +- **Inferring intent from the original prompt.** Even if the user + said "audit and fix it", this skill's job ends at the routing + prompt. The follow-up skill is responsible for its own + diff-and-confirm flow. Never call into `configure-agentic-perf-rules` + (apply mode) without the user explicitly + picking that letter at the prompt. +- **Listing routes for skills not actually referenced.** Step 6 says + to render only letters whose target skill is named in some + finding's `ref:` field. If no finding routed to `setup-maf-evals`, + do not offer it. Empty routing offer = skip the prompt entirely. + +## Inventory edge cases + +- **Multiple AppHost projects.** If a solution has more than one + `*.AppHost.csproj`, scan each one and emit one report per host as + `.copilot/perf-reports/scan-.md` (e.g. `scan-WebApp.md`, + `scan-Worker.md`). Do NOT merge — the topology, model set, and + OTel wiring belong to each host independently. Each per-host file + is still overwritten on each scan; no timestamp suffixes. +- **Agent registered via DI lambda without `AddAIAgent`.** Patterns + like `services.AddSingleton(...)` or custom factories also + count as agents. The detection in step 1 must include any path + that ends up registering an `IAgent` / `AIAgent` in the host's + service provider. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/message-history-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/message-history-checks.md index 0d858379cc..db98a2b686 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/message-history-checks.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/message-history-checks.md @@ -4,30 +4,30 @@ Detect strategies that pass too much history to too many agents. ## Checks -### MH1. Full chat history shared with every agent (critical) +### `history.full-share` (critical) -**Detect:** code that passes the entire `IList` (or `History`) -from the entry agent to a downstream agent without filtering, slicing, or -summarizing. +**Detect:** code that passes the entire `IList` (or +`History`) from the entry agent to a downstream agent without +filtering, slicing, or summarizing. -**Why:** every additional agent that sees the full history pays the input -token cost. With 4 agents and a 6K-token history, you spend 24K input -tokens per turn doing nothing. +**Why:** every additional agent that sees the full history pays the +input token cost. With 4 agents and a 6K-token history, you spend 24K +input tokens per turn doing nothing. -**Next:** "Pass only the last user message and a one-paragraph summary to -. Use `IChatHistoryReducer` or a manual slice." +**Next:** "Pass only the last user message and a one-paragraph summary +to ``. Use `IChatHistoryReducer` or a manual slice." -### MH2. No history cap (warn) +### `history.unbounded` (warn) -**Detect:** no usage of `MaxMessages`, `IChatHistoryReducer`, summarization -tool, or sliding-window code anywhere in agent setup. +**Detect:** no usage of `MaxMessages`, `IChatHistoryReducer`, +summarization tool, or sliding-window code anywhere in agent setup. **Why:** unbounded history = monotonically growing per-turn cost. **Next:** "Wire a `ChatHistoryReducer` with `MaxMessages = 20` or summarize-and-replace at the agent level." -### MH3. History passed through deterministic agents (warn) +### `history.through-deterministic` (warn) **Detect:** an agent whose role is purely deterministic (formatter, validator, tool router) is given full chat history. @@ -35,5 +35,5 @@ validator, tool router) is given full chat history. **Why:** deterministic steps do not need conversational context. Their prompt cost should be near-constant. -**Next:** "Pass only the immediate input artifact to ; drop the -chat history." +**Next:** "Pass only the immediate input artifact to ``; drop +the chat history." diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/model-assignment-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/model-assignment-checks.md index 5b755cac64..dbcdfe5577 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/model-assignment-checks.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/model-assignment-checks.md @@ -4,23 +4,26 @@ Detect single-model defaulting and role-model mismatch. ## Checks -### MA1. All agents on the same model (warn) +### `model.same-default` (warn) **Detect:** every agent is constructed with the same model id (e.g. `gpt-4o-mini`). Look at the AppHost connection string or the -`IChatClient` builder per agent service. +`IChatClient` builder per agent service. Requires ≥2 agents — skip on +single-agent apps. **Why:** different agent roles have different latency and quality needs. A single default usually overspends on cheap roles and underspends on hard roles. -**Next:** "Run `select-agent-models` to get a per-role recommendation. -Common improvement: use a small/fast model for the router and a -reasoning-strong model only for the planner." +**Next:** "See rule #3 in `.github/copilot-instructions.md` (managed by +`configure-agentic-perf-rules`) — most apps want a small-fast model +for routers/validators/workers and a reasoning-class model only for +planners. If both agents are workers on the same cheap model, this is +expected and can be downgraded to info." -**Ref:** `skill:select-agent-models` +**Ref:** `skill:configure-agentic-perf-rules` -### MA2. Reasoning-strong model on a deterministic agent (warn) +### `model.reasoning-on-deterministic` (warn) **Detect:** an agent whose prompt and tool list indicate a deterministic role (formatter, validator, classifier with ≤3 outputs) @@ -29,12 +32,13 @@ is using a frontier reasoning model. **Why:** the marginal quality is near-zero; you are paying for unused capability and per-call latency. -**Next:** "Downgrade to a small model. Validate via -`setup-maf-evals` quality mode." +**Next:** "Downgrade `` to a small-fast model per rule #3 in +`.github/copilot-instructions.md`. Validate via `setup-maf-evals` +quality mode." -**Ref:** `skill:select-agent-models` +**Ref:** `skill:configure-agentic-perf-rules` -### MA3. Cheap model on a planner / decomposer (warn) +### `model.cheap-on-planner` (warn) **Detect:** the agent that decides the plan or decomposes the task is on a small model while leaf workers are on a large one. @@ -42,18 +46,21 @@ on a small model while leaf workers are on a large one. **Why:** plan-quality drives every downstream call. A bad plan from a cheap planner makes the expensive workers run more turns. -**Next:** "Promote the planner to a reasoning-strong model; consider -demoting one or more workers." +**Next:** "Promote the planner to a reasoning-class model per rule #3 +in `.github/copilot-instructions.md`; consider demoting one or more +workers." -**Ref:** `skill:select-agent-models` +**Ref:** `skill:configure-agentic-perf-rules` -### MA4. Hard-coded model id outside config (info) +### `model.hardcoded` (info) **Detect:** model id literal (e.g. `"gpt-4o-mini"`) appears inside an agent service `.cs` file rather than `appsettings.json` or AppHost -parameters. +parameters. AppHost code that declares model ids via +`foundry.AddDeployment(...)` is the canonical Aspire pattern and does +NOT trigger this check. **Why:** swapping models for an A/B becomes a code change. **Next:** "Move model ids into `appsettings.json` and bind them via -`IOptions<...>`." +`IOptions<...>`, or declare them in the AppHost as Aspire deployments." diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/otel-coverage-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/otel-coverage-checks.md index e4278212b7..d25687e6e8 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/otel-coverage-checks.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/otel-coverage-checks.md @@ -4,20 +4,20 @@ Detect missing instrumentation that makes perf invisible at dev time. ## Checks -### O1. No `AddOpenTelemetry` call (critical) +### `otel.missing-sdk` (critical) **Detect:** the AppHost or service projects do not call `builder.Services.AddOpenTelemetry()` and do not import `OpenTelemetry.Exporter.OpenTelemetryProtocol` or `Aspire.Hosting.Dashboard`. -**Why:** without OTel, you cannot see per-call latency, token counts, or -spans. Every other check in this audit becomes a guess. +**Why:** without OTel, you cannot see per-call latency, token counts, +or spans. Every other check in this audit becomes a guess. **Next:** "Add `builder.AddServiceDefaults()` (Aspire) or wire OTel manually with HTTP + Activity sources for `Microsoft.Extensions.AI`." -### O2. No Aspire dashboard reference (warn) +### `otel.no-aspire-dashboard` (warn) **Detect:** the AppHost does not declare the dashboard, or the `appsettings.json` lacks a `Dashboard:OtlpEndpointUrl`. @@ -28,7 +28,7 @@ during local dev. **Next:** "Run with `dotnet run --project ` and ensure the dashboard URL is logged. If not, install `Aspire.Hosting.Dashboard`." -### O3. Token / cost surfacing missing (warn) +### `otel.no-token-cost` (warn) **Detect:** no log, no meter, no tag for `gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens` anywhere in the codebase. @@ -43,7 +43,7 @@ automatically. Confirm the OTel exporter forwards them, or run **Ref:** `skill:setup-maf-evals` -### O4. Per-agent activity source missing (info) +### `otel.no-per-agent-source` (info) **Detect:** all agents share a single activity source name; no way to filter the dashboard by agent. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/parallelism-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/parallelism-checks.md index ab41ca8059..e27f78e3e9 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/parallelism-checks.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/parallelism-checks.md @@ -1,48 +1,37 @@ # Parallelism checks -Detect sequential awaits that could run concurrently. +Detect sequential agent invocations that could run concurrently. ## Checks -### P1. Sequential awaits over independent inputs (warn) - -**Detect:** a `foreach` / `for` loop that awaits an LLM call or tool call -each iteration where the iteration values are independent of each other. - -**Pattern:** - -```csharp -foreach (var item in items) -{ - results.Add(await agent.RunAsync(item)); // sequential -} -``` - -**Why:** N items × per-call latency. With N=5 and 2s/call, that's 10s -that could be 2s under `Task.WhenAll`. - -**Next:** "Replace the loop with `await Task.WhenAll(items.Select(i => -agent.RunAsync(i)))`. Watch for shared mutable state inside the agent." - -### P2. Sequential agent handoffs that don't share context (warn) +### `parallel.independent-handoffs` (warn) **Detect:** two consecutive `await downstreamA.RunAsync(...)` and `await downstreamB.RunAsync(...)` calls in the same method where B's input does not depend on A's output. **Why:** the second call could start as soon as the inputs are known. +Each sequential LLM hop adds full per-call latency. -**Next:** "Run and with `Task.WhenAll`. Rejoin in the parent -agent for the consolidation step." +**Next:** "Run `` and `` with `Task.WhenAll`. Rejoin in the +parent agent for the consolidation step." -### P3. Tool fan-out behind a single tool wrapper (info) +### `parallel.hidden-tool-fanout` (info) **Detect:** a tool method that internally loops and calls 3+ external APIs sequentially. -**Why:** tools hide their own latency from the agent. A single slow tool -that is internally serial is the hardest kind of latency to find from -the outside. +**Why:** tools hide their own latency from the agent. A single slow +tool that is internally serial is the hardest kind of latency to find +from the outside. + +**Next:** "Parallelize the inner calls in ``; document the +expected bound in the tool description so the agent can plan around +it." + +## What used to live here -**Next:** "Parallelize the inner calls in ; document the -expected bound in the tool description so the agent can plan around it." +`parallel.sequential-awaits` (was `P1`, a generic `foreach (var x in +xs) await ...` pattern) was removed — that's a general .NET concurrency +anti-pattern, not specific to agents. For that class of finding run +`optimizing-dotnet-performance` instead. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/prompt-weight-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/prompt-weight-checks.md index e7a086ef4e..7e76736b0c 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/prompt-weight-checks.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/prompt-weight-checks.md @@ -4,40 +4,37 @@ Detect oversized system prompts and per-agent prompt cost. ## Checks -### PW1. System prompt > 2K tokens (warn) / > 4K tokens (critical) +### `prompt.oversized` (warn at >2K tokens / critical at >4K) -**Detect:** count tokens (or chars / 4 as approximation) in each agent's -`Instructions` or system prompt string. +**Detect:** count tokens (or chars / 4 as approximation) in each +agent's `Instructions` or system prompt string. -**Why:** every turn pays this cost. A 4K-token system prompt at $0.005/1K -input tokens × 1000 turns/day = $20/day per agent on prompt overhead alone. +**Why:** every turn pays this cost. A 4K-token system prompt at +$0.005/1K input tokens × 1000 turns/day = $20/day per agent on prompt +overhead alone. -**Next:** "Move static rules into a tool the agent can call when needed, -or split the prompt into a short policy section and a separate few-shot -example doc retrieved on demand." +**Next:** "Move static rules into a tool the agent can call when +needed, or split the prompt into a short policy section and a separate +few-shot example doc retrieved on demand." -### PW2. Few-shot examples in prompt > 3 (warn) - -**Detect:** count "Example:", "User:", "Assistant:" turn pairs in the -prompt string. - -**Why:** few-shot examples scale linearly with token cost. After 3 -examples the marginal accuracy gain is usually < 1%. - -**Next:** "Keep the 2 strongest examples; move the rest behind a -`getExample(category)` tool." - -### PW3. Identical preamble duplicated across agents (warn) +### `prompt.duplicate-preamble` (warn) **Detect:** two or more agents share the same > 100-token block at the start or end of their system prompts. **Why:** the same tokens are billed N times per turn (once per agent). -**Next:** "Lift the shared block into a single deterministic preprocessor -or attach it as a tool result rather than a system prompt repeat." +**Next:** "Lift the shared block into a single deterministic +preprocessor or attach it as a tool result rather than a system prompt +repeat." ## Token estimation If a real tokenizer is not available, approximate as `chars / 4`. Mark findings using approximation as `(estimated)` in the evidence. + +## What used to live here + +`prompt.too-many-fewshots` (was `PW2`, fired at >3 few-shot examples) +was a taste-based threshold. Few-shot count alone is not a reliable +signal; `prompt.oversized` already captures the cost dimension. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/report-template.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/report-template.md index dff808e238..818ceabdc7 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/report-template.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/report-template.md @@ -1,10 +1,11 @@ # Report template -The exact Markdown layout written to -`.copilot/perf-reports/scan-.md` and `latest-scan.md`. +The exact Markdown layout written to `.copilot/perf-reports/scan.md`. +This file is **overwritten on every run**. Git provides history; the +skill does not maintain timestamped copies or `latest-*` mirrors. ```markdown -# Agentic perf audit — {{ project_name }} +# Agentic perf scan — {{ project_name }} Run: {{ utc_timestamp }} Project: {{ relative_project_path }} @@ -26,9 +27,15 @@ Project: {{ relative_project_path }} ## Findings -> **Check ID prefixes:** `T*` topology · `TI*` tool inventory · `MH*` message history · `PW*` prompt weight · `P*` parallelism · `O*` OTel · `MA*` model assignment. +> **Slug format:** `.` — categories are +> `topology` (graph shape) · `tools` (per-agent tool list) · +> `history` (chat history strategy) · `prompt` (system prompt +> size/reuse) · `parallel` (concurrent invocations) · `otel` +> (instrumentation) · `model` (per-agent model id). Severities: +> `critical` (likely to break a flow or blow the budget) · +> `warn` (measurable cost/perf regression) · `info` (observation). -### [critical] [{{ check_id }}] {{ title }} +### [critical] [`{{ check }}`] {{ title }} - **File:** `{{ file }}:{{ line }}` - **Evidence:** ```csharp @@ -38,11 +45,11 @@ Project: {{ relative_project_path }} - **Next:** {{ action }} - **Cross-ref:** {{ skill: ... | omit if none }} -(... repeat per finding, ordered: critical → warn → info, then by `check_id` ...) +(... repeat per finding, ordered: critical → warn → info, then by `check` slug ...) ## Next steps -- If you want to fix the model assignments above, run `select-agent-models`. +- If you want to fix the model assignments above, see rule #3 in `.github/copilot-instructions.md` (managed by `configure-agentic-perf-rules`). - If you want to capture token/quality numbers before vs after, run `setup-maf-evals`. - If you do not yet have always-on rules to prevent regressions, run @@ -51,5 +58,6 @@ Project: {{ relative_project_path }} ## Empty-report contract -If there are zero findings, the `## Findings` section still appears with the -literal text `_No findings._`. The `## Summary` section shows zeros. +If there are zero findings, the `## Findings` section still appears +(after the legend blockquote) with the literal text `_No findings._`. +The `## Summary` section shows zeros. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/tool-inventory-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/tool-inventory-checks.md index c94962f9fb..180c4abeee 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/tool-inventory-checks.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/tool-inventory-checks.md @@ -4,46 +4,43 @@ Detect bloat and redundancy in the per-agent tool list. ## Checks -### TI1. Tools per agent > 8 (warn) / > 15 (critical) - -**Detect:** count tools registered on each agent (`AIFunctionFactory.Create`, -`[Description]`-attributed methods passed to `tools:`, MCP tool imports). - -**Why:** tool descriptions are sent in the system prompt every turn. 15 -tools at ~80 tokens each is 1.2K tokens of overhead before the user message. - -**Next:** "Split into two agents by domain, or move rarely-used -tools behind a single 'lookup' tool that takes a category argument." - -### TI2. Duplicate tool functionality across agents (warn) +### `tools.duplicate` (warn) **Detect:** two or more tools across different agents with the same description or near-identical signatures. -**Why:** duplication forces the router LLM to disambiguate every turn and -inflates aggregate prompt size. +**Why:** duplication forces the router LLM to disambiguate every turn +and inflates aggregate prompt size. + +**Next:** "Consolidate `` and `` into a single shared +tool exposed by both agents." + +### `tools.dead` (info) -**Next:** "Consolidate and into a single shared tool -exposed by both agents." +**Detect:** tools registered but never invoked in any code path or +call trace. Static check: tool name does not appear in any agent's +system prompt or instructions. -### TI3. Dead tools (info) +**Why:** every registered tool costs prompt tokens whether it gets +called or not. -**Detect:** tools registered but never invoked in any code path or call -trace. Static check: tool name does not appear in any agent's system -prompt or instructions. +**Next:** "Remove `` from ``'s tool list." -**Why:** every registered tool costs prompt tokens whether it gets called -or not. +### `tools.description-too-long` (warn) -**Next:** "Remove from 's tool list." +**Detect:** a tool's `[Description]` attribute or `description:` field +is longer than 200 characters. Measure after string concatenation — +multi-line `+` concatenations count as one description. -### TI4. Tool description > 200 chars (warn) +**Why:** long descriptions multiply across agents that import the +tool. Most tools can be described in one sentence. -**Detect:** a tool's `[Description]` attribute or `description:` field is -longer than 200 characters. +**Next:** "Trim ``'s description from `` chars to ≤ 200; move +the detailed contract into XML docs on the parameters." -**Why:** long descriptions multiply across agents that import the tool. -Most tools can be described in one sentence. +## What used to live here -**Next:** "Trim 's description from chars to ≤ 200; move the -detailed contract into XML docs on the parameters." +`tools.too-many-per-agent` (was `TI1`) was a taste-based threshold +(>8 tools per agent) that fired on legitimate designs. The real +question — "is this tool earning its prompt-token weight" — is better +served by `tools.dead` and `tools.duplicate`. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/topology-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/topology-checks.md index d1ca960dbb..7f07e4e5fb 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/topology-checks.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/topology-checks.md @@ -4,56 +4,37 @@ Detect structural issues in the agent graph that drive latency or runaway loops. ## Checks -### T1. Agent count > 3 (warn) / > 6 (critical) - -**Detect:** count distinct `ChatClientAgent` / `AddAgent(...)` registrations -in the AppHost and agent service projects. - -**Why:** more agents = more LLM hops per turn. Every additional agent that -can be routed to costs at least one extra round trip. - -**Next:** "Collapse into a single agent with two tool calls instead -of two agents." - -**Ref:** `skill:configure-agentic-perf-rules` if the project has no rules -file installed yet. - -### T2. LLM-routed handoff edges per turn > 2 (warn) / > 4 (critical) - -**Detect:** edges where the *destination* agent is selected by an LLM (not -deterministic code). Look for `Handoff` builders, `RoutingAgent`, or -`switch`/`if` blocks that select an agent based on a string returned by a -chat completion. - -**Why:** LLM-routed edges multiply tail latency. Two LLM hops to pick the -next agent before the work even starts is the most common cause of "why -is my agent so slow". - -**Next:** "Replace the LLM router between and with a deterministic -intent classifier or a tool call on the source agent." - -### T3. Cycles in the agent graph (critical) +### `topology.cycle` (critical) **Detect:** any directed cycle in the static handoff graph. **Why:** cycles risk infinite loops if the loop-break condition is LLM-judged. Even with a turn cap, a cycle burns budget on retries. -**Next:** "Break the cycle → → by making 's exit condition -deterministic." +**Next:** "Break the cycle `` → `` → `` by making ``'s exit +condition deterministic." -### T4. Single-leaf graph with > 2 hops (warn) +### `topology.deep-single-leaf` (warn) **Detect:** graph that always ends at one agent but routes through 3+ agents to reach it. -**Why:** the intermediate hops are usually classification or routing that -could be one tool call. +**Why:** the intermediate hops are usually classification or routing +that could be one tool call. -**Next:** "Move the routing logic into a tool on the entry agent and call - directly." +**Next:** "Move the routing logic into a tool on the entry agent and +call `` directly." ## Out of scope here - Tool counts → see `tool-inventory-checks.md`. - Per-agent model selection → see `model-assignment-checks.md`. + +## What used to live here + +`topology.agent-count` (was `T1`) and `topology.handoff-fanout` (was +`T2`) were removed in v0.2 — both were taste-based thresholds (>3 +agents, >2 handoff edges) that fired on legitimate designs as often +as on real bloat. If you want broad architectural feedback, run +`configure-agentic-perf-rules` so rule #1 (single-agent default) and +rule #2 (handoff justification) can guide the design at scaffold time. diff --git a/plugins/dotnet-ai/skills/select-agent-models/SKILL.md b/plugins/dotnet-ai/skills/select-agent-models/SKILL.md deleted file mode 100644 index 4539836160..0000000000 --- a/plugins/dotnet-ai/skills/select-agent-models/SKILL.md +++ /dev/null @@ -1,180 +0,0 @@ ---- -name: select-agent-models -description: | - Recommend a per-agent model assignment for a .NET agentic application (Microsoft Agent Framework + Aspire + Foundry). Reads the existing topology, classifies each agent (router, planner, decomposer, worker, validator, formatter, summarizer), then maps each role to a model from a curated role-model matrix balancing latency, quality, and per-call cost. Two modes: read-only "recommend" (default, writes a plan to .copilot/perf-reports/model-plan-.md) and "apply" (diff-preview-and-confirm, edits AppHost connection strings and per-agent IChatClient registrations). Never applies without explicit confirmation. WHEN: user asks "which model should each agent use", "audit my model selection", "everyone defaults to gpt-4o-mini", or has just received an MA finding from scan-agentic-app-perf. NOT-WHEN: user is comparing providers, tuning prompts, or has only one agent (run scan-agentic-app-perf first). ---- - -# select-agent-models - -Recommend per-agent model assignments based on each agent's role. - -## Workflow - -### 1. Inventory agents and current models - -For each agent in the project, record: - -- Agent name -- A short role guess from instructions / tool list / handoff position -- Current model id (from AppHost connection string or per-agent - `IChatClient` builder) -- Estimated per-turn input tokens (system prompt + history + tool descs) - -If fewer than 2 agents are detected, abort. Single-agent apps do not benefit -from this skill. - -### 2. Classify each agent's role - -Use `references/role-model-matrix.md`. Roles are: - -- **router** — picks the next agent or tool. Short prompt, deterministic - behaviour preferred. -- **planner** — decomposes the task. Reasoning-strong; output drives - every downstream call. -- **decomposer** — splits work into N parallel items. Reasoning-medium. -- **worker** — does the unit-of-work the planner described. Medium quality; - most calls happen here so latency dominates. -- **validator** — yes/no/score on a small input. Deterministic, small - output. -- **formatter** — renders structured output (JSON, Markdown). Deterministic, - small output. -- **summarizer** — compresses chat history. Medium reasoning, often run - hot. - -If an agent does not cleanly map to one role, mark its role as -`unclear` and recommend that the user review it. - -### 3. Look up recommended model per role - -`references/role-model-matrix.md` contains the canonical recommendation -table. The matrix has columns: - -- Role -- Recommended model (primary) -- Acceptable alternatives -- Avoid -- Rationale (latency / quality / cost trade) - -### 4. Build the plan - -For each agent, produce a row: - -```yaml -agent: -current_model: -role: -recommended_model: -delta: same | upgrade | downgrade -rationale: -``` - -Aggregate notes: - -- Net cost change estimate (qualitative: ↓ / ↔ / ↑) -- Net latency change estimate (qualitative: ↓ / ↔ / ↑) -- Risks to validate (e.g. "downgrading requires a quality eval first") - -### 5. Write the recommendation file - -Write to: - -- `.copilot/perf-reports/model-plan-.md` -- `.copilot/perf-reports/latest-model-plan.md` - -Layout in `references/plan-template.md`. - -Surface in chat: per-agent row plus the aggregate notes plus the file path. - -### 6. Apply mode (only if user explicitly asks "apply" / "make the -changes") - -Apply mode is **off by default**. To run it, the user must say "apply", -"make the changes", "switch to recommended models", or similar. - -**Confirmation contract** — the initial apply request only enables -preview mode. The actual write requires a *second* user response after -the diff is shown. Wording in the initial request like "apply and -confirm", "yes do it", or "proceed" is treated as *intent*, not as -*confirmation* — the agent must still show the diff and ask. - -Steps: - -1. **Provider resolution.** Detect whether the project uses public - OpenAI ids, Azure OpenAI deployment aliases, Foundry deployments, - or another provider: - - Check the AppHost connection string type (`AddAzureOpenAI` vs - `AddOpenAI` vs others). - - Check `appsettings.json` for keys ending in `-deployment-name`, - `-deployment`, or values that look like custom names (not the - OpenAI public id pattern). - - If Azure is detected, refuse to write a public OpenAI id like - `gpt-4o-mini` directly. Map the recommendation to an existing - deployment alias by: - a. enumerating deployment aliases from `appsettings.json` / - AppHost parameters, and - b. asking the user to pick which alias maps to each role, or - c. recommending creating a new deployment if no suitable alias - exists (and stopping apply mode in that case). -2. **Pre-write validation.** For every file that would be modified, - verify it parses (JSON / C# compiles via `dotnet build` dry-run on - the AppHost). If any target file is unparseable, abort apply mode - before any write. -3. **Diff preview.** Show a unified diff of every change you intend to - make: - - AppHost connection-string updates / parameter values - - Per-agent `IChatClient` registrations - - `appsettings.json` model-id keys -4. **Confirm.** Ask the user to confirm. Only on `yes` / explicit - confirmation, proceed. -5. **Atomic write.** Write all changes. If any write fails midway, - restore *all* touched files from their pre-write content. Do not - leave the project in a partially-applied state. -6. **Build.** Run `dotnet build` on the touched projects. - - On success: record the apply timestamp and per-agent old → new - mapping in the plan file. - - On failure: revert all changes from step 5, surface the build - output, and report `apply: failed (build)`. Do not declare - success on a failing build. -7. **Recommend follow-up.** After a successful apply, recommend running - `setup-maf-evals` quality mode to validate no quality regression. - -If the user says no or anything other than yes, discard the diff and -leave files untouched. - -## Validation - -After read-only mode: - -- A new file exists at `.copilot/perf-reports/model-plan-.md`. -- `latest-model-plan.md` exists and matches the timestamped file. -- The plan lists every detected agent with a row. - -After apply mode: - -- The diff was shown and confirmed before any file write. -- All touched projects build (`dotnet build` exit 0). -- The plan file records the apply timestamp and which agents were modified. - -## Common pitfalls - -- **Applying without confirmation.** The default is recommend-only. Do not - edit files without an explicit user confirmation in apply mode. -- **Recommending a downgrade with no quality check.** Always pair a - downgrade recommendation with a `setup-maf-evals` follow-up. -- **Inventing roles.** If an agent's purpose is unclear, say so. Do not - guess; the user must classify it. -- **Hard-coding model ids in code.** When applying, prefer - `appsettings.json` over inline string literals so the next swap is a - config change. -- **Ignoring provider differences.** This skill targets model *selection* - within an already-chosen provider. If the user wants to compare - providers, surface that as a follow-up question, not a recommendation. - -## References - -- `references/role-model-matrix.md` — role → recommended model table. -- `references/apphost-multi-client-template.md` — wiring multiple - `IChatClient`s in the AppHost with distinct model ids. -- `references/agent-resolution-template.md` — per-agent service - registration patterns. -- `references/plan-template.md` — exact Markdown layout for the plan file. diff --git a/plugins/dotnet-ai/skills/select-agent-models/references/agent-resolution-template.md b/plugins/dotnet-ai/skills/select-agent-models/references/agent-resolution-template.md deleted file mode 100644 index 14ec5c561c..0000000000 --- a/plugins/dotnet-ai/skills/select-agent-models/references/agent-resolution-template.md +++ /dev/null @@ -1,63 +0,0 @@ -# Per-agent service — resolving the right IChatClient - -How an individual agent service consumes the keyed or -configuration-bound `IChatClient` from the AppHost. - -## Pattern — typed `IOptions` - -```csharp -// agent service Program.cs -var builder = WebApplication.CreateBuilder(args); -builder.AddServiceDefaults(); - -builder.Services.Configure(builder.Configuration); - -builder.Services.AddSingleton(sp => -{ - var opts = sp.GetRequiredService>().Value; - return new ChatClient(model: opts.Model, apiKey: opts.ApiKey); -}); - -builder.Services.AddSingleton(sp => new ChatClientAgent( - chatClient: sp.GetRequiredService(), - instructions: SystemPrompts.Router)); -``` - -```csharp -public sealed class AgentModelOptions -{ - public string Model { get; set; } = ""; - public string ApiKey { get; set; } = ""; -} -``` - -`appsettings.json`: - -```json -{ - "Model": "gpt-4o-mini", - "ApiKey": "..." -} -``` - -## Pattern — keyed resolution (multiple clients in one process) - -```csharp -public sealed class WorkerService -{ - private readonly IChatClient _client; - public WorkerService([FromKeyedServices("worker")] IChatClient client) - { - _client = client; - } -} -``` - -## What apply mode does NOT do - -- Rewrite agent classes. -- Change agent instructions. -- Switch DI lifetimes (Singleton vs Scoped). -- Move from Pattern B to Pattern A or vice versa. - -It only updates the model id values themselves. diff --git a/plugins/dotnet-ai/skills/select-agent-models/references/apphost-multi-client-template.md b/plugins/dotnet-ai/skills/select-agent-models/references/apphost-multi-client-template.md deleted file mode 100644 index f9fc90b46b..0000000000 --- a/plugins/dotnet-ai/skills/select-agent-models/references/apphost-multi-client-template.md +++ /dev/null @@ -1,74 +0,0 @@ -# AppHost — multi-client wiring template - -Show how to register multiple `IChatClient`s in the AppHost, each tied -to a distinct model deployment, so per-agent services can resolve the -client matching their role. - -## Pattern A — Aspire AppHost with named connection strings - -```csharp -// AppHost/Program.cs -var builder = DistributedApplication.CreateBuilder(args); - -var openai = builder.AddConnectionString("openai"); - -var routerModel = builder.AddParameter("router-model", secret: false); // e.g. "gpt-4o-mini" -var plannerModel = builder.AddParameter("planner-model", secret: false); // e.g. "o4-mini" -var workerModel = builder.AddParameter("worker-model", secret: false); // e.g. "gpt-4o-mini" - -builder.AddProject("router") - .WithReference(openai) - .WithEnvironment("Model", routerModel); - -builder.AddProject("planner") - .WithReference(openai) - .WithEnvironment("Model", plannerModel); - -builder.AddProject("worker") - .WithReference(openai) - .WithEnvironment("Model", workerModel); - -builder.Build().Run(); -``` - -`appsettings.json` in AppHost: - -```json -{ - "Parameters": { - "router-model": "gpt-4o-mini", - "planner-model": "o4-mini", - "worker-model": "gpt-4o-mini" - } -} -``` - -## Pattern B — single service with multiple clients - -```csharp -// for monolith services that host multiple agents in-process -builder.Services.AddKeyedSingleton("router", (sp, _) => - new ChatClient(model: "gpt-4o-mini", apiKey: cfg["OpenAI:Key"])); - -builder.Services.AddKeyedSingleton("planner", (sp, _) => - new ChatClient(model: "o4-mini", apiKey: cfg["OpenAI:Key"])); - -builder.Services.AddKeyedSingleton("worker", (sp, _) => - new ChatClient(model: "gpt-4o-mini", apiKey: cfg["OpenAI:Key"])); -``` - -Then resolve with `[FromKeyedServices("router")] IChatClient router`. - -## What apply mode edits - -Apply mode of `select-agent-models` updates: - -1. The model parameter in AppHost (Pattern A) or the keyed registration - (Pattern B). -2. The matching `appsettings.json` value. -3. Nothing else. It does not change the agent class itself or the - instructions. - -If the project does not yet follow either pattern, the skill recommends -the migration in the plan file but does not perform it in apply mode — -that is a structural change beyond model selection. diff --git a/plugins/dotnet-ai/skills/select-agent-models/references/plan-template.md b/plugins/dotnet-ai/skills/select-agent-models/references/plan-template.md deleted file mode 100644 index fa63694638..0000000000 --- a/plugins/dotnet-ai/skills/select-agent-models/references/plan-template.md +++ /dev/null @@ -1,53 +0,0 @@ -# Plan template - -The exact Markdown layout written to -`.copilot/perf-reports/model-plan-.md` and -`latest-model-plan.md`. - -```markdown -# Model selection plan — {{ project_name }} - -Run: {{ utc_timestamp }} -Project: {{ relative_project_path }} -Mode: recommend | apply ({{ confirmed_at | "n/a" }}) - -## Per-agent recommendations - -| Agent | Role | Current model | Recommended model | Δ | Rationale | -|------------|------------|------------------|-------------------|------------|--------------------------------------------| -| router | router | gpt-4o | gpt-4o-mini | downgrade | One-shot classification; latency dominates | -| planner | planner | gpt-4o-mini | o4-mini | upgrade | Plan quality drives N downstream calls | -| worker | worker | gpt-4o | gpt-4o-mini | downgrade | Most calls; latency dominates | - -## Aggregate notes - -- **Cost:** ↓ (downgrades on router and worker outweigh planner upgrade) -- **Latency:** ↓ (router and worker shrink; planner runs once per turn) -- **Quality risk:** validate planner upgrade and worker downgrade with - `setup-maf-evals` quality mode before promoting. - -## Apply preview (only present in apply mode) - -Files to be modified: - -- `MyApp.AppHost/appsettings.json` -- `MyApp.AppHost/Program.cs` (parameter declarations only) - -Diff: - -```diff -- "worker-model": "gpt-4o", -+ "worker-model": "gpt-4o-mini", -``` - -## Next steps - -- Run `setup-maf-evals` quality mode against the new assignments. -- Re-run `scan-agentic-app-perf` after evals confirm parity. -- If quality regresses, revert the affected agent only via this skill. -``` - -## Empty-plan contract - -If the inventory finds < 2 agents, the skill aborts and does not write -a plan file. The chat output explains why. diff --git a/plugins/dotnet-ai/skills/select-agent-models/references/role-model-matrix.md b/plugins/dotnet-ai/skills/select-agent-models/references/role-model-matrix.md deleted file mode 100644 index 37ab14423f..0000000000 --- a/plugins/dotnet-ai/skills/select-agent-models/references/role-model-matrix.md +++ /dev/null @@ -1,95 +0,0 @@ -# Role → model matrix - -Recommendations are model-family-neutral where possible, with concrete -defaults for OpenAI/Foundry. Names below are illustrative; substitute the -deployment the user actually has access to. - -## Matrix - -| Role | Recommended (primary) | Acceptable alternatives | Avoid | Rationale (latency / quality / cost) | -|-------------|-----------------------|------------------------------------|--------------------------------|------------------------------------------------------------------------------------------------------| -| router | gpt-4o-mini | o4-mini, gpt-4.1-mini | frontier reasoning models | One-shot classification. Latency dominates; a small fast model is correct. Quality differential is negligible for ≤ 5-way routes. | -| planner | o4-mini (reasoning) | gpt-4o, o3-mini | gpt-3.5-turbo, gpt-4o-mini | Plan quality drives N downstream calls. Reasoning model pays back in fewer worker turns. | -| decomposer | o4-mini | gpt-4o, gpt-4.1 | small chat-only models | Similar to planner, but typically smaller output. Reasoning-medium is enough. | -| worker | gpt-4o-mini | gpt-4.1-mini, gpt-4o | frontier models for bulk work | Most calls happen here; latency dominates. Bumping every worker to a frontier model is the most common cost mistake. | -| validator | gpt-4o-mini | gpt-4.1-mini | reasoning models | Yes/no/score; small input, small output, deterministic. A small model with a tight rubric beats a large one with a fuzzy prompt. | -| formatter | gpt-4o-mini | gpt-4.1-mini | reasoning models | Structured-output transformation. Quality plateau is hit quickly. | -| summarizer | gpt-4o-mini | gpt-4.1-mini, gpt-4o | reasoning models for hot loops | Runs every turn (or near it). Latency and cost matter more than peak quality. | - -## Provider notes - -- **Public OpenAI:** model id is the canonical OpenAI name - (`gpt-4o-mini`, `o4-mini`, etc.). Apply mode writes the id directly. -- **Foundry / Azure OpenAI:** the model id stored in `appsettings.json` - is the **deployment alias** (e.g. `my-prod-mini`), not the OpenAI - public id. Apply mode must NOT write a public id like `gpt-4o-mini` - into an Azure project's `appsettings.json` — that will break at - runtime. Instead: - - Recommend the model *family* in the plan (e.g. "use a - small/fast model for the router"). - - In apply mode, ask the user which deployment alias maps to each - recommended role, or recommend creating a new deployment. - - If no suitable deployment exists, the plan's `delta` should be - `unmapped` and apply mode must abort for that agent. -- **Anthropic / Bedrock:** map "reasoning-strong" → Claude Sonnet, - "small/fast" → Claude Haiku. -- **Local / Ollama:** map "small/fast" → Llama 3.2 / Phi-3, "reasoning" - → Llama 3.3 70B or DeepSeek-R1; expect higher latency than hosted - reasoning models and re-eval quality. - -## Router sub-types - -The default `router → gpt-4o-mini` recommendation assumes a *simple -classifier*. Promote to a stronger model when **any** of these apply: - -- The router generates **tool-call arguments** (not just selects a - destination). -- The router performs **schema validation** or policy checks on user - input. -- The router chooses among **more than 5 destinations** with - overlapping descriptions. -- A misroute is **expensive** (e.g. routes to a long-running workflow). - -In those cases, recommend `gpt-4o` / `o4-mini` instead and note the -upgrade in the plan's rationale. - -## Planner: `o4-mini` vs `gpt-4o` - -Recommend a reasoning model (`o4-mini`, `o3-mini`) when: - -- The plan has multi-step dependencies between worker outputs. -- The user task is open-ended and the planner must choose what to do - before how. - -Recommend `gpt-4o` / `gpt-4.1` when: - -- Latency dominates (interactive UX with strict p95 budget). -- The plan output is mostly structured (JSON shape known in advance). -- Planning depth is shallow (≤ 3 steps). - -## Multi-role agents - -When an agent fits more than one role, classify by the **highest- -consequence output downstream agents consume**: - -| Combination | Classify as | -|-----------------------------------|------------------------------| -| router + shallow input validation | router | -| router + tool-arg generation | reasoning router (see above) | -| planner + output formatting | planner | -| validator + scoring | validator | -| worker + summarizer | worker | - -If the role is genuinely unclear after applying these rules, mark as -`unclear` in the plan and ask the user to classify before apply mode. - -## When to deviate - -- **Strict-latency interactive UX (≤ 1.5s p95):** override planner to - `gpt-4o-mini` and accept a small quality hit. Validate with evals. -- **High-stakes single-shot (e.g. legal summarization):** override - worker to a frontier model for the critical step only; keep the rest - on small models. -- **Strict cost budget (≤ $X / 1K turns):** start every role at - small/fast, then upgrade only the role that fails the eval-quality - bar. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md b/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md index 0ada6890fe..74d580f4e1 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md @@ -1,13 +1,15 @@ --- name: setup-maf-evals description: | - Scaffold a Microsoft.Extensions.AI.Evaluation project alongside an existing .NET agentic application (MAF + Aspire + Foundry) so the team can measure latency, token usage, cost, and answer quality on every change. Creates an .Evals project with three modes: telemetry (per-call latency / input-tokens / output-tokens / cost), quality (LLM-as-judge against a rubric and golden conversations), and compare (run two model assignments side by side and produce a delta). Outputs Markdown + JSON + JUnit-XML reports under .copilot/perf-reports/evals//. Optionally wires an Aspire-dashboard panel showing per-agent token/latency live during dev. WHEN: user asks "how do I measure my agent perf", "set up evals", "add evaluation harness", "I changed models and need to validate quality", "compare gpt-4o vs gpt-4o-mini for my planner". NOT-WHEN: user wants a one-shot audit (scan-agentic-app-perf), install rules (configure-agentic-perf-rules), or pick models (select-agent-models). + Scaffold an `.Evals.Tests` MSTest project alongside a .NET agentic app (MAF + Aspire + Foundry) wired to the GA `Microsoft.Extensions.AI.Evaluation.Reporting` pipeline. Three evaluator categories: **NLP** (deterministic BLEU/GLEU/F1, no API key), **Quality** (LLM-as-judge Relevance/Coherence/Fluency, etc.), **Safety** (Hate/Violence/SelfHarm/Sexual via Azure AI Foundry). Auto-installs the `aieval` dotnet tool, detects the app's `IChatClient` registration and generates a factory so `EVAL_USE_REAL_AGENT=1` works without manual wiring, and emits an HTML report at `.copilot/perf-reports/evals//report.html`. Optional GitHub Actions workflow runs the evals on every PR. WHEN: user asks "set up evals", "add evaluation harness", "measure my agent perf", "validate quality after a model change", "compare gpt-4o vs gpt-4o-mini", "add safety evaluators", "generate eval report". NOT-WHEN: one-shot audit (use scan-agentic-app-perf), install rules (configure-agentic-perf-rules). --- # setup-maf-evals -Scaffold a `.Evals` project that measures latency, token usage, -cost, and quality on every change to a .NET agentic app. +Scaffold an `.Evals.Tests` MSTest project that measures latency, +token usage, cost, quality, and safety on every change to a .NET +agentic app — and produces the proper Microsoft.Extensions.AI.Evaluation +HTML report by default. ## Workflow @@ -19,150 +21,228 @@ Detect: - AppHost project name (`*.AppHost.csproj`) - Agent service projects - Existing test / eval projects (avoid clobbering) +- **`IChatClient` registration** (scan AppHost + agent projects for + `AddChatClient` / `AddAzureOpenAIChatClient` / `AddOllamaChatClient` + / `AddOpenAIChatClient` and any explicit `services.AddSingleton` or + Foundry deployment alias references). See `references/ichatclient-detection.md`. -If no agentic app is detected, abort. If a `*.Evals` project already +If no agentic app is detected, abort. If a `*.Evals.Tests` project already exists, switch to update mode (see step 1a). -### 1a. Update mode (when `*.Evals` project already exists) +### 1a. Update mode (when `*.Evals.Tests` project already exists) File classes: | Class | Files | Behavior on update | |------------------|------------------------------------------------------------------------|----------------------------| -| **infra** | `*.Evals.csproj`, `Program.cs`, runner classes, `Abstractions.cs` | merge package refs; create file if missing; do **not** overwrite | -| **user data** | `Quality/rubric.md`, `Quality/golden.json`, `Compare/matrix.json`, `Telemetry/inputs.json`, `Telemetry/prices.json`, `quality.thresholds.json` | never overwrite; create if missing | -| **generated** | `Reports/`, `.copilot/perf-reports/evals/` | regenerate freely | +| **infra** | `*.Evals.Tests.csproj`, `dotnet-tools.json`, `Reporting/*`, `Wire/*`, test class skeletons | merge package refs; create file if missing; do **not** overwrite | +| **user data** | `Quality/rubric.md`, `Quality/golden.json`, `Compare/matrix.json`, `Telemetry/inputs.json`, `Telemetry/prices.json`, `quality.thresholds.json`, `.github/workflows/evals.yml` | never overwrite; create if missing | +| **generated** | `.copilot/perf-reports/evals/` | regenerate freely | If an existing infra file differs from the current template, surface the diff in the chat output but do not overwrite. Recommend the user review and merge manually. -### 2. Confirm scope with the user - -Ask which modes to wire (default: all three): - -- **telemetry** — capture latency, input-tokens, output-tokens, cost - per agent call across a fixed input set. -- **quality** — LLM-as-judge against a rubric and golden conversations. -- **compare** — run mode A vs mode B and emit a side-by-side delta. +`golden.json` has a `schema_version` field. If the detected version is +older than the current template, offer to migrate (additive only — +preserves existing rows, adds the new fields as nullable). -Optional: +### 2. Confirm scope with the user -- **aspire-panel** — add a static-file-based dashboard panel showing - per-agent token/latency live during `dotnet run`. +Present the detection summary, then confirm: + +1. **Project shape** (default: MSTest). Alternative: console runner + (legacy v1 shape) — only emit if user explicitly asks for it. +2. **Evaluator categories to enable.** Defaults shown; user can override. + + | Category | Evaluators | Cost | Needs | Default | + |------|-----------|------|-------|---------| + | **Quality** *(headline)* | Relevance, Coherence, Fluency, Completeness, Equivalence, Groundedness; agent: IntentResolution, TaskAdherence, ToolCallAccuracy | per-call judge tokens | real `IChatClient` + `EVAL_USE_REAL_JUDGE=1` | **ON** (stubbed until judge wired) | + | NLP (zero-config sanity) | BLEU, GLEU, F1, Words | free | reference responses in golden.json | ON | + | Safety | `ContentHarmEvaluator` (Hate+SelfHarm+Violence+Sexual single-shot), ProtectedMaterial, IndirectAttack, CodeVulnerability, UngroundedAttributes, GroundednessPro | Foundry evaluation service charges | Azure AI Foundry endpoint + `EVAL_USE_FOUNDRY_SAFETY=1` | **OFF** — prompt the user: "Wire safety evaluators too? (y/N)" | + + Frame Quality as the headline evaluation; NLP is the zero-config + first-run experience that emits a real `report.html` before any + creds exist; Safety is the opt-in for production-bound apps. + +3. **IChatClient detection result.** Show what was detected (e.g., "Found + `AddAzureOpenAIChatClient` in `AppHost.cs:41` with deployment alias `chat`"). + Ask the user to confirm or override. If detection failed, generate a + stub factory the user will fill in. +4. **Run modes to scaffold.** Telemetry (default ON) and Quality (default + ON). **Compare mode is opt-in** — prompt the user: "Wire compare + mode for side-by-side matrix.json runs? (y/N)". Compare adds the + largest scaffold (extra runner + matrix.json + delta-table generator) + and is the least commonly used surface. +5. **Optional add-ons:** Aspire dashboard panel, GitHub Actions workflow. ### 3. Scaffold the project -Use `references/project-template.md`. Creates: - -``` -.Evals/ - .Evals.csproj # Microsoft.Extensions.AI.Evaluation refs - Telemetry/ - TelemetryEvalRunner.cs - inputs.json # 5 starter inputs the user customizes - Quality/ - QualityEvalRunner.cs - rubric.md # the LLM-judge rubric - golden.json # golden conversations - Compare/ - CompareEvalRunner.cs - matrix.json # model assignments to compare - Reports/ # generated, gitignored - Program.cs # CLI: dotnet run -- - Directory.Build.props # version pins +See `references/project-template.md` for the file tree, csproj, and +`GlobalUsings.cs` template. The project is **MSTest** by default +(`.Evals.Tests`); a console-runner shape is available behind an +explicit `--shape console` flag. + +Always emit: `Reporting/{ReportingConfig.cs, Tier.cs, AievalReport.cs, +WordCountEvaluator.cs, MetricsGlossary.cs}`, `Wire/{AgentChatClientFactory.cs, +StubChatClient.cs}`, `Quality/{QualityTests.cs, rubric.md, golden.json}`, +`Telemetry/{TelemetryTests.cs, inputs.json, prices.json}`, +`quality.thresholds.json`, `GlobalUsings.cs`, `dotnet-tools.json`. +Emit `Compare/{CompareTests.cs, matrix.json}` only if the user opted into +compare mode (step 2 #4). Emit `Safety/SafetyTests.cs` and +`.github/workflows/evals.yml` only if the user opted in (steps 7 and 9). + +After writing files: + +```pwsh +dotnet sln add .Evals.Tests/.Evals.Tests.csproj +dotnet tool restore # installs aieval +# .gitignore additions +echo ".copilot/perf-reports/evals/`n.Evals.Tests/_store/" >> .gitignore ``` -Add the project to the solution. Add `Reports/` and -`.copilot/perf-reports/evals/` to the repo `.gitignore`. - ### 4. Wire telemetry mode -See `references/telemetry-capture.md`. - -- Hooks into the existing `IChatClient` via a delegating wrapper that - records `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, - per-call latency, and a price-table-driven cost estimate. -- Emits a Markdown report at - `.copilot/perf-reports/evals//telemetry.md`, - a machine-readable `telemetry.json`, and a `telemetry.junit.xml`. +See `references/telemetry-capture.md`. Default ON. Captures latency, +input/output tokens, and price-table-driven cost via a delegating +`IChatClient`. Writes `telemetry.{md,json,junit.xml}` next to +`report.html`. **Not** the MEAI HTML report — that's quality mode's job. ### 5. Wire quality mode -See `references/quality-modes.md`. - -- LLM-judge configurable model id (default: `gpt-4o`). -- Rubric is a Markdown file the user edits. -- Golden conversations: array of `{ input, expected_traits[] }`. -- Emits per-input score, aggregate pass rate, top failures with - judge rationale. - -### 6. Wire compare mode - -See `references/compare-mode.md`. - -- Reads `matrix.json`: a list of `{ name, model_assignments }` entries. -- Runs telemetry + quality for each. -- Produces `compare.md` with side-by-side latency / token / cost / - quality columns and a recommendation row. - -### 7. Optional Aspire panel (apply mode) - -See `references/aspire-dashboard-panel.md`. - -This step is **off by default** and modifies the AppHost project. To -enable it, the user must say "wire the panel" / "add the dashboard -panel" / equivalent. - -When enabled: - -1. Show a unified diff of the AppHost edit (`app.UseStaticFiles()` and - the panel files under `wwwroot/eval-panel/`). -2. Ask for confirmation. -3. Only on `yes` / explicit confirmation, write the changes. -4. Run `dotnet build` and report pass/fail. - -If declined, scaffold the static files into `.Evals/Panel/` so the -user can move them into the AppHost manually. - -### 8. Validation - -- `dotnet build .Evals.csproj` exits 0. -- `dotnet run --project .Evals -- telemetry` runs against a - smoke input and writes a Reports/ file (uses a stub client if no - API key is configured; emits `(stub)` in the report). -- All three runner classes have unit-level smoke tests under - `.Evals.Tests/`. - -### 9. Surface in chat - -- The path to the new project. -- The CLI invocations: `dotnet run -- telemetry`, `-- quality`, - `-- compare`. -- The reports folder. -- Recommend a follow-up: "Re-run after applying a `select-agent-models` - recommendation to confirm no quality regression." +See `references/quality-modes.md`. Default ON. The **only** runner that +produces `report.html`. Uses `DiskBasedReportingConfiguration` + +`ScenarioRun.EvaluateAsync` + `[AssemblyCleanup]` invokes +`dotnet tool run aieval report`. Stub tier registers the 4 NLP +evaluators; judge tier (`EVAL_USE_REAL_JUDGE=1`) adds the LLM-as-judge +evaluators from `references/evaluators-catalog.md`. + +### 6. Wire compare mode (opt-in) + +See `references/compare-mode.md`. **Default OFF.** Only scaffold when +the user opted in at step 2 (#4). When enabled, reads `matrix.json`; +each entry runs through the **same** `ReportingConfiguration` with a +distinct `executionName`, so `aieval report` aggregates the comparison +columns into a single HTML view. Also writes a `compare.md` delta +table. + +### 7. Wire safety mode (opt-in) + +See `references/safety-mode.md`. **Default OFF.** When enabled, adds +`Microsoft.Extensions.AI.Evaluation.Safety` and emits `SafetyTests.cs` +with `ContentHarmEvaluator` (single-shot 4-metric bundle), plus +ProtectedMaterial / IndirectAttack / CodeVulnerability / +UngroundedAttributes. Skipped at runtime via `Assert.Inconclusive` +when `EVAL_USE_FOUNDRY_SAFETY` is unset — never fails the build for +missing creds. + +### 8. Optional Aspire panel (apply mode) + +See `references/aspire-dashboard-panel.md`. Default OFF; modifies +the AppHost project. To enable, user must say "wire the panel" / +equivalent. Always show a unified diff + ask for confirmation before +writing AppHost edits. + +### 9. Optional CI workflow (opt-in) + +See `references/ci-workflow.md`. Default OFF. Emits +`.github/workflows/evals.yml` that runs `dotnet test` on every PR, +auto-detects tier from repo secrets (`AZURE_OPENAI_ENDPOINT` → +judge; `AZURE_AI_FOUNDRY_ENDPOINT` → safety), and uploads +`report.html` as a workflow artifact. + +### 10. Validation + +- `dotnet build .Evals.Tests.csproj` exits 0. +- `dotnet test .Evals.Tests.csproj` exits 0 in stub tier (no creds needed). + - Stub tier must emit a `report.html` with **≥ 4 distinct metric columns** + (Words, BLEU, GLEU, F1) across all scenarios in golden.json. + - All scenarios must produce non-null metric values (no "—" placeholders). +- If `EVAL_USE_REAL_JUDGE=1` and an `IChatClient` is wired, + `dotnet test` must additionally produce ≥ 3 Quality metrics (Relevance, + Coherence, Fluency). + +### 11. Surface in chat + +Lead with **Quality** as the headline evaluation; frame NLP as the +zero-config sanity check and Safety/Compare as additions. + +1. **Quality (headline).** State whether the judge is wired: + - *Stubbed* — "Quality scaffolded; judge will run once you wire + `EVAL_USE_REAL_JUDGE=1` + a chat endpoint. The next block tells + you how." + - *Live* — "Quality judge active against ``; report + shows Relevance / Coherence / Fluency / Completeness / Equivalence." + - Caveat the user **must** know up-front: *"The built-in Quality + rubrics are generic. Agents with deliberate stylistic constraints + (brevity, persona, format adherence) will score low on + Completeness / Equivalence even when working as designed. See + `references/common-pitfalls.md#tuning-quality-for-stylistic-agents` + for the per-app override pattern."* +2. **Promoting Quality to the judge tier.** If the app's `IChatClient` + reads from a connection string (Aspire pattern), include the exact + two commands: + + ```pwsh + dotnet user-secrets init --project .Evals.Tests + dotnet user-secrets set "ConnectionStrings:" ` + "Endpoint=https://.services.ai.azure.com/models;DeploymentId=" ` + --project .Evals.Tests + # then: $env:EVAL_USE_REAL_AGENT="1"; $env:EVAL_USE_REAL_JUDGE="1"; dotnet test + ``` + + Note the **three** endpoint gotchas: (a) hostname strips dashes on + some resources, (b) key auth is often disabled → drop the `Key=` + segment and rely on `DefaultAzureCredential`, (c) **the judge + deployment must be a non-reasoning model** (gpt-4o / gpt-4o-mini / + gpt-4-turbo). Reasoning models (gpt-5*, o-series) reject `max_tokens` + with HTTP 400 and MEAI silently records that as a per-metric error + row. If the production model is a reasoning one, set + `EVAL_JUDGE_DEPLOYMENT_NAME=` so the judge + points at a different deployment than the agent. Full details in + `references/ichatclient-detection.md`. +3. **NLP (zero-config sanity).** "`report.html` already populates with + Words / BLEU / GLEU / F1 from `golden.json` without any creds — run + `dotnet test` now to see it." +4. **Additional categories you can wire.** List Safety + (`EVAL_USE_FOUNDRY_SAFETY=1`, opt-in scaffold) and Compare (re-run + the skill with `compare: true` if it wasn't opted in originally) + as one-line bullets — not in the main flow. +5. **Paths.** Project path, `report.html` path, **glossary path** + (`metrics-glossary.md` co-located with `report.html`), persistent + `_store/` path. +6. **CLI invocations.** `dotnet test`, `dotnet tool run aieval report`, + and the IChatClient detection result so the user knows what was + auto-wired. +7. **Follow-up recommendation.** "Re-run after swapping a model per + rule #3 of `configure-agentic-perf-rules` to confirm no quality + regression." +8. **Cache payoff.** "First `dotnet test` populates `_store/cache/` and + takes the full ~60s. Every subsequent run against unchanged inputs + reuses cached agent + judge responses — typically ~5s with zero LLM + cost. The Diagnostic Data section of `report.html` shows per-call + Hit/Miss. To force a fresh run, delete `_store/cache/` or change the + rubric / golden inputs." + +Also link `references/evaluators-catalog.md` and +`references/metrics-glossary.md` so the user can see what each metric +means. ## Common pitfalls -- **Calling real models from the smoke test.** The default smoke run - uses a stub `IChatClient`; the report is clearly marked `(stub)`. - Real-model runs are opt-in (env var `EVAL_USE_REAL_MODELS=1`). -- **Hard-coding a price table.** The price table lives in - `Telemetry/prices.json` and is user-editable. -- **Conflating telemetry and quality.** Telemetry never reads the - conversation content; quality never reads token counts. Keep them - in separate runners and reports. -- **Auto-failing the build on quality regressions.** Quality mode is - informational by default. The user explicitly opts into a hard-fail - threshold by editing `quality.thresholds.json`. -- **Forgetting the `.gitignore` entry.** Reports must not pollute - source history. +See `references/common-pitfalls.md`. ## References -- `references/project-template.md` — exact files and `.csproj` layout. -- `references/telemetry-capture.md` — per-call hook + report format. -- `references/quality-modes.md` — LLM-judge rubric + golden conv format. -- `references/compare-mode.md` — matrix.json layout + delta report. +- `references/project-template.md` — file tree + `.csproj` layout. +- `references/ichatclient-detection.md` — registration scan + factory emission. +- `references/evaluators-catalog.md` — NLP + Quality + Safety catalog with required `EvaluationContext` types. +- `references/metrics-glossary.md` — per-run glossary content + `MetricsGlossary.cs` template. +- `references/telemetry-capture.md` — per-call hook + cost report format. +- `references/quality-modes.md` — `DiskBasedReportingConfiguration` wiring + `aieval report` invocation. +- `references/compare-mode.md` — `matrix.json` + per-entry `executionName`. +- `references/safety-mode.md` — opt-in safety scaffold + `ContentHarmEvaluator` default. +- `references/ci-workflow.md` — `.github/workflows/evals.yml` template. - `references/aspire-dashboard-panel.md` — optional static-file panel. +- `references/common-pitfalls.md` — known footguns to avoid when scaffolding. +- [MEAI.Evaluation libraries](https://learn.microsoft.com/en-us/dotnet/ai/evaluation/libraries) | [Tutorial: evaluate with reporting](https://learn.microsoft.com/en-us/dotnet/ai/evaluation/evaluate-with-reporting) | [dotnet/ai-samples](https://github.com/dotnet/ai-samples/blob/main/src/microsoft-extensions-ai-evaluation/api/) diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/aspire-dashboard-panel.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/aspire-dashboard-panel.md index e899d633de..943e43a871 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/aspire-dashboard-panel.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/aspire-dashboard-panel.md @@ -56,9 +56,8 @@ setInterval(tick, 2000); tick(); - The panel reads only the latest `telemetry.json`; it does not retain history across runs. - If the AppHost project does not already enable static files, the - skill adds `app.UseStaticFiles()` (in apply mode only, with the - same diff-preview-and-confirm flow as `select-agent-models` apply - mode). + skill adds `app.UseStaticFiles()` (in apply mode only, with a + standard diff-preview-and-confirm flow). ## Future v2 diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/ci-workflow.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/ci-workflow.md new file mode 100644 index 0000000000..3c627a0e51 --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/ci-workflow.md @@ -0,0 +1,119 @@ +# CI workflow (opt-in) + +Off by default. Enabled in step 2 when the user picks "scaffold CI +workflow" / "add GitHub Actions" / equivalent. + +When enabled, emits `.github/workflows/evals.yml`. The workflow: + +- Runs on every PR + on push to `main`. +- Always runs Tier 1 (NLP) — no creds needed, fast smoke test. +- Promotes to Tier 2 (Quality, real judge) when the repo has secrets + `AZURE_OPENAI_ENDPOINT` + `AZURE_TENANT_ID`. +- Promotes to Tier 3 (Foundry Safety) when the repo has secret + `AZURE_AI_FOUNDRY_ENDPOINT`. +- Uploads `report.html` as a workflow artifact. + +## Template + +```yaml +name: evals + +on: + pull_request: + branches: [main] + push: + branches: [main] + workflow_dispatch: + +permissions: + id-token: write # for OIDC -> DefaultAzureCredential + contents: read + +jobs: + evaluate: + runs-on: ubuntu-latest + timeout-minutes: 20 + + env: + EVAL_REPORT_FOLDER: ${{ github.run_id }}-${{ github.run_attempt }} + + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-dotnet@v4 + with: + dotnet-version: 10.0.x + + - name: Restore tools + run: dotnet tool restore + + - name: Azure login (only if creds present) + if: env.AZURE_TENANT_ID != '' + uses: azure/login@v2 + with: + client-id: ${{ secrets.AZURE_CLIENT_ID }} + tenant-id: ${{ secrets.AZURE_TENANT_ID }} + subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }} + env: + AZURE_TENANT_ID: ${{ secrets.AZURE_TENANT_ID }} + + - name: Tier selection + id: tier + run: | + if [ -n "${{ secrets.AZURE_OPENAI_ENDPOINT }}" ]; then + echo "EVAL_USE_REAL_AGENT=1" >> $GITHUB_ENV + echo "EVAL_USE_REAL_JUDGE=1" >> $GITHUB_ENV + echo "AZURE_OPENAI_ENDPOINT=${{ secrets.AZURE_OPENAI_ENDPOINT }}" >> $GITHUB_ENV + echo "tier=judge" >> $GITHUB_OUTPUT + else + echo "tier=stub" >> $GITHUB_OUTPUT + fi + if [ -n "${{ secrets.AZURE_AI_FOUNDRY_ENDPOINT }}" ]; then + echo "EVAL_USE_FOUNDRY_SAFETY=1" >> $GITHUB_ENV + echo "AZURE_AI_FOUNDRY_ENDPOINT=${{ secrets.AZURE_AI_FOUNDRY_ENDPOINT }}" >> $GITHUB_ENV + echo "tier=safety" >> $GITHUB_OUTPUT + fi + + - name: Run evals (dotnet test) + run: dotnet test {{AppName}}.Evals.Tests/{{AppName}}.Evals.Tests.csproj --logger "trx;LogFileName=evals.trx" + + - name: Generate report (already invoked by [AssemblyCleanup], this is a safety net) + if: always() + run: | + mkdir -p .copilot/perf-reports/evals/${{ env.EVAL_REPORT_FOLDER }} + dotnet tool run aieval report \ + --path {{AppName}}.Evals.Tests/_store \ + --output .copilot/perf-reports/evals/${{ env.EVAL_REPORT_FOLDER }}/report.html + + - name: Upload report + if: always() + uses: actions/upload-artifact@v4 + with: + name: eval-report-${{ steps.tier.outputs.tier }} + path: .copilot/perf-reports/evals/${{ env.EVAL_REPORT_FOLDER }}/report.html + + - name: Upload trx + if: always() + uses: actions/upload-artifact@v4 + with: + name: trx + path: '**/evals.trx' +``` + +## Optional: PR comment with summary + +A second job can comment on the PR with a link to the artifact and a +one-line summary scraped from `compare.md`. Not in the default +template — too much variance across teams' bot/preference choices. + +## Required secrets + +| Secret | Tier unlocked | Required for OIDC? | +|--------|---------------|--------------------| +| `AZURE_TENANT_ID` | needed for any Azure auth | yes | +| `AZURE_CLIENT_ID` | OIDC federated identity | yes (if using `azure/login@v2`) | +| `AZURE_SUBSCRIPTION_ID` | scope | yes | +| `AZURE_OPENAI_ENDPOINT` | Tier 2 (judge) | no | +| `AZURE_AI_FOUNDRY_ENDPOINT` | Tier 3 (safety) | no | + +Document these clearly in the chat output when the workflow is scaffolded. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md new file mode 100644 index 0000000000..f52f521b49 --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md @@ -0,0 +1,188 @@ +# Common pitfalls + +Footguns that turned up while building and dogfooding this skill. +Avoid them when scaffolding `.Evals.Tests`. + +## Reporting pipeline + +- **Multiple `[AssemblyCleanup]` methods.** MSTest forbids more than + one `[AssemblyCleanup]` per assembly (UTA014 at discovery time — + every test in the assembly fails to load). The + `MetricsGlossary.WriteGlossary` write must be **chained from + `AievalReport.GenerateReport`'s single `[AssemblyCleanup]`**, not + declared as its own. Wrap the chained call in a `try / catch` so a + glossary-write failure never masks the report. +- **Hand-rolling reports instead of using the Reporting pipeline.** + The whole point of GA `Microsoft.Extensions.AI.Evaluation.Reporting` + 10.7.0 is `DiskBasedReportingConfiguration` + `aieval`. Never write + a hand-rolled markdown report and call it the "quality report" — + that's an MEAI report (HTML) vs a cost/latency capture (markdown). +- **Treating telemetry / compare / quality as separate report + streams.** Compare mode goes through `ReportingConfiguration` with + a distinct `executionName` per matrix entry, so `aieval report` + aggregates them into the same HTML. +- **Forgetting the per-run `metrics-glossary.md`.** The aieval HTML is + data-bound JSON; it shows numbers but no definitions. Co-locate + `metrics-glossary.md` (tier-aware) so a first-time reader can decode + the columns. The `Reporting/MetricsGlossary.cs` template handles + this — don't strip it. +- **Misreading "Cache Miss" in the Diagnostic Data section.** With the + default template (`enableResponseCaching: true`, agent resolved via + `run.ChatConfiguration!.ChatClient`, no per-run `executionName`), + the **first** `dotnet test` run shows Miss everywhere (cache empty) + and **subsequent runs against unchanged inputs show Hit everywhere** + in ~5s with zero LLM cost. Persistent Miss after run 1 means one of: + (a) the rubric / golden / scenario inputs changed, (b) the judge + model or chat options changed, (c) `_store/cache/` was deleted, or + (d) something is bypassing the cache — most commonly calling + `Wire.ResolveAgentClient()` (uncached) instead of + `run.ChatConfiguration!.ChatClient`, OR passing a fresh + `executionName` to `DiskBasedReportingConfiguration.Create(...)`. + Hit/Miss never affects correctness; it just tells you whether the + LLM was actually called this run. + +## Clients (agent vs judge vs stub) + +- **Calling real models from the default test run.** Stub tier uses + `StubChatClient`; the report banner is clearly marked + `(stub IChatClient)`. Real-model runs are opt-in via three + independent env vars. +- **Conflating agent and judge clients.** Two different `IChatClient` + roles. The skill exposes them as two independent env vars + (`EVAL_USE_REAL_AGENT`, `EVAL_USE_REAL_JUDGE`) — one can be real + while the other is stubbed. +- **Auto-detected factory throwing a generic NRE on missing config.** + When the app uses Aspire orchestration, `dotnet test` runs outside + the AppHost and `ConnectionStrings:` is unset. The factory + template in `ichatclient-detection.md` wraps DI resolution in a + `try / catch` that throws a friendly `InvalidOperationException` + naming the connection-string key and the user-secrets command. Don't + strip this when adapting the template. +- **User-secrets silently not loading in `dotnet test`.** `dotnet test` + runs under `testhost.exe` as the entry assembly, so + `Host.CreateApplicationBuilder()` does NOT pick up the secrets store + bound to your test project's `UserSecretsId`. The factory must call + `builder.Configuration.AddUserSecrets(typeof(...).Assembly, optional: true)` + explicitly. Without it the secret is set on disk but the friendly NRE + still fires. +- **`services.ai.azure.com` hostname strips dashes.** Resource + `foundry-abc` resolves to host `foundryabc.services.ai.azure.com` (no + dash). Authoritative endpoints are in + `az cognitiveservices account show -n -g --query properties.endpoints`. + Use the `AI Foundry API` or `Azure AI Model Inference API` entries — + not the legacy `properties.endpoint` value, which points at the + `cognitiveservices.azure.com` hostname that 404s for the `/models` route. +- **Key-based auth disabled (`403`).** Foundry resources provisioned by + Aspire/azd usually set `disableLocalAuth=true`. Drop the `Key=` + segment from the connection string and rely on `DefaultAzureCredential` + (`az login` + a Cognitive Services User role assignment on the + resource). The Aspire `AddAzureChatCompletionsClient` registration + picks the credential automatically when the key is absent. +- **Reasoning models (gpt-5, gpt-5-mini, o1, o3) reject `max_tokens`.** + The Azure.AI.Inference SDK still sends `max_tokens`; reasoning models + require `max_completion_tokens` and return 400 + `unsupported_parameter`. **The MEAI Quality evaluators swallow the + 400 and record it as a per-metric error row, so tests pass but every + Quality column is an error.** Pick a non-reasoning judge model + (gpt-4o, gpt-4o-mini, gpt-4-turbo). Tip: when picking a Foundry + deployment to point `ConnectionStrings:` at, check + `az cognitiveservices account deployment list -n -g + --query "[].{name:name, model:properties.model.name}" -o tsv` + and avoid any deployment whose model is `gpt-5*` or `o*`. When the + agent uses a reasoning model in production, set + `EVAL_JUDGE_DEPLOYMENT_NAME=` so the judge + client points at a compatible deployment while the agent client + keeps the production model. + +## Evaluators + +- **Wiring 4 separate safety evaluators.** Use `ContentHarmEvaluator` + for the Hate / SelfHarm / Violence / Sexual bundle — single Foundry + call, 4 metrics back. The 4 individual evaluators + (`HateAndUnfairnessEvaluator` etc.) are a strict subset. +- **Putting `RelevanceTruthAndCompletenessEvaluator` in the default + set.** Marked experimental upstream; not part of the v2 default. +- **Forgetting `EvaluationContext` for NLP evaluators.** BLEU / GLEU + need `BLEUEvaluatorContext(IEnumerable)`; F1 needs + `F1EvaluatorContext(string)`. If `golden.json` lacks + `reference_response`, NLP evaluators emit `(no reference)` and the + scenario shows blanks in the report. + +### Tuning Quality for stylistic agents + +The built-in Quality evaluators (`RelevanceEvaluator`, +`CoherenceEvaluator`, `FluencyEvaluator`, `CompletenessEvaluator`, +`EquivalenceEvaluator`) use **generic rubrics baked into the +evaluator's judge prompt**. They don't know about *your* agent's +contract. + +- **`CompletenessEvaluator` explicitly rewards thoroughness.** Any + agent whose contract is "compress / summarize / dumb-down" (ELI5, + TL;DR summarizers, tweet-length responders, structured-extractor + agents) will score 1-2 even when working perfectly. The score is + measuring conformance to a generic "answer thoroughly" rubric, not + conformance to your agent's contract. +- **`EquivalenceEvaluator` rewards lexical/semantic match to a + reference.** If your `golden.json` references are full-form + explanations but your agent emits paraphrases or stylized variants, + Equivalence will flag drift that isn't a real regression. +- **`CoherenceEvaluator` and `FluencyEvaluator`** penalize + fragmentary / one-line / heavily-structured outputs (JSON-only, + bullet-only, single-sentence). Agents that respond in a strict + format will be marked down. +- **Only `RelevanceEvaluator` is largely intent-agnostic** — it + checks "did the response address the query" without preferring + long-form. It's the one Quality metric that's mostly safe to leave + on for any agent. + +**Three remediation patterns, in increasing order of effort:** + +1. **Drop the offending evaluators per app.** Edit + `Reporting/ReportingConfig.cs` and remove `CompletenessEvaluator` + / `EquivalenceEvaluator` from the judge-tier evaluator list. + Keep Relevance + Coherence + Fluency. Document the choice in a + `// per-app: ELI5 contract → drop Completeness` comment so future + re-scaffolds don't add them back. +2. **Rewrite goldens in the agent's voice.** Edit `Quality/golden.json` + so `reference_response` is itself ELI5-style (or tweet-style, or + bullet-style). Equivalence becomes meaningful again; Completeness + still complains. +3. **Add a custom rubric-driven evaluator.** Write a `RubricEvaluator` + that reads `Quality/rubric.md` and asks the judge to score the + response against *your* criteria ("Is the explanation appropriate + for a 5-year-old? Score 1-5."). See + `references/evaluators-catalog.md#custom-rubric-driven-evaluator` + for the template. This is the right long-term answer for any + non-generic agent. + +> **Surface this in the chat output on first scaffold.** The user +> shouldn't have to interpret bad Quality scores against a bad-fit +> rubric and conclude their agent is broken. Step 11 of `SKILL.md` +> calls this out explicitly in the Quality headline block. + +## Configuration + +- **Hard-coding a price table.** Lives in `Telemetry/prices.json`, + user-editable. Costs change. +- **Auto-failing the build on quality regressions.** Quality mode is + informational by default. Users opt into a hard-fail by editing + `quality.thresholds.json` (which maps to real MEAI metric names like + `Relevance` / `BLEU` / `F1` and `EvaluationRating` levels), then + setting `hard_fail: true`. +- **Wrong `Microsoft.Extensions.Hosting` version.** Must be `10.0.1` + (not `10.0.0`) to satisfy the transitive constraint from + `Microsoft.Agents.AI.Hosting` 1.x. Pinning `10.0.0` produces + `NU1605` (treated as error in the agentic-app project graph). +- **Forgetting `.gitignore` entries.** Must include both + `.copilot/perf-reports/evals/` and `.Evals.Tests/_store/`. + Otherwise reports pollute history and the persistent `_store/` + blocks PR diffs. + +## Update mode + +- **Overwriting `Quality/rubric.md` or `Quality/golden.json`.** These + are user data — never overwrite. The skill's update-mode behaviour + table in step 1a of `SKILL.md` is the source of truth. +- **Migrating `golden.json` schema destructively.** Migration is + additive only: new fields go in as nullable, existing rows are + preserved. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/compare-mode.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/compare-mode.md index 05d89b8571..94d5f60c6f 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/compare-mode.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/compare-mode.md @@ -1,89 +1,99 @@ # Compare mode -Run telemetry + quality for two or more model assignments and emit a -side-by-side delta. +Compare mode runs the quality + telemetry pipeline against **multiple +model assignments** and produces a side-by-side report. Crucially, it +goes through the **same `DiskBasedReportingConfiguration`** so +`aieval report` aggregates the comparison into a single HTML view. -## `Compare/matrix.json` +## `matrix.json` ```json -[ - { - "name": "baseline", - "model_assignments": { - "router": "gpt-4o-mini", - "planner": "gpt-4o", - "worker": "gpt-4o-mini" +{ + "schema_version": 2, + "entries": [ + { + "name": "baseline-all-mini", + "model_assignments": { + "receptionist": "gpt-4o-mini", + "behavioural": "gpt-4o-mini", + "technical": "gpt-4o-mini", + "summariser": "gpt-4o-mini" + } + }, + { + "name": "interviewers-upgraded", + "model_assignments": { + "receptionist": "gpt-4o-mini", + "behavioural": "gpt-4o", + "technical": "gpt-4o", + "summariser": "gpt-4o-mini" + } } - }, - { - "name": "candidate", - "model_assignments": { - "router": "gpt-4o-mini", - "planner": "o4-mini", - "worker": "gpt-4o-mini" - } - } -] + ] +} ``` -## Runner behaviour - -For each entry in the matrix: - -1. Apply the `model_assignments` in-process (override the - per-agent `IChatClient` registrations; do NOT modify - `appsettings.json`). -2. Run telemetry mode against `Telemetry/inputs.json`. -3. Run quality mode against `Quality/golden.json`. -4. Capture both reports labeled by `name`. - -Then diff: - -- Latency: per-agent delta (ms) and aggregate delta. -- Tokens: input/output deltas per agent. -- Cost: aggregate delta (USD). -- Quality: pass-rate delta and per-input score delta. - -## Report — `compare.md` - -```markdown -# Compare — {{ utc_timestamp }} - -Variants: {{ name_list }} - -## Latency (avg ms per agent) - -| Agent | baseline | candidate | Δ | -|---------|----------|-----------|--------| -| router | 340 | 342 | +2 | -| planner | 1240 | 980 | -260 | -| worker | 410 | 405 | -5 | +## Test shape + +Each matrix entry becomes one row of a parameterised test. Each entry +uses a **distinct `executionName`** so `aieval report` shows the +comparison side-by-side. + +```csharp +[TestClass] +public sealed class CompareTests +{ + public static IEnumerable Matrix() => + MatrixLoader.Load().Select(e => new object[] { e }); + + [TestMethod, DynamicData(nameof(Matrix), DynamicDataSourceType.Method)] + public async Task RunEntry(MatrixEntry entry) + { + // executionName scoped per entry so aieval report groups columns by it. + var reporting = DiskBasedReportingConfiguration.Create( + storageRootPath: ReportingConfig.StorageRoot, + evaluators: ReportingConfig.EvaluatorList(), + chatConfiguration: new ChatConfiguration( + Wire.ResolveJudgeClient(Wire.ResolveAgentClient(entry.ModelAssignments))), + enableResponseCaching: true, + // Stable per-entry name (NOT prefixed with a per-run timestamp) + // so re-running compare reuses the cache for unchanged entries. + executionName: $"compare-{entry.Name}"); + + foreach (var g in GoldenLoader.Load()) + { + var scenarioName = $"Compare.{entry.Name}.{g.Id}"; + await using var run = await reporting.CreateScenarioRunAsync(scenarioName); + // ... same shape as QualityTests + } + } +} +``` -## Token cost (USD per 1K turns, projected) +## Override the per-agent model id -| Variant | Cost | Δ | -|-----------|--------|----------| -| baseline | $5.34 | — | -| candidate | $4.18 | -$1.16 | +`Wire.ResolveAgentClient(IDictionary overrides)` is the +extension point. The generated factory (`AgentChatClientFactory`) +exposes an overload accepting per-agent model assignments — useful when +the app uses multiple deployment aliases or supports model swapping +via `ChatOptions.ModelId`. -## Quality (pass rate) +## Compare-specific report -| Variant | Pass | Δ | -|-----------|------|-------| -| baseline | 92% | — | -| candidate | 90% | -2% | +`compare.md` (still emitted, in addition to the aggregated +`report.html`): -## Recommendation +| name | avg ms | in tok | out tok | $ | mean quality | mean BLEU | +|------|--------|--------|---------|---|--------------|-----------| +| baseline-all-mini | 432 | 380 | 210 | 0.0021 | 3.8 | 0.31 | +| interviewers-upgraded | 891 | 380 | 245 | 0.0210 | 4.4 | 0.42 | -candidate ⟶ -22% cost, -260ms planner latency, -2pp quality. -If quality bar is "no regression", reject. If quality bar is -"≥ 88%", accept. -``` +| Recommendation | +|----------------| +| `interviewers-upgraded`: +0.6 quality at +9.4× cost. Promote only if quality bar requires it. | -## Constraints +The recommendation row is rule-based: -- Compare mode never edits `appsettings.json`. -- Compare mode never makes a recommendation by itself; it states the - trade in plain terms and leaves the decision to the user. -- For ≥ 3 variants, the table grows columns; the recommendation row - picks the variant with the best cost-quality frontier (Pareto). +- If cost increases > 3× and quality delta < 0.3 → **do not promote**. +- If cost increases ≤ 1.5× and quality delta ≥ 0.5 → **promote**. +- Otherwise → **manual review**. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/dotnet-tools-manifest.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/dotnet-tools-manifest.md new file mode 100644 index 0000000000..c8e50335d4 --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/dotnet-tools-manifest.md @@ -0,0 +1,50 @@ +# `dotnet-tools.json` manifest + +The skill scaffolds a **local** tool manifest. Global install is +explicitly avoided — pinning the tool version to the project makes +evals reproducible across machines and CI runs. + +## Why local + +- Reproducible: every clone gets the exact same `aieval` version. +- CI-friendly: `dotnet tool restore` in the workflow is one line. +- No PATH conflicts with developers who may have other versions installed. + +## Template + +```json +{ + "version": 1, + "isRoot": true, + "tools": { + "microsoft.extensions.ai.evaluation.console": { + "version": "10.7.0", + "commands": ["aieval"], + "rollForward": false + } + } +} +``` + +Place at `.Evals.Tests/dotnet-tools.json` (not the repo root) so +the manifest follows the project. The skill emits `dotnet tool restore` +as the next step in the chat output after scaffolding. + +## Why `rollForward: false` + +The output of `aieval report` is consumed by humans visually and by +artifact comparison in CI. A silent rollForward of the tool to a +newer version can change report layout, charts, or metric grouping — +breaking trend comparisons and visual diffs. + +If a user wants to upgrade the tool, the skill should emit a +`dotnet tool update microsoft.extensions.ai.evaluation.console` +command in the chat output (with a note: "this may change report +layout"). + +## Conflict with existing manifest + +If `dotnet-tools.json` already exists at the project (or any parent +directory), the skill **merges** the `aieval` entry rather than +overwriting. If a different version of `aieval` is already pinned, +surface the diff in chat output and require user confirmation. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md new file mode 100644 index 0000000000..127968f217 --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md @@ -0,0 +1,271 @@ +# Evaluators catalog + +Full catalog of evaluators wired by `setup-maf-evals`, grouped by tier. +The skill defaults to Tier 1 (NLP) always on, Tier 2 (Quality) on but +gated by `EVAL_USE_REAL_JUDGE`, Tier 3 (Safety) off unless opted in. + +Source: [learn.microsoft.com/en-us/dotnet/ai/evaluation/libraries](https://learn.microsoft.com/en-us/dotnet/ai/evaluation/libraries) + +## Tier 1 — NLP (deterministic, no LLM) + +Package: `Microsoft.Extensions.AI.Evaluation.NLP` (preview 10.7.0). + +| Evaluator | Metric | Context type needed | Notes | +|-----------|--------|---------------------|-------| +| `BLEUEvaluator` | `BLEU` | `BLEUEvaluatorContext(IEnumerable references)` | n-gram overlap with one or more reference strings | +| `GLEUEvaluator` | `GLEU` | `GLEUEvaluatorContext(IEnumerable references)` | sentence-level BLEU variant | +| `F1Evaluator` | `F1` | `F1EvaluatorContext(string groundTruth)` | unigram-level F1 | + +Plus a built-in custom evaluator the skill always scaffolds: + +| Evaluator | Metric | Why | +|-----------|--------|-----| +| `WordCountEvaluator` (custom) | `Words` | Sanity check: response is non-empty and reasonable length. Same pattern as the Learn doc tutorial. | + +### `WordCountEvaluator` reference implementation + +Scaffold this file verbatim (the Learn-doc canonical pattern) into +`Reporting/WordCountEvaluator.cs`. It runs in stub tier with no API key. + +```csharp +public sealed class WordCountEvaluator : IEvaluator +{ + public const string MetricName = "Words"; + public IReadOnlyCollection EvaluationMetricNames { get; } = [MetricName]; + + public ValueTask EvaluateAsync( + IEnumerable messages, + ChatResponse modelResponse, + ChatConfiguration? chatConfiguration = null, + IEnumerable? additionalContext = null, + CancellationToken cancellationToken = default) + { + var text = modelResponse?.Text ?? string.Empty; + var count = text.Split( + new[] { ' ', '\t', '\r', '\n' }, + StringSplitOptions.RemoveEmptyEntries).Length; + + var metric = new NumericMetric(MetricName, value: count) + { + Interpretation = count switch + { + < 5 => new EvaluationMetricInterpretation(EvaluationRating.Poor, reason: "Response too short"), + > 500 => new EvaluationMetricInterpretation(EvaluationRating.Average, reason: "Response very long"), + _ => new EvaluationMetricInterpretation(EvaluationRating.Good), + } + }; + return new ValueTask(new EvaluationResult(metric)); + } +} +``` + +## Tier 2 — Quality (LLM-as-judge) + +Package: `Microsoft.Extensions.AI.Evaluation.Quality` (GA 10.7.0). + +Requires `EVAL_USE_REAL_JUDGE=1` and a real `IChatClient`. The skill +wires the following by default: + +| Evaluator | Metric | Context type | Notes | +|-----------|--------|--------------|-------| +| `RelevanceEvaluator` | `Relevance` | none | how relevant is the response to the query | +| `CoherenceEvaluator` | `Coherence` | none | logical, orderly presentation | +| `FluencyEvaluator` | `Fluency` | none | grammar, readability | +| `CompletenessEvaluator` | `Completeness` | `CompletenessEvaluatorContext(string groundTruth)` | comprehensive and accurate | +| `EquivalenceEvaluator` | `Equivalence` | `EquivalenceEvaluatorContext(string groundTruth)` | similarity vs ground truth wrt query | +| `GroundednessEvaluator` | `Groundedness` | `GroundednessEvaluatorContext(string context)` | alignment with given context | + +Agent-focused (added when `*.AppHost.csproj` is detected, indicating +this is an agentic app and not just a chat completion app): + +| Evaluator | Metric | Context type | Notes | +|-----------|--------|--------------|-------| +| `IntentResolutionEvaluator` | `Intent Resolution` | none | identifies + resolves user intent | +| `TaskAdherenceEvaluator` | `Task Adherence` | none | sticks to assigned task | +| `ToolCallAccuracyEvaluator` | `Tool Call Accuracy` | `ToolCallAccuracyEvaluatorContext(...)` | uses tools correctly | + +**Not wired by default:** `RelevanceTruthAndCompletenessEvaluator` +(marked experimental in upstream docs), `RetrievalEvaluator` (specific +to RAG pipelines — separate skill territory). + +## Tier 3 — Safety (Foundry Evaluation service) + +Package: `Microsoft.Extensions.AI.Evaluation.Safety` (preview 10.7.0). + +Off by default. Enabled when user opts in during step 2 of the +workflow. Requires `EVAL_USE_FOUNDRY_SAFETY=1` and an Azure AI Foundry +endpoint (the skill prompts for `AZURE_AI_FOUNDRY_ENDPOINT` + Entra credentials). + +**Always wire the bundle, not the 4 separate evaluators:** + +| Evaluator | Metrics produced | Notes | +|-----------|------------------|-------| +| `ContentHarmEvaluator` | `Hate And Unfairness`, `Self Harm`, `Violence`, `Sexual` | **Single-shot — one Foundry call returns all 4 metrics.** Always prefer this over the 4 separate evaluators below. | + +Additional safety evaluators (each wired as separate `[TestMethod]`): + +| Evaluator | Metric | Notes | +|-----------|--------|-------| +| `ProtectedMaterialEvaluator` | `Protected Material` | copyrighted material in output | +| `IndirectAttackEvaluator` | `Indirect Attack` | prompt-injection-style indirect attacks | +| `CodeVulnerabilityEvaluator` | `Code Vulnerability` | vulnerable code in output | +| `UngroundedAttributesEvaluator` | `Ungrounded Attributes` | inferred human attributes | +| `GroundednessProEvaluator` | `Groundedness Pro` | fine-tuned Foundry-hosted groundedness check | + +The 4 separate evaluators (`HateAndUnfairnessEvaluator`, +`SelfHarmEvaluator`, `ViolenceEvaluator`, `SexualEvaluator`) are +**not** scaffolded — they're a strict subset of `ContentHarmEvaluator` +and cost 4× more Foundry calls for the same metrics. + +## Threshold mapping + +`quality.thresholds.json` maps **real MEAI metric names** to minimum +`EvaluationRating` enum values: + +```json +{ + "schema_version": 2, + "hard_fail": false, + "thresholds": { + "Relevance": { "min_rating": "Good" }, + "Coherence": { "min_rating": "Good" }, + "Fluency": { "min_rating": "Average" }, + "Groundedness":{ "min_rating": "Good" }, + "BLEU": { "min_value": 0.20 }, + "F1": { "min_value": 0.30 }, + "Words": { "min_value": 5, "max_value": 500 } + } +} +``` + +`hard_fail: true` makes any below-threshold metric fail the test. +Default `false` makes the test pass-through informational — failures +show in the report only. + +## Custom rubric-driven evaluator + +The built-in Quality evaluators (Relevance, Coherence, Fluency, +Completeness, Equivalence) judge against generic rubrics baked into +the evaluator's judge prompt. They cannot read `Quality/rubric.md`. +That makes them ill-fitting for agents with deliberate stylistic +constraints (ELI5 / summarizer / strict-format / persona-bound +agents) — see `common-pitfalls.md#tuning-quality-for-stylistic-agents`. + +For those agents, scaffold a `RubricEvaluator` that reads +`Quality/rubric.md` and asks the judge to score against *your* +criteria. The pattern is shape-identical to `WordCountEvaluator` — +it just delegates to the judge chat client. + +```csharp +// Reporting/RubricEvaluator.cs +// Custom rubric-driven evaluator. Reads Quality/rubric.md and asks +// the judge to score the response against per-app criteria. Emits a +// single "RubricFit" metric (numeric 1-5) plus a free-text rationale. +public sealed class RubricEvaluator : IEvaluator +{ + public const string MetricName = "RubricFit"; + public IReadOnlyCollection EvaluationMetricNames { get; } = [MetricName]; + + private readonly string _rubric; + + public RubricEvaluator(string rubricMarkdown) => _rubric = rubricMarkdown; + + public static RubricEvaluator FromFile(string path) => + new(File.ReadAllText(path)); + + public async ValueTask EvaluateAsync( + IEnumerable messages, + ChatResponse modelResponse, + ChatConfiguration? chatConfiguration = null, + IEnumerable? additionalContext = null, + CancellationToken cancellationToken = default) + { + if (chatConfiguration?.ChatClient is null) + { + // Judge tier not active — return a stubbed Inconclusive metric. + return new EvaluationResult(new NumericMetric(MetricName) + { + Interpretation = new EvaluationMetricInterpretation( + EvaluationRating.Inconclusive, reason: "Judge not wired.") + }); + } + + var userQuery = string.Join("\n", messages.Select(m => m.Text)); + var responseText = modelResponse.Text ?? string.Empty; + + // NOTE: $$"""...""" (double-$) raw string. Inside, single { } is literal + // (matters for the JSON braces below), and {{var}} is interpolation. + // Plain $"""...""" would reject `{{ ... }}` as literal with CS9006. + var prompt = $$""" + You are scoring an assistant response against the rubric below. + + ## Rubric + {{_rubric}} + + ## User query + {{userQuery}} + + ## Assistant response + {{responseText}} + + Respond with strict JSON: { "score": <1-5 int>, "rationale": "<1-2 sentences>" }. + 5 = perfectly satisfies every rubric clause. 1 = ignores the rubric. + """; + + var judge = await chatConfiguration.ChatClient.GetResponseAsync( + prompt, cancellationToken: cancellationToken).ConfigureAwait(false); + + // Tolerant parse: fall back to Inconclusive on bad JSON. + var (score, rationale) = TryParse(judge.Text ?? ""); + + var metric = new NumericMetric(MetricName, value: score) + { + Interpretation = new EvaluationMetricInterpretation( + rating: score switch { >= 4 => EvaluationRating.Good, + 3 => EvaluationRating.Average, + _ => EvaluationRating.Poor }, + reason: rationale) + }; + return new EvaluationResult(metric); + } + + private static (int score, string rationale) TryParse(string raw) + { + try + { + using var doc = JsonDocument.Parse(raw.Trim().Trim('`', ' ', '\n', '\r')); + return (doc.RootElement.GetProperty("score").GetInt32(), + doc.RootElement.GetProperty("rationale").GetString() ?? ""); + } + catch { return (3, "Judge response was unparseable; defaulted to Average."); } + } +} +``` + +Wire it from `Reporting/ReportingConfig.cs`'s judge-tier evaluator +list **alongside** Relevance/Coherence/Fluency (drop Completeness + +Equivalence per the pitfall guidance): + +```csharp +// In ReportingConfig.ForQuality(), when EvalEnv.UseRealJudge: +evaluators.Add(new RelevanceEvaluator()); +evaluators.Add(new CoherenceEvaluator()); +evaluators.Add(new FluencyEvaluator()); +evaluators.Add(RubricEvaluator.FromFile( + Path.Combine(AppContext.BaseDirectory, "Quality", "rubric.md"))); +// CompletenessEvaluator + EquivalenceEvaluator deliberately dropped +// for this app — see common-pitfalls.md tuning section. +``` + +Ensure `Quality/rubric.md` is copied to output by adding to the csproj: + +```xml + + + +``` + +This `RubricFit` metric will appear in `report.html` alongside the +built-ins, and the rationale shows in the per-scenario detail drawer. +Update `Reporting/MetricsGlossary.cs`'s `QualityEntries` constant to +add a one-line `RubricFit` definition pointing at `Quality/rubric.md`. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md new file mode 100644 index 0000000000..00363d0f8a --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md @@ -0,0 +1,214 @@ +# IChatClient detection + +The skill auto-detects how the target app registers its `IChatClient` +and emits `Wire/AgentChatClientFactory.cs` so `EVAL_USE_REAL_AGENT=1` +works without further code. Detection result is surfaced in step 2 +(scope confirmation) and can be overridden by the user. + +## Patterns to scan for + +Scan **`*.AppHost.csproj` directory** and **all `*.Agent*.csproj` +directories**. Match (case-insensitive, multi-line): + +| Pattern (regex-ish) | Inferred client | Example file:line | +|---------------------|-----------------|--------------------| +| `AddAzureOpenAIChatClient\s*\(` or `AddAzureOpenAIClient\s*\(` | Azure OpenAI | `AppHost.cs`, `Program.cs` | +| `AddOpenAIChatClient\s*\(` | OpenAI direct | `Program.cs` | +| `AddOllamaChatClient\s*\(` | Ollama | `Program.cs` | +| `AddAIInference\s*\(` (Foundry deployment alias) | Azure AI Foundry | `AppHost.cs` | +| `AddAzureChatCompletionsClient\s*\([^)]*\)\s*\.AddChatClient\s*\(` | Aspire `Aspire.Azure.AI.Inference` (Foundry-routed) | `Program.cs` | +| `services\.AddSingleton` (any explicit registration) | custom | varies | +| `\.AsIChatClient\(\)` (after an SDK client) | manual wrap | varies | + +> The `AddAzureChatCompletionsClient(...).AddChatClient(...)` chain is the +> standard Aspire 13.2 way of wiring an `IChatClient` against a Foundry chat +> deployment. The argument is the **connection-string name**, which Aspire's +> AppHost populates automatically (`AddDeployment("chat", ...)` -> connection +> string `chat`). The factory mirrors both calls verbatim. + +Capture the deployment alias / model id literal if present (e.g., +`builder.AddAIInference("chat", "gpt-4o-mini")` → alias `chat`). + +## What to emit + +### Case A — exactly one registration found + +Emit a factory that resolves from the host: + +```csharp +// Wire/AgentChatClientFactory.cs +namespace {{AppName}}.Evals.Tests; + +internal static class AgentChatClientFactory +{ + /// + /// Resolves the same IChatClient the app uses, by building a minimal + /// host that mirrors the app's DI registration. + /// Detected: {{DetectionSummary}} at {{File}}:{{Line}} + /// + public static IChatClient Create() + { + var builder = Host.CreateApplicationBuilder(); + + // In test hosts (`dotnet test`), the entry assembly is testhost.exe so + // user-secrets are NOT auto-loaded by CreateApplicationBuilder. + // Add them explicitly from THIS assembly's UserSecretsId. + builder.Configuration.AddUserSecrets(typeof(AgentChatClientFactory).Assembly, optional: true); + + // {{InsertDetectedRegistrationCallVerbatim}} + var host = builder.Build(); + try + { + return host.Services.GetRequiredService(); + } + catch (InvalidOperationException ex) + { + throw new InvalidOperationException( + "EVAL_USE_REAL_AGENT=1 but IChatClient could not be resolved. " + + "The detected registration ({{DetectionSummary}}) reads connection " + + "string \"{{ConnStrName}}\" from configuration. Aspire's AppHost " + + "populates this at runtime, but `dotnet test` runs standalone. " + + "Wire one of:\n" + + " # Key-based auth (only if the resource has it enabled):\n" + + " dotnet user-secrets set \"ConnectionStrings:{{ConnStrName}}\" " + + "\"Endpoint=https://.services.ai.azure.com/models;Key=;DeploymentId={{ConnStrName}}\" --project {{AppName}}.Evals.Tests\n" + + " # Entra-ID auth (DefaultAzureCredential — works when key auth is disabled):\n" + + " dotnet user-secrets set \"ConnectionStrings:{{ConnStrName}}\" " + + "\"Endpoint=https://.services.ai.azure.com/models;DeploymentId={{ConnStrName}}\" --project {{AppName}}.Evals.Tests\n" + + "Note: the hostname strips dashes from the resource name " + + "(resource `foundry-abc` -> host `foundryabc.services.ai.azure.com`).\n" + + "Get the exact endpoint from `azd env get-values` or " + + "`az cognitiveservices account show -n -g --query properties.endpoints`.", + ex); + } + } +} +``` + +Where `{{InsertDetectedRegistrationCallVerbatim}}` is the literal call +copied from the detection source (with any required `using`s in scope +via `GlobalUsings.cs`), and `{{ConnStrName}}` is the connection-string +literal extracted from the call (e.g., the `"chat"` argument). + +### Case B — multiple registrations found + +Emit the same factory but with a comment listing all candidates, and +have the chat output ask the user to pick one before write. Do **not** +write the file until confirmation. + +### Case C — no registration found + +Emit a stub factory the user fills in: + +```csharp +// Wire/AgentChatClientFactory.cs +namespace {{AppName}}.Evals.Tests; + +internal static class AgentChatClientFactory +{ + /// + /// Auto-detection failed: no IChatClient registration found in + /// AppHost or agent projects. Wire your client manually. + /// + public static IChatClient Create() => + throw new NotImplementedException( + "Wire your IChatClient here. See https://learn.microsoft.com/en-us/dotnet/ai/microsoft-extensions-ai"); +} +``` + +## Runtime selection + +`ReportingConfig` picks agent client based on tier: + +```csharp +internal static IChatClient ResolveAgentClient() => + EvalEnv.UseRealAgent ? AgentChatClientFactory.Create() : new StubChatClient(); +``` + +And by default the judge client is the **same** instance as the agent +client. This saves a duplicate Azure credential setup AND lets the +MEAI response cache serve both the agent call and the judge call from +one cache (because `QualityTests` uses `run.ChatConfiguration!.ChatClient` +for the agent call — that's the cached wrapper around the shared +instance). The user can override by setting `EVAL_JUDGE_DEPLOYMENT_NAME` +to a different deployment alias when (e.g.) the production model is a +reasoning model that can't be used as a judge. Trade-off: with the +override active, `run.ChatConfiguration!.ChatClient` becomes the +**judge** client, so the agent call would silently use the judge +model. To avoid that, `QualityTests` falls back to +`Wire.ResolveAgentClient()` (uncached) for agent calls whenever the +override is set; cache hits then apply only to judge calls. + +## Connection-string setup for standalone test runs + +When the target app uses Aspire orchestration, the AppHost populates +`ConnectionStrings:` automatically. **`dotnet test` runs outside +the AppHost and does not get this for free.** Surface this in the chat +output any time the detected pattern reads from a connection string +(`AddAzureChatCompletionsClient`, `AddAIInference`, `AddOllamaChatClient`, +or `AddAzureOpenAIChatClient` without an explicit endpoint). + +> **Important — the factory must opt into user-secrets explicitly.** In a +> `dotnet test` process the entry assembly is `testhost.exe`, so the +> `dotnet user-secrets` payload tied to *your* `UserSecretsId` is NOT auto +> loaded by `Host.CreateApplicationBuilder()`. The Case A template above +> calls `builder.Configuration.AddUserSecrets(typeof(...).Assembly)` for +> this reason. Without that line, the secret is set on disk but never read. + +Recommend the user wire one of: + +```pwsh +# 0 — one-time: bind the test project to a secrets store +dotnet user-secrets init --project .Evals.Tests + +# Option A — Key-based auth (only if the resource has key auth enabled) +dotnet user-secrets set "ConnectionStrings:" ` + "Endpoint=https://.services.ai.azure.com/models;Key=;DeploymentId=" ` + --project .Evals.Tests + +# Option B — Entra-ID auth (DefaultAzureCredential; works when key auth disabled) +dotnet user-secrets set "ConnectionStrings:" ` + "Endpoint=https://.services.ai.azure.com/models;DeploymentId=" ` + --project .Evals.Tests +# requires `az login` and a Cognitive Services User / Azure AI User role +# on the resource for the signed-in identity. + +# Option C — env var (works in CI without a secrets file) +$env:ConnectionStrings__ = "Endpoint=https://...;DeploymentId=" +``` + +**Two endpoint gotchas to call out in chat:** + +1. The `services.ai.azure.com/models` hostname **strips dashes** from the + resource name. Resource `foundry-abc` -> host `foundryabc.services.ai.azure.com`. + Use `az cognitiveservices account show -n -g --query properties.endpoints` + to see all valid endpoint hostnames (`AI Foundry API` / `Azure AI Model Inference API`). +2. If the resource has `disableLocalAuth=true` (common on Foundry resources + provisioned by Aspire/azd), key-based auth returns `403 Key based authentication + is disabled for this resource`. Drop the `Key=` segment and use Entra (Option B). + +For Foundry-routed clients the connection string is what `azd env get-values` +prints for `connectionString` against the deployment resource. Document this +in the chat output along with the tier banner so the user doesn't see a +silent NRE on first real-agent run. + +## Chat output (step 2) + +When detection succeeds, surface as: + +``` +IChatClient detection: + - AddAzureOpenAIChatClient at AppHost.cs:41 + - Deployment alias: "chat" → gpt-4o-mini + - Will generate Wire/AgentChatClientFactory.cs that resolves it from the host. + +Override [y/N]? +``` + +When detection fails: + +``` +IChatClient detection: no registration found. + - Will generate a stub factory you'll need to fill in. + - Tier 2 (Quality) and Tier 3 (Safety) will be skipped until wired. +``` diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/metrics-glossary.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/metrics-glossary.md new file mode 100644 index 0000000000..476c995158 --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/metrics-glossary.md @@ -0,0 +1,247 @@ +# Metrics glossary + +Source of truth for the per-run `metrics-glossary.md` artifact that the +scaffolded `.Evals.Tests/Reporting/MetricsGlossary.cs` writes next +to `report.html`. The aieval HTML report is data-bound JSON — it shows +numbers but no definitions — so we co-locate this glossary with the +report so a first-time reader has a one-page cheat-sheet. + +The skill emits **only the entries for evaluators that actually ran** in +the active tier, so a stub-tier user sees Words/BLEU/GLEU/F1 and +nothing else. + +## NLP tier (deterministic, no LLM) + +### `Words` +- **Custom evaluator** scaffolded by this skill (see `evaluators-catalog.md`). +- **What it measures:** raw token count of the response text. +- **Scale:** integer ≥ 0. +- **Interpretation:** `< 5` → Poor (response too short / empty). `5–500` → Good. `> 500` → Average (response unusually long). +- **Trust for:** sanity-checking that the model produced *anything* and isn't running away with a 10-page essay. +- **Don't trust for:** quality. A 50-word wrong answer scores the same as a 50-word correct answer. + +### `BLEU` — Bilingual Evaluation Understudy +- **What it measures:** n-gram (1-4) overlap between the response and one or more reference strings, with a brevity penalty. +- **Scale:** 0.0 – 1.0. +- **Interpretation:** ~0.0–0.1 weak overlap, often paraphrased; ~0.1–0.3 normal for free-form generation; ~0.3–0.5 strong; > 0.5 near-quotation. +- **Trust for:** "is the response in the same lexical neighbourhood as the reference?" — useful as a regression signal when the reference is canonical. +- **Don't trust for:** semantic correctness. A correct paraphrase scores low; an incorrect copy-paste of reference fragments scores high. +- **Reference:** [`BLEUEvaluator` in MEAI.Evaluation.NLP](https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.ai.evaluation.nlp.bleuevaluator). + +### `GLEU` — Google-BLEU +- **What it measures:** sentence-level BLEU variant. Symmetric — penalises both missing reference n-grams and extra invented ones. +- **Scale:** 0.0 – 1.0. +- **Interpretation:** same buckets as BLEU; GLEU usually tracks BLEU but is less brittle on short outputs. +- **Trust for:** the same use cases as BLEU when individual scenarios are short (1-2 sentences) and BLEU's brevity penalty would be misleading. +- **Don't trust for:** any semantic claim. Same caveats as BLEU. +- **Reference:** [`GLEUEvaluator`](https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.ai.evaluation.nlp.gleuevaluator). + +### `F1` — Token-level F1 +- **What it measures:** harmonic mean of unigram precision and recall against the ground-truth string. Order-insensitive. +- **Scale:** 0.0 – 1.0. +- **Interpretation:** ~0.0–0.2 mostly-disjoint vocab; 0.3–0.5 typical for free-form generation; > 0.6 strong word-level match. +- **Trust for:** QA / extraction-style scenarios where the *set of words* matters more than phrasing (SQuAD-style benchmarks use this). +- **Don't trust for:** word-order-sensitive tasks (e.g., "yes" vs "no" answers buried in long output) — the F1 score will be high even if the polarity is wrong. +- **Reference:** [`F1Evaluator`](https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.ai.evaluation.nlp.f1evaluator). + +> **NLP-tier headline:** all three of BLEU/GLEU/F1 are *lexical* metrics. +> They're cheap, deterministic, and free, but they cannot tell you the +> response is *correct* — only that it's *similar in wording*. Treat them +> as a regression early-warning, not a quality verdict. + +## Quality tier (LLM-as-judge — `EVAL_USE_REAL_JUDGE=1`) + +All Quality metrics produce a 1-5 `EvaluationRating` (Poor → Excellent) +plus a free-text rationale from the judge model. + +### `Relevance` +- **What it measures:** does the response actually address the user's query? +- **Trust for:** catching off-topic regressions (model started monologuing about a different topic). +- **Don't trust for:** factual correctness — a *relevantly wrong* answer can score high. + +### `Coherence` +- **What it measures:** is the response logically structured / orderly / easy to follow? +- **Trust for:** detecting rambling, contradictory, or logically broken outputs. +- **Don't trust for:** depth or correctness — coherent nonsense scores high. + +### `Fluency` +- **What it measures:** grammar, readability, naturalness. +- **Trust for:** detecting broken-English / token-soup outputs from undertrained or quantised models. +- **Don't trust for:** anything beyond surface text quality. A fluent lie scores high. + +### `Completeness` +- **What it measures:** how comprehensive and accurate the response is, given a reference (`CompletenessEvaluatorContext(groundTruth)`). +- **Trust for:** catching responses that are correct but partial (covered 2 of 4 required points). +- **Don't trust for:** brevity-as-a-feature scenarios — a short-but-correct answer can score lower than a long-and-padded one. + +### `Equivalence` +- **What it measures:** semantic similarity between response and ground truth in the context of the original query. +- **Trust for:** distinguishing "right answer, different words" (high) from "wrong answer" (low). Better than BLEU/F1 when paraphrasing is acceptable. +- **Don't trust for:** any case where the ground truth is itself ambiguous or one of multiple valid answers. + +### `Groundedness` +- **What it measures:** alignment between the response and a supplied source-of-truth context (`GroundednessEvaluatorContext(context)`). +- **Trust for:** RAG pipelines — flags when the model invented facts not in the retrieved context. +- **Don't trust for:** open-ended chat without a context document. + +### Agentic-only + +The skill wires these only when an `*.AppHost.csproj` is detected. + +- **`Intent Resolution`** — did the model identify and resolve the user's actual intent (vs answering a related-but-different question)? +- **`Task Adherence`** — did the model stick to the assigned task or wander into other agent territory? +- **`Tool Call Accuracy`** — did the model invoke the right tools with the right arguments? Requires `ToolCallAccuracyEvaluatorContext`. + +> **Quality-tier headline:** these are LLM-judge subjective scores. They +> drift across judge model versions. Pin the judge model in +> `quality.thresholds.json` if you want comparable scores across runs. + +## Safety tier (Foundry — `EVAL_USE_FOUNDRY_SAFETY=1`) + +All Safety evaluators return a 1-5 severity (1 = safe, 5 = severe harm) +plus a confidence score. Each metric is an *output classifier* — it +inspects the model's response, not the input prompt. + +### `Hate And Unfairness`, `Self Harm`, `Violence`, `Sexual` +- Wired together as the single-shot `ContentHarmEvaluator` (one Foundry call → all 4 metrics). +- **Trust for:** detecting harmful content in agent outputs. +- **Don't trust for:** input-side filtering — these never see the user's prompt. Pair with input-side Azure AI Content Safety for full coverage. + +### `Protected Material` +- Detects copyrighted text / song lyrics / book passages reproduced in the output. + +### `Indirect Attack` +- Detects prompt-injection-style content that would indicate the model picked up an indirect attack from retrieved content or tool output. Closest thing to an input-side check available in this tier. + +### `Code Vulnerability` +- Flags vulnerable patterns in code the model emitted (SQL injection, hardcoded credentials, weak crypto, etc.). + +### `Ungrounded Attributes` +- Detects inferred human attributes (race, gender, age, religion, etc.) in the response that weren't in the input. + +### `Groundedness Pro` +- Foundry-hosted fine-tuned groundedness evaluator. More accurate than the open-source `GroundednessEvaluator` but costs a Foundry call per scenario. + +> **Safety-tier headline:** all safety metrics are *output classifiers*. +> They protect downstream consumers from the agent's outputs; they do not +> protect the agent from its inputs. + +## Quick reference card + +| Metric | Tier | Scale | Needs ground truth? | Needs LLM? | +|--------|------|-------|---------------------|-----------| +| Words | NLP | int | no | no | +| BLEU | NLP | 0-1 | yes (references) | no | +| GLEU | NLP | 0-1 | yes (references) | no | +| F1 | NLP | 0-1 | yes (one string) | no | +| Relevance / Coherence / Fluency | Quality | 1-5 | no | yes (judge) | +| Completeness / Equivalence | Quality | 1-5 | yes (one string) | yes (judge) | +| Groundedness | Quality | 1-5 | yes (context) | yes (judge) | +| Intent Resolution / Task Adherence | Quality (agentic) | 1-5 | no | yes (judge) | +| Tool Call Accuracy | Quality (agentic) | 1-5 | yes (expected_tool_calls) | yes (judge) | +| Content Harm bundle (4 metrics) | Safety | 1-5 severity | no | yes (Foundry) | +| Protected / Indirect / Code Vuln. / Ungrounded | Safety | 1-5 severity | varies | yes (Foundry) | +| Groundedness Pro | Safety | 1-5 | yes (context) | yes (Foundry) | + +## `MetricsGlossary.cs` template + +The scaffolded `.Evals.Tests/Reporting/MetricsGlossary.cs` writes +the tier-relevant slice of this glossary next to `report.html` after +each `dotnet test` run. + +> **MSTest constraint:** an assembly may declare **only one** +> `[AssemblyCleanup]` method. The skill emits `MetricsGlossary` as a +> plain static class (no `[TestClass]`, no `[AssemblyCleanup]`) and +> chains `MetricsGlossary.WriteGlossary()` from +> `AievalReport.GenerateReport`'s single `[AssemblyCleanup]`. +> Wrap the call in a `try/catch` so a glossary-write failure never +> masks the report. + +```csharp +internal static class MetricsGlossary +{ + public static void WriteGlossary() + { + var outDir = Path.Combine( + RepoRoot.Find(), ".copilot", "perf-reports", "evals", EvalEnv.ReportFolder); + Directory.CreateDirectory(outDir); + var path = Path.Combine(outDir, "metrics-glossary.md"); + + var sb = new StringBuilder(); + sb.AppendLine($"# Metrics glossary — {EvalEnv.Tier} tier"); + sb.AppendLine(); + sb.AppendLine($"Generated: {DateTime.UtcNow:O}"); + sb.AppendLine(); + sb.AppendLine($"Companion to `report.html` in this folder. The aieval HTML report shows numbers; this file explains them."); + sb.AppendLine(); + + sb.AppendLine(NlpEntries); + if (EvalEnv.UseRealJudge) sb.AppendLine(QualityEntries); + if (EvalEnv.UseFoundrySafety) sb.AppendLine(SafetyEntries); + + sb.AppendLine(); + sb.AppendLine("> Source: setup-maf-evals references/metrics-glossary.md"); + + File.WriteAllText(path, sb.ToString()); + Console.WriteLine($"[MetricsGlossary] {path}"); + } + + private const string NlpEntries = """ + ## NLP tier (deterministic, no LLM) + - **Words** (int): response length sanity check. <5 too short, 5-500 ok, >500 long. + - **BLEU** (0-1): n-gram overlap with reference(s). 0.1-0.3 normal, >0.3 strong, >0.5 near-quotation. *Lexical, not semantic.* + - **GLEU** (0-1): sentence-level BLEU; better for short outputs. Same buckets as BLEU. + - **F1** (0-1): unigram token F1 vs ground-truth. 0.3-0.5 typical, >0.6 strong word-level match. Order-insensitive. + + > Headline: NLP metrics measure wording similarity, not correctness. Use for regression early-warning, not as quality verdicts. + """; + + private const string QualityEntries = """ + ## Quality tier (LLM-as-judge) + Each rated 1-5 (Poor → Excellent) with a free-text rationale. + - **Relevance**: addresses the user's query. Catches off-topic regressions. + - **Coherence**: logically structured. Catches rambling/contradictory outputs. + - **Fluency**: grammar/readability. Catches broken-English outputs. + - **Completeness** (needs reference): comprehensive and accurate. + - **Equivalence** (needs reference): semantic similarity in context of the query. + - **Groundedness** (needs context): aligned with supplied source-of-truth. + - **Intent Resolution / Task Adherence / Tool Call Accuracy** (agentic only). + + > Headline: judge scores drift across model versions. Pin the judge model for comparable runs. + """; + + private const string SafetyEntries = """ + ## Safety tier (Foundry) + Each rated 1-5 severity (1 safe → 5 severe). + - **ContentHarm bundle** (single-shot, 4 metrics): Hate-And-Unfairness, Self-Harm, Violence, Sexual. + - **Protected Material**: copyrighted text reproduced. + - **Indirect Attack**: prompt-injection content from retrieved/tool data. + - **Code Vulnerability**: vulnerable code patterns (SQLi, weak crypto, etc.). + - **Ungrounded Attributes**: inferred human attributes not in input. + - **Groundedness Pro**: Foundry-hosted fine-tuned groundedness check. + + > Headline: all safety metrics inspect *outputs*, not inputs. Pair with Azure AI Content Safety on the request side for full coverage. + """; +} +``` + +Then in `Reporting/AievalReport.cs` (the assembly's single +`[AssemblyCleanup]` host): + +```csharp +[TestClass] +public static class AievalReport +{ + [AssemblyCleanup] + public static void GenerateReport() + { + // ... aieval invocation ... + + try { MetricsGlossary.WriteGlossary(); } + catch (Exception ex) { Console.Error.WriteLine($"[MetricsGlossary] Failed: {ex.Message}"); } + } +} +``` + +The skill should emit both files verbatim — change the `private const` +strings only if upstream MEAI changes a metric definition. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/project-template.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/project-template.md index f0003db43f..9bcbb88ac7 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/project-template.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/project-template.md @@ -1,156 +1,137 @@ -# Project template — `.Evals` +# Project template (MSTest shape) -The exact files written when scaffolding the eval harness. +The scaffold creates `.Evals.Tests` as a **MSTest** project. This +matches the +[upstream Learn doc tutorial](https://learn.microsoft.com/en-us/dotnet/ai/evaluation/evaluate-with-reporting) +and the [dotnet/ai-samples evaluation unit tests](https://github.com/dotnet/ai-samples/blob/main/src/microsoft-extensions-ai-evaluation/api/), +which is the canonical pattern for `Microsoft.Extensions.AI.Evaluation`. -## `.Evals.csproj` +A console-runner shape is available behind an explicit +`--shape console` flag for users who can't take an MSTest dependency, +but is no longer the default — every CI system understands `dotnet test` +out of the box, and Test Explorer integration is automatic. -Use the actual current package versions from NuGet at scaffold time -(query `nuget.org` or `dotnet package search`). The versions below -reflect the latest stable family at the time of writing -(`Microsoft.Extensions.AI.Evaluation` 10.x is GA on nuget.org); query -`dotnet package search "Microsoft.Extensions.AI.Evaluation"` and bump -to the latest stable when you scaffold. +## File tree + +``` +.Evals.Tests/ + .Evals.Tests.csproj + dotnet-tools.json + GlobalUsings.cs + Reporting/ + ReportingConfig.cs # DiskBasedReportingConfiguration factory; tier-aware evaluator list + Tier.cs # EvalTier enum + EvalEnv reader + AievalReport.cs # [AssemblyCleanup] that invokes the dotnet tool + Wire/ + AgentChatClientFactory.cs # auto-generated from IChatClient detection + StubChatClient.cs # used when EVAL_USE_REAL_AGENT is unset + Telemetry/ + TelemetryTests.cs + inputs.json + prices.json + Quality/ + QualityTests.cs + rubric.md + golden.json + Compare/ + CompareTests.cs + matrix.json + Safety/ # only if user opted in + SafetyTests.cs + quality.thresholds.json +.github/ + workflows/ + evals.yml # optional, opt-in +``` + +`.gitignore` additions (idempotent): + +``` +# setup-maf-evals +.copilot/perf-reports/evals/ +.Evals.Tests/_store/ +``` + +## `.csproj` template ```xml - - Exe net10.0 enable enable - {{ AppName }}.Evals + false + {{AppName}}.Evals.Tests - - - - - - + + + + + + + + + + + + + + + + + + + + - + - - - - - + - ``` -If the consumer repo uses Central Package Management -(`Directory.Packages.props` with `ManagePackageVersionsCentrally=true`), -omit the `Version=` attributes and add matching `` -entries to the central props file instead. +**Why these versions:** -The `ProjectReference` line points to the agent service the evals will -exercise (typically the agent service, not the AppHost). If the repo -has multiple agent service projects, generate one `ProjectReference` -per project and let the runner classes select which agent to invoke. +- `Microsoft.Extensions.AI.Evaluation.{Reporting,Quality,Console}` are GA at `10.7.0`. +- `Microsoft.Extensions.AI.Evaluation.{NLP,Safety}` are still preview at `10.7.0-preview.1.26309.5`. NLP is opt-in-on; Safety is opt-in-off. +- `Microsoft.Extensions.Hosting` and `Microsoft.Extensions.Configuration.*` must be `10.0.1` (not `10.0.0`) to satisfy the transitive constraint from `Microsoft.Agents.AI.Hosting`. Pinning `10.0.0` produces `NU1605`. -## `Abstractions.cs` (generated alongside Program.cs) - -```csharp -public interface IEvalRunner -{ - Task RunAsync(CancellationToken ct = default); -} - -public sealed record EvalReport( - bool Success, - string OneLineSummary, - string ReportDirectory); -``` - -The three runner classes (`TelemetryEvalRunner`, `QualityEvalRunner`, -`CompareEvalRunner`) each implement `IEvalRunner` and write their -report files under `ReportDirectory`. See `telemetry-capture.md`, -`quality-modes.md`, and `compare-mode.md` for each runner's body. - -## `Program.cs` - -```csharp -var mode = args.FirstOrDefault() ?? "telemetry"; -IEvalRunner runner = mode switch -{ - "telemetry" => new TelemetryEvalRunner(), - "quality" => new QualityEvalRunner(), - "compare" => new CompareEvalRunner(), - _ => throw new ArgumentException($"Unknown mode: {mode}") -}; - -var report = await runner.RunAsync(); -Console.WriteLine(report.OneLineSummary); -return report.Success ? 0 : 1; -``` - -## `Telemetry/inputs.json` (starter) - -```json -[ - { "id": "smoke-1", "user_message": "Say hi in five words." }, - { "id": "smoke-2", "user_message": "Plan a one-day trip to Seattle." }, - { "id": "smoke-3", "user_message": "Summarize the previous answer in 20 words." } -] -``` - -## `Telemetry/prices.json` (starter — user edits) +## `dotnet-tools.json` ```json { - "gpt-4o-mini": { "input_per_1k": 0.00015, "output_per_1k": 0.00060 }, - "gpt-4o": { "input_per_1k": 0.00250, "output_per_1k": 0.01000 }, - "o4-mini": { "input_per_1k": 0.00110, "output_per_1k": 0.00440 } + "version": 1, + "isRoot": true, + "tools": { + "microsoft.extensions.ai.evaluation.console": { + "version": "10.7.0", + "commands": ["aieval"], + "rollForward": false + } + } } ``` -## `Quality/rubric.md` (starter) - -```markdown -# Quality rubric - -Score each response on a 1-5 scale per trait. The judge model returns -a JSON object: `{ trait: score, ... }`. - -- **on_topic** — answers the user's actual question. -- **concise** — no filler; under 4 sentences unless asked otherwise. -- **safe** — no PII leakage, no instructions to violate policy. -- **format_correct** — Markdown / JSON shape matches the contract. - -Pass threshold (default): mean ≥ 4.0 AND no trait < 3. -``` - -## `Quality/golden.json` (starter) - -```json -[ - { - "id": "g1", - "user_message": "Plan a one-day trip to Seattle.", - "expected_traits": ["on_topic", "concise", "format_correct"] - } -] -``` +After scaffold: `dotnet tool restore` (the skill runs this automatically). -## `Compare/matrix.json` (starter) +## `GlobalUsings.cs` -```json -[ - { - "name": "baseline", - "model_assignments": { "router": "gpt-4o-mini", "planner": "gpt-4o", "worker": "gpt-4o-mini" } - }, - { - "name": "candidate", - "model_assignments": { "router": "gpt-4o-mini", "planner": "o4-mini", "worker": "gpt-4o-mini" } - } -] +```csharp +global using Microsoft.Extensions.AI; +global using Microsoft.Extensions.AI.Evaluation; +global using Microsoft.Extensions.AI.Evaluation.Reporting; +global using Microsoft.Extensions.AI.Evaluation.Reporting.Storage; +global using Microsoft.Extensions.AI.Evaluation.NLP; +global using Microsoft.Extensions.AI.Evaluation.Quality; +global using Microsoft.Extensions.Configuration; // AddUserSecrets in AgentChatClientFactory +global using Microsoft.Extensions.DependencyInjection; // GetRequiredService in AgentChatClientFactory +global using Microsoft.Extensions.Hosting; // Host.CreateApplicationBuilder +global using Microsoft.VisualStudio.TestTools.UnitTesting; ``` diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/quality-modes.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/quality-modes.md index 41f5981324..e9df0de7b9 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/quality-modes.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/quality-modes.md @@ -1,95 +1,253 @@ -# Quality modes — LLM-judge +# Quality mode -How quality mode runs the agent against `golden.json` and asks a judge -model to score each response per the rubric. +`Quality/QualityTests.cs` is the MSTest class that actually drives +`Microsoft.Extensions.AI.Evaluation.Reporting`. It's the **only** +runner that produces `report.html`. -## Runner +## Response cache (read first) + +`DiskBasedReportingConfiguration.Create(..., enableResponseCaching: true)` +wraps the supplied `IChatClient` with a content-addressable cache stored +under `_store/cache/`. The first `dotnet test` run populates it; every +subsequent run against the same scenarios is **near-instant with zero +LLM cost** because both the agent call and the judge call are served +from disk. + +Two rules MUST be followed to make the cache work: + +1. **The agent call must go through `run.ChatConfiguration!.ChatClient`** + (NOT `Wire.ResolveAgentClient()` directly). The run-scoped client is + the cached wrapper. Calling the factory directly bypasses the cache. +2. **Do not pass a per-run `executionName`** to `Create(...)`. The + `executionName` is part of the cache scope; a fresh timestamp per run + guarantees misses. Use a separate `EvalEnv.ReportFolder` value for + the report-output directory if you want per-run history. + +If both rules are followed, the only legitimate reasons to see Miss in +the report's Diagnostic Data section are: (a) the cache directory was +deleted, (b) the rubric / golden / scenario input changed, (c) the +judge model or chat options changed, or (d) it's the first run. + +## Pipeline + +``` +[ClassInitialize] → build ReportingConfig (tier-aware evaluator list) +[TestMethod] → one per golden.json entry + ├─ CreateScenarioRunAsync(scenarioName) + ├─ get cached IChatClient from run.ChatConfiguration + ├─ get agent response (cached) + ├─ build per-evaluator EvaluationContext (BLEU refs, F1 ground truth, ...) + └─ scenarioRun.EvaluateAsync(messages, response, contexts) // judge calls cached +[AssemblyCleanup] → dotnet tool run aieval report --path _store --output /report.html +``` + +## Reporting config (sketch) ```csharp -public sealed class QualityEvalRunner : IEvalRunner +// Reporting/ReportingConfig.cs +internal static class ReportingConfig { - public async Task RunAsync() - { - var rubric = await File.ReadAllTextAsync("Quality/rubric.md"); - var golden = await JsonSerializer.DeserializeAsync( - File.OpenRead("Quality/golden.json")); + public static readonly string StorageRoot = + Path.Combine(RepoRoot.Find(), "_store"); - var judge = new ChatClientBuilder() - .UseFunctionInvocation() - .Build(new ChatClient(model: Config.JudgeModel, apiKey: Config.JudgeApiKey)); - - var rows = new List(); - foreach (var g in golden!) + public static ReportingConfiguration ForQuality() + { + // ONE client serves both the agent call and the judge call. MEAI wraps + // it with the response cache when handed to ChatConfiguration, so the + // *agent* call gets cached too when QualityTests calls it through + // `run.ChatConfiguration!.ChatClient` (see "Test class" below). + // On re-runs against unchanged inputs, the entire run is a cache hit + // and finishes in seconds with zero LLM cost. Override the judge with + // a separate model via EVAL_JUDGE_DEPLOYMENT_NAME (advanced). + var agent = Wire.ResolveAgentClient(); + var judge = Wire.ResolveJudgeClient(agent); + + var evaluators = new List + { + // Tier 1 — always on, deterministic + new WordCountEvaluator(), + new BLEUEvaluator(), + new GLEUEvaluator(), + new F1Evaluator(), + }; + + if (EvalEnv.UseRealJudge) { - var actual = await Agent.RunAsync(g.UserMessage); - var verdict = await Judge(judge, rubric, g, actual); - rows.Add(new QualityRow(g.Id, verdict.Scores, verdict.PassFail, verdict.Rationale)); + // Tier 2 — needs real judge + evaluators.Add(new RelevanceEvaluator()); + evaluators.Add(new CoherenceEvaluator()); + evaluators.Add(new FluencyEvaluator()); + evaluators.Add(new CompletenessEvaluator()); + evaluators.Add(new EquivalenceEvaluator()); + evaluators.Add(new GroundednessEvaluator()); + + if (AgenticAppDetected) + { + evaluators.Add(new IntentResolutionEvaluator()); + evaluators.Add(new TaskAdherenceEvaluator()); + evaluators.Add(new ToolCallAccuracyEvaluator()); + } } - return EvalReport.FromQuality(rows); + // executionName: deliberately omitted. MEAI's default keeps the cache + // scope stable across runs so re-runs hit. The report folder + // timestamp lives separately in EvalEnv.ReportFolder. + return DiskBasedReportingConfiguration.Create( + storageRootPath: StorageRoot, + evaluators: evaluators, + chatConfiguration: new ChatConfiguration(judge), + enableResponseCaching: true); } } ``` -## Judge prompt skeleton +## Test class (sketch) -``` -You are a quality judge. Score the assistant response on each trait -in the rubric (1-5). Return ONLY a JSON object of the form: +```csharp +[TestClass] +public sealed class QualityTests +{ + private static ReportingConfiguration s_reporting = null!; -{ "scores": { "": , ... }, "rationale": "" } + [ClassInitialize] + public static void Init(TestContext _) => + s_reporting = ReportingConfig.ForQuality(); -Rubric: -{{ rubric_md }} + public static IEnumerable Golden() => + GoldenLoader.Load().Select(g => new object[] { g }); -User asked: -{{ user_message }} + [TestMethod, DynamicData(nameof(Golden), DynamicDataSourceType.Method)] + public async Task Evaluate(GoldenItem g) + { + var scenarioName = $"{nameof(QualityTests)}.{g.Id}"; + await using var run = await s_reporting.CreateScenarioRunAsync(scenarioName); + + // IMPORTANT: use the run's ChatClient, NOT Wire.ResolveAgentClient(), + // for the agent call. The run-scoped client is wrapped with MEAI's + // response cache (set up by ReportingConfig.ForQuality), so identical + // inputs across runs return cached responses instead of paying for + // a fresh LLM call. Calling AgentChatClientFactory directly bypasses + // the cache and guarantees a cache miss on every judge call too + // (judge cache key includes the agent response, which then varies). + // + // EDGE CASE: when EVAL_JUDGE_DEPLOYMENT_NAME splits judge from agent, + // run.ChatConfiguration.ChatClient IS the judge client — using it as + // the agent would silently call the wrong model. Fall back to the + // uncached agent factory in that case (judge calls still cache). + var agent = string.IsNullOrEmpty( + Environment.GetEnvironmentVariable("EVAL_JUDGE_DEPLOYMENT_NAME")) + ? run.ChatConfiguration!.ChatClient + : Wire.ResolveAgentClient(); + var messages = new List + { + new(ChatRole.System, RubricLoader.SystemPrompt()), + new(ChatRole.User, g.UserMessage), + }; + var response = await agent.GetResponseAsync(messages); -Assistant replied: -{{ actual_response }} + var contexts = new List(); + if (!string.IsNullOrEmpty(g.ReferenceResponse)) + { + contexts.Add(new BLEUEvaluatorContext(new[] { g.ReferenceResponse })); + contexts.Add(new GLEUEvaluatorContext(new[] { g.ReferenceResponse })); + contexts.Add(new F1EvaluatorContext(g.ReferenceResponse)); + contexts.Add(new EquivalenceEvaluatorContext(g.ReferenceResponse)); + contexts.Add(new CompletenessEvaluatorContext(g.ReferenceResponse)); + } + if (!string.IsNullOrEmpty(g.Context)) + contexts.Add(new GroundednessEvaluatorContext(g.Context)); -Required traits to score: -{{ expected_traits }} + var result = await run.EvaluateAsync(messages, response, contexts); + Thresholds.ApplyOrLog(result, g.Id); // hard_fail in JSON => Assert.Fail + } +} ``` -## Pass/fail - -`quality.thresholds.json` (optional, user-edited): +## Report generation -```json +```csharp +// Reporting/AievalReport.cs +[TestClass] +public static class AievalReport { - "mean_score_min": 4.0, - "per_trait_min": 3, - "fail_on_threshold_breach": false + [AssemblyCleanup] + public static void GenerateReport() + { + var outDir = Path.Combine( + RepoRoot.Find(), ".copilot", "perf-reports", "evals", + EvalEnv.ReportFolder); + Directory.CreateDirectory(outDir); + var html = Path.Combine(outDir, "report.html"); + + var psi = new ProcessStartInfo("dotnet", + $"tool run aieval report --path \"{ReportingConfig.StorageRoot}\" --output \"{html}\"") + { + WorkingDirectory = RepoRoot.Find(), + RedirectStandardOutput = true, + RedirectStandardError = true, + UseShellExecute = false, + CreateNoWindow = true, + }; + using var p = Process.Start(psi)!; + p.WaitForExit(); + TestContext.Out?.WriteLine($"Eval report: {html}"); + } } ``` -If `fail_on_threshold_breach: false` (default), the runner exits 0 -even on quality regressions and marks them in the report. Set to -`true` to gate CI. - -## Report — `quality.md` - -```markdown -# Quality — {{ utc_timestamp }} +## EvalEnv (sketch, in `Reporting/Tier.cs`) -Judge: {{ judge_model }} | Inputs: {{ count }} | Pass rate: {{ pct }} - -| Id | Mean | on_topic | concise | safe | format | Pass | Rationale | -|----|------|----------|---------|------|--------|------|---------------------------------| -| g1 | 4.5 | 5 | 4 | 5 | 4 | ✅ | Concise with clear sections. | -| g2 | 3.0 | 4 | 2 | 3 | 3 | ❌ | Rambling intro; over 6 sentences. | +```csharp +internal static class EvalEnv +{ + public static bool UseRealAgent => + Environment.GetEnvironmentVariable("EVAL_USE_REAL_AGENT") == "1"; + + public static bool UseRealJudge => + Environment.GetEnvironmentVariable("EVAL_USE_REAL_JUDGE") == "1"; + + public static bool UseFoundrySafety => + Environment.GetEnvironmentVariable("EVAL_USE_FOUNDRY_SAFETY") == "1"; + + public static string Tier => + UseFoundrySafety ? "Safety" : UseRealJudge ? "Judge" : "Stub"; + + // Per-run timestamp ONLY for the report output folder under + // .copilot/perf-reports/evals//. NOT passed to MEAI's + // ReportingConfiguration — that would scope the response cache per + // run and defeat caching. Override with EVAL_REPORT_FOLDER in CI to + // align with a build number. + public static readonly string ReportFolder = + Environment.GetEnvironmentVariable("EVAL_REPORT_FOLDER") + ?? DateTime.UtcNow.ToString("yyyyMMdd-HHmmss"); +} +``` -## Failures +## `golden.json` schema (v2) -### g2 (3.0) -- judge rationale: ... -- actual response (truncated): ... +```json +{ + "schema_version": 2, + "scenarios": [ + { + "id": "g-receptionist-greeting", + "user_message": "Hi, I'd like to start an interview.", + "reference_response": "Hello! I'd be happy to start an interview with you. What role are you preparing for?", + "context": null, + "expected_traits": ["on_topic", "safe", "format_correct"], + "expected_tool_calls": null + } + ] +} ``` -## Cost considerations +- `reference_response`: required for BLEU/GLEU/F1/Equivalence/Completeness. +- `context`: required for Groundedness. Free text providing the + source-of-truth context the response should be grounded in. +- `expected_traits`: free-form labels surfaced in the report rubric + view. Read by the LLM judge. +- `expected_tool_calls`: required for `ToolCallAccuracyEvaluator`. -Each quality run pays for: agent calls (real or stub) + judge calls -(always real). Expect roughly 1 judge call per input. Use a smaller -judge model (e.g. `gpt-4o-mini`) for early iterations and switch up -when stabilizing. +Migration from v1 (no `schema_version`, no `reference_response`): the +skill adds the fields as `null` so existing tests don't fail. NLP +evaluators emit `(no reference)` when null. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/safety-mode.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/safety-mode.md new file mode 100644 index 0000000000..2b38264d3b --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/safety-mode.md @@ -0,0 +1,89 @@ +# Safety mode (opt-in) + +Off by default. Enabled in step 2 of the workflow when the user picks +"include Safety tier" / says "wire safety" / equivalent. + +When enabled, the skill: + +1. Adds `Microsoft.Extensions.AI.Evaluation.Safety` (preview 10.7.0) to the csproj. +2. Generates `Safety/SafetyTests.cs` using `ContentHarmEvaluator` as the + default bundle (4 metrics in 1 Foundry call), plus + `ProtectedMaterialEvaluator`, `IndirectAttackEvaluator`, + `CodeVulnerabilityEvaluator`, `UngroundedAttributesEvaluator`, and + optionally `GroundednessProEvaluator`. +3. Adds the required Azure AI Foundry endpoint config keys to + `quality.thresholds.json` and surfaces them in the chat output. + +## Runtime gating + +Safety tests must **never** fail the build when Foundry creds are +missing — they're an opt-in capability. The pattern: + +```csharp +[TestClass] +public sealed class SafetyTests +{ + [ClassInitialize] + public static void Init(TestContext _) + { + if (!EvalEnv.UseFoundrySafety) + Assert.Inconclusive( + "Safety tier disabled. Set EVAL_USE_FOUNDRY_SAFETY=1 and " + + "AZURE_AI_FOUNDRY_ENDPOINT to enable."); + } + + public static IEnumerable Golden() => + GoldenLoader.Load().Select(g => new object[] { g }); + + [TestMethod, DynamicData(nameof(Golden), DynamicDataSourceType.Method)] + public async Task ContentHarm(GoldenItem g) + { + var reporting = ReportingConfig.ForSafety(); // separate config — Foundry chat client + await using var run = await reporting.CreateScenarioRunAsync($"Safety.ContentHarm.{g.Id}"); + var agent = Wire.ResolveAgentClient(); + var messages = new List { new(ChatRole.User, g.UserMessage) }; + var response = await agent.GetResponseAsync(messages); + await run.EvaluateAsync(messages, response); // ContentHarmEvaluator returns all 4 metrics + } + + // Repeat for ProtectedMaterial / IndirectAttack / CodeVulnerability / etc. +} +``` + +## Why `ContentHarmEvaluator` (not 4 separate) + +From the upstream docs: + +> ContentHarmEvaluator provides single-shot evaluation for the four +> metrics supported by HateAndUnfairnessEvaluator, SelfHarmEvaluator, +> ViolenceEvaluator, and SexualEvaluator. + +That's **1 Foundry call instead of 4** for the same metric set. Always +wire `ContentHarmEvaluator` unless the user has a strict reason to +isolate one harm category. + +## Config keys surfaced in chat + +When Safety tier is enabled, the skill output adds: + +``` +Safety tier enabled. To activate at runtime: + export EVAL_USE_FOUNDRY_SAFETY=1 + export AZURE_AI_FOUNDRY_ENDPOINT=https://.cognitiveservices.azure.com + az login --tenant # DefaultAzureCredential + +Safety tests are MARKED INCONCLUSIVE (not failed) when the env vars are unset, +so your default `dotnet test` run will not break. +``` + +## What Safety evaluators do **not** cover + +Document this explicitly in the rubric: safety evaluators are *output* +classifiers, not *input* classifiers. They do not protect the agent +from receiving harmful prompts — for that, use a separate input filter +(e.g., Azure AI Content Safety on the request side). + +`IndirectAttackEvaluator` is the closest to an input-side check; it +looks for prompt-injection-style content in the *response* that would +indicate the model picked up an indirect attack from retrieved content +or tool output. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/telemetry-capture.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/telemetry-capture.md index 1aa8148537..7c2ee3d44f 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/telemetry-capture.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/telemetry-capture.md @@ -1,83 +1,111 @@ -# Telemetry capture +# Telemetry mode -How telemetry mode wraps the existing `IChatClient` and writes per-call -records to disk. +Telemetry mode captures **latency, input tokens, output tokens, and +cost** per agent call across a fixed input set. It is **not** the same +as the MEAI eval report — it produces a separate cost/latency capture. -## Wrapper +## Why separate from quality + +Quality mode answers "is the response any good?" via +`Microsoft.Extensions.AI.Evaluation.Reporting` and produces +`report.html`. + +Telemetry mode answers "how much does it cost and how slow is it?" +via a delegating `IChatClient` wrapper. Different question, different +artifact. Conflating them is the #1 most common scaffolding mistake. + +## Artifacts (each in `.copilot/perf-reports/evals//`) + +- `telemetry.md` — human-readable per-input table. +- `telemetry.json` — machine-readable for CI scraping. +- `telemetry.junit.xml` — for test-result dashboards. + +These are **distinct** from `report.html` (which only quality mode +writes). The skill output should never refer to `telemetry.md` as +"the eval report." + +## Test shape ```csharp -public sealed class TelemetryChatClient(IChatClient inner, TelemetrySink sink) : IChatClient +[TestClass] +public sealed class TelemetryTests { - public async Task GetResponseAsync(IList messages, - ChatOptions? options = null, CancellationToken ct = default) + public static IEnumerable Inputs() => + InputsLoader.Load().Select(i => new object[] { i }); + + [TestMethod, DynamicData(nameof(Inputs), DynamicDataSourceType.Method)] + public async Task Capture(TelemetryInput input) { + var inner = Wire.ResolveAgentClient(); + var wrapped = new TelemetryCapturingChatClient(inner, PriceTable.Load()); + + var messages = new List { new(ChatRole.User, input.Text) }; var sw = Stopwatch.StartNew(); - var resp = await inner.GetResponseAsync(messages, options, ct); + var response = await wrapped.GetResponseAsync(messages); sw.Stop(); - sink.Record(new TelemetryRecord( - AgentName: options?.AdditionalProperties?["agent"] as string ?? "unknown", - Model: options?.ModelId ?? "unknown", - InputTokens: resp.Usage?.InputTokenCount ?? 0, - OutputTokens: resp.Usage?.OutputTokenCount ?? 0, + TelemetryStore.Record(new TelemetryRecord( + AgentName: input.Agent, + Model: response.ModelId ?? "unknown", + InputTokens: response.Usage?.InputTokenCount ?? 0, + OutputTokens: response.Usage?.OutputTokenCount ?? 0, LatencyMs: sw.ElapsedMilliseconds, - CostUsd: PriceTable.Cost(options?.ModelId, resp.Usage))); - - return resp; + CostUsd: wrapped.LastCostUsd)); } + + [ClassCleanup] + public static void WriteReports() => TelemetryStore.FlushTo( + Path.Combine(RepoRoot.Find(), ".copilot", "perf-reports", "evals", + ReportingConfig.ExecutionName)); } ``` -Register in DI as a decorator over the real client. +## Delegating client (sketch) + +```csharp +internal sealed class TelemetryCapturingChatClient(IChatClient inner, PriceTable prices) : IChatClient +{ + public decimal LastCostUsd { get; private set; } + + public async Task GetResponseAsync( + IEnumerable messages, ChatOptions? options = null, + CancellationToken cancellationToken = default) + { + var resp = await inner.GetResponseAsync(messages, options, cancellationToken); + var usage = resp.Usage; + LastCostUsd = prices.Cost(resp.ModelId, + usage?.InputTokenCount ?? 0, usage?.OutputTokenCount ?? 0); + return resp; + } -## Report — `telemetry.md` + // GetStreamingResponseAsync delegates similarly; usage parsed off the last update. -```markdown -# Telemetry — {{ utc_timestamp }} + public object? GetService(Type serviceType, object? serviceKey = null) => + inner.GetService(serviceType, serviceKey); -Inputs: {{ count }} Stub mode: {{ true | false }} + public void Dispose() => inner.Dispose(); +} +``` -| Agent | Model | Calls | Avg ms | p95 ms | Avg in tok | Avg out tok | Cost (USD) | -|-----------|--------------|-------|--------|--------|------------|-------------|------------| -| router | gpt-4o-mini | 12 | 340 | 520 | 180 | 22 | $0.00031 | -| planner | gpt-4o | 6 | 1240 | 1880 | 1100 | 260 | $0.00385 | -| worker | gpt-4o-mini | 18 | 410 | 640 | 320 | 180 | $0.00118 | +## `inputs.json` -Total cost: $0.00534 +```json +[ + { "agent": "receptionist", "text": "Hi" }, + { "agent": "behavioural", "text": "Tell me about a tough project" }, + { "agent": "technical", "text": "How would you throttle requests?" }, + { "agent": "summariser", "text": "Wrap up the interview" } +] ``` -## Machine-readable — `telemetry.json` +## `prices.json` ```json { - "timestamp": "2026-06-15T17:00:00Z", - "stub": false, - "records": [ - { "agent": "router", "model": "gpt-4o-mini", "input_tokens": 178, "output_tokens": 21, "latency_ms": 332, "cost_usd": 0.0000256 } - ], - "aggregate": { "calls": 36, "total_cost_usd": 0.00534 } + "gpt-4o-mini": { "input_per_1k": 0.00015, "output_per_1k": 0.0006 }, + "gpt-4o": { "input_per_1k": 0.0025, "output_per_1k": 0.01 }, + "o4-mini": { "input_per_1k": 0.003, "output_per_1k": 0.012 } } ``` -## JUnit-XML — `telemetry.junit.xml` - -Standard JUnit suite where each test case is one input id, marked -passed if the run succeeded (no thrown exception). Latency/token -metrics are emitted as `` per test case so CI can pick -them up. - -## Stub mode - -Two independent toggles control whether real models are called: - -- `EVAL_USE_REAL_AGENT` (default `0`) — when `0`, the wrapper - short-circuits the agent-under-test client and returns a deterministic - canned response. Telemetry numbers reflect the stub, marked `(stub)`. -- `EVAL_USE_REAL_JUDGE` (default `0`) — when `0`, quality mode skips - the real judge call and emits per-input scores of `null` with a - rationale of `"(stub) judge disabled"`. The pass-rate row reports - `(stub)`. - -Compare mode honors both toggles independently. Setting only -`EVAL_USE_REAL_AGENT=1` is a valid local-dev configuration: real -agent calls, no judge cost. +Edit freely — costs change. The price table is **never** baked into source. diff --git a/tests/dotnet-ai/agentic-perf-reviewer/eval.yaml b/tests/dotnet-ai/agentic-perf-reviewer/eval.yaml new file mode 100644 index 0000000000..647e1dc9ce --- /dev/null +++ b/tests/dotnet-ai/agentic-perf-reviewer/eval.yaml @@ -0,0 +1,105 @@ +name: agentic-perf-reviewer +required_agents: + - agentic-perf-reviewer + +# Routing-discrimination scenarios for the agent. +# These exercise whether the agent description matches the right prompts +# AND correctly punts non-agentic .NET perf questions to other agents. +# Static / no-fixture tests — the assertions are over which agent the +# host's description-matcher routes the prompt to. + +scenarios: + # ─── Should route TO agentic-perf-reviewer ───────────────────────── + + - name: prompt-mentions-agentic-app-and-slowness + prompt: | + My agentic .NET app feels slow during multi-agent handoffs. + Can you review it? + assertions: + - type: agent_invoked + name: agentic-perf-reviewer + + - name: prompt-mentions-aspire-foundry-perf-review + prompt: | + Review the perf of my Aspire + Foundry agent project at + ./MyApp for cost and latency issues. + assertions: + - type: agent_invoked + name: agentic-perf-reviewer + + - name: prompt-mentions-topology-and-model-selection + prompt: | + Audit my Microsoft Agent Framework topology and per-agent + model selection — I think we have too many handoffs. + assertions: + - type: agent_invoked + name: agentic-perf-reviewer + + - name: prompt-after-non-trivial-topology-change + prompt: | + I just added two new agents and an LLM-routed handoff edge. + Anything we should worry about before merging? + assertions: + - type: agent_invoked + name: agentic-perf-reviewer + + # ─── Should NOT route to agentic-perf-reviewer ───────────────────── + + - name: plain-dotnet-allocation-question-routes-to-perf-agent + prompt: | + My .NET service has high LOH allocations from string concatenation + in a hot loop. How do I fix? + assertions: + - type: agent_invoked + name: optimizing-dotnet-performance + # explicitly NOT agentic-perf-reviewer per the description's + # "Do NOT use for non-agentic .NET performance reviews" carve-out + + - name: linq-hot-path-question-routes-to-perf-agent + prompt: | + Replace this LINQ-heavy hot path with something faster. + assertions: + - type: agent_invoked + name: optimizing-dotnet-performance + + - name: async-anti-pattern-question-routes-to-perf-agent + prompt: | + I'm seeing thread-pool starvation. Where are my sync-over-async bugs? + assertions: + - type: agent_invoked + name: optimizing-dotnet-performance + + - name: non-dotnet-project-no-invocation + prompt: | + Review my Python LangChain agent app for perf issues. + assertions: + - type: agent_invoked + name: none + # the description scopes to .NET MAF + Aspire + Foundry; + # Python LangChain should not match. + + # ─── Should route to a child skill, NOT the agent ────────────────── + + - name: install-perf-rules-directly-routes-to-skill + prompt: | + Install the agentic perf rules into my project's copilot instructions. + assertions: + - type: skill_invoked + name: configure-agentic-perf-rules + # direct rules-install requests should hit the skill, not the + # umbrella agent. The agent's description says "review ... + + # orchestrates", but pure install requests skip the review pass. + + - name: pick-models-directly-routes-to-skill + prompt: | + Which model should each agent in my workflow use? + assertions: + - type: skill_invoked + name: select-agent-models + + - name: wire-evals-directly-routes-to-skill + prompt: | + Set up evals for my agent app. + assertions: + - type: skill_invoked + name: setup-maf-evals diff --git a/tests/dotnet-ai/configure-agentic-perf-rules/eval.yaml b/tests/dotnet-ai/configure-agentic-perf-rules/eval.yaml index cd99ffb476..2a23305c90 100644 --- a/tests/dotnet-ai/configure-agentic-perf-rules/eval.yaml +++ b/tests/dotnet-ai/configure-agentic-perf-rules/eval.yaml @@ -38,7 +38,7 @@ scenarios: value: "END: managed by configure-agentic-perf-rules" - type: "file_contains" path: ".github/copilot-instructions.md" - value: "agent_count_max" + value: "per_turn_input_token_warn" - type: "file_contains" path: ".github/copilot-instructions.md" value: "Agent count" @@ -113,14 +113,12 @@ scenarios: content: | # Project notes - + ## Agentic Performance Rules ```yaml thresholds: - agent_count_max: 3 - llm_routed_edges_max_per_turn: 2 per_turn_input_token_warn: 8000 per_turn_output_token_warn: 2000 baseline_token_increase_warn_pct: 20 @@ -133,7 +131,7 @@ scenarios: assertions: - type: "file_contains" path: ".github/copilot-instructions.md" - value: "v0.1.0" + value: "v0.3.0" - type: "exit_success" rubric: - "Detected the existing managed block at the current skill version" @@ -153,8 +151,6 @@ scenarios: ```yaml thresholds: - agent_count_max: 3 - llm_routed_edges_max_per_turn: 2 per_turn_input_token_warn: 25000 per_turn_output_token_warn: 2000 baseline_token_increase_warn_pct: 20 @@ -176,6 +172,44 @@ scenarios: - "Showed the user a diff or summary of what changed before applying" timeout: 360 + - name: "Update from v0.2.0 — drops deprecated agent_count_max and llm_routed_edges_max_per_turn with warning" + prompt: "Update the agentic-perf rules in this project to the latest version." + setup: + files: + - path: ".github/copilot-instructions.md" + content: | + + + ## Agentic Performance Rules + + ```yaml + thresholds: + agent_count_max: 5 + llm_routed_edges_max_per_turn: 3 + per_turn_input_token_warn: 12000 + per_turn_output_token_warn: 2000 + baseline_token_increase_warn_pct: 20 + unbounded_history_warn: true + ``` + + (older content) + + + assertions: + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "per_turn_input_token_warn: 12000" + - type: "exit_success" + rubric: + - "Detected the existing managed block at v0.2.0" + - "Replaced the block with the current-version template (v0.3.0+)" + - "Preserved per_turn_input_token_warn: 12000 (user override) — did NOT reset to default" + - "Dropped the deprecated `agent_count_max` key from the rendered YAML (no longer a threshold in current schema)" + - "Dropped the deprecated `llm_routed_edges_max_per_turn` key from the rendered YAML" + - "Emitted a chat warning naming each dropped key (`agent_count_max`, `llm_routed_edges_max_per_turn`) so the user has an audit trail" + - "Did NOT silently retain the dropped keys in the new managed block" + timeout: 360 + - name: "AGENTS.md stub when both instructions files exist" prompt: "Install the agentic-perf rules. The project uses both AGENTS.md and .github/copilot-instructions.md." setup: diff --git a/tests/dotnet-ai/setup-maf-evals/eval.yaml b/tests/dotnet-ai/setup-maf-evals/eval.yaml index f2a6704d22..9a985f5b84 100644 --- a/tests/dotnet-ai/setup-maf-evals/eval.yaml +++ b/tests/dotnet-ai/setup-maf-evals/eval.yaml @@ -3,35 +3,179 @@ required_skills: - setup-maf-evals scenarios: - - name: scaffold-evals-project-fresh + - name: scaffold-evals-tests-project-fresh prompt: | - Run setup-maf-evals on the project at ./fixture. Wire all three modes - (telemetry, quality, compare). Skip the Aspire panel. + Run setup-maf-evals on the project at ./fixture. Wire telemetry, + quality, and compare modes. Skip Safety, Aspire panel, and CI workflow. setup: files: - - path: fixture/MyApp.sln + - path: fixture/MyApp.slnx content: | - Microsoft Visual Studio Solution File, Format Version 12.00 + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj content: | net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: | + var builder = DistributedApplication.CreateBuilder(args); + builder.AddAzureOpenAIChatClient("chat", "gpt-4o-mini"); + builder.Build().Run(); - path: fixture/MyApp.Coach/MyApp.Coach.csproj content: | net10.0 assertions: - type: file_exists - path: fixture/MyApp.Evals/MyApp.Evals.csproj + path: fixture/MyApp.Evals.Tests/MyApp.Evals.Tests.csproj + - type: file_exists + path: fixture/MyApp.Evals.Tests/dotnet-tools.json + - type: file_exists + path: fixture/MyApp.Evals.Tests/Reporting/ReportingConfig.cs - type: file_exists - path: fixture/MyApp.Evals/Telemetry/inputs.json + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs - type: file_exists - path: fixture/MyApp.Evals/Quality/rubric.md + path: fixture/MyApp.Evals.Tests/Quality/QualityTests.cs - type: file_exists - path: fixture/MyApp.Evals/Quality/golden.json + path: fixture/MyApp.Evals.Tests/Quality/golden.json - type: file_exists - path: fixture/MyApp.Evals/Compare/matrix.json + path: fixture/MyApp.Evals.Tests/Telemetry/TelemetryTests.cs + - type: file_exists + path: fixture/MyApp.Evals.Tests/Compare/CompareTests.cs + - type: file_contains + path: fixture/MyApp.Evals.Tests/MyApp.Evals.Tests.csproj + text: "Microsoft.Extensions.AI.Evaluation.Reporting" + - type: file_contains + path: fixture/MyApp.Evals.Tests/MyApp.Evals.Tests.csproj + text: "Microsoft.Extensions.AI.Evaluation.NLP" + - type: file_contains + path: fixture/MyApp.Evals.Tests/MyApp.Evals.Tests.csproj + text: "MSTest" + - type: file_contains + path: fixture/MyApp.Evals.Tests/dotnet-tools.json + text: "microsoft.extensions.ai.evaluation.console" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Quality/QualityTests.cs + text: "DiskBasedReportingConfiguration" - type: file_contains - path: fixture/MyApp.Evals/MyApp.Evals.csproj - text: "Microsoft.Extensions.AI.Evaluation" + path: fixture/MyApp.Evals.Tests/Quality/QualityTests.cs + text: "BLEUEvaluator" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Quality/golden.json + text: "reference_response" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Quality/golden.json + text: "schema_version" + - type: file_contains + path: fixture/.gitignore + text: ".copilot/perf-reports/evals/" + - type: file_contains + path: fixture/.gitignore + text: "_store/" + + - name: scaffold-with-safety-tier + prompt: | + Run setup-maf-evals on the project at ./fixture. Include the Safety tier. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: | + var builder = DistributedApplication.CreateBuilder(args); + builder.AddAzureOpenAIChatClient("chat", "gpt-4o-mini"); + assertions: + - type: file_exists + path: fixture/MyApp.Evals.Tests/Safety/SafetyTests.cs + - type: file_contains + path: fixture/MyApp.Evals.Tests/Safety/SafetyTests.cs + text: "ContentHarmEvaluator" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Safety/SafetyTests.cs + text: "Assert.Inconclusive" + - type: file_contains + path: fixture/MyApp.Evals.Tests/MyApp.Evals.Tests.csproj + text: "Microsoft.Extensions.AI.Evaluation.Safety" + + - name: scaffold-with-ci-workflow + prompt: | + Run setup-maf-evals on the project at ./fixture. Include the GitHub Actions workflow. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: "var b = DistributedApplication.CreateBuilder(args); b.AddAzureOpenAIChatClient(\"chat\", \"gpt-4o-mini\");" + assertions: + - type: file_exists + path: fixture/.github/workflows/evals.yml + - type: file_contains + path: fixture/.github/workflows/evals.yml + text: "aieval report" + - type: file_contains + path: fixture/.github/workflows/evals.yml + text: "upload-artifact" + + - name: ichatclient-detection-azure-openai + prompt: | + Run setup-maf-evals on the project at ./fixture. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: | + var builder = DistributedApplication.CreateBuilder(args); + builder.AddAzureOpenAIChatClient("chat", "gpt-4o-mini"); + assertions: + - type: file_exists + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs + - type: file_contains + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs + text: "AddAzureOpenAIChatClient" + - type: output_contains + text: "AppHost.cs" + + - name: ichatclient-detection-missing-emits-stub + prompt: | + Run setup-maf-evals on the project at ./fixture. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: "var b = DistributedApplication.CreateBuilder(args); b.Build().Run();" + assertions: + - type: file_contains + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs + text: "NotImplementedException" + - type: output_contains + text: "no registration found" - name: skip-when-no-app-host prompt: | @@ -45,42 +189,171 @@ scenarios: - type: output_contains text: "agentic" - - name: telemetry-stub-run-produces-report + - name: update-mode-preserves-user-data prompt: | - Run setup-maf-evals on the project at ./fixture, then run - "dotnet run --project MyApp.Evals -- telemetry" with EVAL_USE_REAL_MODELS unset. + Run setup-maf-evals on the project at ./fixture. A MyApp.Evals.Tests project + already exists with a custom rubric and golden.json. Do not overwrite either. setup: files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj content: | net10.0 - - path: fixture/MyApp.Coach/MyApp.Coach.csproj + - path: fixture/MyApp.Evals.Tests/MyApp.Evals.Tests.csproj content: | net10.0 + - path: fixture/MyApp.Evals.Tests/Quality/rubric.md + content: | + # Custom rubric — DO NOT OVERWRITE + - my_trait: ... + - path: fixture/MyApp.Evals.Tests/Quality/golden.json + content: | + [{"id": "custom", "user_message": "preserve me", "expected_traits": ["unique"]}] assertions: - type: file_contains - path: fixture/.gitignore - text: ".copilot/perf-reports/evals/" + path: fixture/MyApp.Evals.Tests/Quality/rubric.md + text: "DO NOT OVERWRITE" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Quality/golden.json + text: "preserve me" + + - name: tier-banner-surfaces-in-chat-output + prompt: | + Run setup-maf-evals on the project at ./fixture. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: "var b = DistributedApplication.CreateBuilder(args); b.AddAzureOpenAIChatClient(\"chat\", \"gpt-4o-mini\");" + assertions: + - type: output_contains + text: "EVAL_USE_REAL_AGENT" + - type: output_contains + text: "EVAL_USE_REAL_JUDGE" + - type: output_contains + text: "EVAL_USE_FOUNDRY_SAFETY" + + - name: scaffold-emits-metrics-glossary + prompt: | + Run setup-maf-evals on the project at ./fixture. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: "var b = DistributedApplication.CreateBuilder(args); b.AddAzureOpenAIChatClient(\"chat\", \"gpt-4o-mini\");" + assertions: + - type: file_exists + path: fixture/MyApp.Evals.Tests/Reporting/MetricsGlossary.cs + - type: file_contains + path: fixture/MyApp.Evals.Tests/Reporting/MetricsGlossary.cs + text: "metrics-glossary.md" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Reporting/MetricsGlossary.cs + text: "[AssemblyCleanup]" - type: output_contains - text: "stub" + text: "metrics-glossary.md" - - name: update-mode-preserves-user-edits + - name: factory-emits-friendly-secrets-diagnostic prompt: | - Run setup-maf-evals on the project at ./fixture. A MyApp.Evals project - already exists with a custom rubric. Do not overwrite it. + Run setup-maf-evals on the project at ./fixture. setup: files: + - path: fixture/MyApp.slnx + content: | + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj content: | net10.0 - - path: fixture/MyApp.Evals/MyApp.Evals.csproj + - path: fixture/MyApp.AppHost/AppHost.cs + content: | + var builder = DistributedApplication.CreateBuilder(args); + builder.AddAzureChatCompletionsClient("chat").AddChatClient("chat"); + assertions: + - type: file_contains + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs + text: "dotnet user-secrets" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs + text: "ConnectionStrings" + + - name: factory-loads-user-secrets-explicitly + prompt: | + Run setup-maf-evals on the project at ./fixture. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj content: | net10.0 - - path: fixture/MyApp.Evals/Quality/rubric.md + - path: fixture/MyApp.AppHost/AppHost.cs content: | - # Custom rubric — DO NOT OVERWRITE - - my_trait: ... + var builder = DistributedApplication.CreateBuilder(args); + builder.AddAzureChatCompletionsClient("chat").AddChatClient("chat"); assertions: + # Without explicit AddUserSecrets, dotnet test never reads the user-secrets store + # because the entry assembly is testhost.exe, not the test project. - type: file_contains - path: fixture/MyApp.Evals/Quality/rubric.md - text: "DO NOT OVERWRITE" + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs + text: "AddUserSecrets" + + - name: pitfalls-doc-warns-reasoning-models-reject-max-tokens + prompt: | + Read references/common-pitfalls.md from the setup-maf-evals skill. + assertions: + - type: file_contains + path: ../../../plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md + text: "max_tokens" + - type: file_contains + path: ../../../plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md + text: "EVAL_JUDGE_DEPLOYMENT_NAME" + + - name: pitfalls-doc-warns-stylistic-agents-fail-completeness + prompt: | + Read references/common-pitfalls.md and references/evaluators-catalog.md from setup-maf-evals. + assertions: + - type: file_contains + path: ../../../plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md + text: "Tuning Quality for stylistic agents" + - type: file_contains + path: ../../../plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md + text: "CompletenessEvaluator" + - type: file_contains + path: ../../../plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md + text: "RubricEvaluator" + + - name: compare-mode-is-opt-in-not-default + prompt: | + Read SKILL.md for setup-maf-evals; what's the default for compare mode? + assertions: + - type: file_contains + path: ../../../plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md + text: "Wire compare mode (opt-in)" + - type: file_contains + path: ../../../plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md + text: "Compare mode is opt-in" +