From d44b42f14c2a34e509993abf896366f79ecc875c Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Wed, 17 Jun 2026 15:31:38 -0700 Subject: [PATCH 01/18] setup-maf-evals: v2 spec rewrite (MSTest shape, 3 tiers, IChatClient detection) Rewrites SKILL.md and references to specify the v2 overhaul: - Replaces hand-rolled markdown runner with a Microsoft.Extensions.AI.Evaluation.Reporting + `aieval` HTML report pipeline (GA 10.7.0). - Switches the scaffolded project from a console runner to an MSTest `.Evals.Tests` project matching the canonical Learn-docs pattern. - Categorizes evaluators into three independent tiers (NLP / Quality / Safety) with separate env knobs (EVAL_USE_REAL_AGENT, EVAL_USE_REAL_JUDGE, EVAL_USE_FOUNDRY_SAFETY). - Adds IChatClient auto-detection so generated AgentChatClientFactory wires to the app's existing chat client registration. - Adds opt-in Safety tier (ContentHarmEvaluator via Azure AI Foundry) and opt-in GitHub Actions workflow. - New eval.yaml has 8 scenarios covering scaffold, update-mode preservation, IChatClient detection, reporting wiring, schema v2, CI workflow, safety opt-in. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../dotnet-ai/skills/setup-maf-evals/SKILL.md | 255 ++++++++++++------ .../setup-maf-evals/references/ci-workflow.md | 119 ++++++++ .../references/compare-mode.md | 154 ++++++----- .../references/dotnet-tools-manifest.md | 50 ++++ .../references/evaluators-catalog.md | 106 ++++++++ .../references/ichatclient-detection.md | 114 ++++++++ .../references/project-template.md | 222 +++++++-------- .../references/quality-modes.md | 214 ++++++++++----- .../setup-maf-evals/references/safety-mode.md | 89 ++++++ .../references/telemetry-capture.md | 140 ++++++---- tests/dotnet-ai/setup-maf-evals/eval.yaml | 222 ++++++++++++--- 11 files changed, 1260 insertions(+), 425 deletions(-) create mode 100644 plugins/dotnet-ai/skills/setup-maf-evals/references/ci-workflow.md create mode 100644 plugins/dotnet-ai/skills/setup-maf-evals/references/dotnet-tools-manifest.md create mode 100644 plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md create mode 100644 plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md create mode 100644 plugins/dotnet-ai/skills/setup-maf-evals/references/safety-mode.md diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md b/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md index 0ada6890fe..767a3d802d 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md @@ -1,13 +1,15 @@ --- name: setup-maf-evals description: | - Scaffold a Microsoft.Extensions.AI.Evaluation project alongside an existing .NET agentic application (MAF + Aspire + Foundry) so the team can measure latency, token usage, cost, and answer quality on every change. Creates an .Evals project with three modes: telemetry (per-call latency / input-tokens / output-tokens / cost), quality (LLM-as-judge against a rubric and golden conversations), and compare (run two model assignments side by side and produce a delta). Outputs Markdown + JSON + JUnit-XML reports under .copilot/perf-reports/evals//. Optionally wires an Aspire-dashboard panel showing per-agent token/latency live during dev. WHEN: user asks "how do I measure my agent perf", "set up evals", "add evaluation harness", "I changed models and need to validate quality", "compare gpt-4o vs gpt-4o-mini for my planner". NOT-WHEN: user wants a one-shot audit (scan-agentic-app-perf), install rules (configure-agentic-perf-rules), or pick models (select-agent-models). + Scaffold an `.Evals.Tests` MSTest project alongside a .NET agentic app (MAF + Aspire + Foundry) wired to the GA `Microsoft.Extensions.AI.Evaluation.Reporting` pipeline. Three evaluator categories: **NLP** (deterministic BLEU/GLEU/F1, no API key), **Quality** (LLM-as-judge Relevance/Coherence/Fluency, etc.), **Safety** (Hate/Violence/SelfHarm/Sexual via Azure AI Foundry). Auto-installs the `aieval` dotnet tool, detects the app's `IChatClient` registration and generates a factory so `EVAL_USE_REAL_AGENT=1` works without manual wiring, and emits an HTML report at `.copilot/perf-reports/evals//report.html`. Optional GitHub Actions workflow runs the evals on every PR. WHEN: user asks "set up evals", "add evaluation harness", "measure my agent perf", "validate quality after a model change", "compare gpt-4o vs gpt-4o-mini", "add safety evaluators", "generate eval report". NOT-WHEN: one-shot audit (use scan-agentic-app-perf), install rules (configure-agentic-perf-rules), pick models (select-agent-models). --- # setup-maf-evals -Scaffold a `.Evals` project that measures latency, token usage, -cost, and quality on every change to a .NET agentic app. +Scaffold an `.Evals.Tests` MSTest project that measures latency, +token usage, cost, quality, and safety on every change to a .NET +agentic app — and produces the proper Microsoft.Extensions.AI.Evaluation +HTML report by default. ## Workflow @@ -19,100 +21,152 @@ Detect: - AppHost project name (`*.AppHost.csproj`) - Agent service projects - Existing test / eval projects (avoid clobbering) +- **`IChatClient` registration** (scan AppHost + agent projects for + `AddChatClient` / `AddAzureOpenAIChatClient` / `AddOllamaChatClient` + / `AddOpenAIChatClient` and any explicit `services.AddSingleton` or + Foundry deployment alias references). See `references/ichatclient-detection.md`. -If no agentic app is detected, abort. If a `*.Evals` project already +If no agentic app is detected, abort. If a `*.Evals.Tests` project already exists, switch to update mode (see step 1a). -### 1a. Update mode (when `*.Evals` project already exists) +### 1a. Update mode (when `*.Evals.Tests` project already exists) File classes: | Class | Files | Behavior on update | |------------------|------------------------------------------------------------------------|----------------------------| -| **infra** | `*.Evals.csproj`, `Program.cs`, runner classes, `Abstractions.cs` | merge package refs; create file if missing; do **not** overwrite | -| **user data** | `Quality/rubric.md`, `Quality/golden.json`, `Compare/matrix.json`, `Telemetry/inputs.json`, `Telemetry/prices.json`, `quality.thresholds.json` | never overwrite; create if missing | -| **generated** | `Reports/`, `.copilot/perf-reports/evals/` | regenerate freely | +| **infra** | `*.Evals.Tests.csproj`, `dotnet-tools.json`, `Reporting/*`, `Wire/*`, test class skeletons | merge package refs; create file if missing; do **not** overwrite | +| **user data** | `Quality/rubric.md`, `Quality/golden.json`, `Compare/matrix.json`, `Telemetry/inputs.json`, `Telemetry/prices.json`, `quality.thresholds.json`, `.github/workflows/evals.yml` | never overwrite; create if missing | +| **generated** | `.copilot/perf-reports/evals/` | regenerate freely | If an existing infra file differs from the current template, surface the diff in the chat output but do not overwrite. Recommend the user review and merge manually. +`golden.json` has a `schema_version` field. If the detected version is +older than the current template, offer to migrate (additive only — +preserves existing rows, adds the new fields as nullable). + ### 2. Confirm scope with the user -Ask which modes to wire (default: all three): +Present the detection summary, then confirm: -- **telemetry** — capture latency, input-tokens, output-tokens, cost - per agent call across a fixed input set. -- **quality** — LLM-as-judge against a rubric and golden conversations. -- **compare** — run mode A vs mode B and emit a side-by-side delta. +1. **Project shape** (default: MSTest). Alternative: console runner + (legacy v1 shape) — only emit if user explicitly asks for it. +2. **Evaluator tiers to enable.** Defaults shown; user can override. -Optional: + | Tier | Evaluators | Cost | Needs | + |------|-----------|------|-------| + | 1 — NLP (default ON) | BLEU, GLEU, F1, Words | free | reference responses in golden.json | + | 2 — Quality (default ON, but stubbed) | Relevance, Coherence, Fluency, Completeness, Equivalence, Groundedness; agent: IntentResolution, TaskAdherence, ToolCallAccuracy | per-call judge tokens | real `IChatClient` + `EVAL_USE_REAL_JUDGE=1` | + | 3 — Safety (default OFF) | `ContentHarmEvaluator` (Hate+SelfHarm+Violence+Sexual single-shot), ProtectedMaterial, IndirectAttack, CodeVulnerability, UngroundedAttributes, GroundednessPro | Foundry evaluation service charges | Azure AI Foundry endpoint + `EVAL_USE_FOUNDRY_SAFETY=1` | -- **aspire-panel** — add a static-file-based dashboard panel showing - per-agent token/latency live during `dotnet run`. +3. **IChatClient detection result.** Show what was detected (e.g., "Found + `AddAzureOpenAIChatClient` in `AppHost.cs:41` with deployment alias `chat`"). + Ask the user to confirm or override. If detection failed, generate a + stub factory the user will fill in. +4. **Run modes to scaffold** (telemetry / quality / compare). Default: all three. +5. **Optional add-ons:** Aspire dashboard panel, GitHub Actions workflow. ### 3. Scaffold the project Use `references/project-template.md`. Creates: ``` -.Evals/ - .Evals.csproj # Microsoft.Extensions.AI.Evaluation refs +.Evals.Tests/ + .Evals.Tests.csproj # MSTest + MEAI.Evaluation refs (Reporting, NLP, Quality, optional Safety) + Reporting/ + ReportingConfig.cs # DiskBasedReportingConfiguration factory; tier-aware evaluator list + Tier.cs # EVAL_USE_REAL_AGENT / EVAL_USE_REAL_JUDGE / EVAL_USE_FOUNDRY_SAFETY enum + Wire/ + AgentChatClientFactory.cs # auto-generated from IChatClient detection (step 1) + StubChatClient.cs # used when EVAL_USE_REAL_AGENT is unset Telemetry/ - TelemetryEvalRunner.cs - inputs.json # 5 starter inputs the user customizes + TelemetryTests.cs # [TestMethod] per input + inputs.json + prices.json Quality/ - QualityEvalRunner.cs - rubric.md # the LLM-judge rubric - golden.json # golden conversations + QualityTests.cs # [TestMethod] per golden scenario; NLP always, Quality when judge on + rubric.md + golden.json # schema_version, user_message, reference_response, expected_traits, optional context, optional expected_tool_calls Compare/ - CompareEvalRunner.cs - matrix.json # model assignments to compare - Reports/ # generated, gitignored - Program.cs # CLI: dotnet run -- - Directory.Build.props # version pins + CompareTests.cs # [DataRow] per matrix entry; distinct executionName per entry + matrix.json + Safety/ # only emitted if user opted in + SafetyTests.cs # ContentHarmEvaluator + ProtectedMaterial + IndirectAttack + quality.thresholds.json # per-metric (Relevance / Coherence / BLEU / ...) -> minimum EvaluationRating + GlobalUsings.cs + dotnet-tools.json # NEW — pins aieval (Microsoft.Extensions.AI.Evaluation.Console, GA) + .github/workflows/evals.yml # OPTIONAL — only if user opted in ``` -Add the project to the solution. Add `Reports/` and -`.copilot/perf-reports/evals/` to the repo `.gitignore`. +After writing files: + +1. `dotnet sln add .Evals.Tests/.Evals.Tests.csproj` +2. Update `.gitignore`: append `.copilot/perf-reports/evals/` and `.Evals.Tests/_store/` if missing. +3. `dotnet tool restore` (installs `aieval`). ### 4. Wire telemetry mode See `references/telemetry-capture.md`. -- Hooks into the existing `IChatClient` via a delegating wrapper that - records `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, - per-call latency, and a price-table-driven cost estimate. +- `TelemetryTests` hooks into the resolved `IChatClient` via a + delegating wrapper that records `gen_ai.usage.input_tokens`, + `gen_ai.usage.output_tokens`, per-call latency, and a + price-table-driven cost estimate. - Emits a Markdown report at `.copilot/perf-reports/evals//telemetry.md`, a machine-readable `telemetry.json`, and a `telemetry.junit.xml`. +- **Note:** Telemetry mode is *not* an MEAI eval report. It's a + cost/latency capture. The HTML report comes from quality mode. ### 5. Wire quality mode See `references/quality-modes.md`. -- LLM-judge configurable model id (default: `gpt-4o`). -- Rubric is a Markdown file the user edits. -- Golden conversations: array of `{ input, expected_traits[] }`. -- Emits per-input score, aggregate pass rate, top failures with - judge rationale. +- `QualityTests` uses `DiskBasedReportingConfiguration` + + `ScenarioRun.EvaluateAsync` — the actual MEAI reporting pipeline. +- Stub tier registers `WordCountEvaluator`, `BLEUEvaluator`, + `GLEUEvaluator`, `F1Evaluator`. Judge tier adds the LLM-judge + evaluators listed in step 2. +- Each evaluator gets its required `EvaluationContext` from + `golden.json` (`BLEUEvaluatorContext(references)`, + `F1EvaluatorContext(groundTruth)`, etc.). +- After all `[TestMethod]`s run, an `[AssemblyCleanup]` invokes + `dotnet tool run aieval report --path _store --output .copilot/perf-reports/evals//report.html`. ### 6. Wire compare mode See `references/compare-mode.md`. -- Reads `matrix.json`: a list of `{ name, model_assignments }` entries. -- Runs telemetry + quality for each. -- Produces `compare.md` with side-by-side latency / token / cost / - quality columns and a recommendation row. +- `CompareTests` uses `[DynamicData]` to feed each `matrix.json` entry + to a single test method. +- Each entry gets its own `executionName` so `aieval report` aggregates + the comparison view automatically. +- Produces `compare.md` (side-by-side latency / token / cost / quality + per matrix entry) **in addition to** the unified HTML report. + +### 7. Wire safety mode (opt-in) -### 7. Optional Aspire panel (apply mode) +See `references/safety-mode.md`. -See `references/aspire-dashboard-panel.md`. +Off by default. When enabled in step 2: -This step is **off by default** and modifies the AppHost project. To -enable it, the user must say "wire the panel" / "add the dashboard -panel" / equivalent. +- Adds `Microsoft.Extensions.AI.Evaluation.Safety` package. +- Generates `SafetyTests` using `ContentHarmEvaluator` (covers Hate + + SelfHarm + Violence + Sexual in one Foundry call), plus + `ProtectedMaterialEvaluator`, `IndirectAttackEvaluator`, + `CodeVulnerabilityEvaluator`, `UngroundedAttributesEvaluator`, and + optionally `GroundednessProEvaluator`. +- Skipped at runtime via `Assert.Inconclusive` if + `EVAL_USE_FOUNDRY_SAFETY` is unset — never fails the build for + missing creds. + +### 8. Optional Aspire panel (apply mode) + +See `references/aspire-dashboard-panel.md`. Off by default; modifies +the AppHost project. To enable, user must say "wire the panel" / +equivalent. When enabled: @@ -122,47 +176,90 @@ When enabled: 3. Only on `yes` / explicit confirmation, write the changes. 4. Run `dotnet build` and report pass/fail. -If declined, scaffold the static files into `.Evals/Panel/` so the -user can move them into the AppHost manually. +If declined, scaffold the static files into `.Evals.Tests/Panel/` +so the user can move them into the AppHost manually. + +### 9. Optional CI workflow (opt-in) + +See `references/ci-workflow.md`. Off by default. + +When enabled, emits `.github/workflows/evals.yml`: + +- Runs `dotnet test` on every PR. +- Checks for repo secrets `AZURE_OPENAI_ENDPOINT` and `AZURE_TENANT_ID`; + if present, sets `EVAL_USE_REAL_JUDGE=1`. Otherwise stub tier. +- Runs `dotnet tool run aieval report` and uploads `report.html` as a + build artifact (PR comment optional). + +### 10. Validation + +- `dotnet build .Evals.Tests.csproj` exits 0. +- `dotnet test .Evals.Tests.csproj` exits 0 in stub tier (no creds needed). + - Stub tier must emit a `report.html` with **≥ 4 distinct metric columns** + (Words, BLEU, GLEU, F1) across all scenarios in golden.json. + - All scenarios must produce non-null metric values (no "—" placeholders). +- If `EVAL_USE_REAL_JUDGE=1` and an `IChatClient` is wired, + `dotnet test` must additionally produce ≥ 3 Quality metrics (Relevance, + Coherence, Fluency). -### 8. Validation +### 11. Surface in chat -- `dotnet build .Evals.csproj` exits 0. -- `dotnet run --project .Evals -- telemetry` runs against a - smoke input and writes a Reports/ file (uses a stub client if no - API key is configured; emits `(stub)` in the report). -- All three runner classes have unit-level smoke tests under - `.Evals.Tests/`. +Print a 3-block summary: -### 9. Surface in chat +1. **Tier banner.** Which tier is active (Stub / Judge / Foundry-Safety) + and the exact env-var commands to upgrade. +2. **Paths.** Project path, HTML report path, persistent `_store/` path. +3. **CLI invocations.** `dotnet test`, `dotnet tool run aieval report`, + and the IChatClient detection result so the user knows what was + auto-wired. +4. **Follow-up recommendation.** "Re-run after applying a + `select-agent-models` recommendation to confirm no quality + regression." -- The path to the new project. -- The CLI invocations: `dotnet run -- telemetry`, `-- quality`, - `-- compare`. -- The reports folder. -- Recommend a follow-up: "Re-run after applying a `select-agent-models` - recommendation to confirm no quality regression." +Also link `references/evaluators-catalog.md` so the user can see what +each metric means. ## Common pitfalls -- **Calling real models from the smoke test.** The default smoke run - uses a stub `IChatClient`; the report is clearly marked `(stub)`. - Real-model runs are opt-in (env var `EVAL_USE_REAL_MODELS=1`). -- **Hard-coding a price table.** The price table lives in - `Telemetry/prices.json` and is user-editable. -- **Conflating telemetry and quality.** Telemetry never reads the - conversation content; quality never reads token counts. Keep them - in separate runners and reports. +- **Hand-rolling reports instead of using the Reporting pipeline.** + The whole point of GA 10.7.0 is `DiskBasedReportingConfiguration` + + `aieval`. Never write a hand-rolled markdown report and call it the + "quality report" — that's an MEAI report (HTML) vs a cost/latency + capture (markdown). +- **Calling real models from the default test run.** Stub tier uses + `StubChatClient`; report is clearly marked `(stub IChatClient)`. + Real-model runs are opt-in via the three env vars. +- **Conflating agent and judge clients.** They're two different + `IChatClient` roles. The skill exposes them as two independent env + vars (`EVAL_USE_REAL_AGENT`, `EVAL_USE_REAL_JUDGE`) — one can be + real while the other is stubbed. +- **Hard-coding a price table.** Lives in `Telemetry/prices.json`, + user-editable. +- **Wiring 4 separate safety evaluators.** Use `ContentHarmEvaluator` + for the Hate/SelfHarm/Violence/Sexual bundle — single Foundry call, + 4 metrics back. - **Auto-failing the build on quality regressions.** Quality mode is - informational by default. The user explicitly opts into a hard-fail - threshold by editing `quality.thresholds.json`. -- **Forgetting the `.gitignore` entry.** Reports must not pollute - source history. + informational by default. Users opt into a hard-fail by editing + `quality.thresholds.json` (which maps to real MEAI metric names like + `Relevance` / `BLEU` / `F1` and `EvaluationRating` levels). +- **Forgetting `.gitignore` entries.** Must include both + `.copilot/perf-reports/evals/` and `.Evals.Tests/_store/`. +- **Treating telemetry/compare/quality as separate report streams.** + Compare mode goes through `ReportingConfiguration` with a distinct + `executionName` per matrix entry, so `aieval report` aggregates them + into the same HTML. ## References -- `references/project-template.md` — exact files and `.csproj` layout. -- `references/telemetry-capture.md` — per-call hook + report format. -- `references/quality-modes.md` — LLM-judge rubric + golden conv format. -- `references/compare-mode.md` — matrix.json layout + delta report. +- `references/project-template.md` — exact files and `.csproj` layout (MSTest shape). +- `references/ichatclient-detection.md` — how to scan AppHost + agent for `IChatClient` registration and emit `AgentChatClientFactory.cs`. +- `references/evaluators-catalog.md` — full catalog of NLP + Quality + Safety evaluators with required `EvaluationContext` types and which tier they belong to. +- `references/telemetry-capture.md` — per-call hook + cost report format. Calls out: this is NOT the MEAI HTML report. +- `references/quality-modes.md` — `DiskBasedReportingConfiguration` wiring, tier-based evaluator registration, `aieval report` invocation. +- `references/compare-mode.md` — `matrix.json` layout, `[DynamicData]` test shape, per-entry `executionName`. +- `references/safety-mode.md` — opt-in safety scaffold, Foundry runtime check, `ContentHarmEvaluator` default. +- `references/ci-workflow.md` — `.github/workflows/evals.yml` template. - `references/aspire-dashboard-panel.md` — optional static-file panel. +- [Microsoft.Extensions.AI.Evaluation libraries](https://learn.microsoft.com/en-us/dotnet/ai/evaluation/libraries) — upstream catalog of evaluators. +- [Tutorial: evaluate with reporting](https://learn.microsoft.com/en-us/dotnet/ai/evaluation/evaluate-with-reporting) — canonical MSTest pattern. +- [dotnet/ai-samples → microsoft-extensions-ai-evaluation/api](https://github.com/dotnet/ai-samples/blob/main/src/microsoft-extensions-ai-evaluation/api/) — canonical unit-test examples. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/ci-workflow.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/ci-workflow.md new file mode 100644 index 0000000000..f1ec46e10a --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/ci-workflow.md @@ -0,0 +1,119 @@ +# CI workflow (opt-in) + +Off by default. Enabled in step 2 when the user picks "scaffold CI +workflow" / "add GitHub Actions" / equivalent. + +When enabled, emits `.github/workflows/evals.yml`. The workflow: + +- Runs on every PR + on push to `main`. +- Always runs Tier 1 (NLP) — no creds needed, fast smoke test. +- Promotes to Tier 2 (Quality, real judge) when the repo has secrets + `AZURE_OPENAI_ENDPOINT` + `AZURE_TENANT_ID`. +- Promotes to Tier 3 (Foundry Safety) when the repo has secret + `AZURE_AI_FOUNDRY_ENDPOINT`. +- Uploads `report.html` as a workflow artifact. + +## Template + +```yaml +name: evals + +on: + pull_request: + branches: [main] + push: + branches: [main] + workflow_dispatch: + +permissions: + id-token: write # for OIDC -> DefaultAzureCredential + contents: read + +jobs: + evaluate: + runs-on: ubuntu-latest + timeout-minutes: 20 + + env: + EVAL_EXECUTION_NAME: ${{ github.run_id }}-${{ github.run_attempt }} + + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-dotnet@v4 + with: + dotnet-version: 10.0.x + + - name: Restore tools + run: dotnet tool restore + + - name: Azure login (only if creds present) + if: env.AZURE_TENANT_ID != '' + uses: azure/login@v2 + with: + client-id: ${{ secrets.AZURE_CLIENT_ID }} + tenant-id: ${{ secrets.AZURE_TENANT_ID }} + subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }} + env: + AZURE_TENANT_ID: ${{ secrets.AZURE_TENANT_ID }} + + - name: Tier selection + id: tier + run: | + if [ -n "${{ secrets.AZURE_OPENAI_ENDPOINT }}" ]; then + echo "EVAL_USE_REAL_AGENT=1" >> $GITHUB_ENV + echo "EVAL_USE_REAL_JUDGE=1" >> $GITHUB_ENV + echo "AZURE_OPENAI_ENDPOINT=${{ secrets.AZURE_OPENAI_ENDPOINT }}" >> $GITHUB_ENV + echo "tier=judge" >> $GITHUB_OUTPUT + else + echo "tier=stub" >> $GITHUB_OUTPUT + fi + if [ -n "${{ secrets.AZURE_AI_FOUNDRY_ENDPOINT }}" ]; then + echo "EVAL_USE_FOUNDRY_SAFETY=1" >> $GITHUB_ENV + echo "AZURE_AI_FOUNDRY_ENDPOINT=${{ secrets.AZURE_AI_FOUNDRY_ENDPOINT }}" >> $GITHUB_ENV + echo "tier=safety" >> $GITHUB_OUTPUT + fi + + - name: Run evals (dotnet test) + run: dotnet test {{AppName}}.Evals.Tests/{{AppName}}.Evals.Tests.csproj --logger "trx;LogFileName=evals.trx" + + - name: Generate report (already invoked by [AssemblyCleanup], this is a safety net) + if: always() + run: | + mkdir -p .copilot/perf-reports/evals/${{ env.EVAL_EXECUTION_NAME }} + dotnet tool run aieval report \ + --path {{AppName}}.Evals.Tests/_store \ + --output .copilot/perf-reports/evals/${{ env.EVAL_EXECUTION_NAME }}/report.html + + - name: Upload report + if: always() + uses: actions/upload-artifact@v4 + with: + name: eval-report-${{ steps.tier.outputs.tier }} + path: .copilot/perf-reports/evals/${{ env.EVAL_EXECUTION_NAME }}/report.html + + - name: Upload trx + if: always() + uses: actions/upload-artifact@v4 + with: + name: trx + path: '**/evals.trx' +``` + +## Optional: PR comment with summary + +A second job can comment on the PR with a link to the artifact and a +one-line summary scraped from `compare.md`. Not in the default +template — too much variance across teams' bot/preference choices. + +## Required secrets + +| Secret | Tier unlocked | Required for OIDC? | +|--------|---------------|--------------------| +| `AZURE_TENANT_ID` | needed for any Azure auth | yes | +| `AZURE_CLIENT_ID` | OIDC federated identity | yes (if using `azure/login@v2`) | +| `AZURE_SUBSCRIPTION_ID` | scope | yes | +| `AZURE_OPENAI_ENDPOINT` | Tier 2 (judge) | no | +| `AZURE_AI_FOUNDRY_ENDPOINT` | Tier 3 (safety) | no | + +Document these clearly in the chat output when the workflow is scaffolded. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/compare-mode.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/compare-mode.md index 05d89b8571..142ad26e63 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/compare-mode.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/compare-mode.md @@ -1,89 +1,97 @@ # Compare mode -Run telemetry + quality for two or more model assignments and emit a -side-by-side delta. +Compare mode runs the quality + telemetry pipeline against **multiple +model assignments** and produces a side-by-side report. Crucially, it +goes through the **same `DiskBasedReportingConfiguration`** so +`aieval report` aggregates the comparison into a single HTML view. -## `Compare/matrix.json` +## `matrix.json` ```json -[ - { - "name": "baseline", - "model_assignments": { - "router": "gpt-4o-mini", - "planner": "gpt-4o", - "worker": "gpt-4o-mini" +{ + "schema_version": 2, + "entries": [ + { + "name": "baseline-all-mini", + "model_assignments": { + "receptionist": "gpt-4o-mini", + "behavioural": "gpt-4o-mini", + "technical": "gpt-4o-mini", + "summariser": "gpt-4o-mini" + } + }, + { + "name": "interviewers-upgraded", + "model_assignments": { + "receptionist": "gpt-4o-mini", + "behavioural": "gpt-4o", + "technical": "gpt-4o", + "summariser": "gpt-4o-mini" + } } - }, - { - "name": "candidate", - "model_assignments": { - "router": "gpt-4o-mini", - "planner": "o4-mini", - "worker": "gpt-4o-mini" - } - } -] + ] +} ``` -## Runner behaviour - -For each entry in the matrix: - -1. Apply the `model_assignments` in-process (override the - per-agent `IChatClient` registrations; do NOT modify - `appsettings.json`). -2. Run telemetry mode against `Telemetry/inputs.json`. -3. Run quality mode against `Quality/golden.json`. -4. Capture both reports labeled by `name`. - -Then diff: - -- Latency: per-agent delta (ms) and aggregate delta. -- Tokens: input/output deltas per agent. -- Cost: aggregate delta (USD). -- Quality: pass-rate delta and per-input score delta. - -## Report — `compare.md` - -```markdown -# Compare — {{ utc_timestamp }} - -Variants: {{ name_list }} - -## Latency (avg ms per agent) - -| Agent | baseline | candidate | Δ | -|---------|----------|-----------|--------| -| router | 340 | 342 | +2 | -| planner | 1240 | 980 | -260 | -| worker | 410 | 405 | -5 | +## Test shape + +Each matrix entry becomes one row of a parameterised test. Each entry +uses a **distinct `executionName`** so `aieval report` shows the +comparison side-by-side. + +```csharp +[TestClass] +public sealed class CompareTests +{ + public static IEnumerable Matrix() => + MatrixLoader.Load().Select(e => new object[] { e }); + + [TestMethod, DynamicData(nameof(Matrix), DynamicDataSourceType.Method)] + public async Task RunEntry(MatrixEntry entry) + { + // executionName scoped per entry so aieval report groups columns by it. + var reporting = DiskBasedReportingConfiguration.Create( + storageRootPath: ReportingConfig.StorageRoot, + evaluators: ReportingConfig.EvaluatorList(), + chatConfiguration: new ChatConfiguration( + Wire.ResolveJudgeClient(Wire.ResolveAgentClient(entry.ModelAssignments))), + enableResponseCaching: true, + executionName: $"{ReportingConfig.ExecutionName}-{entry.Name}"); + + foreach (var g in GoldenLoader.Load()) + { + var scenarioName = $"Compare.{entry.Name}.{g.Id}"; + await using var run = await reporting.CreateScenarioRunAsync(scenarioName); + // ... same shape as QualityTests + } + } +} +``` -## Token cost (USD per 1K turns, projected) +## Override the per-agent model id -| Variant | Cost | Δ | -|-----------|--------|----------| -| baseline | $5.34 | — | -| candidate | $4.18 | -$1.16 | +`Wire.ResolveAgentClient(IDictionary overrides)` is the +extension point. The generated factory (`AgentChatClientFactory`) +exposes an overload accepting per-agent model assignments — useful when +the app uses multiple deployment aliases or supports model swapping +via `ChatOptions.ModelId`. -## Quality (pass rate) +## Compare-specific report -| Variant | Pass | Δ | -|-----------|------|-------| -| baseline | 92% | — | -| candidate | 90% | -2% | +`compare.md` (still emitted, in addition to the aggregated +`report.html`): -## Recommendation +| name | avg ms | in tok | out tok | $ | mean quality | mean BLEU | +|------|--------|--------|---------|---|--------------|-----------| +| baseline-all-mini | 432 | 380 | 210 | 0.0021 | 3.8 | 0.31 | +| interviewers-upgraded | 891 | 380 | 245 | 0.0210 | 4.4 | 0.42 | -candidate ⟶ -22% cost, -260ms planner latency, -2pp quality. -If quality bar is "no regression", reject. If quality bar is -"≥ 88%", accept. -``` +| Recommendation | +|----------------| +| `interviewers-upgraded`: +0.6 quality at +9.4× cost. Promote only if quality bar requires it. | -## Constraints +The recommendation row is rule-based: -- Compare mode never edits `appsettings.json`. -- Compare mode never makes a recommendation by itself; it states the - trade in plain terms and leaves the decision to the user. -- For ≥ 3 variants, the table grows columns; the recommendation row - picks the variant with the best cost-quality frontier (Pareto). +- If cost increases > 3× and quality delta < 0.3 → **do not promote**. +- If cost increases ≤ 1.5× and quality delta ≥ 0.5 → **promote**. +- Otherwise → **manual review**. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/dotnet-tools-manifest.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/dotnet-tools-manifest.md new file mode 100644 index 0000000000..c8e50335d4 --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/dotnet-tools-manifest.md @@ -0,0 +1,50 @@ +# `dotnet-tools.json` manifest + +The skill scaffolds a **local** tool manifest. Global install is +explicitly avoided — pinning the tool version to the project makes +evals reproducible across machines and CI runs. + +## Why local + +- Reproducible: every clone gets the exact same `aieval` version. +- CI-friendly: `dotnet tool restore` in the workflow is one line. +- No PATH conflicts with developers who may have other versions installed. + +## Template + +```json +{ + "version": 1, + "isRoot": true, + "tools": { + "microsoft.extensions.ai.evaluation.console": { + "version": "10.7.0", + "commands": ["aieval"], + "rollForward": false + } + } +} +``` + +Place at `.Evals.Tests/dotnet-tools.json` (not the repo root) so +the manifest follows the project. The skill emits `dotnet tool restore` +as the next step in the chat output after scaffolding. + +## Why `rollForward: false` + +The output of `aieval report` is consumed by humans visually and by +artifact comparison in CI. A silent rollForward of the tool to a +newer version can change report layout, charts, or metric grouping — +breaking trend comparisons and visual diffs. + +If a user wants to upgrade the tool, the skill should emit a +`dotnet tool update microsoft.extensions.ai.evaluation.console` +command in the chat output (with a note: "this may change report +layout"). + +## Conflict with existing manifest + +If `dotnet-tools.json` already exists at the project (or any parent +directory), the skill **merges** the `aieval` entry rather than +overwriting. If a different version of `aieval` is already pinned, +surface the diff in chat output and require user confirmation. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md new file mode 100644 index 0000000000..44dc3020f1 --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md @@ -0,0 +1,106 @@ +# Evaluators catalog + +Full catalog of evaluators wired by `setup-maf-evals`, grouped by tier. +The skill defaults to Tier 1 (NLP) always on, Tier 2 (Quality) on but +gated by `EVAL_USE_REAL_JUDGE`, Tier 3 (Safety) off unless opted in. + +Source: [learn.microsoft.com/en-us/dotnet/ai/evaluation/libraries](https://learn.microsoft.com/en-us/dotnet/ai/evaluation/libraries) + +## Tier 1 — NLP (deterministic, no LLM) + +Package: `Microsoft.Extensions.AI.Evaluation.NLP` (preview 10.7.0). + +| Evaluator | Metric | Context type needed | Notes | +|-----------|--------|---------------------|-------| +| `BLEUEvaluator` | `BLEU` | `BLEUEvaluatorContext(IEnumerable references)` | n-gram overlap with one or more reference strings | +| `GLEUEvaluator` | `GLEU` | `GLEUEvaluatorContext(IEnumerable references)` | sentence-level BLEU variant | +| `F1Evaluator` | `F1` | `F1EvaluatorContext(string groundTruth)` | unigram-level F1 | + +Plus a built-in custom evaluator the skill always scaffolds: + +| Evaluator | Metric | Why | +|-----------|--------|-----| +| `WordCountEvaluator` (custom) | `Words` | Sanity check: response is non-empty and reasonable length. Same pattern as the Learn doc tutorial. | + +## Tier 2 — Quality (LLM-as-judge) + +Package: `Microsoft.Extensions.AI.Evaluation.Quality` (GA 10.7.0). + +Requires `EVAL_USE_REAL_JUDGE=1` and a real `IChatClient`. The skill +wires the following by default: + +| Evaluator | Metric | Context type | Notes | +|-----------|--------|--------------|-------| +| `RelevanceEvaluator` | `Relevance` | none | how relevant is the response to the query | +| `CoherenceEvaluator` | `Coherence` | none | logical, orderly presentation | +| `FluencyEvaluator` | `Fluency` | none | grammar, readability | +| `CompletenessEvaluator` | `Completeness` | `CompletenessEvaluatorContext(string groundTruth)` | comprehensive and accurate | +| `EquivalenceEvaluator` | `Equivalence` | `EquivalenceEvaluatorContext(string groundTruth)` | similarity vs ground truth wrt query | +| `GroundednessEvaluator` | `Groundedness` | `GroundednessEvaluatorContext(string context)` | alignment with given context | + +Agent-focused (added when `*.AppHost.csproj` is detected, indicating +this is an agentic app and not just a chat completion app): + +| Evaluator | Metric | Context type | Notes | +|-----------|--------|--------------|-------| +| `IntentResolutionEvaluator` | `Intent Resolution` | none | identifies + resolves user intent | +| `TaskAdherenceEvaluator` | `Task Adherence` | none | sticks to assigned task | +| `ToolCallAccuracyEvaluator` | `Tool Call Accuracy` | `ToolCallAccuracyEvaluatorContext(...)` | uses tools correctly | + +**Not wired by default:** `RelevanceTruthAndCompletenessEvaluator` +(marked experimental in upstream docs), `RetrievalEvaluator` (specific +to RAG pipelines — separate skill territory). + +## Tier 3 — Safety (Foundry Evaluation service) + +Package: `Microsoft.Extensions.AI.Evaluation.Safety` (preview 10.7.0). + +Off by default. Enabled when user opts in during step 2 of the +workflow. Requires `EVAL_USE_FOUNDRY_SAFETY=1` and an Azure AI Foundry +endpoint (the skill prompts for `AZURE_AI_FOUNDRY_ENDPOINT` + Entra credentials). + +**Always wire the bundle, not the 4 separate evaluators:** + +| Evaluator | Metrics produced | Notes | +|-----------|------------------|-------| +| `ContentHarmEvaluator` | `Hate And Unfairness`, `Self Harm`, `Violence`, `Sexual` | **Single-shot — one Foundry call returns all 4 metrics.** Always prefer this over the 4 separate evaluators below. | + +Additional safety evaluators (each wired as separate `[TestMethod]`): + +| Evaluator | Metric | Notes | +|-----------|--------|-------| +| `ProtectedMaterialEvaluator` | `Protected Material` | copyrighted material in output | +| `IndirectAttackEvaluator` | `Indirect Attack` | prompt-injection-style indirect attacks | +| `CodeVulnerabilityEvaluator` | `Code Vulnerability` | vulnerable code in output | +| `UngroundedAttributesEvaluator` | `Ungrounded Attributes` | inferred human attributes | +| `GroundednessProEvaluator` | `Groundedness Pro` | fine-tuned Foundry-hosted groundedness check | + +The 4 separate evaluators (`HateAndUnfairnessEvaluator`, +`SelfHarmEvaluator`, `ViolenceEvaluator`, `SexualEvaluator`) are +**not** scaffolded — they're a strict subset of `ContentHarmEvaluator` +and cost 4× more Foundry calls for the same metrics. + +## Threshold mapping + +`quality.thresholds.json` maps **real MEAI metric names** to minimum +`EvaluationRating` enum values: + +```json +{ + "schema_version": 2, + "hard_fail": false, + "thresholds": { + "Relevance": { "min_rating": "Good" }, + "Coherence": { "min_rating": "Good" }, + "Fluency": { "min_rating": "Average" }, + "Groundedness":{ "min_rating": "Good" }, + "BLEU": { "min_value": 0.20 }, + "F1": { "min_value": 0.30 }, + "Words": { "min_value": 5, "max_value": 500 } + } +} +``` + +`hard_fail: true` makes any below-threshold metric fail the test. +Default `false` makes the test pass-through informational — failures +show in the report only. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md new file mode 100644 index 0000000000..6d2f1357c0 --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md @@ -0,0 +1,114 @@ +# IChatClient detection + +The skill auto-detects how the target app registers its `IChatClient` +and emits `Wire/AgentChatClientFactory.cs` so `EVAL_USE_REAL_AGENT=1` +works without further code. Detection result is surfaced in step 2 +(scope confirmation) and can be overridden by the user. + +## Patterns to scan for + +Scan **`*.AppHost.csproj` directory** and **all `*.Agent*.csproj` +directories**. Match (case-insensitive, multi-line): + +| Pattern (regex-ish) | Inferred client | Example file:line | +|---------------------|-----------------|--------------------| +| `AddAzureOpenAIChatClient\s*\(` or `AddAzureOpenAIClient\s*\(` | Azure OpenAI | `AppHost.cs`, `Program.cs` | +| `AddOpenAIChatClient\s*\(` | OpenAI direct | `Program.cs` | +| `AddOllamaChatClient\s*\(` | Ollama | `Program.cs` | +| `AddAIInference\s*\(` (Foundry deployment alias) | Azure AI Foundry | `AppHost.cs` | +| `services\.AddSingleton` (any explicit registration) | custom | varies | +| `\.AsIChatClient\(\)` (after an SDK client) | manual wrap | varies | + +Capture the deployment alias / model id literal if present (e.g., +`builder.AddAIInference("chat", "gpt-4o-mini")` → alias `chat`). + +## What to emit + +### Case A — exactly one registration found + +Emit a factory that resolves from the host: + +```csharp +// Wire/AgentChatClientFactory.cs +namespace {{AppName}}.Evals.Tests; + +internal static class AgentChatClientFactory +{ + /// + /// Resolves the same IChatClient the app uses, by building a minimal + /// host that mirrors the app's DI registration. + /// Detected: {{DetectionSummary}} at {{File}}:{{Line}} + /// + public static IChatClient Create() + { + var builder = Host.CreateApplicationBuilder(); + // {{InsertDetectedRegistrationCallVerbatim}} + var host = builder.Build(); + return host.Services.GetRequiredService(); + } +} +``` + +Where `{{InsertDetectedRegistrationCallVerbatim}}` is the literal call +copied from the detection source (with any required `using`s in scope +via `GlobalUsings.cs`). + +### Case B — multiple registrations found + +Emit the same factory but with a comment listing all candidates, and +have the chat output ask the user to pick one before write. Do **not** +write the file until confirmation. + +### Case C — no registration found + +Emit a stub factory the user fills in: + +```csharp +// Wire/AgentChatClientFactory.cs +namespace {{AppName}}.Evals.Tests; + +internal static class AgentChatClientFactory +{ + /// + /// Auto-detection failed: no IChatClient registration found in + /// AppHost or agent projects. Wire your client manually. + /// + public static IChatClient Create() => + throw new NotImplementedException( + "Wire your IChatClient here. See https://learn.microsoft.com/en-us/dotnet/ai/microsoft-extensions-ai"); +} +``` + +## Runtime selection + +`ReportingConfig` picks agent client based on tier: + +```csharp +internal static IChatClient ResolveAgentClient() => + EvalEnv.UseRealAgent ? AgentChatClientFactory.Create() : new StubChatClient(); +``` + +And by default the judge client is the **same** instance (saves a +duplicate Azure credential setup). The user can override by setting +`EVAL_JUDGE_DEPLOYMENT_NAME` to a different deployment alias. + +## Chat output (step 2) + +When detection succeeds, surface as: + +``` +IChatClient detection: + - AddAzureOpenAIChatClient at AppHost.cs:41 + - Deployment alias: "chat" → gpt-4o-mini + - Will generate Wire/AgentChatClientFactory.cs that resolves it from the host. + +Override [y/N]? +``` + +When detection fails: + +``` +IChatClient detection: no registration found. + - Will generate a stub factory you'll need to fill in. + - Tier 2 (Quality) and Tier 3 (Safety) will be skipped until wired. +``` diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/project-template.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/project-template.md index f0003db43f..c84b7ec20a 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/project-template.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/project-template.md @@ -1,156 +1,134 @@ -# Project template — `.Evals` +# Project template (MSTest shape) -The exact files written when scaffolding the eval harness. +The scaffold creates `.Evals.Tests` as a **MSTest** project. This +matches the +[upstream Learn doc tutorial](https://learn.microsoft.com/en-us/dotnet/ai/evaluation/evaluate-with-reporting) +and the [dotnet/ai-samples evaluation unit tests](https://github.com/dotnet/ai-samples/blob/main/src/microsoft-extensions-ai-evaluation/api/), +which is the canonical pattern for `Microsoft.Extensions.AI.Evaluation`. -## `.Evals.csproj` +A console-runner shape is available behind an explicit +`--shape console` flag for users who can't take an MSTest dependency, +but is no longer the default — every CI system understands `dotnet test` +out of the box, and Test Explorer integration is automatic. -Use the actual current package versions from NuGet at scaffold time -(query `nuget.org` or `dotnet package search`). The versions below -reflect the latest stable family at the time of writing -(`Microsoft.Extensions.AI.Evaluation` 10.x is GA on nuget.org); query -`dotnet package search "Microsoft.Extensions.AI.Evaluation"` and bump -to the latest stable when you scaffold. +## File tree + +``` +.Evals.Tests/ + .Evals.Tests.csproj + dotnet-tools.json + GlobalUsings.cs + Reporting/ + ReportingConfig.cs # DiskBasedReportingConfiguration factory; tier-aware evaluator list + Tier.cs # EvalTier enum + EvalEnv reader + AievalReport.cs # [AssemblyCleanup] that invokes the dotnet tool + Wire/ + AgentChatClientFactory.cs # auto-generated from IChatClient detection + StubChatClient.cs # used when EVAL_USE_REAL_AGENT is unset + Telemetry/ + TelemetryTests.cs + inputs.json + prices.json + Quality/ + QualityTests.cs + rubric.md + golden.json + Compare/ + CompareTests.cs + matrix.json + Safety/ # only if user opted in + SafetyTests.cs + quality.thresholds.json +.github/ + workflows/ + evals.yml # optional, opt-in +``` + +`.gitignore` additions (idempotent): + +``` +# setup-maf-evals +.copilot/perf-reports/evals/ +.Evals.Tests/_store/ +``` + +## `.csproj` template ```xml - - Exe net10.0 enable enable - {{ AppName }}.Evals + false + {{AppName}}.Evals.Tests - - - - - - + + + + + + + + + + + + + + + + + + + + - + - - - - - + - ``` -If the consumer repo uses Central Package Management -(`Directory.Packages.props` with `ManagePackageVersionsCentrally=true`), -omit the `Version=` attributes and add matching `` -entries to the central props file instead. +**Why these versions:** -The `ProjectReference` line points to the agent service the evals will -exercise (typically the agent service, not the AppHost). If the repo -has multiple agent service projects, generate one `ProjectReference` -per project and let the runner classes select which agent to invoke. +- `Microsoft.Extensions.AI.Evaluation.{Reporting,Quality,Console}` are GA at `10.7.0`. +- `Microsoft.Extensions.AI.Evaluation.{NLP,Safety}` are still preview at `10.7.0-preview.1.26309.5`. NLP is opt-in-on; Safety is opt-in-off. +- `Microsoft.Extensions.Hosting` and `Microsoft.Extensions.Configuration.*` must be `10.0.1` (not `10.0.0`) to satisfy the transitive constraint from `Microsoft.Agents.AI.Hosting`. Pinning `10.0.0` produces `NU1605`. -## `Abstractions.cs` (generated alongside Program.cs) - -```csharp -public interface IEvalRunner -{ - Task RunAsync(CancellationToken ct = default); -} - -public sealed record EvalReport( - bool Success, - string OneLineSummary, - string ReportDirectory); -``` - -The three runner classes (`TelemetryEvalRunner`, `QualityEvalRunner`, -`CompareEvalRunner`) each implement `IEvalRunner` and write their -report files under `ReportDirectory`. See `telemetry-capture.md`, -`quality-modes.md`, and `compare-mode.md` for each runner's body. - -## `Program.cs` - -```csharp -var mode = args.FirstOrDefault() ?? "telemetry"; -IEvalRunner runner = mode switch -{ - "telemetry" => new TelemetryEvalRunner(), - "quality" => new QualityEvalRunner(), - "compare" => new CompareEvalRunner(), - _ => throw new ArgumentException($"Unknown mode: {mode}") -}; - -var report = await runner.RunAsync(); -Console.WriteLine(report.OneLineSummary); -return report.Success ? 0 : 1; -``` - -## `Telemetry/inputs.json` (starter) - -```json -[ - { "id": "smoke-1", "user_message": "Say hi in five words." }, - { "id": "smoke-2", "user_message": "Plan a one-day trip to Seattle." }, - { "id": "smoke-3", "user_message": "Summarize the previous answer in 20 words." } -] -``` - -## `Telemetry/prices.json` (starter — user edits) +## `dotnet-tools.json` ```json { - "gpt-4o-mini": { "input_per_1k": 0.00015, "output_per_1k": 0.00060 }, - "gpt-4o": { "input_per_1k": 0.00250, "output_per_1k": 0.01000 }, - "o4-mini": { "input_per_1k": 0.00110, "output_per_1k": 0.00440 } + "version": 1, + "isRoot": true, + "tools": { + "microsoft.extensions.ai.evaluation.console": { + "version": "10.7.0", + "commands": ["aieval"], + "rollForward": false + } + } } ``` -## `Quality/rubric.md` (starter) - -```markdown -# Quality rubric - -Score each response on a 1-5 scale per trait. The judge model returns -a JSON object: `{ trait: score, ... }`. - -- **on_topic** — answers the user's actual question. -- **concise** — no filler; under 4 sentences unless asked otherwise. -- **safe** — no PII leakage, no instructions to violate policy. -- **format_correct** — Markdown / JSON shape matches the contract. - -Pass threshold (default): mean ≥ 4.0 AND no trait < 3. -``` - -## `Quality/golden.json` (starter) - -```json -[ - { - "id": "g1", - "user_message": "Plan a one-day trip to Seattle.", - "expected_traits": ["on_topic", "concise", "format_correct"] - } -] -``` +After scaffold: `dotnet tool restore` (the skill runs this automatically). -## `Compare/matrix.json` (starter) +## `GlobalUsings.cs` -```json -[ - { - "name": "baseline", - "model_assignments": { "router": "gpt-4o-mini", "planner": "gpt-4o", "worker": "gpt-4o-mini" } - }, - { - "name": "candidate", - "model_assignments": { "router": "gpt-4o-mini", "planner": "o4-mini", "worker": "gpt-4o-mini" } - } -] +```csharp +global using Microsoft.Extensions.AI; +global using Microsoft.Extensions.AI.Evaluation; +global using Microsoft.Extensions.AI.Evaluation.Reporting; +global using Microsoft.Extensions.AI.Evaluation.Reporting.Storage; +global using Microsoft.Extensions.AI.Evaluation.NLP; +global using Microsoft.Extensions.AI.Evaluation.Quality; +global using Microsoft.VisualStudio.TestTools.UnitTesting; ``` diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/quality-modes.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/quality-modes.md index 41f5981324..eecb89b281 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/quality-modes.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/quality-modes.md @@ -1,95 +1,181 @@ -# Quality modes — LLM-judge +# Quality mode -How quality mode runs the agent against `golden.json` and asks a judge -model to score each response per the rubric. +`Quality/QualityTests.cs` is the MSTest class that actually drives +`Microsoft.Extensions.AI.Evaluation.Reporting`. It's the **only** +runner that produces `report.html`. -## Runner +## Pipeline + +``` +[ClassInitialize] → build ReportingConfig (tier-aware evaluator list) +[TestMethod] → one per golden.json entry + ├─ CreateScenarioRunAsync(scenarioName) + ├─ resolve IChatClient (real or stub per EVAL_USE_REAL_AGENT) + ├─ get agent response + ├─ build per-evaluator EvaluationContext (BLEU refs, F1 ground truth, ...) + └─ scenarioRun.EvaluateAsync(messages, response, contexts) +[AssemblyCleanup] → dotnet tool run aieval report --path _store --output /report.html +``` + +## Reporting config (sketch) ```csharp -public sealed class QualityEvalRunner : IEvalRunner +// Reporting/ReportingConfig.cs +internal static class ReportingConfig { - public async Task RunAsync() - { - var rubric = await File.ReadAllTextAsync("Quality/rubric.md"); - var golden = await JsonSerializer.DeserializeAsync( - File.OpenRead("Quality/golden.json")); + public static readonly string StorageRoot = + Path.Combine(RepoRoot.Find(), "_store"); - var judge = new ChatClientBuilder() - .UseFunctionInvocation() - .Build(new ChatClient(model: Config.JudgeModel, apiKey: Config.JudgeApiKey)); + public static readonly string ExecutionName = + Environment.GetEnvironmentVariable("EVAL_EXECUTION_NAME") + ?? DateTime.UtcNow.ToString("yyyyMMdd-HHmmss"); - var rows = new List(); - foreach (var g in golden!) + public static ReportingConfiguration ForQuality() + { + var agent = Wire.ResolveAgentClient(); + var judge = Wire.ResolveJudgeClient(agent); + + var evaluators = new List { - var actual = await Agent.RunAsync(g.UserMessage); - var verdict = await Judge(judge, rubric, g, actual); - rows.Add(new QualityRow(g.Id, verdict.Scores, verdict.PassFail, verdict.Rationale)); + // Tier 1 — always on, deterministic + new WordCountEvaluator(), + new BLEUEvaluator(), + new GLEUEvaluator(), + new F1Evaluator(), + }; + + if (EvalEnv.UseRealJudge) + { + // Tier 2 — needs real judge + evaluators.Add(new RelevanceEvaluator()); + evaluators.Add(new CoherenceEvaluator()); + evaluators.Add(new FluencyEvaluator()); + evaluators.Add(new CompletenessEvaluator()); + evaluators.Add(new EquivalenceEvaluator()); + evaluators.Add(new GroundednessEvaluator()); + + if (AgenticAppDetected) + { + evaluators.Add(new IntentResolutionEvaluator()); + evaluators.Add(new TaskAdherenceEvaluator()); + evaluators.Add(new ToolCallAccuracyEvaluator()); + } } - return EvalReport.FromQuality(rows); + return DiskBasedReportingConfiguration.Create( + storageRootPath: StorageRoot, + evaluators: evaluators, + chatConfiguration: new ChatConfiguration(judge), + enableResponseCaching: true, + executionName: ExecutionName); } } ``` -## Judge prompt skeleton +## Test class (sketch) -``` -You are a quality judge. Score the assistant response on each trait -in the rubric (1-5). Return ONLY a JSON object of the form: +```csharp +[TestClass] +public sealed class QualityTests +{ + private static ReportingConfiguration s_reporting = null!; -{ "scores": { "": , ... }, "rationale": "" } + [ClassInitialize] + public static void Init(TestContext _) => + s_reporting = ReportingConfig.ForQuality(); -Rubric: -{{ rubric_md }} + public static IEnumerable Golden() => + GoldenLoader.Load().Select(g => new object[] { g }); -User asked: -{{ user_message }} + [TestMethod, DynamicData(nameof(Golden), DynamicDataSourceType.Method)] + public async Task Evaluate(GoldenItem g) + { + var scenarioName = $"{nameof(QualityTests)}.{g.Id}"; + await using var run = await s_reporting.CreateScenarioRunAsync(scenarioName); -Assistant replied: -{{ actual_response }} + var agent = Wire.ResolveAgentClient(); + var messages = new List + { + new(ChatRole.System, RubricLoader.SystemPrompt()), + new(ChatRole.User, g.UserMessage), + }; + var response = await agent.GetResponseAsync(messages); -Required traits to score: -{{ expected_traits }} -``` + var contexts = new List(); + if (!string.IsNullOrEmpty(g.ReferenceResponse)) + { + contexts.Add(new BLEUEvaluatorContext(new[] { g.ReferenceResponse })); + contexts.Add(new GLEUEvaluatorContext(new[] { g.ReferenceResponse })); + contexts.Add(new F1EvaluatorContext(g.ReferenceResponse)); + contexts.Add(new EquivalenceEvaluatorContext(g.ReferenceResponse)); + contexts.Add(new CompletenessEvaluatorContext(g.ReferenceResponse)); + } + if (!string.IsNullOrEmpty(g.Context)) + contexts.Add(new GroundednessEvaluatorContext(g.Context)); -## Pass/fail + var result = await run.EvaluateAsync(messages, response, contexts); + Thresholds.ApplyOrLog(result, g.Id); // hard_fail in JSON => Assert.Fail + } +} +``` -`quality.thresholds.json` (optional, user-edited): +## Report generation -```json +```csharp +// Reporting/AievalReport.cs +[TestClass] +public static class AievalReport { - "mean_score_min": 4.0, - "per_trait_min": 3, - "fail_on_threshold_breach": false + [AssemblyCleanup] + public static void GenerateReport() + { + var outDir = Path.Combine( + RepoRoot.Find(), ".copilot", "perf-reports", "evals", + ReportingConfig.ExecutionName); + Directory.CreateDirectory(outDir); + var html = Path.Combine(outDir, "report.html"); + + var psi = new ProcessStartInfo("dotnet", + $"tool run aieval report --path \"{ReportingConfig.StorageRoot}\" --output \"{html}\"") + { + WorkingDirectory = RepoRoot.Find(), + RedirectStandardOutput = true, + RedirectStandardError = true, + UseShellExecute = false, + CreateNoWindow = true, + }; + using var p = Process.Start(psi)!; + p.WaitForExit(); + TestContext.Out?.WriteLine($"Eval report: {html}"); + } } ``` -If `fail_on_threshold_breach: false` (default), the runner exits 0 -even on quality regressions and marks them in the report. Set to -`true` to gate CI. - -## Report — `quality.md` - -```markdown -# Quality — {{ utc_timestamp }} +## `golden.json` schema (v2) -Judge: {{ judge_model }} | Inputs: {{ count }} | Pass rate: {{ pct }} - -| Id | Mean | on_topic | concise | safe | format | Pass | Rationale | -|----|------|----------|---------|------|--------|------|---------------------------------| -| g1 | 4.5 | 5 | 4 | 5 | 4 | ✅ | Concise with clear sections. | -| g2 | 3.0 | 4 | 2 | 3 | 3 | ❌ | Rambling intro; over 6 sentences. | - -## Failures - -### g2 (3.0) -- judge rationale: ... -- actual response (truncated): ... +```json +{ + "schema_version": 2, + "scenarios": [ + { + "id": "g-receptionist-greeting", + "user_message": "Hi, I'd like to start an interview.", + "reference_response": "Hello! I'd be happy to start an interview with you. What role are you preparing for?", + "context": null, + "expected_traits": ["on_topic", "safe", "format_correct"], + "expected_tool_calls": null + } + ] +} ``` -## Cost considerations +- `reference_response`: required for BLEU/GLEU/F1/Equivalence/Completeness. +- `context`: required for Groundedness. Free text providing the + source-of-truth context the response should be grounded in. +- `expected_traits`: free-form labels surfaced in the report rubric + view. Read by the LLM judge. +- `expected_tool_calls`: required for `ToolCallAccuracyEvaluator`. -Each quality run pays for: agent calls (real or stub) + judge calls -(always real). Expect roughly 1 judge call per input. Use a smaller -judge model (e.g. `gpt-4o-mini`) for early iterations and switch up -when stabilizing. +Migration from v1 (no `schema_version`, no `reference_response`): the +skill adds the fields as `null` so existing tests don't fail. NLP +evaluators emit `(no reference)` when null. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/safety-mode.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/safety-mode.md new file mode 100644 index 0000000000..2b38264d3b --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/safety-mode.md @@ -0,0 +1,89 @@ +# Safety mode (opt-in) + +Off by default. Enabled in step 2 of the workflow when the user picks +"include Safety tier" / says "wire safety" / equivalent. + +When enabled, the skill: + +1. Adds `Microsoft.Extensions.AI.Evaluation.Safety` (preview 10.7.0) to the csproj. +2. Generates `Safety/SafetyTests.cs` using `ContentHarmEvaluator` as the + default bundle (4 metrics in 1 Foundry call), plus + `ProtectedMaterialEvaluator`, `IndirectAttackEvaluator`, + `CodeVulnerabilityEvaluator`, `UngroundedAttributesEvaluator`, and + optionally `GroundednessProEvaluator`. +3. Adds the required Azure AI Foundry endpoint config keys to + `quality.thresholds.json` and surfaces them in the chat output. + +## Runtime gating + +Safety tests must **never** fail the build when Foundry creds are +missing — they're an opt-in capability. The pattern: + +```csharp +[TestClass] +public sealed class SafetyTests +{ + [ClassInitialize] + public static void Init(TestContext _) + { + if (!EvalEnv.UseFoundrySafety) + Assert.Inconclusive( + "Safety tier disabled. Set EVAL_USE_FOUNDRY_SAFETY=1 and " + + "AZURE_AI_FOUNDRY_ENDPOINT to enable."); + } + + public static IEnumerable Golden() => + GoldenLoader.Load().Select(g => new object[] { g }); + + [TestMethod, DynamicData(nameof(Golden), DynamicDataSourceType.Method)] + public async Task ContentHarm(GoldenItem g) + { + var reporting = ReportingConfig.ForSafety(); // separate config — Foundry chat client + await using var run = await reporting.CreateScenarioRunAsync($"Safety.ContentHarm.{g.Id}"); + var agent = Wire.ResolveAgentClient(); + var messages = new List { new(ChatRole.User, g.UserMessage) }; + var response = await agent.GetResponseAsync(messages); + await run.EvaluateAsync(messages, response); // ContentHarmEvaluator returns all 4 metrics + } + + // Repeat for ProtectedMaterial / IndirectAttack / CodeVulnerability / etc. +} +``` + +## Why `ContentHarmEvaluator` (not 4 separate) + +From the upstream docs: + +> ContentHarmEvaluator provides single-shot evaluation for the four +> metrics supported by HateAndUnfairnessEvaluator, SelfHarmEvaluator, +> ViolenceEvaluator, and SexualEvaluator. + +That's **1 Foundry call instead of 4** for the same metric set. Always +wire `ContentHarmEvaluator` unless the user has a strict reason to +isolate one harm category. + +## Config keys surfaced in chat + +When Safety tier is enabled, the skill output adds: + +``` +Safety tier enabled. To activate at runtime: + export EVAL_USE_FOUNDRY_SAFETY=1 + export AZURE_AI_FOUNDRY_ENDPOINT=https://.cognitiveservices.azure.com + az login --tenant # DefaultAzureCredential + +Safety tests are MARKED INCONCLUSIVE (not failed) when the env vars are unset, +so your default `dotnet test` run will not break. +``` + +## What Safety evaluators do **not** cover + +Document this explicitly in the rubric: safety evaluators are *output* +classifiers, not *input* classifiers. They do not protect the agent +from receiving harmful prompts — for that, use a separate input filter +(e.g., Azure AI Content Safety on the request side). + +`IndirectAttackEvaluator` is the closest to an input-side check; it +looks for prompt-injection-style content in the *response* that would +indicate the model picked up an indirect attack from retrieved content +or tool output. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/telemetry-capture.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/telemetry-capture.md index 1aa8148537..7c2ee3d44f 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/telemetry-capture.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/telemetry-capture.md @@ -1,83 +1,111 @@ -# Telemetry capture +# Telemetry mode -How telemetry mode wraps the existing `IChatClient` and writes per-call -records to disk. +Telemetry mode captures **latency, input tokens, output tokens, and +cost** per agent call across a fixed input set. It is **not** the same +as the MEAI eval report — it produces a separate cost/latency capture. -## Wrapper +## Why separate from quality + +Quality mode answers "is the response any good?" via +`Microsoft.Extensions.AI.Evaluation.Reporting` and produces +`report.html`. + +Telemetry mode answers "how much does it cost and how slow is it?" +via a delegating `IChatClient` wrapper. Different question, different +artifact. Conflating them is the #1 most common scaffolding mistake. + +## Artifacts (each in `.copilot/perf-reports/evals//`) + +- `telemetry.md` — human-readable per-input table. +- `telemetry.json` — machine-readable for CI scraping. +- `telemetry.junit.xml` — for test-result dashboards. + +These are **distinct** from `report.html` (which only quality mode +writes). The skill output should never refer to `telemetry.md` as +"the eval report." + +## Test shape ```csharp -public sealed class TelemetryChatClient(IChatClient inner, TelemetrySink sink) : IChatClient +[TestClass] +public sealed class TelemetryTests { - public async Task GetResponseAsync(IList messages, - ChatOptions? options = null, CancellationToken ct = default) + public static IEnumerable Inputs() => + InputsLoader.Load().Select(i => new object[] { i }); + + [TestMethod, DynamicData(nameof(Inputs), DynamicDataSourceType.Method)] + public async Task Capture(TelemetryInput input) { + var inner = Wire.ResolveAgentClient(); + var wrapped = new TelemetryCapturingChatClient(inner, PriceTable.Load()); + + var messages = new List { new(ChatRole.User, input.Text) }; var sw = Stopwatch.StartNew(); - var resp = await inner.GetResponseAsync(messages, options, ct); + var response = await wrapped.GetResponseAsync(messages); sw.Stop(); - sink.Record(new TelemetryRecord( - AgentName: options?.AdditionalProperties?["agent"] as string ?? "unknown", - Model: options?.ModelId ?? "unknown", - InputTokens: resp.Usage?.InputTokenCount ?? 0, - OutputTokens: resp.Usage?.OutputTokenCount ?? 0, + TelemetryStore.Record(new TelemetryRecord( + AgentName: input.Agent, + Model: response.ModelId ?? "unknown", + InputTokens: response.Usage?.InputTokenCount ?? 0, + OutputTokens: response.Usage?.OutputTokenCount ?? 0, LatencyMs: sw.ElapsedMilliseconds, - CostUsd: PriceTable.Cost(options?.ModelId, resp.Usage))); - - return resp; + CostUsd: wrapped.LastCostUsd)); } + + [ClassCleanup] + public static void WriteReports() => TelemetryStore.FlushTo( + Path.Combine(RepoRoot.Find(), ".copilot", "perf-reports", "evals", + ReportingConfig.ExecutionName)); } ``` -Register in DI as a decorator over the real client. +## Delegating client (sketch) + +```csharp +internal sealed class TelemetryCapturingChatClient(IChatClient inner, PriceTable prices) : IChatClient +{ + public decimal LastCostUsd { get; private set; } + + public async Task GetResponseAsync( + IEnumerable messages, ChatOptions? options = null, + CancellationToken cancellationToken = default) + { + var resp = await inner.GetResponseAsync(messages, options, cancellationToken); + var usage = resp.Usage; + LastCostUsd = prices.Cost(resp.ModelId, + usage?.InputTokenCount ?? 0, usage?.OutputTokenCount ?? 0); + return resp; + } -## Report — `telemetry.md` + // GetStreamingResponseAsync delegates similarly; usage parsed off the last update. -```markdown -# Telemetry — {{ utc_timestamp }} + public object? GetService(Type serviceType, object? serviceKey = null) => + inner.GetService(serviceType, serviceKey); -Inputs: {{ count }} Stub mode: {{ true | false }} + public void Dispose() => inner.Dispose(); +} +``` -| Agent | Model | Calls | Avg ms | p95 ms | Avg in tok | Avg out tok | Cost (USD) | -|-----------|--------------|-------|--------|--------|------------|-------------|------------| -| router | gpt-4o-mini | 12 | 340 | 520 | 180 | 22 | $0.00031 | -| planner | gpt-4o | 6 | 1240 | 1880 | 1100 | 260 | $0.00385 | -| worker | gpt-4o-mini | 18 | 410 | 640 | 320 | 180 | $0.00118 | +## `inputs.json` -Total cost: $0.00534 +```json +[ + { "agent": "receptionist", "text": "Hi" }, + { "agent": "behavioural", "text": "Tell me about a tough project" }, + { "agent": "technical", "text": "How would you throttle requests?" }, + { "agent": "summariser", "text": "Wrap up the interview" } +] ``` -## Machine-readable — `telemetry.json` +## `prices.json` ```json { - "timestamp": "2026-06-15T17:00:00Z", - "stub": false, - "records": [ - { "agent": "router", "model": "gpt-4o-mini", "input_tokens": 178, "output_tokens": 21, "latency_ms": 332, "cost_usd": 0.0000256 } - ], - "aggregate": { "calls": 36, "total_cost_usd": 0.00534 } + "gpt-4o-mini": { "input_per_1k": 0.00015, "output_per_1k": 0.0006 }, + "gpt-4o": { "input_per_1k": 0.0025, "output_per_1k": 0.01 }, + "o4-mini": { "input_per_1k": 0.003, "output_per_1k": 0.012 } } ``` -## JUnit-XML — `telemetry.junit.xml` - -Standard JUnit suite where each test case is one input id, marked -passed if the run succeeded (no thrown exception). Latency/token -metrics are emitted as `` per test case so CI can pick -them up. - -## Stub mode - -Two independent toggles control whether real models are called: - -- `EVAL_USE_REAL_AGENT` (default `0`) — when `0`, the wrapper - short-circuits the agent-under-test client and returns a deterministic - canned response. Telemetry numbers reflect the stub, marked `(stub)`. -- `EVAL_USE_REAL_JUDGE` (default `0`) — when `0`, quality mode skips - the real judge call and emits per-input scores of `null` with a - rationale of `"(stub) judge disabled"`. The pass-rate row reports - `(stub)`. - -Compare mode honors both toggles independently. Setting only -`EVAL_USE_REAL_AGENT=1` is a valid local-dev configuration: real -agent calls, no judge cost. +Edit freely — costs change. The price table is **never** baked into source. diff --git a/tests/dotnet-ai/setup-maf-evals/eval.yaml b/tests/dotnet-ai/setup-maf-evals/eval.yaml index f2a6704d22..c6600ec809 100644 --- a/tests/dotnet-ai/setup-maf-evals/eval.yaml +++ b/tests/dotnet-ai/setup-maf-evals/eval.yaml @@ -3,84 +3,244 @@ required_skills: - setup-maf-evals scenarios: - - name: scaffold-evals-project-fresh + - name: scaffold-evals-tests-project-fresh prompt: | - Run setup-maf-evals on the project at ./fixture. Wire all three modes - (telemetry, quality, compare). Skip the Aspire panel. + Run setup-maf-evals on the project at ./fixture. Wire telemetry, + quality, and compare modes. Skip Safety, Aspire panel, and CI workflow. setup: files: - - path: fixture/MyApp.sln + - path: fixture/MyApp.slnx content: | - Microsoft Visual Studio Solution File, Format Version 12.00 + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj content: | net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: | + var builder = DistributedApplication.CreateBuilder(args); + builder.AddAzureOpenAIChatClient("chat", "gpt-4o-mini"); + builder.Build().Run(); - path: fixture/MyApp.Coach/MyApp.Coach.csproj content: | net10.0 assertions: - type: file_exists - path: fixture/MyApp.Evals/MyApp.Evals.csproj + path: fixture/MyApp.Evals.Tests/MyApp.Evals.Tests.csproj + - type: file_exists + path: fixture/MyApp.Evals.Tests/dotnet-tools.json + - type: file_exists + path: fixture/MyApp.Evals.Tests/Reporting/ReportingConfig.cs - type: file_exists - path: fixture/MyApp.Evals/Telemetry/inputs.json + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs - type: file_exists - path: fixture/MyApp.Evals/Quality/rubric.md + path: fixture/MyApp.Evals.Tests/Quality/QualityTests.cs - type: file_exists - path: fixture/MyApp.Evals/Quality/golden.json + path: fixture/MyApp.Evals.Tests/Quality/golden.json - type: file_exists - path: fixture/MyApp.Evals/Compare/matrix.json + path: fixture/MyApp.Evals.Tests/Telemetry/TelemetryTests.cs + - type: file_exists + path: fixture/MyApp.Evals.Tests/Compare/CompareTests.cs + - type: file_contains + path: fixture/MyApp.Evals.Tests/MyApp.Evals.Tests.csproj + text: "Microsoft.Extensions.AI.Evaluation.Reporting" + - type: file_contains + path: fixture/MyApp.Evals.Tests/MyApp.Evals.Tests.csproj + text: "Microsoft.Extensions.AI.Evaluation.NLP" + - type: file_contains + path: fixture/MyApp.Evals.Tests/MyApp.Evals.Tests.csproj + text: "MSTest" + - type: file_contains + path: fixture/MyApp.Evals.Tests/dotnet-tools.json + text: "microsoft.extensions.ai.evaluation.console" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Quality/QualityTests.cs + text: "DiskBasedReportingConfiguration" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Quality/QualityTests.cs + text: "BLEUEvaluator" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Quality/golden.json + text: "reference_response" - type: file_contains - path: fixture/MyApp.Evals/MyApp.Evals.csproj - text: "Microsoft.Extensions.AI.Evaluation" + path: fixture/MyApp.Evals.Tests/Quality/golden.json + text: "schema_version" + - type: file_contains + path: fixture/.gitignore + text: ".copilot/perf-reports/evals/" + - type: file_contains + path: fixture/.gitignore + text: "_store/" - - name: skip-when-no-app-host + - name: scaffold-with-safety-tier prompt: | - Run setup-maf-evals on the project at ./fixture. + Run setup-maf-evals on the project at ./fixture. Include the Safety tier. setup: files: - - path: fixture/PlainApi/PlainApi.csproj + - path: fixture/MyApp.slnx content: | - net10.0 + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: | + var builder = DistributedApplication.CreateBuilder(args); + builder.AddAzureOpenAIChatClient("chat", "gpt-4o-mini"); assertions: - - type: output_contains - text: "agentic" + - type: file_exists + path: fixture/MyApp.Evals.Tests/Safety/SafetyTests.cs + - type: file_contains + path: fixture/MyApp.Evals.Tests/Safety/SafetyTests.cs + text: "ContentHarmEvaluator" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Safety/SafetyTests.cs + text: "Assert.Inconclusive" + - type: file_contains + path: fixture/MyApp.Evals.Tests/MyApp.Evals.Tests.csproj + text: "Microsoft.Extensions.AI.Evaluation.Safety" - - name: telemetry-stub-run-produces-report + - name: scaffold-with-ci-workflow prompt: | - Run setup-maf-evals on the project at ./fixture, then run - "dotnet run --project MyApp.Evals -- telemetry" with EVAL_USE_REAL_MODELS unset. + Run setup-maf-evals on the project at ./fixture. Include the GitHub Actions workflow. setup: files: + - path: fixture/MyApp.slnx + content: | + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj content: | net10.0 - - path: fixture/MyApp.Coach/MyApp.Coach.csproj + - path: fixture/MyApp.AppHost/AppHost.cs + content: "var b = DistributedApplication.CreateBuilder(args); b.AddAzureOpenAIChatClient(\"chat\", \"gpt-4o-mini\");" + assertions: + - type: file_exists + path: fixture/.github/workflows/evals.yml + - type: file_contains + path: fixture/.github/workflows/evals.yml + text: "aieval report" + - type: file_contains + path: fixture/.github/workflows/evals.yml + text: "upload-artifact" + + - name: ichatclient-detection-azure-openai + prompt: | + Run setup-maf-evals on the project at ./fixture. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj content: | net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: | + var builder = DistributedApplication.CreateBuilder(args); + builder.AddAzureOpenAIChatClient("chat", "gpt-4o-mini"); assertions: + - type: file_exists + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs - type: file_contains - path: fixture/.gitignore - text: ".copilot/perf-reports/evals/" + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs + text: "AddAzureOpenAIChatClient" - type: output_contains - text: "stub" + text: "AppHost.cs" - - name: update-mode-preserves-user-edits + - name: ichatclient-detection-missing-emits-stub prompt: | - Run setup-maf-evals on the project at ./fixture. A MyApp.Evals project - already exists with a custom rubric. Do not overwrite it. + Run setup-maf-evals on the project at ./fixture. setup: files: + - path: fixture/MyApp.slnx + content: | + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj content: | net10.0 - - path: fixture/MyApp.Evals/MyApp.Evals.csproj + - path: fixture/MyApp.AppHost/AppHost.cs + content: "var b = DistributedApplication.CreateBuilder(args); b.Build().Run();" + assertions: + - type: file_contains + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs + text: "NotImplementedException" + - type: output_contains + text: "no registration found" + + - name: skip-when-no-app-host + prompt: | + Run setup-maf-evals on the project at ./fixture. + setup: + files: + - path: fixture/PlainApi/PlainApi.csproj + content: | + net10.0 + assertions: + - type: output_contains + text: "agentic" + + - name: update-mode-preserves-user-data + prompt: | + Run setup-maf-evals on the project at ./fixture. A MyApp.Evals.Tests project + already exists with a custom rubric and golden.json. Do not overwrite either. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.Evals.Tests/MyApp.Evals.Tests.csproj content: | net10.0 - - path: fixture/MyApp.Evals/Quality/rubric.md + - path: fixture/MyApp.Evals.Tests/Quality/rubric.md content: | # Custom rubric — DO NOT OVERWRITE - my_trait: ... + - path: fixture/MyApp.Evals.Tests/Quality/golden.json + content: | + [{"id": "custom", "user_message": "preserve me", "expected_traits": ["unique"]}] assertions: - type: file_contains - path: fixture/MyApp.Evals/Quality/rubric.md + path: fixture/MyApp.Evals.Tests/Quality/rubric.md text: "DO NOT OVERWRITE" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Quality/golden.json + text: "preserve me" + + - name: tier-banner-surfaces-in-chat-output + prompt: | + Run setup-maf-evals on the project at ./fixture. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: "var b = DistributedApplication.CreateBuilder(args); b.AddAzureOpenAIChatClient(\"chat\", \"gpt-4o-mini\");" + assertions: + - type: output_contains + text: "EVAL_USE_REAL_AGENT" + - type: output_contains + text: "EVAL_USE_REAL_JUDGE" + - type: output_contains + text: "EVAL_USE_FOUNDRY_SAFETY" From 6a166c7b4ecddd7932201b599db566282f3baa26 Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Thu, 18 Jun 2026 12:27:24 -0700 Subject: [PATCH 02/18] setup-maf-evals: dogfood findings from ELI5Agent run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three gaps surfaced when scaffolding ELI5Agent.Evals.Tests against an Aspire 13.2 + Foundry app: 1. **Detection table missing the Aspire Inference pattern.** `builder.AddAzureChatCompletionsClient(\"chat\").AddChatClient(\"chat\")` is the standard Aspire 13.2 way of wiring an IChatClient against a Foundry chat deployment, and was not in the v2 detection table. Added to ichatclient-detection.md with a note that the argument is the connection-string name. 2. **Connection-string setup not surfaced for standalone runs.** When the app uses Aspire orchestration, `ConnectionStrings:` is populated by the AppHost — but `dotnet test` runs outside the host and gets a silent missing-config NRE on first real-agent run. Added a `Connection-string setup for standalone test runs` section to ichatclient-detection.md that surfaces both user-secrets and env-var setup paths and points to `azd env get-values`. 3. **WordCountEvaluator implementation not pinned.** The catalog called the evaluator out as `always scaffolded (custom)` but never gave a verbatim template. Added the Learn-doc canonical implementation to evaluators-catalog.md so every scaffold gets the same (correct) IEvaluator skeleton. Validated end-to-end against ELI5Agent: scaffolded ELI5Agent.Evals.Tests, `dotnet test` exits 0 in stub tier with 4 scenarios x 4 metrics (Words/BLEU/GLEU/F1) producing a 670 KB report.html. Real-judge tier deferred (no Azure creds set up locally for ELI5Agent — itself the trigger for finding 2). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../references/evaluators-catalog.md | 37 +++++++++++++++++++ .../references/ichatclient-detection.md | 32 ++++++++++++++++ 2 files changed, 69 insertions(+) diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md index 44dc3020f1..ca6a90dea3 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md @@ -22,6 +22,43 @@ Plus a built-in custom evaluator the skill always scaffolds: |-----------|--------|-----| | `WordCountEvaluator` (custom) | `Words` | Sanity check: response is non-empty and reasonable length. Same pattern as the Learn doc tutorial. | +### `WordCountEvaluator` reference implementation + +Scaffold this file verbatim (the Learn-doc canonical pattern) into +`Reporting/WordCountEvaluator.cs`. It runs in stub tier with no API key. + +```csharp +public sealed class WordCountEvaluator : IEvaluator +{ + public const string MetricName = "Words"; + public IReadOnlyCollection EvaluationMetricNames { get; } = [MetricName]; + + public ValueTask EvaluateAsync( + IEnumerable messages, + ChatResponse modelResponse, + ChatConfiguration? chatConfiguration = null, + IEnumerable? additionalContext = null, + CancellationToken cancellationToken = default) + { + var text = modelResponse?.Text ?? string.Empty; + var count = text.Split( + new[] { ' ', '\t', '\r', '\n' }, + StringSplitOptions.RemoveEmptyEntries).Length; + + var metric = new NumericMetric(MetricName, value: count) + { + Interpretation = count switch + { + < 5 => new EvaluationMetricInterpretation(EvaluationRating.Poor, reason: "Response too short"), + > 500 => new EvaluationMetricInterpretation(EvaluationRating.Average, reason: "Response very long"), + _ => new EvaluationMetricInterpretation(EvaluationRating.Good), + } + }; + return new ValueTask(new EvaluationResult(metric)); + } +} +``` + ## Tier 2 — Quality (LLM-as-judge) Package: `Microsoft.Extensions.AI.Evaluation.Quality` (GA 10.7.0). diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md index 6d2f1357c0..7fcb3570f3 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md @@ -16,9 +16,16 @@ directories**. Match (case-insensitive, multi-line): | `AddOpenAIChatClient\s*\(` | OpenAI direct | `Program.cs` | | `AddOllamaChatClient\s*\(` | Ollama | `Program.cs` | | `AddAIInference\s*\(` (Foundry deployment alias) | Azure AI Foundry | `AppHost.cs` | +| `AddAzureChatCompletionsClient\s*\([^)]*\)\s*\.AddChatClient\s*\(` | Aspire `Aspire.Azure.AI.Inference` (Foundry-routed) | `Program.cs` | | `services\.AddSingleton` (any explicit registration) | custom | varies | | `\.AsIChatClient\(\)` (after an SDK client) | manual wrap | varies | +> The `AddAzureChatCompletionsClient(...).AddChatClient(...)` chain is the +> standard Aspire 13.2 way of wiring an `IChatClient` against a Foundry chat +> deployment. The argument is the **connection-string name**, which Aspire's +> AppHost populates automatically (`AddDeployment("chat", ...)` -> connection +> string `chat`). The factory mirrors both calls verbatim. + Capture the deployment alias / model id literal if present (e.g., `builder.AddAIInference("chat", "gpt-4o-mini")` → alias `chat`). @@ -92,6 +99,31 @@ And by default the judge client is the **same** instance (saves a duplicate Azure credential setup). The user can override by setting `EVAL_JUDGE_DEPLOYMENT_NAME` to a different deployment alias. +## Connection-string setup for standalone test runs + +When the target app uses Aspire orchestration, the AppHost populates +`ConnectionStrings:` automatically. **`dotnet test` runs outside +the AppHost and does not get this for free.** Surface this in the chat +output any time the detected pattern reads from a connection string +(`AddAzureChatCompletionsClient`, `AddAIInference`, `AddOllamaChatClient`, +or `AddAzureOpenAIChatClient` without an explicit endpoint). + +Recommend the user wire one of: + +```pwsh +# Option A — user secrets (recommended for local dev) +dotnet user-secrets init --project .Evals.Tests +dotnet user-secrets set "ConnectionStrings:" "Endpoint=https://...;Key=..." --project .Evals.Tests + +# Option B — env var (works in CI) +$env:ConnectionStrings__ = "Endpoint=https://...;Key=..." +``` + +For Foundry-routed clients the connection string is what `azd env get-values` +prints for `connectionString` against the deployment resource. Document this +in the chat output along with the tier banner so the user doesn't see a +silent NRE on first real-agent run. + ## Chat output (step 2) When detection succeeds, surface as: From 8255ef82bd24e8e0d5755ef3589b322f71c0a7fa Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Thu, 18 Jun 2026 13:11:02 -0700 Subject: [PATCH 03/18] setup-maf-evals: per-run metrics glossary + friendly user-secrets diagnostic 1. New references/metrics-glossary.md authored as the source of truth for metric definitions/scales/thresholds across NLP, Quality, and Safety tiers. Includes the canonical Reporting/MetricsGlossary.cs template that the skill emits into .Evals.Tests/Reporting/. 2. Factory template in references/ichatclient-detection.md now wraps DI resolution in try/catch and throws a friendly InvalidOperationException naming the connection-string key + the exact 'dotnet user-secrets set' command + the env-var alternative + 'azd env get-values' pointer. Replaces the silent NRE that ELI5Agent dogfooding hit when EVAL_USE_REAL_AGENT=1 with no creds set up. 3. Two new eval.yaml scenarios bind the new behavior: - scaffold-emits-metrics-glossary: asserts MetricsGlossary.cs exists, references metrics-glossary.md, has [AssemblyCleanup] - factory-emits-friendly-secrets-diagnostic: asserts AgentChatClientFactory mentions 'dotnet user-secrets' + 'ConnectionStrings' 4. Captured two MSTest constraints that turned up while implementing the glossary writer: - quality-modes.md ReportingConfig: ExecutionName must be cached at class load, NOT re-evaluated per call (otherwise AievalReport and MetricsGlossary land in different timestamped folders 5 s apart) - metrics-glossary.md template now declares MetricsGlossary as a plain static class; MSTest forbids more than one [AssemblyCleanup] per assembly (UTA014). Glossary write is chained from AievalReport's single AssemblyCleanup, wrapped in try/catch so a glossary-write failure doesn't mask the report. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../references/ichatclient-detection.md | 23 +- .../references/metrics-glossary.md | 247 ++++++++++++++++++ .../references/quality-modes.md | 3 + tests/dotnet-ai/setup-maf-evals/eval.yaml | 52 ++++ 4 files changed, 323 insertions(+), 2 deletions(-) create mode 100644 plugins/dotnet-ai/skills/setup-maf-evals/references/metrics-glossary.md diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md index 7fcb3570f3..0d2318d994 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md @@ -51,14 +51,33 @@ internal static class AgentChatClientFactory var builder = Host.CreateApplicationBuilder(); // {{InsertDetectedRegistrationCallVerbatim}} var host = builder.Build(); - return host.Services.GetRequiredService(); + try + { + return host.Services.GetRequiredService(); + } + catch (InvalidOperationException ex) + { + throw new InvalidOperationException( + "EVAL_USE_REAL_AGENT=1 but IChatClient could not be resolved. " + + "The detected registration ({{DetectionSummary}}) reads connection " + + "string \"{{ConnStrName}}\" from configuration. Aspire's AppHost " + + "populates this at runtime, but `dotnet test` runs standalone. " + + "Wire one of:\n" + + " dotnet user-secrets set \"ConnectionStrings:{{ConnStrName}}\" " + + "\"Endpoint=https://...;Key=...\" --project {{AppName}}.Evals.Tests\n" + + "or set the env var:\n" + + " $env:ConnectionStrings__{{ConnStrName}} = \"Endpoint=...;Key=...\"\n" + + "Get the value from `azd env get-values` against the deployment resource.", + ex); + } } } ``` Where `{{InsertDetectedRegistrationCallVerbatim}}` is the literal call copied from the detection source (with any required `using`s in scope -via `GlobalUsings.cs`). +via `GlobalUsings.cs`), and `{{ConnStrName}}` is the connection-string +literal extracted from the call (e.g., the `"chat"` argument). ### Case B — multiple registrations found diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/metrics-glossary.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/metrics-glossary.md new file mode 100644 index 0000000000..3ed5432217 --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/metrics-glossary.md @@ -0,0 +1,247 @@ +# Metrics glossary + +Source of truth for the per-run `metrics-glossary.md` artifact that the +scaffolded `.Evals.Tests/Reporting/MetricsGlossary.cs` writes next +to `report.html`. The aieval HTML report is data-bound JSON — it shows +numbers but no definitions — so we co-locate this glossary with the +report so a first-time reader has a one-page cheat-sheet. + +The skill emits **only the entries for evaluators that actually ran** in +the active tier, so a stub-tier user sees Words/BLEU/GLEU/F1 and +nothing else. + +## NLP tier (deterministic, no LLM) + +### `Words` +- **Custom evaluator** scaffolded by this skill (see `evaluators-catalog.md`). +- **What it measures:** raw token count of the response text. +- **Scale:** integer ≥ 0. +- **Interpretation:** `< 5` → Poor (response too short / empty). `5–500` → Good. `> 500` → Average (response unusually long). +- **Trust for:** sanity-checking that the model produced *anything* and isn't running away with a 10-page essay. +- **Don't trust for:** quality. A 50-word wrong answer scores the same as a 50-word correct answer. + +### `BLEU` — Bilingual Evaluation Understudy +- **What it measures:** n-gram (1-4) overlap between the response and one or more reference strings, with a brevity penalty. +- **Scale:** 0.0 – 1.0. +- **Interpretation:** ~0.0–0.1 weak overlap, often paraphrased; ~0.1–0.3 normal for free-form generation; ~0.3–0.5 strong; > 0.5 near-quotation. +- **Trust for:** "is the response in the same lexical neighbourhood as the reference?" — useful as a regression signal when the reference is canonical. +- **Don't trust for:** semantic correctness. A correct paraphrase scores low; an incorrect copy-paste of reference fragments scores high. +- **Reference:** [`BLEUEvaluator` in MEAI.Evaluation.NLP](https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.ai.evaluation.nlp.bleuevaluator). + +### `GLEU` — Google-BLEU +- **What it measures:** sentence-level BLEU variant. Symmetric — penalises both missing reference n-grams and extra invented ones. +- **Scale:** 0.0 – 1.0. +- **Interpretation:** same buckets as BLEU; GLEU usually tracks BLEU but is less brittle on short outputs. +- **Trust for:** the same use cases as BLEU when individual scenarios are short (1-2 sentences) and BLEU's brevity penalty would be misleading. +- **Don't trust for:** any semantic claim. Same caveats as BLEU. +- **Reference:** [`GLEUEvaluator`](https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.ai.evaluation.nlp.gleuevaluator). + +### `F1` — Token-level F1 +- **What it measures:** harmonic mean of unigram precision and recall against the ground-truth string. Order-insensitive. +- **Scale:** 0.0 – 1.0. +- **Interpretation:** ~0.0–0.2 mostly-disjoint vocab; 0.3–0.5 typical for free-form generation; > 0.6 strong word-level match. +- **Trust for:** QA / extraction-style scenarios where the *set of words* matters more than phrasing (SQuAD-style benchmarks use this). +- **Don't trust for:** word-order-sensitive tasks (e.g., "yes" vs "no" answers buried in long output) — the F1 score will be high even if the polarity is wrong. +- **Reference:** [`F1Evaluator`](https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.ai.evaluation.nlp.f1evaluator). + +> **NLP-tier headline:** all three of BLEU/GLEU/F1 are *lexical* metrics. +> They're cheap, deterministic, and free, but they cannot tell you the +> response is *correct* — only that it's *similar in wording*. Treat them +> as a regression early-warning, not a quality verdict. + +## Quality tier (LLM-as-judge — `EVAL_USE_REAL_JUDGE=1`) + +All Quality metrics produce a 1-5 `EvaluationRating` (Poor → Excellent) +plus a free-text rationale from the judge model. + +### `Relevance` +- **What it measures:** does the response actually address the user's query? +- **Trust for:** catching off-topic regressions (model started monologuing about a different topic). +- **Don't trust for:** factual correctness — a *relevantly wrong* answer can score high. + +### `Coherence` +- **What it measures:** is the response logically structured / orderly / easy to follow? +- **Trust for:** detecting rambling, contradictory, or logically broken outputs. +- **Don't trust for:** depth or correctness — coherent nonsense scores high. + +### `Fluency` +- **What it measures:** grammar, readability, naturalness. +- **Trust for:** detecting broken-English / token-soup outputs from undertrained or quantised models. +- **Don't trust for:** anything beyond surface text quality. A fluent lie scores high. + +### `Completeness` +- **What it measures:** how comprehensive and accurate the response is, given a reference (`CompletenessEvaluatorContext(groundTruth)`). +- **Trust for:** catching responses that are correct but partial (covered 2 of 4 required points). +- **Don't trust for:** brevity-as-a-feature scenarios — a short-but-correct answer can score lower than a long-and-padded one. + +### `Equivalence` +- **What it measures:** semantic similarity between response and ground truth in the context of the original query. +- **Trust for:** distinguishing "right answer, different words" (high) from "wrong answer" (low). Better than BLEU/F1 when paraphrasing is acceptable. +- **Don't trust for:** any case where the ground truth is itself ambiguous or one of multiple valid answers. + +### `Groundedness` +- **What it measures:** alignment between the response and a supplied source-of-truth context (`GroundednessEvaluatorContext(context)`). +- **Trust for:** RAG pipelines — flags when the model invented facts not in the retrieved context. +- **Don't trust for:** open-ended chat without a context document. + +### Agentic-only + +The skill wires these only when an `*.AppHost.csproj` is detected. + +- **`Intent Resolution`** — did the model identify and resolve the user's actual intent (vs answering a related-but-different question)? +- **`Task Adherence`** — did the model stick to the assigned task or wander into other agent territory? +- **`Tool Call Accuracy`** — did the model invoke the right tools with the right arguments? Requires `ToolCallAccuracyEvaluatorContext`. + +> **Quality-tier headline:** these are LLM-judge subjective scores. They +> drift across judge model versions. Pin the judge model in +> `quality.thresholds.json` if you want comparable scores across runs. + +## Safety tier (Foundry — `EVAL_USE_FOUNDRY_SAFETY=1`) + +All Safety evaluators return a 1-5 severity (1 = safe, 5 = severe harm) +plus a confidence score. Each metric is an *output classifier* — it +inspects the model's response, not the input prompt. + +### `Hate And Unfairness`, `Self Harm`, `Violence`, `Sexual` +- Wired together as the single-shot `ContentHarmEvaluator` (one Foundry call → all 4 metrics). +- **Trust for:** detecting harmful content in agent outputs. +- **Don't trust for:** input-side filtering — these never see the user's prompt. Pair with input-side Azure AI Content Safety for full coverage. + +### `Protected Material` +- Detects copyrighted text / song lyrics / book passages reproduced in the output. + +### `Indirect Attack` +- Detects prompt-injection-style content that would indicate the model picked up an indirect attack from retrieved content or tool output. Closest thing to an input-side check available in this tier. + +### `Code Vulnerability` +- Flags vulnerable patterns in code the model emitted (SQL injection, hardcoded credentials, weak crypto, etc.). + +### `Ungrounded Attributes` +- Detects inferred human attributes (race, gender, age, religion, etc.) in the response that weren't in the input. + +### `Groundedness Pro` +- Foundry-hosted fine-tuned groundedness evaluator. More accurate than the open-source `GroundednessEvaluator` but costs a Foundry call per scenario. + +> **Safety-tier headline:** all safety metrics are *output classifiers*. +> They protect downstream consumers from the agent's outputs; they do not +> protect the agent from its inputs. + +## Quick reference card + +| Metric | Tier | Scale | Needs ground truth? | Needs LLM? | +|--------|------|-------|---------------------|-----------| +| Words | NLP | int | no | no | +| BLEU | NLP | 0-1 | yes (references) | no | +| GLEU | NLP | 0-1 | yes (references) | no | +| F1 | NLP | 0-1 | yes (one string) | no | +| Relevance / Coherence / Fluency | Quality | 1-5 | no | yes (judge) | +| Completeness / Equivalence | Quality | 1-5 | yes (one string) | yes (judge) | +| Groundedness | Quality | 1-5 | yes (context) | yes (judge) | +| Intent Resolution / Task Adherence | Quality (agentic) | 1-5 | no | yes (judge) | +| Tool Call Accuracy | Quality (agentic) | 1-5 | yes (expected_tool_calls) | yes (judge) | +| Content Harm bundle (4 metrics) | Safety | 1-5 severity | no | yes (Foundry) | +| Protected / Indirect / Code Vuln. / Ungrounded | Safety | 1-5 severity | varies | yes (Foundry) | +| Groundedness Pro | Safety | 1-5 | yes (context) | yes (Foundry) | + +## `MetricsGlossary.cs` template + +The scaffolded `.Evals.Tests/Reporting/MetricsGlossary.cs` writes +the tier-relevant slice of this glossary next to `report.html` after +each `dotnet test` run. + +> **MSTest constraint:** an assembly may declare **only one** +> `[AssemblyCleanup]` method. The skill emits `MetricsGlossary` as a +> plain static class (no `[TestClass]`, no `[AssemblyCleanup]`) and +> chains `MetricsGlossary.WriteGlossary()` from +> `AievalReport.GenerateReport`'s single `[AssemblyCleanup]`. +> Wrap the call in a `try/catch` so a glossary-write failure never +> masks the report. + +```csharp +internal static class MetricsGlossary +{ + public static void WriteGlossary() + { + var outDir = Path.Combine( + RepoRoot.Find(), ".copilot", "perf-reports", "evals", EvalEnv.ExecutionName); + Directory.CreateDirectory(outDir); + var path = Path.Combine(outDir, "metrics-glossary.md"); + + var sb = new StringBuilder(); + sb.AppendLine($"# Metrics glossary — {EvalEnv.Tier} tier"); + sb.AppendLine(); + sb.AppendLine($"Generated: {DateTime.UtcNow:O}"); + sb.AppendLine(); + sb.AppendLine($"Companion to `report.html` in this folder. The aieval HTML report shows numbers; this file explains them."); + sb.AppendLine(); + + sb.AppendLine(NlpEntries); + if (EvalEnv.UseRealJudge) sb.AppendLine(QualityEntries); + if (EvalEnv.UseFoundrySafety) sb.AppendLine(SafetyEntries); + + sb.AppendLine(); + sb.AppendLine("> Source: setup-maf-evals references/metrics-glossary.md"); + + File.WriteAllText(path, sb.ToString()); + Console.WriteLine($"[MetricsGlossary] {path}"); + } + + private const string NlpEntries = """ + ## NLP tier (deterministic, no LLM) + - **Words** (int): response length sanity check. <5 too short, 5-500 ok, >500 long. + - **BLEU** (0-1): n-gram overlap with reference(s). 0.1-0.3 normal, >0.3 strong, >0.5 near-quotation. *Lexical, not semantic.* + - **GLEU** (0-1): sentence-level BLEU; better for short outputs. Same buckets as BLEU. + - **F1** (0-1): unigram token F1 vs ground-truth. 0.3-0.5 typical, >0.6 strong word-level match. Order-insensitive. + + > Headline: NLP metrics measure wording similarity, not correctness. Use for regression early-warning, not as quality verdicts. + """; + + private const string QualityEntries = """ + ## Quality tier (LLM-as-judge) + Each rated 1-5 (Poor → Excellent) with a free-text rationale. + - **Relevance**: addresses the user's query. Catches off-topic regressions. + - **Coherence**: logically structured. Catches rambling/contradictory outputs. + - **Fluency**: grammar/readability. Catches broken-English outputs. + - **Completeness** (needs reference): comprehensive and accurate. + - **Equivalence** (needs reference): semantic similarity in context of the query. + - **Groundedness** (needs context): aligned with supplied source-of-truth. + - **Intent Resolution / Task Adherence / Tool Call Accuracy** (agentic only). + + > Headline: judge scores drift across model versions. Pin the judge model for comparable runs. + """; + + private const string SafetyEntries = """ + ## Safety tier (Foundry) + Each rated 1-5 severity (1 safe → 5 severe). + - **ContentHarm bundle** (single-shot, 4 metrics): Hate-And-Unfairness, Self-Harm, Violence, Sexual. + - **Protected Material**: copyrighted text reproduced. + - **Indirect Attack**: prompt-injection content from retrieved/tool data. + - **Code Vulnerability**: vulnerable code patterns (SQLi, weak crypto, etc.). + - **Ungrounded Attributes**: inferred human attributes not in input. + - **Groundedness Pro**: Foundry-hosted fine-tuned groundedness check. + + > Headline: all safety metrics inspect *outputs*, not inputs. Pair with Azure AI Content Safety on the request side for full coverage. + """; +} +``` + +Then in `Reporting/AievalReport.cs` (the assembly's single +`[AssemblyCleanup]` host): + +```csharp +[TestClass] +public static class AievalReport +{ + [AssemblyCleanup] + public static void GenerateReport() + { + // ... aieval invocation ... + + try { MetricsGlossary.WriteGlossary(); } + catch (Exception ex) { Console.Error.WriteLine($"[MetricsGlossary] Failed: {ex.Message}"); } + } +} +``` + +The skill should emit both files verbatim — change the `private const` +strings only if upstream MEAI changes a metric definition. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/quality-modes.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/quality-modes.md index eecb89b281..d11111d573 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/quality-modes.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/quality-modes.md @@ -26,6 +26,9 @@ internal static class ReportingConfig public static readonly string StorageRoot = Path.Combine(RepoRoot.Find(), "_store"); + // Resolved once at class load — must NOT re-evaluate DateTime.UtcNow per + // call, otherwise AievalReport and MetricsGlossary land in different + // timestamped output folders. public static readonly string ExecutionName = Environment.GetEnvironmentVariable("EVAL_EXECUTION_NAME") ?? DateTime.UtcNow.ToString("yyyyMMdd-HHmmss"); diff --git a/tests/dotnet-ai/setup-maf-evals/eval.yaml b/tests/dotnet-ai/setup-maf-evals/eval.yaml index c6600ec809..3d41826c73 100644 --- a/tests/dotnet-ai/setup-maf-evals/eval.yaml +++ b/tests/dotnet-ai/setup-maf-evals/eval.yaml @@ -244,3 +244,55 @@ scenarios: text: "EVAL_USE_REAL_JUDGE" - type: output_contains text: "EVAL_USE_FOUNDRY_SAFETY" + + - name: scaffold-emits-metrics-glossary + prompt: | + Run setup-maf-evals on the project at ./fixture. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: "var b = DistributedApplication.CreateBuilder(args); b.AddAzureOpenAIChatClient(\"chat\", \"gpt-4o-mini\");" + assertions: + - type: file_exists + path: fixture/MyApp.Evals.Tests/Reporting/MetricsGlossary.cs + - type: file_contains + path: fixture/MyApp.Evals.Tests/Reporting/MetricsGlossary.cs + text: "metrics-glossary.md" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Reporting/MetricsGlossary.cs + text: "[AssemblyCleanup]" + - type: output_contains + text: "metrics-glossary.md" + + - name: factory-emits-friendly-secrets-diagnostic + prompt: | + Run setup-maf-evals on the project at ./fixture. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: | + var builder = DistributedApplication.CreateBuilder(args); + builder.AddAzureChatCompletionsClient("chat").AddChatClient("chat"); + assertions: + - type: file_contains + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs + text: "dotnet user-secrets" + - type: file_contains + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs + text: "ConnectionStrings" From 012a9a63a196279a76ad960f70bdb24cd0e6e93b Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Thu, 18 Jun 2026 13:11:22 -0700 Subject: [PATCH 04/18] setup-maf-evals: slim SKILL.md by moving prose into references/ Spec was creeping toward the validator's 'comprehensive' threshold (3,574 BPE tokens / 14,238 chars / 265 lines). Pulled prose-heavy sections into references and kept SKILL.md focused on decision-relevant content. - Step 3 (Scaffold): replaced 27-line file tree with a one-sentence summary + link to project-template.md. Kept the post-write powershell as a fenced code block so the agent has concrete commands. - Steps 4-9 (telemetry/quality/compare/safety/panel/CI): collapsed bulleted prose into 3-4 line stubs that retain the decision facts (default ON/OFF, opt-in semantics, env knob) and link the corresponding reference. - Step 11 (Surface in chat): added the metrics-glossary.md path to the 'Paths' bullet and to the trailing 'see also' line. - ## Common pitfalls: extracted to references/common-pitfalls.md (also adds two new entries from this work: the multi-AssemblyCleanup MSTest constraint and the AgentChatClientFactory friendly-NRE pattern). SKILL.md keeps a one-line link. - ## References: trimmed each bullet to one short line; consolidated the three external links onto a single line. Net effect: SKILL.md 14,238 -> 10,449 chars, 265 -> 189 lines, 3,574 -> 2,672 BPE tokens (-25%). Validator still passes; eval.yaml scenarios are unaffected (assertions are file-based, not prose-based). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../dotnet-ai/skills/setup-maf-evals/SKILL.md | 203 ++++++------------ .../references/common-pitfalls.md | 87 ++++++++ 2 files changed, 152 insertions(+), 138 deletions(-) create mode 100644 plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md b/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md index 767a3d802d..a16145a8dc 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md @@ -70,126 +70,76 @@ Present the detection summary, then confirm: ### 3. Scaffold the project -Use `references/project-template.md`. Creates: - -``` -.Evals.Tests/ - .Evals.Tests.csproj # MSTest + MEAI.Evaluation refs (Reporting, NLP, Quality, optional Safety) - Reporting/ - ReportingConfig.cs # DiskBasedReportingConfiguration factory; tier-aware evaluator list - Tier.cs # EVAL_USE_REAL_AGENT / EVAL_USE_REAL_JUDGE / EVAL_USE_FOUNDRY_SAFETY enum - Wire/ - AgentChatClientFactory.cs # auto-generated from IChatClient detection (step 1) - StubChatClient.cs # used when EVAL_USE_REAL_AGENT is unset - Telemetry/ - TelemetryTests.cs # [TestMethod] per input - inputs.json - prices.json - Quality/ - QualityTests.cs # [TestMethod] per golden scenario; NLP always, Quality when judge on - rubric.md - golden.json # schema_version, user_message, reference_response, expected_traits, optional context, optional expected_tool_calls - Compare/ - CompareTests.cs # [DataRow] per matrix entry; distinct executionName per entry - matrix.json - Safety/ # only emitted if user opted in - SafetyTests.cs # ContentHarmEvaluator + ProtectedMaterial + IndirectAttack - quality.thresholds.json # per-metric (Relevance / Coherence / BLEU / ...) -> minimum EvaluationRating - GlobalUsings.cs - dotnet-tools.json # NEW — pins aieval (Microsoft.Extensions.AI.Evaluation.Console, GA) - .github/workflows/evals.yml # OPTIONAL — only if user opted in -``` +See `references/project-template.md` for the file tree, csproj, and +`GlobalUsings.cs` template. The project is **MSTest** by default +(`.Evals.Tests`); a console-runner shape is available behind an +explicit `--shape console` flag. + +Always emit: `Reporting/{ReportingConfig.cs, Tier.cs, AievalReport.cs, +WordCountEvaluator.cs, MetricsGlossary.cs}`, `Wire/{AgentChatClientFactory.cs, +StubChatClient.cs}`, `Quality/{QualityTests.cs, rubric.md, golden.json}`, +`Telemetry/{TelemetryTests.cs, inputs.json, prices.json}`, +`Compare/{CompareTests.cs, matrix.json}`, `quality.thresholds.json`, +`GlobalUsings.cs`, `dotnet-tools.json`. Emit `Safety/SafetyTests.cs` and +`.github/workflows/evals.yml` only if the user opted in (steps 7 and 9). After writing files: -1. `dotnet sln add .Evals.Tests/.Evals.Tests.csproj` -2. Update `.gitignore`: append `.copilot/perf-reports/evals/` and `.Evals.Tests/_store/` if missing. -3. `dotnet tool restore` (installs `aieval`). +```pwsh +dotnet sln add .Evals.Tests/.Evals.Tests.csproj +dotnet tool restore # installs aieval +# .gitignore additions +echo ".copilot/perf-reports/evals/`n.Evals.Tests/_store/" >> .gitignore +``` ### 4. Wire telemetry mode -See `references/telemetry-capture.md`. - -- `TelemetryTests` hooks into the resolved `IChatClient` via a - delegating wrapper that records `gen_ai.usage.input_tokens`, - `gen_ai.usage.output_tokens`, per-call latency, and a - price-table-driven cost estimate. -- Emits a Markdown report at - `.copilot/perf-reports/evals//telemetry.md`, - a machine-readable `telemetry.json`, and a `telemetry.junit.xml`. -- **Note:** Telemetry mode is *not* an MEAI eval report. It's a - cost/latency capture. The HTML report comes from quality mode. +See `references/telemetry-capture.md`. Default ON. Captures latency, +input/output tokens, and price-table-driven cost via a delegating +`IChatClient`. Writes `telemetry.{md,json,junit.xml}` next to +`report.html`. **Not** the MEAI HTML report — that's quality mode's job. ### 5. Wire quality mode -See `references/quality-modes.md`. - -- `QualityTests` uses `DiskBasedReportingConfiguration` + - `ScenarioRun.EvaluateAsync` — the actual MEAI reporting pipeline. -- Stub tier registers `WordCountEvaluator`, `BLEUEvaluator`, - `GLEUEvaluator`, `F1Evaluator`. Judge tier adds the LLM-judge - evaluators listed in step 2. -- Each evaluator gets its required `EvaluationContext` from - `golden.json` (`BLEUEvaluatorContext(references)`, - `F1EvaluatorContext(groundTruth)`, etc.). -- After all `[TestMethod]`s run, an `[AssemblyCleanup]` invokes - `dotnet tool run aieval report --path _store --output .copilot/perf-reports/evals//report.html`. +See `references/quality-modes.md`. Default ON. The **only** runner that +produces `report.html`. Uses `DiskBasedReportingConfiguration` + +`ScenarioRun.EvaluateAsync` + `[AssemblyCleanup]` invokes +`dotnet tool run aieval report`. Stub tier registers the 4 NLP +evaluators; judge tier (`EVAL_USE_REAL_JUDGE=1`) adds the LLM-as-judge +evaluators from `references/evaluators-catalog.md`. ### 6. Wire compare mode -See `references/compare-mode.md`. - -- `CompareTests` uses `[DynamicData]` to feed each `matrix.json` entry - to a single test method. -- Each entry gets its own `executionName` so `aieval report` aggregates - the comparison view automatically. -- Produces `compare.md` (side-by-side latency / token / cost / quality - per matrix entry) **in addition to** the unified HTML report. +See `references/compare-mode.md`. Default ON. Reads `matrix.json`, +each entry runs through the **same** `ReportingConfiguration` with a +distinct `executionName`, so `aieval report` aggregates the comparison +columns into a single HTML view. Also writes a `compare.md` delta +table. ### 7. Wire safety mode (opt-in) -See `references/safety-mode.md`. - -Off by default. When enabled in step 2: - -- Adds `Microsoft.Extensions.AI.Evaluation.Safety` package. -- Generates `SafetyTests` using `ContentHarmEvaluator` (covers Hate + - SelfHarm + Violence + Sexual in one Foundry call), plus - `ProtectedMaterialEvaluator`, `IndirectAttackEvaluator`, - `CodeVulnerabilityEvaluator`, `UngroundedAttributesEvaluator`, and - optionally `GroundednessProEvaluator`. -- Skipped at runtime via `Assert.Inconclusive` if - `EVAL_USE_FOUNDRY_SAFETY` is unset — never fails the build for - missing creds. +See `references/safety-mode.md`. **Default OFF.** When enabled, adds +`Microsoft.Extensions.AI.Evaluation.Safety` and emits `SafetyTests.cs` +with `ContentHarmEvaluator` (single-shot 4-metric bundle), plus +ProtectedMaterial / IndirectAttack / CodeVulnerability / +UngroundedAttributes. Skipped at runtime via `Assert.Inconclusive` +when `EVAL_USE_FOUNDRY_SAFETY` is unset — never fails the build for +missing creds. ### 8. Optional Aspire panel (apply mode) -See `references/aspire-dashboard-panel.md`. Off by default; modifies +See `references/aspire-dashboard-panel.md`. Default OFF; modifies the AppHost project. To enable, user must say "wire the panel" / -equivalent. - -When enabled: - -1. Show a unified diff of the AppHost edit (`app.UseStaticFiles()` and - the panel files under `wwwroot/eval-panel/`). -2. Ask for confirmation. -3. Only on `yes` / explicit confirmation, write the changes. -4. Run `dotnet build` and report pass/fail. - -If declined, scaffold the static files into `.Evals.Tests/Panel/` -so the user can move them into the AppHost manually. +equivalent. Always show a unified diff + ask for confirmation before +writing AppHost edits. ### 9. Optional CI workflow (opt-in) -See `references/ci-workflow.md`. Off by default. - -When enabled, emits `.github/workflows/evals.yml`: - -- Runs `dotnet test` on every PR. -- Checks for repo secrets `AZURE_OPENAI_ENDPOINT` and `AZURE_TENANT_ID`; - if present, sets `EVAL_USE_REAL_JUDGE=1`. Otherwise stub tier. -- Runs `dotnet tool run aieval report` and uploads `report.html` as a - build artifact (PR comment optional). +See `references/ci-workflow.md`. Default OFF. Emits +`.github/workflows/evals.yml` that runs `dotnet test` on every PR, +auto-detects tier from repo secrets (`AZURE_OPENAI_ENDPOINT` → +judge; `AZURE_AI_FOUNDRY_ENDPOINT` → safety), and uploads +`report.html` as a workflow artifact. ### 10. Validation @@ -208,7 +158,9 @@ Print a 3-block summary: 1. **Tier banner.** Which tier is active (Stub / Judge / Foundry-Safety) and the exact env-var commands to upgrade. -2. **Paths.** Project path, HTML report path, persistent `_store/` path. +2. **Paths.** Project path, HTML report path, **glossary path** + (`metrics-glossary.md` co-located with `report.html`), persistent + `_store/` path. 3. **CLI invocations.** `dotnet test`, `dotnet tool run aieval report`, and the IChatClient detection result so the user knows what was auto-wired. @@ -216,50 +168,25 @@ Print a 3-block summary: `select-agent-models` recommendation to confirm no quality regression." -Also link `references/evaluators-catalog.md` so the user can see what -each metric means. +Also link `references/evaluators-catalog.md` and +`references/metrics-glossary.md` so the user can see what each metric +means. ## Common pitfalls -- **Hand-rolling reports instead of using the Reporting pipeline.** - The whole point of GA 10.7.0 is `DiskBasedReportingConfiguration` + - `aieval`. Never write a hand-rolled markdown report and call it the - "quality report" — that's an MEAI report (HTML) vs a cost/latency - capture (markdown). -- **Calling real models from the default test run.** Stub tier uses - `StubChatClient`; report is clearly marked `(stub IChatClient)`. - Real-model runs are opt-in via the three env vars. -- **Conflating agent and judge clients.** They're two different - `IChatClient` roles. The skill exposes them as two independent env - vars (`EVAL_USE_REAL_AGENT`, `EVAL_USE_REAL_JUDGE`) — one can be - real while the other is stubbed. -- **Hard-coding a price table.** Lives in `Telemetry/prices.json`, - user-editable. -- **Wiring 4 separate safety evaluators.** Use `ContentHarmEvaluator` - for the Hate/SelfHarm/Violence/Sexual bundle — single Foundry call, - 4 metrics back. -- **Auto-failing the build on quality regressions.** Quality mode is - informational by default. Users opt into a hard-fail by editing - `quality.thresholds.json` (which maps to real MEAI metric names like - `Relevance` / `BLEU` / `F1` and `EvaluationRating` levels). -- **Forgetting `.gitignore` entries.** Must include both - `.copilot/perf-reports/evals/` and `.Evals.Tests/_store/`. -- **Treating telemetry/compare/quality as separate report streams.** - Compare mode goes through `ReportingConfiguration` with a distinct - `executionName` per matrix entry, so `aieval report` aggregates them - into the same HTML. +See `references/common-pitfalls.md`. ## References -- `references/project-template.md` — exact files and `.csproj` layout (MSTest shape). -- `references/ichatclient-detection.md` — how to scan AppHost + agent for `IChatClient` registration and emit `AgentChatClientFactory.cs`. -- `references/evaluators-catalog.md` — full catalog of NLP + Quality + Safety evaluators with required `EvaluationContext` types and which tier they belong to. -- `references/telemetry-capture.md` — per-call hook + cost report format. Calls out: this is NOT the MEAI HTML report. -- `references/quality-modes.md` — `DiskBasedReportingConfiguration` wiring, tier-based evaluator registration, `aieval report` invocation. -- `references/compare-mode.md` — `matrix.json` layout, `[DynamicData]` test shape, per-entry `executionName`. -- `references/safety-mode.md` — opt-in safety scaffold, Foundry runtime check, `ContentHarmEvaluator` default. +- `references/project-template.md` — file tree + `.csproj` layout. +- `references/ichatclient-detection.md` — registration scan + factory emission. +- `references/evaluators-catalog.md` — NLP + Quality + Safety catalog with required `EvaluationContext` types. +- `references/metrics-glossary.md` — per-run glossary content + `MetricsGlossary.cs` template. +- `references/telemetry-capture.md` — per-call hook + cost report format. +- `references/quality-modes.md` — `DiskBasedReportingConfiguration` wiring + `aieval report` invocation. +- `references/compare-mode.md` — `matrix.json` + per-entry `executionName`. +- `references/safety-mode.md` — opt-in safety scaffold + `ContentHarmEvaluator` default. - `references/ci-workflow.md` — `.github/workflows/evals.yml` template. - `references/aspire-dashboard-panel.md` — optional static-file panel. -- [Microsoft.Extensions.AI.Evaluation libraries](https://learn.microsoft.com/en-us/dotnet/ai/evaluation/libraries) — upstream catalog of evaluators. -- [Tutorial: evaluate with reporting](https://learn.microsoft.com/en-us/dotnet/ai/evaluation/evaluate-with-reporting) — canonical MSTest pattern. -- [dotnet/ai-samples → microsoft-extensions-ai-evaluation/api](https://github.com/dotnet/ai-samples/blob/main/src/microsoft-extensions-ai-evaluation/api/) — canonical unit-test examples. +- `references/common-pitfalls.md` — known footguns to avoid when scaffolding. +- [MEAI.Evaluation libraries](https://learn.microsoft.com/en-us/dotnet/ai/evaluation/libraries) | [Tutorial: evaluate with reporting](https://learn.microsoft.com/en-us/dotnet/ai/evaluation/evaluate-with-reporting) | [dotnet/ai-samples](https://github.com/dotnet/ai-samples/blob/main/src/microsoft-extensions-ai-evaluation/api/) diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md new file mode 100644 index 0000000000..2cc24c84c2 --- /dev/null +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md @@ -0,0 +1,87 @@ +# Common pitfalls + +Footguns that turned up while building and dogfooding this skill. +Avoid them when scaffolding `.Evals.Tests`. + +## Reporting pipeline + +- **Multiple `[AssemblyCleanup]` methods.** MSTest forbids more than + one `[AssemblyCleanup]` per assembly (UTA014 at discovery time — + every test in the assembly fails to load). The + `MetricsGlossary.WriteGlossary` write must be **chained from + `AievalReport.GenerateReport`'s single `[AssemblyCleanup]`**, not + declared as its own. Wrap the chained call in a `try / catch` so a + glossary-write failure never masks the report. +- **Hand-rolling reports instead of using the Reporting pipeline.** + The whole point of GA `Microsoft.Extensions.AI.Evaluation.Reporting` + 10.7.0 is `DiskBasedReportingConfiguration` + `aieval`. Never write + a hand-rolled markdown report and call it the "quality report" — + that's an MEAI report (HTML) vs a cost/latency capture (markdown). +- **Treating telemetry / compare / quality as separate report + streams.** Compare mode goes through `ReportingConfiguration` with + a distinct `executionName` per matrix entry, so `aieval report` + aggregates them into the same HTML. +- **Forgetting the per-run `metrics-glossary.md`.** The aieval HTML is + data-bound JSON; it shows numbers but no definitions. Co-locate + `metrics-glossary.md` (tier-aware) so a first-time reader can decode + the columns. The `Reporting/MetricsGlossary.cs` template handles + this — don't strip it. + +## Clients (agent vs judge vs stub) + +- **Calling real models from the default test run.** Stub tier uses + `StubChatClient`; the report banner is clearly marked + `(stub IChatClient)`. Real-model runs are opt-in via three + independent env vars. +- **Conflating agent and judge clients.** Two different `IChatClient` + roles. The skill exposes them as two independent env vars + (`EVAL_USE_REAL_AGENT`, `EVAL_USE_REAL_JUDGE`) — one can be real + while the other is stubbed. +- **Auto-detected factory throwing a generic NRE on missing config.** + When the app uses Aspire orchestration, `dotnet test` runs outside + the AppHost and `ConnectionStrings:` is unset. The factory + template in `ichatclient-detection.md` wraps DI resolution in a + `try / catch` that throws a friendly `InvalidOperationException` + naming the connection-string key and the user-secrets command. Don't + strip this when adapting the template. + +## Evaluators + +- **Wiring 4 separate safety evaluators.** Use `ContentHarmEvaluator` + for the Hate / SelfHarm / Violence / Sexual bundle — single Foundry + call, 4 metrics back. The 4 individual evaluators + (`HateAndUnfairnessEvaluator` etc.) are a strict subset. +- **Putting `RelevanceTruthAndCompletenessEvaluator` in the default + set.** Marked experimental upstream; not part of the v2 default. +- **Forgetting `EvaluationContext` for NLP evaluators.** BLEU / GLEU + need `BLEUEvaluatorContext(IEnumerable)`; F1 needs + `F1EvaluatorContext(string)`. If `golden.json` lacks + `reference_response`, NLP evaluators emit `(no reference)` and the + scenario shows blanks in the report. + +## Configuration + +- **Hard-coding a price table.** Lives in `Telemetry/prices.json`, + user-editable. Costs change. +- **Auto-failing the build on quality regressions.** Quality mode is + informational by default. Users opt into a hard-fail by editing + `quality.thresholds.json` (which maps to real MEAI metric names like + `Relevance` / `BLEU` / `F1` and `EvaluationRating` levels), then + setting `hard_fail: true`. +- **Wrong `Microsoft.Extensions.Hosting` version.** Must be `10.0.1` + (not `10.0.0`) to satisfy the transitive constraint from + `Microsoft.Agents.AI.Hosting` 1.x. Pinning `10.0.0` produces + `NU1605` (treated as error in the agentic-app project graph). +- **Forgetting `.gitignore` entries.** Must include both + `.copilot/perf-reports/evals/` and `.Evals.Tests/_store/`. + Otherwise reports pollute history and the persistent `_store/` + blocks PR diffs. + +## Update mode + +- **Overwriting `Quality/rubric.md` or `Quality/golden.json`.** These + are user data — never overwrite. The skill's update-mode behaviour + table in step 1a of `SKILL.md` is the source of truth. +- **Migrating `golden.json` schema destructively.** Migration is + additive only: new fields go in as nullable, existing rows are + preserved. From cc57ef05bb89f653b1bce576fc74e382ebc864cc Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Thu, 18 Jun 2026 13:47:49 -0700 Subject: [PATCH 05/18] setup-maf-evals: factory must explicitly load user-secrets in test hosts Dogfood findings against ELI5Agent surfaced 3 real failure modes when promoting from stub to judge tier: 1. user-secrets silently not loading: dotnet test runs under testhost.exe as the entry assembly, so Host.CreateApplicationBuilder() does NOT pick up the secrets store keyed off the test project's UserSecretsId. Fix: factory template now calls builder.Configuration.AddUserSecrets(typeof(...).Assembly, optional: true). 2. services.ai.azure.com hostname strips dashes from the resource name (foundry-abc -> foundryabc.services.ai.azure.com). The legacy properties.endpoint value points at cognitiveservices.azure.com which 404s for the /models route. 3. Foundry resources provisioned by Aspire/azd usually have disableLocalAuth=true. Key-based auth returns 403; drop the Key= segment and rely on DefaultAzureCredential. Updates: - references/ichatclient-detection.md: factory template adds AddUserSecrets; diagnostic message now lists Entra and Key options + endpoint gotchas. - references/project-template.md: GlobalUsings.cs adds Microsoft.Extensions .Configuration / .DependencyInjection / .Hosting (needed by the factory). - references/common-pitfalls.md: 3 new entries covering the above. - SKILL.md: step 11 now surfaces the exact 2-command judge-promotion path. - tests/dotnet-ai/setup-maf-evals/eval.yaml: new factory-loads-user-secrets-explicitly scenario. Verified end-to-end: ELI5Agent judge tier 4/4 passing in 2m 24s, report.html shows 9 metric families (Words/BLEU/GLEU/F1 + Relevance/ Coherence/Fluency/Completeness/Equivalence). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../dotnet-ai/skills/setup-maf-evals/SKILL.md | 16 +++++- .../references/common-pitfalls.md | 20 +++++++ .../references/ichatclient-detection.md | 55 ++++++++++++++++--- .../references/project-template.md | 3 + tests/dotnet-ai/setup-maf-evals/eval.yaml | 25 +++++++++ 5 files changed, 110 insertions(+), 9 deletions(-) diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md b/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md index a16145a8dc..6011f0b619 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md @@ -164,7 +164,21 @@ Print a 3-block summary: 3. **CLI invocations.** `dotnet test`, `dotnet tool run aieval report`, and the IChatClient detection result so the user knows what was auto-wired. -4. **Follow-up recommendation.** "Re-run after applying a +4. **Promoting to the judge tier.** If the app's `IChatClient` reads from + a connection string (Aspire pattern), include the exact two commands: + + ```pwsh + dotnet user-secrets init --project .Evals.Tests + dotnet user-secrets set "ConnectionStrings:" ` + "Endpoint=https://.services.ai.azure.com/models;DeploymentId=" ` + --project .Evals.Tests + # then: $env:EVAL_USE_REAL_AGENT="1"; $env:EVAL_USE_REAL_JUDGE="1"; dotnet test + ``` + + Note the two endpoint gotchas (dash stripping; key auth often disabled + → drop the `Key=` segment and rely on `DefaultAzureCredential`). Full + details in `references/ichatclient-detection.md`. +5. **Follow-up recommendation.** "Re-run after applying a `select-agent-models` recommendation to confirm no quality regression." diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md index 2cc24c84c2..c71cf89fc2 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md @@ -44,6 +44,26 @@ Avoid them when scaffolding `.Evals.Tests`. `try / catch` that throws a friendly `InvalidOperationException` naming the connection-string key and the user-secrets command. Don't strip this when adapting the template. +- **User-secrets silently not loading in `dotnet test`.** `dotnet test` + runs under `testhost.exe` as the entry assembly, so + `Host.CreateApplicationBuilder()` does NOT pick up the secrets store + bound to your test project's `UserSecretsId`. The factory must call + `builder.Configuration.AddUserSecrets(typeof(...).Assembly, optional: true)` + explicitly. Without it the secret is set on disk but the friendly NRE + still fires. +- **`services.ai.azure.com` hostname strips dashes.** Resource + `foundry-abc` resolves to host `foundryabc.services.ai.azure.com` (no + dash). Authoritative endpoints are in + `az cognitiveservices account show -n -g --query properties.endpoints`. + Use the `AI Foundry API` or `Azure AI Model Inference API` entries — + not the legacy `properties.endpoint` value, which points at the + `cognitiveservices.azure.com` hostname that 404s for the `/models` route. +- **Key-based auth disabled (`403`).** Foundry resources provisioned by + Aspire/azd usually set `disableLocalAuth=true`. Drop the `Key=` + segment from the connection string and rely on `DefaultAzureCredential` + (`az login` + a Cognitive Services User role assignment on the + resource). The Aspire `AddAzureChatCompletionsClient` registration + picks the credential automatically when the key is absent. ## Evaluators diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md index 0d2318d994..1f51db9af2 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md @@ -49,6 +49,12 @@ internal static class AgentChatClientFactory public static IChatClient Create() { var builder = Host.CreateApplicationBuilder(); + + // In test hosts (`dotnet test`), the entry assembly is testhost.exe so + // user-secrets are NOT auto-loaded by CreateApplicationBuilder. + // Add them explicitly from THIS assembly's UserSecretsId. + builder.Configuration.AddUserSecrets(typeof(AgentChatClientFactory).Assembly, optional: true); + // {{InsertDetectedRegistrationCallVerbatim}} var host = builder.Build(); try @@ -63,11 +69,16 @@ internal static class AgentChatClientFactory "string \"{{ConnStrName}}\" from configuration. Aspire's AppHost " + "populates this at runtime, but `dotnet test` runs standalone. " + "Wire one of:\n" + + " # Key-based auth (only if the resource has it enabled):\n" + " dotnet user-secrets set \"ConnectionStrings:{{ConnStrName}}\" " + - "\"Endpoint=https://...;Key=...\" --project {{AppName}}.Evals.Tests\n" + - "or set the env var:\n" + - " $env:ConnectionStrings__{{ConnStrName}} = \"Endpoint=...;Key=...\"\n" + - "Get the value from `azd env get-values` against the deployment resource.", + "\"Endpoint=https://.services.ai.azure.com/models;Key=;DeploymentId={{ConnStrName}}\" --project {{AppName}}.Evals.Tests\n" + + " # Entra-ID auth (DefaultAzureCredential — works when key auth is disabled):\n" + + " dotnet user-secrets set \"ConnectionStrings:{{ConnStrName}}\" " + + "\"Endpoint=https://.services.ai.azure.com/models;DeploymentId={{ConnStrName}}\" --project {{AppName}}.Evals.Tests\n" + + "Note: the hostname strips dashes from the resource name " + + "(resource `foundry-abc` -> host `foundryabc.services.ai.azure.com`).\n" + + "Get the exact endpoint from `azd env get-values` or " + + "`az cognitiveservices account show -n -g --query properties.endpoints`.", ex); } } @@ -127,17 +138,45 @@ output any time the detected pattern reads from a connection string (`AddAzureChatCompletionsClient`, `AddAIInference`, `AddOllamaChatClient`, or `AddAzureOpenAIChatClient` without an explicit endpoint). +> **Important — the factory must opt into user-secrets explicitly.** In a +> `dotnet test` process the entry assembly is `testhost.exe`, so the +> `dotnet user-secrets` payload tied to *your* `UserSecretsId` is NOT auto +> loaded by `Host.CreateApplicationBuilder()`. The Case A template above +> calls `builder.Configuration.AddUserSecrets(typeof(...).Assembly)` for +> this reason. Without that line, the secret is set on disk but never read. + Recommend the user wire one of: ```pwsh -# Option A — user secrets (recommended for local dev) +# 0 — one-time: bind the test project to a secrets store dotnet user-secrets init --project .Evals.Tests -dotnet user-secrets set "ConnectionStrings:" "Endpoint=https://...;Key=..." --project .Evals.Tests -# Option B — env var (works in CI) -$env:ConnectionStrings__ = "Endpoint=https://...;Key=..." +# Option A — Key-based auth (only if the resource has key auth enabled) +dotnet user-secrets set "ConnectionStrings:" ` + "Endpoint=https://.services.ai.azure.com/models;Key=;DeploymentId=" ` + --project .Evals.Tests + +# Option B — Entra-ID auth (DefaultAzureCredential; works when key auth disabled) +dotnet user-secrets set "ConnectionStrings:" ` + "Endpoint=https://.services.ai.azure.com/models;DeploymentId=" ` + --project .Evals.Tests +# requires `az login` and a Cognitive Services User / Azure AI User role +# on the resource for the signed-in identity. + +# Option C — env var (works in CI without a secrets file) +$env:ConnectionStrings__ = "Endpoint=https://...;DeploymentId=" ``` +**Two endpoint gotchas to call out in chat:** + +1. The `services.ai.azure.com/models` hostname **strips dashes** from the + resource name. Resource `foundry-abc` -> host `foundryabc.services.ai.azure.com`. + Use `az cognitiveservices account show -n -g --query properties.endpoints` + to see all valid endpoint hostnames (`AI Foundry API` / `Azure AI Model Inference API`). +2. If the resource has `disableLocalAuth=true` (common on Foundry resources + provisioned by Aspire/azd), key-based auth returns `403 Key based authentication + is disabled for this resource`. Drop the `Key=` segment and use Entra (Option B). + For Foundry-routed clients the connection string is what `azd env get-values` prints for `connectionString` against the deployment resource. Document this in the chat output along with the tier banner so the user doesn't see a diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/project-template.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/project-template.md index c84b7ec20a..9bcbb88ac7 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/project-template.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/project-template.md @@ -130,5 +130,8 @@ global using Microsoft.Extensions.AI.Evaluation.Reporting; global using Microsoft.Extensions.AI.Evaluation.Reporting.Storage; global using Microsoft.Extensions.AI.Evaluation.NLP; global using Microsoft.Extensions.AI.Evaluation.Quality; +global using Microsoft.Extensions.Configuration; // AddUserSecrets in AgentChatClientFactory +global using Microsoft.Extensions.DependencyInjection; // GetRequiredService in AgentChatClientFactory +global using Microsoft.Extensions.Hosting; // Host.CreateApplicationBuilder global using Microsoft.VisualStudio.TestTools.UnitTesting; ``` diff --git a/tests/dotnet-ai/setup-maf-evals/eval.yaml b/tests/dotnet-ai/setup-maf-evals/eval.yaml index 3d41826c73..2915e8c7a8 100644 --- a/tests/dotnet-ai/setup-maf-evals/eval.yaml +++ b/tests/dotnet-ai/setup-maf-evals/eval.yaml @@ -296,3 +296,28 @@ scenarios: - type: file_contains path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs text: "ConnectionStrings" + + - name: factory-loads-user-secrets-explicitly + prompt: | + Run setup-maf-evals on the project at ./fixture. + setup: + files: + - path: fixture/MyApp.slnx + content: | + + + + - path: fixture/MyApp.AppHost/MyApp.AppHost.csproj + content: | + net10.0 + - path: fixture/MyApp.AppHost/AppHost.cs + content: | + var builder = DistributedApplication.CreateBuilder(args); + builder.AddAzureChatCompletionsClient("chat").AddChatClient("chat"); + assertions: + # Without explicit AddUserSecrets, dotnet test never reads the user-secrets store + # because the entry assembly is testhost.exe, not the test project. + - type: file_contains + path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs + text: "AddUserSecrets" + From b7ca9037751592f38679a161e0d47ce3ece2417a Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Thu, 18 Jun 2026 13:55:44 -0700 Subject: [PATCH 06/18] setup-maf-evals: warn about reasoning models rejecting max_tokens Dogfooding the Judge tier against ELI5Agent's gpt-5-mini judge surfaced a silent failure: reasoning models (gpt-5*, o-series) reject the max_tokens parameter that Azure.AI.Inference still sends, returning HTTP 400 unsupported_parameter. The MEAI Quality evaluators swallow the 400 and record it as a per-metric error row -- tests exit 0 but every Quality column is an error. Pitfalls doc now covers: - which model families are affected (gpt-5*, o1/o3/o-series) - how the failure manifests (test pass + errors in report.html, not a hard test failure) - the workaround (pick a non-reasoning judge: gpt-4o / gpt-4o-mini / gpt-4-turbo; if the agent uses reasoning, set EVAL_JUDGE_DEPLOYMENT_NAME= to split agent and judge deployments) - az CLI snippet to list deployment model families SKILL.md step 11's judge-promotion block now flags this alongside the existing dash/key-auth gotchas. eval.yaml gets a smoke assertion that common-pitfalls.md mentions both max_tokens and the env-var workaround. Verified end-to-end: re-pointed ELI5Agent at a gpt-4o-mini Foundry deployment, cleared _store/, re-ran judge tier: 4/4 passing in 44s, clean report (no max_tokens / unsupported_parameter strings anywhere), all 9 metric families populate (4 NLP + Relevance / Coherence / Fluency / Completeness / Equivalence). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md | 13 ++++++++++--- .../setup-maf-evals/references/common-pitfalls.md | 15 +++++++++++++++ tests/dotnet-ai/setup-maf-evals/eval.yaml | 11 +++++++++++ 3 files changed, 36 insertions(+), 3 deletions(-) diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md b/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md index 6011f0b619..83bdf51d3e 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md @@ -175,9 +175,16 @@ Print a 3-block summary: # then: $env:EVAL_USE_REAL_AGENT="1"; $env:EVAL_USE_REAL_JUDGE="1"; dotnet test ``` - Note the two endpoint gotchas (dash stripping; key auth often disabled - → drop the `Key=` segment and rely on `DefaultAzureCredential`). Full - details in `references/ichatclient-detection.md`. + Note the **three** endpoint gotchas: (a) hostname strips dashes on + some resources, (b) key auth is often disabled → drop the `Key=` + segment and rely on `DefaultAzureCredential`, (c) **the judge + deployment must be a non-reasoning model** (gpt-4o / gpt-4o-mini / + gpt-4-turbo). Reasoning models (gpt-5*, o-series) reject `max_tokens` + with HTTP 400 and MEAI silently records that as a per-metric error + row. If the production model is a reasoning one, set + `EVAL_JUDGE_DEPLOYMENT_NAME=` so the judge + points at a different deployment than the agent. Full details in + `references/ichatclient-detection.md`. 5. **Follow-up recommendation.** "Re-run after applying a `select-agent-models` recommendation to confirm no quality regression." diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md index c71cf89fc2..2d0d0c406e 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md @@ -64,6 +64,21 @@ Avoid them when scaffolding `.Evals.Tests`. (`az login` + a Cognitive Services User role assignment on the resource). The Aspire `AddAzureChatCompletionsClient` registration picks the credential automatically when the key is absent. +- **Reasoning models (gpt-5, gpt-5-mini, o1, o3) reject `max_tokens`.** + The Azure.AI.Inference SDK still sends `max_tokens`; reasoning models + require `max_completion_tokens` and return 400 + `unsupported_parameter`. **The MEAI Quality evaluators swallow the + 400 and record it as a per-metric error row, so tests pass but every + Quality column is an error.** Pick a non-reasoning judge model + (gpt-4o, gpt-4o-mini, gpt-4-turbo). Tip: when picking a Foundry + deployment to point `ConnectionStrings:` at, check + `az cognitiveservices account deployment list -n -g + --query "[].{name:name, model:properties.model.name}" -o tsv` + and avoid any deployment whose model is `gpt-5*` or `o*`. When the + agent uses a reasoning model in production, set + `EVAL_JUDGE_DEPLOYMENT_NAME=` so the judge + client points at a compatible deployment while the agent client + keeps the production model. ## Evaluators diff --git a/tests/dotnet-ai/setup-maf-evals/eval.yaml b/tests/dotnet-ai/setup-maf-evals/eval.yaml index 2915e8c7a8..c717bd582b 100644 --- a/tests/dotnet-ai/setup-maf-evals/eval.yaml +++ b/tests/dotnet-ai/setup-maf-evals/eval.yaml @@ -321,3 +321,14 @@ scenarios: path: fixture/MyApp.Evals.Tests/Wire/AgentChatClientFactory.cs text: "AddUserSecrets" + - name: pitfalls-doc-warns-reasoning-models-reject-max-tokens + prompt: | + Read references/common-pitfalls.md from the setup-maf-evals skill. + assertions: + - type: file_contains + path: ../../../plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md + text: "max_tokens" + - type: file_contains + path: ../../../plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md + text: "EVAL_JUDGE_DEPLOYMENT_NAME" + From 4695899d7afa5a53360d631cb24942951c06a852 Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Mon, 22 Jun 2026 13:02:53 -0700 Subject: [PATCH 07/18] setup-maf-evals: quality-first chat output + rubric-driven evaluator + compare opt-in Three changes from a dogfood-driven review: 1. Compare mode -> opt-in (step 2 #4 + step 6). Largest scaffold for least-common surface; users opt in if they actually need matrix.json side-by-side. Scaffold step now only emits Compare/* on opt-in. 2. Step 11 chat output restructured to lead with Quality as the headline evaluation. NLP framed as the zero-config sanity check (still on by default so first \dotnet test\ produces a real report). Safety + Compare listed as additional categories below. Step 2 tier table re-ordered with Quality on top and a 'Default' column for clarity. 3. New common-pitfalls section 'Tuning Quality for stylistic agents' covering the case where ELI5 / summarizer / strict-format / persona agents get punished by generic rubrics in CompletenessEvaluator and EquivalenceEvaluator. Includes three remediation patterns: - drop offending evaluators per app - rewrite goldens in agent voice - custom rubric-driven evaluator evaluators-catalog.md now has the full RubricEvaluator template (reads Quality/rubric.md, judges via the active IChatClient, emits a RubricFit numeric metric with rationale). Step 11 surfaces this caveat on first scaffold so users don't see bad Quality scores against a bad-fit rubric and conclude their agent is broken. eval.yaml adds three smoke assertions: - compare-mode-is-opt-in-not-default - pitfalls-doc-warns-stylistic-agents-fail-completeness Validator green (9 skills + 1 agent + 1 plugin). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../dotnet-ai/skills/setup-maf-evals/SKILL.md | 77 +++++++---- .../references/common-pitfalls.md | 52 ++++++++ .../references/evaluators-catalog.md | 125 ++++++++++++++++++ tests/dotnet-ai/setup-maf-evals/eval.yaml | 25 ++++ 4 files changed, 255 insertions(+), 24 deletions(-) diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md b/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md index 83bdf51d3e..9f77873082 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md @@ -53,19 +53,27 @@ Present the detection summary, then confirm: 1. **Project shape** (default: MSTest). Alternative: console runner (legacy v1 shape) — only emit if user explicitly asks for it. -2. **Evaluator tiers to enable.** Defaults shown; user can override. +2. **Evaluator categories to enable.** Defaults shown; user can override. - | Tier | Evaluators | Cost | Needs | - |------|-----------|------|-------| - | 1 — NLP (default ON) | BLEU, GLEU, F1, Words | free | reference responses in golden.json | - | 2 — Quality (default ON, but stubbed) | Relevance, Coherence, Fluency, Completeness, Equivalence, Groundedness; agent: IntentResolution, TaskAdherence, ToolCallAccuracy | per-call judge tokens | real `IChatClient` + `EVAL_USE_REAL_JUDGE=1` | - | 3 — Safety (default OFF) | `ContentHarmEvaluator` (Hate+SelfHarm+Violence+Sexual single-shot), ProtectedMaterial, IndirectAttack, CodeVulnerability, UngroundedAttributes, GroundednessPro | Foundry evaluation service charges | Azure AI Foundry endpoint + `EVAL_USE_FOUNDRY_SAFETY=1` | + | Category | Evaluators | Cost | Needs | Default | + |------|-----------|------|-------|---------| + | **Quality** *(headline)* | Relevance, Coherence, Fluency, Completeness, Equivalence, Groundedness; agent: IntentResolution, TaskAdherence, ToolCallAccuracy | per-call judge tokens | real `IChatClient` + `EVAL_USE_REAL_JUDGE=1` | **ON** (stubbed until judge wired) | + | NLP (zero-config sanity) | BLEU, GLEU, F1, Words | free | reference responses in golden.json | ON | + | Safety | `ContentHarmEvaluator` (Hate+SelfHarm+Violence+Sexual single-shot), ProtectedMaterial, IndirectAttack, CodeVulnerability, UngroundedAttributes, GroundednessPro | Foundry evaluation service charges | Azure AI Foundry endpoint + `EVAL_USE_FOUNDRY_SAFETY=1` | **OFF** — prompt the user: "Wire safety evaluators too? (y/N)" | + + Frame Quality as the headline evaluation; NLP is the zero-config + first-run experience that emits a real `report.html` before any + creds exist; Safety is the opt-in for production-bound apps. 3. **IChatClient detection result.** Show what was detected (e.g., "Found `AddAzureOpenAIChatClient` in `AppHost.cs:41` with deployment alias `chat`"). Ask the user to confirm or override. If detection failed, generate a stub factory the user will fill in. -4. **Run modes to scaffold** (telemetry / quality / compare). Default: all three. +4. **Run modes to scaffold.** Telemetry (default ON) and Quality (default + ON). **Compare mode is opt-in** — prompt the user: "Wire compare + mode for side-by-side matrix.json runs? (y/N)". Compare adds the + largest scaffold (extra runner + matrix.json + delta-table generator) + and is the least commonly used surface. 5. **Optional add-ons:** Aspire dashboard panel, GitHub Actions workflow. ### 3. Scaffold the project @@ -79,8 +87,9 @@ Always emit: `Reporting/{ReportingConfig.cs, Tier.cs, AievalReport.cs, WordCountEvaluator.cs, MetricsGlossary.cs}`, `Wire/{AgentChatClientFactory.cs, StubChatClient.cs}`, `Quality/{QualityTests.cs, rubric.md, golden.json}`, `Telemetry/{TelemetryTests.cs, inputs.json, prices.json}`, -`Compare/{CompareTests.cs, matrix.json}`, `quality.thresholds.json`, -`GlobalUsings.cs`, `dotnet-tools.json`. Emit `Safety/SafetyTests.cs` and +`quality.thresholds.json`, `GlobalUsings.cs`, `dotnet-tools.json`. +Emit `Compare/{CompareTests.cs, matrix.json}` only if the user opted into +compare mode (step 2 #4). Emit `Safety/SafetyTests.cs` and `.github/workflows/evals.yml` only if the user opted in (steps 7 and 9). After writing files: @@ -108,9 +117,10 @@ produces `report.html`. Uses `DiskBasedReportingConfiguration` + evaluators; judge tier (`EVAL_USE_REAL_JUDGE=1`) adds the LLM-as-judge evaluators from `references/evaluators-catalog.md`. -### 6. Wire compare mode +### 6. Wire compare mode (opt-in) -See `references/compare-mode.md`. Default ON. Reads `matrix.json`, +See `references/compare-mode.md`. **Default OFF.** Only scaffold when +the user opted in at step 2 (#4). When enabled, reads `matrix.json`; each entry runs through the **same** `ReportingConfiguration` with a distinct `executionName`, so `aieval report` aggregates the comparison columns into a single HTML view. Also writes a `compare.md` delta @@ -154,18 +164,24 @@ judge; `AZURE_AI_FOUNDRY_ENDPOINT` → safety), and uploads ### 11. Surface in chat -Print a 3-block summary: - -1. **Tier banner.** Which tier is active (Stub / Judge / Foundry-Safety) - and the exact env-var commands to upgrade. -2. **Paths.** Project path, HTML report path, **glossary path** - (`metrics-glossary.md` co-located with `report.html`), persistent - `_store/` path. -3. **CLI invocations.** `dotnet test`, `dotnet tool run aieval report`, - and the IChatClient detection result so the user knows what was - auto-wired. -4. **Promoting to the judge tier.** If the app's `IChatClient` reads from - a connection string (Aspire pattern), include the exact two commands: +Lead with **Quality** as the headline evaluation; frame NLP as the +zero-config sanity check and Safety/Compare as additions. + +1. **Quality (headline).** State whether the judge is wired: + - *Stubbed* — "Quality scaffolded; judge will run once you wire + `EVAL_USE_REAL_JUDGE=1` + a chat endpoint. The next block tells + you how." + - *Live* — "Quality judge active against ``; report + shows Relevance / Coherence / Fluency / Completeness / Equivalence." + - Caveat the user **must** know up-front: *"The built-in Quality + rubrics are generic. Agents with deliberate stylistic constraints + (brevity, persona, format adherence) will score low on + Completeness / Equivalence even when working as designed. See + `references/common-pitfalls.md#tuning-quality-for-stylistic-agents` + for the per-app override pattern."* +2. **Promoting Quality to the judge tier.** If the app's `IChatClient` + reads from a connection string (Aspire pattern), include the exact + two commands: ```pwsh dotnet user-secrets init --project .Evals.Tests @@ -185,7 +201,20 @@ Print a 3-block summary: `EVAL_JUDGE_DEPLOYMENT_NAME=` so the judge points at a different deployment than the agent. Full details in `references/ichatclient-detection.md`. -5. **Follow-up recommendation.** "Re-run after applying a +3. **NLP (zero-config sanity).** "`report.html` already populates with + Words / BLEU / GLEU / F1 from `golden.json` without any creds — run + `dotnet test` now to see it." +4. **Additional categories you can wire.** List Safety (`EVAL_USE_FOUNDRY_SAFETY=1` + + opt-in scaffold) and Compare (re-run the skill with `compare: true` + if it wasn't opted in originally) as one-line bullets — not in the + main flow. +5. **Paths.** Project path, `report.html` path, **glossary path** + (`metrics-glossary.md` co-located with `report.html`), persistent + `_store/` path. +6. **CLI invocations.** `dotnet test`, `dotnet tool run aieval report`, + and the IChatClient detection result so the user knows what was + auto-wired. +7. **Follow-up recommendation.** "Re-run after applying a `select-agent-models` recommendation to confirm no quality regression." diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md index 2d0d0c406e..97a849a017 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md @@ -94,6 +94,58 @@ Avoid them when scaffolding `.Evals.Tests`. `reference_response`, NLP evaluators emit `(no reference)` and the scenario shows blanks in the report. +### Tuning Quality for stylistic agents + +The built-in Quality evaluators (`RelevanceEvaluator`, +`CoherenceEvaluator`, `FluencyEvaluator`, `CompletenessEvaluator`, +`EquivalenceEvaluator`) use **generic rubrics baked into the +evaluator's judge prompt**. They don't know about *your* agent's +contract. + +- **`CompletenessEvaluator` explicitly rewards thoroughness.** Any + agent whose contract is "compress / summarize / dumb-down" (ELI5, + TL;DR summarizers, tweet-length responders, structured-extractor + agents) will score 1-2 even when working perfectly. The score is + measuring conformance to a generic "answer thoroughly" rubric, not + conformance to your agent's contract. +- **`EquivalenceEvaluator` rewards lexical/semantic match to a + reference.** If your `golden.json` references are full-form + explanations but your agent emits paraphrases or stylized variants, + Equivalence will flag drift that isn't a real regression. +- **`CoherenceEvaluator` and `FluencyEvaluator`** penalize + fragmentary / one-line / heavily-structured outputs (JSON-only, + bullet-only, single-sentence). Agents that respond in a strict + format will be marked down. +- **Only `RelevanceEvaluator` is largely intent-agnostic** — it + checks "did the response address the query" without preferring + long-form. It's the one Quality metric that's mostly safe to leave + on for any agent. + +**Three remediation patterns, in increasing order of effort:** + +1. **Drop the offending evaluators per app.** Edit + `Reporting/ReportingConfig.cs` and remove `CompletenessEvaluator` + / `EquivalenceEvaluator` from the judge-tier evaluator list. + Keep Relevance + Coherence + Fluency. Document the choice in a + `// per-app: ELI5 contract → drop Completeness` comment so future + re-scaffolds don't add them back. +2. **Rewrite goldens in the agent's voice.** Edit `Quality/golden.json` + so `reference_response` is itself ELI5-style (or tweet-style, or + bullet-style). Equivalence becomes meaningful again; Completeness + still complains. +3. **Add a custom rubric-driven evaluator.** Write a `RubricEvaluator` + that reads `Quality/rubric.md` and asks the judge to score the + response against *your* criteria ("Is the explanation appropriate + for a 5-year-old? Score 1-5."). See + `references/evaluators-catalog.md#custom-rubric-driven-evaluator` + for the template. This is the right long-term answer for any + non-generic agent. + +> **Surface this in the chat output on first scaffold.** The user +> shouldn't have to interpret bad Quality scores against a bad-fit +> rubric and conclude their agent is broken. Step 11 of `SKILL.md` +> calls this out explicitly in the Quality headline block. + ## Configuration - **Hard-coding a price table.** Lives in `Telemetry/prices.json`, diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md index ca6a90dea3..cd7c43b9fc 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md @@ -141,3 +141,128 @@ and cost 4× more Foundry calls for the same metrics. `hard_fail: true` makes any below-threshold metric fail the test. Default `false` makes the test pass-through informational — failures show in the report only. + +## Custom rubric-driven evaluator + +The built-in Quality evaluators (Relevance, Coherence, Fluency, +Completeness, Equivalence) judge against generic rubrics baked into +the evaluator's judge prompt. They cannot read `Quality/rubric.md`. +That makes them ill-fitting for agents with deliberate stylistic +constraints (ELI5 / summarizer / strict-format / persona-bound +agents) — see `common-pitfalls.md#tuning-quality-for-stylistic-agents`. + +For those agents, scaffold a `RubricEvaluator` that reads +`Quality/rubric.md` and asks the judge to score against *your* +criteria. The pattern is shape-identical to `WordCountEvaluator` — +it just delegates to the judge chat client. + +```csharp +// Reporting/RubricEvaluator.cs +// Custom rubric-driven evaluator. Reads Quality/rubric.md and asks +// the judge to score the response against per-app criteria. Emits a +// single "RubricFit" metric (numeric 1-5) plus a free-text rationale. +public sealed class RubricEvaluator : IEvaluator +{ + public const string MetricName = "RubricFit"; + public IReadOnlyCollection EvaluationMetricNames { get; } = [MetricName]; + + private readonly string _rubric; + + public RubricEvaluator(string rubricMarkdown) => _rubric = rubricMarkdown; + + public static RubricEvaluator FromFile(string path) => + new(File.ReadAllText(path)); + + public async ValueTask EvaluateAsync( + IEnumerable messages, + ChatResponse modelResponse, + ChatConfiguration? chatConfiguration = null, + IEnumerable? additionalContext = null, + CancellationToken cancellationToken = default) + { + if (chatConfiguration?.ChatClient is null) + { + // Judge tier not active — return a stubbed Inconclusive metric. + return new EvaluationResult(new NumericMetric(MetricName) + { + Interpretation = new EvaluationMetricInterpretation( + EvaluationRating.Inconclusive, reason: "Judge not wired.") + }); + } + + var userQuery = string.Join("\n", messages.Select(m => m.Text)); + var responseText = modelResponse.Text ?? string.Empty; + + var prompt = $""" + You are scoring an assistant response against the rubric below. + + ## Rubric + {_rubric} + + ## User query + {userQuery} + + ## Assistant response + {responseText} + + Respond with strict JSON: {{ "score": <1-5 int>, "rationale": "<1-2 sentences>" }}. + 5 = perfectly satisfies every rubric clause. 1 = ignores the rubric. + """; + + var judge = await chatConfiguration.ChatClient.GetResponseAsync( + prompt, cancellationToken: cancellationToken).ConfigureAwait(false); + + // Tolerant parse: fall back to Inconclusive on bad JSON. + var (score, rationale) = TryParse(judge.Text ?? ""); + + var metric = new NumericMetric(MetricName, value: score) + { + Interpretation = new EvaluationMetricInterpretation( + rating: score switch { >= 4 => EvaluationRating.Good, + 3 => EvaluationRating.Average, + _ => EvaluationRating.Poor }, + reason: rationale) + }; + return new EvaluationResult(metric); + } + + private static (int score, string rationale) TryParse(string raw) + { + try + { + using var doc = JsonDocument.Parse(raw.Trim().Trim('`', ' ', '\n', '\r')); + return (doc.RootElement.GetProperty("score").GetInt32(), + doc.RootElement.GetProperty("rationale").GetString() ?? ""); + } + catch { return (3, "Judge response was unparseable; defaulted to Average."); } + } +} +``` + +Wire it from `Reporting/ReportingConfig.cs`'s judge-tier evaluator +list **alongside** Relevance/Coherence/Fluency (drop Completeness + +Equivalence per the pitfall guidance): + +```csharp +// In ReportingConfig.ForQuality(), when EvalEnv.UseRealJudge: +evaluators.Add(new RelevanceEvaluator()); +evaluators.Add(new CoherenceEvaluator()); +evaluators.Add(new FluencyEvaluator()); +evaluators.Add(RubricEvaluator.FromFile( + Path.Combine(AppContext.BaseDirectory, "Quality", "rubric.md"))); +// CompletenessEvaluator + EquivalenceEvaluator deliberately dropped +// for this app — see common-pitfalls.md tuning section. +``` + +Ensure `Quality/rubric.md` is copied to output by adding to the csproj: + +```xml + + + +``` + +This `RubricFit` metric will appear in `report.html` alongside the +built-ins, and the rationale shows in the per-scenario detail drawer. +Update `Reporting/MetricsGlossary.cs`'s `QualityEntries` constant to +add a one-line `RubricFit` definition pointing at `Quality/rubric.md`. diff --git a/tests/dotnet-ai/setup-maf-evals/eval.yaml b/tests/dotnet-ai/setup-maf-evals/eval.yaml index c717bd582b..9a985f5b84 100644 --- a/tests/dotnet-ai/setup-maf-evals/eval.yaml +++ b/tests/dotnet-ai/setup-maf-evals/eval.yaml @@ -332,3 +332,28 @@ scenarios: path: ../../../plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md text: "EVAL_JUDGE_DEPLOYMENT_NAME" + - name: pitfalls-doc-warns-stylistic-agents-fail-completeness + prompt: | + Read references/common-pitfalls.md and references/evaluators-catalog.md from setup-maf-evals. + assertions: + - type: file_contains + path: ../../../plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md + text: "Tuning Quality for stylistic agents" + - type: file_contains + path: ../../../plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md + text: "CompletenessEvaluator" + - type: file_contains + path: ../../../plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md + text: "RubricEvaluator" + + - name: compare-mode-is-opt-in-not-default + prompt: | + Read SKILL.md for setup-maf-evals; what's the default for compare mode? + assertions: + - type: file_contains + path: ../../../plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md + text: "Wire compare mode (opt-in)" + - type: file_contains + path: ../../../plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md + text: "Compare mode is opt-in" + From 40606ef2d86262cbdfcb4d182111ad6941db5c2f Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Mon, 22 Jun 2026 13:14:01 -0700 Subject: [PATCH 08/18] setup-maf-evals: fix RubricEvaluator template raw-string interpolation (CS9006) Found by dogfooding the v2 template against ELI5Agent: the prompt block used single-dollar interpolated raw string with {{ }} attempting to escape literal JSON braces. C# raw strings reject {{ as literal in single-dollar mode (CS9006: 'does not start with enough $ characters to allow this many consecutive opening braces as content'). Fix: switch to double-dollar raw string $$"""...""" where literal { is fine and {{var}} is interpolation. Added a comment in the template explaining the choice so users don't "simplify" it back. End-to-end validated against ELI5Agent: - Compiles clean. - 4/4 judge-tier tests pass in 58s with the new evaluator wired. - RubricFit column appears in report.html with substantive rationales. - Score comparison (same agent, same responses): generic Relevance 3.25, Coherence 3.00, Fluency 3.50 vs RubricFit (ELI5-tuned rubric) 5.00 / 5 -- exactly the remediation outcome documented in common-pitfalls. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../references/evaluators-catalog.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md index cd7c43b9fc..127968f217 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/evaluators-catalog.md @@ -193,19 +193,22 @@ public sealed class RubricEvaluator : IEvaluator var userQuery = string.Join("\n", messages.Select(m => m.Text)); var responseText = modelResponse.Text ?? string.Empty; - var prompt = $""" + // NOTE: $$"""...""" (double-$) raw string. Inside, single { } is literal + // (matters for the JSON braces below), and {{var}} is interpolation. + // Plain $"""...""" would reject `{{ ... }}` as literal with CS9006. + var prompt = $$""" You are scoring an assistant response against the rubric below. ## Rubric - {_rubric} + {{_rubric}} ## User query - {userQuery} + {{userQuery}} ## Assistant response - {responseText} + {{responseText}} - Respond with strict JSON: {{ "score": <1-5 int>, "rationale": "<1-2 sentences>" }}. + Respond with strict JSON: { "score": <1-5 int>, "rationale": "<1-2 sentences>" }. 5 = perfectly satisfies every rubric clause. 1 = ignores the rubric. """; From dbd46ef131c525ca55aff6abfd868453257cbb55 Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Mon, 22 Jun 2026 13:24:09 -0700 Subject: [PATCH 09/18] docs(setup-maf-evals): explain Cache Miss in Diagnostic Data Document that 'Cache Miss' on every call is expected in the real-agent + real-judge workflow because the judge cache key includes the agent's response, which varies run-to-run. Also note that executionName scopes the cache. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../setup-maf-evals/references/common-pitfalls.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md index 97a849a017..9b25c98e57 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md @@ -26,6 +26,19 @@ Avoid them when scaffolding `.Evals.Tests`. `metrics-glossary.md` (tier-aware) so a first-time reader can decode the columns. The `Reporting/MetricsGlossary.cs` template handles this — don't strip it. +- **Misreading "Cache Miss" in the Diagnostic Data section.** With + `enableResponseCaching: true`, the report's per-call `cacheHit` flag + records whether the judge response cache was hit. Expect **Miss on + every call** in the `real agent + real judge` workflow — the cache + key is the judge's input prompt, which includes the agent's response, + and a live LLM agent's output varies run-to-run even at low + temperature. Hit/Miss is informational; it does not affect + correctness or scores. The cache pays off when you (a) capture agent + responses to fixtures and re-evaluate them while iterating on + rubrics, or (b) run a stub agent with deterministic output. To get + hits across runs you ALSO need to pin `executionName` (the cache is + scoped per execution name; a fresh timestamp per run guarantees + misses regardless of input). ## Clients (agent vs judge vs stub) From 29591582a37948964fe40b01f0ca66561cda1859 Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Mon, 22 Jun 2026 13:36:07 -0700 Subject: [PATCH 10/18] feat(setup-maf-evals): route agent call through cached client Three changes that make the MEAI response cache actually pay off across runs (verified on ELI5Agent: cold 39s/0 hits -> warm 15s/20 hits with the same scenarios): 1. QualityTests uses run.ChatConfiguration!.ChatClient (the cached wrapper) for the agent call by default, only falling back to the uncached factory when EVAL_JUDGE_DEPLOYMENT_NAME splits judge from agent. Previously every run created a fresh uncached agent client and the agent's varying output guaranteed judge cache misses too. 2. Drop the executionName argument from DiskBasedReportingConfiguration.Create. executionName is part of the cache scope -- passing a fresh per-run timestamp guaranteed misses regardless of input. Report folder timestamping moves to the new EvalEnv.ReportFolder (EVAL_REPORT_FOLDER env var). 3. SKILL.md step 11 + common-pitfalls.md rewritten to explain the cache payoff: first run ~60s populates _store/cache/, every subsequent run against unchanged inputs ~5s with zero LLM cost. Compare mode keeps a stable per-entry executionName for cache reuse across compare runs. ichatclient-detection.md documents the override-splits-cache trade-off. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../dotnet-ai/skills/setup-maf-evals/SKILL.md | 14 ++- .../setup-maf-evals/references/ci-workflow.md | 8 +- .../references/common-pitfalls.md | 27 +++--- .../references/compare-mode.md | 4 +- .../references/ichatclient-detection.md | 16 ++- .../references/metrics-glossary.md | 2 +- .../references/quality-modes.md | 97 ++++++++++++++++--- 7 files changed, 128 insertions(+), 40 deletions(-) diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md b/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md index 9f77873082..e3da25f1e3 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md @@ -204,10 +204,10 @@ zero-config sanity check and Safety/Compare as additions. 3. **NLP (zero-config sanity).** "`report.html` already populates with Words / BLEU / GLEU / F1 from `golden.json` without any creds — run `dotnet test` now to see it." -4. **Additional categories you can wire.** List Safety (`EVAL_USE_FOUNDRY_SAFETY=1` - + opt-in scaffold) and Compare (re-run the skill with `compare: true` - if it wasn't opted in originally) as one-line bullets — not in the - main flow. +4. **Additional categories you can wire.** List Safety + (`EVAL_USE_FOUNDRY_SAFETY=1`, opt-in scaffold) and Compare (re-run + the skill with `compare: true` if it wasn't opted in originally) + as one-line bullets — not in the main flow. 5. **Paths.** Project path, `report.html` path, **glossary path** (`metrics-glossary.md` co-located with `report.html`), persistent `_store/` path. @@ -217,6 +217,12 @@ zero-config sanity check and Safety/Compare as additions. 7. **Follow-up recommendation.** "Re-run after applying a `select-agent-models` recommendation to confirm no quality regression." +8. **Cache payoff.** "First `dotnet test` populates `_store/cache/` and + takes the full ~60s. Every subsequent run against unchanged inputs + reuses cached agent + judge responses — typically ~5s with zero LLM + cost. The Diagnostic Data section of `report.html` shows per-call + Hit/Miss. To force a fresh run, delete `_store/cache/` or change the + rubric / golden inputs." Also link `references/evaluators-catalog.md` and `references/metrics-glossary.md` so the user can see what each metric diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/ci-workflow.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/ci-workflow.md index f1ec46e10a..3c627a0e51 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/ci-workflow.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/ci-workflow.md @@ -35,7 +35,7 @@ jobs: timeout-minutes: 20 env: - EVAL_EXECUTION_NAME: ${{ github.run_id }}-${{ github.run_attempt }} + EVAL_REPORT_FOLDER: ${{ github.run_id }}-${{ github.run_attempt }} steps: - uses: actions/checkout@v4 @@ -80,17 +80,17 @@ jobs: - name: Generate report (already invoked by [AssemblyCleanup], this is a safety net) if: always() run: | - mkdir -p .copilot/perf-reports/evals/${{ env.EVAL_EXECUTION_NAME }} + mkdir -p .copilot/perf-reports/evals/${{ env.EVAL_REPORT_FOLDER }} dotnet tool run aieval report \ --path {{AppName}}.Evals.Tests/_store \ - --output .copilot/perf-reports/evals/${{ env.EVAL_EXECUTION_NAME }}/report.html + --output .copilot/perf-reports/evals/${{ env.EVAL_REPORT_FOLDER }}/report.html - name: Upload report if: always() uses: actions/upload-artifact@v4 with: name: eval-report-${{ steps.tier.outputs.tier }} - path: .copilot/perf-reports/evals/${{ env.EVAL_EXECUTION_NAME }}/report.html + path: .copilot/perf-reports/evals/${{ env.EVAL_REPORT_FOLDER }}/report.html - name: Upload trx if: always() diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md index 9b25c98e57..f52f521b49 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/common-pitfalls.md @@ -26,19 +26,20 @@ Avoid them when scaffolding `.Evals.Tests`. `metrics-glossary.md` (tier-aware) so a first-time reader can decode the columns. The `Reporting/MetricsGlossary.cs` template handles this — don't strip it. -- **Misreading "Cache Miss" in the Diagnostic Data section.** With - `enableResponseCaching: true`, the report's per-call `cacheHit` flag - records whether the judge response cache was hit. Expect **Miss on - every call** in the `real agent + real judge` workflow — the cache - key is the judge's input prompt, which includes the agent's response, - and a live LLM agent's output varies run-to-run even at low - temperature. Hit/Miss is informational; it does not affect - correctness or scores. The cache pays off when you (a) capture agent - responses to fixtures and re-evaluate them while iterating on - rubrics, or (b) run a stub agent with deterministic output. To get - hits across runs you ALSO need to pin `executionName` (the cache is - scoped per execution name; a fresh timestamp per run guarantees - misses regardless of input). +- **Misreading "Cache Miss" in the Diagnostic Data section.** With the + default template (`enableResponseCaching: true`, agent resolved via + `run.ChatConfiguration!.ChatClient`, no per-run `executionName`), + the **first** `dotnet test` run shows Miss everywhere (cache empty) + and **subsequent runs against unchanged inputs show Hit everywhere** + in ~5s with zero LLM cost. Persistent Miss after run 1 means one of: + (a) the rubric / golden / scenario inputs changed, (b) the judge + model or chat options changed, (c) `_store/cache/` was deleted, or + (d) something is bypassing the cache — most commonly calling + `Wire.ResolveAgentClient()` (uncached) instead of + `run.ChatConfiguration!.ChatClient`, OR passing a fresh + `executionName` to `DiskBasedReportingConfiguration.Create(...)`. + Hit/Miss never affects correctness; it just tells you whether the + LLM was actually called this run. ## Clients (agent vs judge vs stub) diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/compare-mode.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/compare-mode.md index 142ad26e63..94d5f60c6f 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/compare-mode.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/compare-mode.md @@ -56,7 +56,9 @@ public sealed class CompareTests chatConfiguration: new ChatConfiguration( Wire.ResolveJudgeClient(Wire.ResolveAgentClient(entry.ModelAssignments))), enableResponseCaching: true, - executionName: $"{ReportingConfig.ExecutionName}-{entry.Name}"); + // Stable per-entry name (NOT prefixed with a per-run timestamp) + // so re-running compare reuses the cache for unchanged entries. + executionName: $"compare-{entry.Name}"); foreach (var g in GoldenLoader.Load()) { diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md index 1f51db9af2..00363d0f8a 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/ichatclient-detection.md @@ -125,9 +125,19 @@ internal static IChatClient ResolveAgentClient() => EvalEnv.UseRealAgent ? AgentChatClientFactory.Create() : new StubChatClient(); ``` -And by default the judge client is the **same** instance (saves a -duplicate Azure credential setup). The user can override by setting -`EVAL_JUDGE_DEPLOYMENT_NAME` to a different deployment alias. +And by default the judge client is the **same** instance as the agent +client. This saves a duplicate Azure credential setup AND lets the +MEAI response cache serve both the agent call and the judge call from +one cache (because `QualityTests` uses `run.ChatConfiguration!.ChatClient` +for the agent call — that's the cached wrapper around the shared +instance). The user can override by setting `EVAL_JUDGE_DEPLOYMENT_NAME` +to a different deployment alias when (e.g.) the production model is a +reasoning model that can't be used as a judge. Trade-off: with the +override active, `run.ChatConfiguration!.ChatClient` becomes the +**judge** client, so the agent call would silently use the judge +model. To avoid that, `QualityTests` falls back to +`Wire.ResolveAgentClient()` (uncached) for agent calls whenever the +override is set; cache hits then apply only to judge calls. ## Connection-string setup for standalone test runs diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/metrics-glossary.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/metrics-glossary.md index 3ed5432217..476c995158 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/metrics-glossary.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/metrics-glossary.md @@ -163,7 +163,7 @@ internal static class MetricsGlossary public static void WriteGlossary() { var outDir = Path.Combine( - RepoRoot.Find(), ".copilot", "perf-reports", "evals", EvalEnv.ExecutionName); + RepoRoot.Find(), ".copilot", "perf-reports", "evals", EvalEnv.ReportFolder); Directory.CreateDirectory(outDir); var path = Path.Combine(outDir, "metrics-glossary.md"); diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/quality-modes.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/quality-modes.md index d11111d573..e9df0de7b9 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/quality-modes.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/quality-modes.md @@ -4,16 +4,40 @@ `Microsoft.Extensions.AI.Evaluation.Reporting`. It's the **only** runner that produces `report.html`. +## Response cache (read first) + +`DiskBasedReportingConfiguration.Create(..., enableResponseCaching: true)` +wraps the supplied `IChatClient` with a content-addressable cache stored +under `_store/cache/`. The first `dotnet test` run populates it; every +subsequent run against the same scenarios is **near-instant with zero +LLM cost** because both the agent call and the judge call are served +from disk. + +Two rules MUST be followed to make the cache work: + +1. **The agent call must go through `run.ChatConfiguration!.ChatClient`** + (NOT `Wire.ResolveAgentClient()` directly). The run-scoped client is + the cached wrapper. Calling the factory directly bypasses the cache. +2. **Do not pass a per-run `executionName`** to `Create(...)`. The + `executionName` is part of the cache scope; a fresh timestamp per run + guarantees misses. Use a separate `EvalEnv.ReportFolder` value for + the report-output directory if you want per-run history. + +If both rules are followed, the only legitimate reasons to see Miss in +the report's Diagnostic Data section are: (a) the cache directory was +deleted, (b) the rubric / golden / scenario input changed, (c) the +judge model or chat options changed, or (d) it's the first run. + ## Pipeline ``` [ClassInitialize] → build ReportingConfig (tier-aware evaluator list) [TestMethod] → one per golden.json entry ├─ CreateScenarioRunAsync(scenarioName) - ├─ resolve IChatClient (real or stub per EVAL_USE_REAL_AGENT) - ├─ get agent response + ├─ get cached IChatClient from run.ChatConfiguration + ├─ get agent response (cached) ├─ build per-evaluator EvaluationContext (BLEU refs, F1 ground truth, ...) - └─ scenarioRun.EvaluateAsync(messages, response, contexts) + └─ scenarioRun.EvaluateAsync(messages, response, contexts) // judge calls cached [AssemblyCleanup] → dotnet tool run aieval report --path _store --output /report.html ``` @@ -26,15 +50,15 @@ internal static class ReportingConfig public static readonly string StorageRoot = Path.Combine(RepoRoot.Find(), "_store"); - // Resolved once at class load — must NOT re-evaluate DateTime.UtcNow per - // call, otherwise AievalReport and MetricsGlossary land in different - // timestamped output folders. - public static readonly string ExecutionName = - Environment.GetEnvironmentVariable("EVAL_EXECUTION_NAME") - ?? DateTime.UtcNow.ToString("yyyyMMdd-HHmmss"); - public static ReportingConfiguration ForQuality() { + // ONE client serves both the agent call and the judge call. MEAI wraps + // it with the response cache when handed to ChatConfiguration, so the + // *agent* call gets cached too when QualityTests calls it through + // `run.ChatConfiguration!.ChatClient` (see "Test class" below). + // On re-runs against unchanged inputs, the entire run is a cache hit + // and finishes in seconds with zero LLM cost. Override the judge with + // a separate model via EVAL_JUDGE_DEPLOYMENT_NAME (advanced). var agent = Wire.ResolveAgentClient(); var judge = Wire.ResolveJudgeClient(agent); @@ -65,12 +89,14 @@ internal static class ReportingConfig } } + // executionName: deliberately omitted. MEAI's default keeps the cache + // scope stable across runs so re-runs hit. The report folder + // timestamp lives separately in EvalEnv.ReportFolder. return DiskBasedReportingConfiguration.Create( storageRootPath: StorageRoot, evaluators: evaluators, chatConfiguration: new ChatConfiguration(judge), - enableResponseCaching: true, - executionName: ExecutionName); + enableResponseCaching: true); } } ``` @@ -96,7 +122,22 @@ public sealed class QualityTests var scenarioName = $"{nameof(QualityTests)}.{g.Id}"; await using var run = await s_reporting.CreateScenarioRunAsync(scenarioName); - var agent = Wire.ResolveAgentClient(); + // IMPORTANT: use the run's ChatClient, NOT Wire.ResolveAgentClient(), + // for the agent call. The run-scoped client is wrapped with MEAI's + // response cache (set up by ReportingConfig.ForQuality), so identical + // inputs across runs return cached responses instead of paying for + // a fresh LLM call. Calling AgentChatClientFactory directly bypasses + // the cache and guarantees a cache miss on every judge call too + // (judge cache key includes the agent response, which then varies). + // + // EDGE CASE: when EVAL_JUDGE_DEPLOYMENT_NAME splits judge from agent, + // run.ChatConfiguration.ChatClient IS the judge client — using it as + // the agent would silently call the wrong model. Fall back to the + // uncached agent factory in that case (judge calls still cache). + var agent = string.IsNullOrEmpty( + Environment.GetEnvironmentVariable("EVAL_JUDGE_DEPLOYMENT_NAME")) + ? run.ChatConfiguration!.ChatClient + : Wire.ResolveAgentClient(); var messages = new List { new(ChatRole.System, RubricLoader.SystemPrompt()), @@ -134,7 +175,7 @@ public static class AievalReport { var outDir = Path.Combine( RepoRoot.Find(), ".copilot", "perf-reports", "evals", - ReportingConfig.ExecutionName); + EvalEnv.ReportFolder); Directory.CreateDirectory(outDir); var html = Path.Combine(outDir, "report.html"); @@ -154,6 +195,34 @@ public static class AievalReport } ``` +## EvalEnv (sketch, in `Reporting/Tier.cs`) + +```csharp +internal static class EvalEnv +{ + public static bool UseRealAgent => + Environment.GetEnvironmentVariable("EVAL_USE_REAL_AGENT") == "1"; + + public static bool UseRealJudge => + Environment.GetEnvironmentVariable("EVAL_USE_REAL_JUDGE") == "1"; + + public static bool UseFoundrySafety => + Environment.GetEnvironmentVariable("EVAL_USE_FOUNDRY_SAFETY") == "1"; + + public static string Tier => + UseFoundrySafety ? "Safety" : UseRealJudge ? "Judge" : "Stub"; + + // Per-run timestamp ONLY for the report output folder under + // .copilot/perf-reports/evals//. NOT passed to MEAI's + // ReportingConfiguration — that would scope the response cache per + // run and defeat caching. Override with EVAL_REPORT_FOLDER in CI to + // align with a build number. + public static readonly string ReportFolder = + Environment.GetEnvironmentVariable("EVAL_REPORT_FOLDER") + ?? DateTime.UtcNow.ToString("yyyyMMdd-HHmmss"); +} +``` + ## `golden.json` schema (v2) ```json From ee3199186121563009ad3a24eaf1007599e2048f Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Mon, 22 Jun 2026 14:02:38 -0700 Subject: [PATCH 11/18] docs(dotnet-ai): add common-pitfalls + check-id glossary across perf skills MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Polish pass after dogfooding 3 perf skills against ELI5Agent (1-agent MAF/Aspire/Foundry app): - scan-agentic-app-perf: new check-id-glossary.md (24 check codes, severity, per-category file links, cross-skill route table) written to .copilot/perf-reports/ alongside scan reports. Mirrors the metrics-glossary.md pattern from setup-maf-evals v2. - scan / configure / select: new references/common-pitfalls.md capturing dogfooded false-positive patterns (MA4 in AppHost is expected, MA1 false-fires on 1-agent apps, O3 in MEAI source is normal, sentinel parsing must fail-closed, role classification edge cases, etc). - agentic-perf-reviewer.agent.md: inline Common pitfalls section (agents have no references/ folder by convention). - tests/dotnet-ai/agentic-perf-reviewer/eval.yaml: 11 routing- discrimination scenarios (4 should-invoke, 4 should-defer-to- optimizing-dotnet-performance, 3 should-route-to-child-skill). Validator: ✅ 9 skills + 1 agent + 1 plugin pass. Markdownlint: ✅ 0 errors on changed files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../agents/agentic-perf-reviewer.agent.md | 54 +++++++++ .../references/common-pitfalls.md | 100 +++++++++++++++++ .../skills/scan-agentic-app-perf/SKILL.md | 6 + .../references/check-id-glossary.md | 65 +++++++++++ .../references/common-pitfalls.md | 99 +++++++++++++++++ .../references/common-pitfalls.md | 86 ++++++++++++++ .../dotnet-ai/agentic-perf-reviewer/eval.yaml | 105 ++++++++++++++++++ 7 files changed, 515 insertions(+) create mode 100644 plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/common-pitfalls.md create mode 100644 plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-id-glossary.md create mode 100644 plugins/dotnet-ai/skills/scan-agentic-app-perf/references/common-pitfalls.md create mode 100644 plugins/dotnet-ai/skills/select-agent-models/references/common-pitfalls.md create mode 100644 tests/dotnet-ai/agentic-perf-reviewer/eval.yaml diff --git a/plugins/dotnet-ai/agents/agentic-perf-reviewer.agent.md b/plugins/dotnet-ai/agents/agentic-perf-reviewer.agent.md index 96b45cfd74..78c12cf3f6 100644 --- a/plugins/dotnet-ai/agents/agentic-perf-reviewer.agent.md +++ b/plugins/dotnet-ai/agents/agentic-perf-reviewer.agent.md @@ -126,3 +126,57 @@ Keep reports concise and actionable. - `select-agent-models` — per-agent model recommendations. - `setup-maf-evals` — telemetry / quality / compare harness. - `configure-agentic-perf-rules` — install always-on rules. + +## Common pitfalls + +Real-world failure modes for this agent — observed across the four +target apps used during dogfooding. + +- **Routing to this agent when the project is non-agentic.** The + description string explicitly carves out plain .NET perf reviews + (allocations, async, LINQ, serialization, hot-path optimization) → + `optimizing-dotnet-performance`. If Pass 1 detection finds no + AppHost AND no `Microsoft.Agents.AI` reference AND no + `IChatClient` builders, abort cleanly. Do not "audit" a plain + ASP.NET API or a console app — surface the wrong-agent message + and recommend the .NET perf agent instead. +- **Skipping Pass 2 because Pass 1 looks "fine".** Even when Pass 1 + finds nothing alarming, Pass 2 (running `scan-agentic-app-perf`) + is mandatory. The scan has 24 checks the human eye misses (e.g. + T3 cycles in `WorkflowBuilder`, MH1 full-history sharing buried + in `WithChatOptions`). Quick-triage exit is only when the user + explicitly says "quick triage only". +- **Citing findings from memory.** Always re-read + `.copilot/perf-reports/scan-.md` between Pass 2 and Pass 3. + Token-stream context drift will mangle line numbers and code + snippets if you cite from working memory. Open the file path, + re-read each finding's `file:line`, then write the synthesis. +- **Pre-confirming a fix on the user's behalf.** When the user says + "audit and apply the fixes" in one turn, treat it as INTENT but + not as CONFIRMATION. The invoked skill (e.g. `select-agent-models` + apply mode, `configure-agentic-perf-rules`) must present its OWN + diff and obtain its OWN confirmation before any write. Pre- + confirming with "since you said apply, I'll go ahead" is a real + source of regressions. +- **Recommending a model downgrade without an eval gate.** Any Pass + 3 action that says "downgrade Agent X from gpt-4o to gpt-4o-mini" + must be paired with "validate via `setup-maf-evals` quality mode + before shipping". Apparent free wins on cost frequently regress + quality on edge cases. +- **Producing numeric estimates without evidence.** Pass 1's + qualitative-only rule applies to the entire agent. Never write + "this will save 40% latency" without a `setup-maf-evals` compare + report you can cite. Replace with "should lower per-turn token + cost" or "may reduce critical-path latency". +- **Single-agent apps reaching Pass 2.** Pass 2 will route to + `select-agent-models`, which then aborts (single-agent apps don't + benefit from per-agent model selection). When Pass 1 detects only + 1 agent, skip the `select-agent-models` route in Pass 3's + follow-up offer — keep only the routes that actually apply + (typically `setup-maf-evals` + `configure-agentic-perf-rules`). +- **Forgetting `configure-agentic-perf-rules` when no managed + block exists.** Pass 2 step 3 mandates this check. If the project + has no `.github/copilot-instructions.md` managed block, list it + as a Pass 3 follow-up regardless of other findings — installing + rules is the only durable way to prevent future regressions + between reviews. diff --git a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/common-pitfalls.md b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/common-pitfalls.md new file mode 100644 index 0000000000..e4047ba743 --- /dev/null +++ b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/common-pitfalls.md @@ -0,0 +1,100 @@ +# Common pitfalls + +Real-world failure modes for `configure-agentic-perf-rules` — observed +during dogfooding (interview-coach v2 first install, behavioral coach +no-op re-run, ELI5Agent fresh install). + +## Sentinel parsing (FAIL CLOSED, always) + +- **Lenient sentinel matching.** The BEGIN and END regexes in step 2 + are intentionally strict (anchored, full-line, exact version + pattern). Do not "fix up" a malformed sentinel by partial-matching + or by accepting trailing whitespace beyond what `\s*$` allows. + Auto-repair of corrupted sentinels is forbidden — the user might + have intentionally renamed the block while migrating, and silent + repair would clobber that. +- **Refusing to edit on multiple BEGIN/END.** If the file contains + more than one BEGIN or END, abort with a chat message that names + every offending line number. Multiple managed blocks usually mean + a merge conflict went unresolved; appending a third would make it + worse. +- **Out-of-order sentinels.** BEGIN must appear before its matching + END. If you find END before BEGIN, treat as malformed and abort — + do not swap them. + +## Threshold preservation + +- **Losing user-edited threshold values on update.** Step 2's + "Threshold preservation algorithm" is the contract: parse the + existing `thresholds:` map into `prev_user`, overlay the new + default map, override each known key from `prev_user`. Skipping + this step and writing the new defaults verbatim is the most + user-visible regression this skill can ship. +- **Silently dropping unknown keys.** If `prev_user` has a key not + in `new_defaults` (e.g. a deprecated threshold), drop it AND emit + a chat warning naming the dropped key. Silent drops break audit + trails — the user needs to know their override no longer applies. +- **Type-validating with `Convert.ToInt32` instead of strict parse.** + `agent_count_max: "three"` should fail validation, not coerce to + some default. Use strict numeric parsing; on failure, keep the + default and warn. + +## Target-file selection + +- **Writing to `AGENTS.md` when `.github/copilot-instructions.md` + exists.** The order in step 2's target-file table is the spec: + `.github/copilot-instructions.md` is the primary destination; + `AGENTS.md` gets a stub pointer. The reverse only happens during + the explicit migration path (existing block in AGENTS.md, none in + copilot-instructions.md). +- **Writing to both files.** "Both have a block" is the abort path — + the user must consolidate manually. Writing the rule prose into + two files would split the source of truth and let the two copies + drift on the next update. +- **Creating `.github/` outside the project root.** If neither file + exists, create `.github/copilot-instructions.md` under the resolved + project root (the directory containing the `.sln`/`.slnx`/`*.AppHost.csproj`). + Never write to a parent directory or a sibling project. + +## Path safety + +- **Following symlinks out of the project root.** Step 2's "Path + safety" rule requires resolving the absolute path AND ensuring it + still starts with the project-root prefix after symlink resolution. + Some CI environments place repos under symlinks; a naive resolve + can end up writing to the symlink target outside the workspace. +- **Accepting `..` in paths.** Reject any path containing `..` + segments before normalization, unless the post-normalization path + is still inside the project root. The simplest safe check: + `Path.GetFullPath(target).StartsWith(Path.GetFullPath(projectRoot))`. + +## Cross-tool stub on AGENTS.md + +- **Re-adding the stub on every run.** The stub is one line: + `> Agentic-perf rules for this project live in .github/copilot-instructions.md (managed by configure-agentic-perf-rules).` + Check whether that exact line is already present before appending; + re-running the skill should not grow the file by one line each time. +- **Replacing user prose in AGENTS.md with the stub.** AGENTS.md + often contains real onboarding prose the user wrote. Append the + stub at the bottom if missing; never overwrite existing content. + +## Version handling + +- **Refusing to downgrade is correct.** If the file has a newer + version than this skill, abort cleanly with a chat message naming + both versions. Do not "merge" or "convert" — that's how data loss + happens. +- **Comparing versions as strings.** "v0.1.10" sorts before "v0.1.2" + lexically. Parse into semver triples and compare numerically. + +## Idempotency + +- **No-op path must actually be a no-op.** When the block is present, + same version, and structurally valid, the skill must not touch the + file at all — not even to rewrite identical content. Tooling and + git status both rely on "no change on second run". Verify by + comparing the SHA256 hash before/after; they should match exactly. +- **Tracking "already installed" silently.** Even on a no-op, the + chat output should say "configure-agentic-perf-rules v0.1.0 block + already current — no changes". Quiet no-ops make the user think + the skill didn't run. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md index cff5c4f788..a88bc9f562 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md @@ -94,6 +94,11 @@ Sort findings by severity (critical → warn → info), then by `check_id` - `.copilot/perf-reports/scan-.md` (timestamped, kept) - `.copilot/perf-reports/latest-scan.md` (overwritten each run) +- `.copilot/perf-reports/check-id-glossary.md` (overwritten each run) + — a one-line-per-code reference card so first-time readers of a + report can decode `T1`/`TI3`/`MA4` without opening the skill repo. + Source: copy the "Reference card" section verbatim from + `references/check-id-glossary.md`. See `references/report-template.md` for the exact layout. @@ -185,4 +190,5 @@ After running: - `references/parallelism-checks.md` — sequential calls that could fan out. - `references/otel-coverage-checks.md` — Aspire dashboard, token/cost telemetry. - `references/model-assignment-checks.md` — single-model defaulting, role mismatch. +- `references/check-id-glossary.md` — the reference card written alongside each report. - `references/report-template.md` — exact Markdown layout for the report. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-id-glossary.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-id-glossary.md new file mode 100644 index 0000000000..9db38254f9 --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-id-glossary.md @@ -0,0 +1,65 @@ +# Check-ID glossary + +This file is the source of truth for the check-ID reference card that +`scan-agentic-app-perf` writes alongside every report (parallel to the +metrics-glossary pattern in `setup-maf-evals`). + +When the skill runs, it copies this file's "Reference card" section +verbatim to `.copilot/perf-reports/check-id-glossary.md` so a first-time +report reader can decode the codes without opening the skill repo. + +## Reference card + +> **Prefixes:** `T*` topology · `TI*` tool inventory · `MH*` message +> history · `PW*` prompt weight · `P*` parallelism · `O*` OTel · `MA*` +> model assignment. +> +> **Severity:** `critical` = likely to break a flow or blow the budget · +> `warn` = measurable cost/perf regression · `info` = observation only. + +| ID | Sev | Title | Full check | +|-----|----------|-------------------------------------------------------------|------------| +| T1 | warn | Agent count > 3 | `topology-checks.md#t1` | +| T2 | warn | LLM-routed handoff edges per turn > 2 | `topology-checks.md#t2` | +| T3 | critical | Cycles in the agent graph | `topology-checks.md#t3` | +| T4 | warn | Single-leaf graph with > 2 hops | `topology-checks.md#t4` | +| TI1 | warn | Tools per agent > 8 | `tool-inventory-checks.md#ti1` | +| TI2 | warn | Duplicate tool functionality across agents | `tool-inventory-checks.md#ti2` | +| TI3 | info | Dead tools (declared, never invoked) | `tool-inventory-checks.md#ti3` | +| TI4 | warn | Tool description > 200 chars | `tool-inventory-checks.md#ti4` | +| MH1 | critical | Full chat history shared with every agent | `message-history-checks.md#mh1` | +| MH2 | warn | No history cap (unbounded growth) | `message-history-checks.md#mh2` | +| MH3 | warn | History passed through deterministic agents | `message-history-checks.md#mh3` | +| PW1 | warn | System prompt > 2K tokens | `prompt-weight-checks.md#pw1` | +| PW2 | warn | Few-shot examples in prompt > 3 | `prompt-weight-checks.md#pw2` | +| PW3 | warn | Identical preamble duplicated across agents | `prompt-weight-checks.md#pw3` | +| P1 | warn | Sequential awaits over independent inputs | `parallelism-checks.md#p1` | +| P2 | warn | Sequential agent handoffs that don't share context | `parallelism-checks.md#p2` | +| P3 | info | Tool fan-out behind a single tool wrapper | `parallelism-checks.md#p3` | +| O1 | critical | No `AddOpenTelemetry` call | `otel-coverage-checks.md#o1` | +| O2 | warn | No Aspire dashboard reference | `otel-coverage-checks.md#o2` | +| O3 | warn | Token / cost surfacing missing | `otel-coverage-checks.md#o3` | +| O4 | info | Per-agent activity source missing | `otel-coverage-checks.md#o4` | +| MA1 | warn | All agents on the same model | `model-assignment-checks.md#ma1` | +| MA2 | warn | Reasoning-strong model on a deterministic agent | `model-assignment-checks.md#ma2` | +| MA3 | warn | Cheap model on a planner / decomposer | `model-assignment-checks.md#ma3` | +| MA4 | info | Hard-coded model id outside config | `model-assignment-checks.md#ma4` | + +## Cross-skill routes embedded in `ref:` fields + +| `ref:` value | Skill to run next | +|-----------------------------------|----------------------------------| +| `skill:select-agent-models` | Per-agent model recommendations | +| `skill:setup-maf-evals` | Wire eval reports + telemetry | +| `skill:configure-agentic-perf-rules` | Install always-on rules block | + +## Notes for skill implementation + +- When generating a report, write a copy of the "Reference card" section + (the table and the prefix/severity legend immediately above it) to + `.copilot/perf-reports/check-id-glossary.md`. Overwrite each run; the + content is static within a skill version and does not depend on findings. +- The glossary file is per-repo (one file regardless of how many runs); + the timestamped `scan-.md` reports link to it relatively. +- Keep this table in lockstep with the per-category reference files. Adding + a new check ID in `topology-checks.md` REQUIRES a row here. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/common-pitfalls.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/common-pitfalls.md new file mode 100644 index 0000000000..dc7f7877db --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/common-pitfalls.md @@ -0,0 +1,99 @@ +# Common pitfalls + +Real-world failure modes for `scan-agentic-app-perf` — observed during +dogfooding (interview-coach v1/v2, ELI5Agent, behavioral-interview-coach). + +## Output discipline + +- **Editing source code.** This skill is read-only. The ONLY write + paths it owns are `.copilot/perf-reports/scan-.md`, + `latest-scan.md`, and `check-id-glossary.md`. Never touch + `.gitignore`, source files, config files, AppHost, or + `*.csproj`. If a check tempts you to fix the issue inline, stop and + add it as a finding instead — the user opts into the fix through + the step-6 routing prompt. +- **Surfacing more than 3 critical findings in chat.** Step 5 caps + chat output at 3 critical findings + counts + report path. Anything + beyond that goes in the report file only. Pasting the entire + report into chat defeats the "single file you can re-open" UX. +- **Forgetting `latest-scan.md`.** The timestamped report is for + history; `latest-scan.md` is what tooling and humans will actually + open first. They must be byte-for-byte identical for the same run. + +## Evidence integrity + +- **Hallucinating findings.** Every finding must cite a real file + and (where applicable) a real line. Before adding a finding, re-open + the cited file and verify the snippet exists at the cited line — that + is the "evidence gate" in step 3. If you can't point to evidence, + drop the finding. A 5-finding report with verified citations beats + a 15-finding report with two hallucinations every time. +- **Citing line ranges without re-reading.** Files drift between + the inventory pass (step 1) and the per-category checks (step 2). + Re-read the cited region right before writing each finding, not + before the entire batch. +- **Absence-of-X findings without searched-pattern evidence.** When + a finding fires because something is *missing* (e.g. O3 "no token + surfacing"), the `evidence` field MUST list the exact files and + patterns that were searched — not a snippet. This is what lets the + user reproduce the negative result. + +## False positives we keep seeing + +- **MA4 (hard-coded model id) firing on AppHost code.** MA4 is about + model ids living in *agent service* files (`*.Agent/Program.cs` + etc.) where swapping requires a code change. Model ids declared in + the AppHost via `foundry.AddDeployment("chat", FoundryModel.OpenAI.Gpt4oMini)` + are the **canonical Aspire-native place** — that's not a defect. + When you encounter this pattern, either suppress MA4 entirely or + downgrade to `info` with a "no action required" `next:`. +- **MA1 (single-model) on single-agent apps.** MA1 explicitly assumes + ≥2 agents (different roles, different needs). Do not fire on + 1-agent apps; the check is trivially "satisfied" with one model. +- **O3 (no token surfacing) when `Microsoft.Extensions.AI` activity + source is registered.** MEAI emits `gen_ai.usage.*` activity tags + automatically; if `AddSource("Microsoft.Extensions.AI")` is wired in + OTel and an OTLP exporter is configured, O3 should NOT fire. The + check is for codebases that strip the source or wrap MEAI behind + custom infrastructure that loses the tags. + +## Per-check sharpening + +- **T2 (handoff edges per turn).** Count statically-resolvable edges + only. Don't try to simulate dynamic LLM-routed handoff at scan + time; the count is "edges declared in `CreateHandoffBuilderWith`", + not "edges executed". +- **PW1 (system prompt > 2K tokens).** Use a rough token estimator + (chars/4) or `cl100k_base` if available. Never claim an exact + token count without naming the encoder you used. +- **TI2 (duplicate tool functionality).** Compare tool *descriptions* + (the `[Description("...")]` attribute), not method names. Two tools + with the same method name but different descriptions are usually + fine; two tools with different names and a copy-pasted description + are usually a refactor candidate. + +## Routing offer (step 6) + +- **Inferring intent from the original prompt.** Even if the user + said "audit and fix it", this skill's job ends at the routing + prompt. The follow-up skill is responsible for its own + diff-and-confirm flow. Never call into `configure-agentic-perf-rules` + or `select-agent-models` (apply mode) without the user explicitly + picking that letter at the prompt. +- **Listing routes for skills not actually referenced.** Step 6 says + to render only letters whose target skill is named in some + finding's `ref:` field. If no finding routed to `setup-maf-evals`, + do not offer it. Empty routing offer = skip the prompt entirely. + +## Inventory edge cases + +- **Multiple AppHost projects.** If a solution has more than one + `*.AppHost.csproj`, scan each one and emit one report per host + with the host name in the filename + (`scan--.md`). Do NOT merge — the topology, model set, + and OTel wiring belong to each host independently. +- **Agent registered via DI lambda without `AddAIAgent`.** Patterns + like `services.AddSingleton(...)` or custom factories also + count as agents. The detection in step 1 must include any path + that ends up registering an `IAgent` / `AIAgent` in the host's + service provider. diff --git a/plugins/dotnet-ai/skills/select-agent-models/references/common-pitfalls.md b/plugins/dotnet-ai/skills/select-agent-models/references/common-pitfalls.md new file mode 100644 index 0000000000..40ab8c27ac --- /dev/null +++ b/plugins/dotnet-ai/skills/select-agent-models/references/common-pitfalls.md @@ -0,0 +1,86 @@ +# Common pitfalls + +Real-world failure modes for `select-agent-models` — observed during +dogfooding (interview-coach v2, behavioral coach, ELI5Agent abort path). + +## Applicability + +- **Running on single-agent apps.** Step 1 requires ≥2 agents. + Single-agent apps trivially don't benefit — there's no role + diversity to optimize across. Abort cleanly with a chat message + ("ELI5Agent has 1 agent — `select-agent-models` does not apply. + Use `scan-agentic-app-perf` for a single-agent check"). Do NOT + write a model-plan file. +- **Running on apps without distinct model literals.** If every + agent already resolves the same `IChatClient` from DI (Aspire's + default), there's no per-agent override surface to recommend + against. Still produce a plan — but mark the apply mode as + "requires opting into per-agent `AddChatClient()` first". + +## Role classification + +- **Forcing a role on every agent.** When an agent's instructions + and tool list don't cleanly map to one of the seven roles, mark + the role as `unclear` and explicitly ask the user. Do not silently + default to "worker" — that produces wrong recommendations the user + doesn't realize are wrong. +- **Classifying by agent name alone.** Names lie. `"PlannerAgent"` + might actually be a router; `"FastWorker"` might be doing planning. + Classify by: + 1. The system prompt's verbs (decompose / pick / validate / format / summarize), + 2. The tool list (no tools → not a worker), + 3. The handoff position (root agent → likely router or planner). +- **Treating multi-role agents as one.** If an agent is doing both + planning and validation, surface it as "role: unclear — appears + to mix planning and validation; consider splitting". Don't pick + one role and ignore the other. + +## Apply mode + +- **Applying without a diff preview.** "Apply" mode MUST show a + unified diff of the changes (AppHost connection-string updates + + per-agent `IChatClient` registration changes) AND wait for explicit + user confirmation before any write. The diff is the user's last + chance to catch a wrong role classification before it ships. +- **Editing files outside `*.AppHost/` and `*.Agent*/`.** The only + files this skill edits in apply mode are AppHost code (deployment + declarations + connection strings) and agent service code (the + `AddChatClient` call site). `appsettings.json` model overrides are + an advanced opt-in, not a default write target. +- **Skipping the rollback hint.** After a successful apply, the chat + output must include "Revert with `git checkout -- `" + naming the exact files touched. Apply mode is a destructive + operation; the user needs to know the undo path. + +## Plan output + +- **Recommending downgrades without an eval gate.** Any `delta: + downgrade` row should have a `risks` line that explicitly says + "validate via `setup-maf-evals` quality mode before shipping". + Downgrades that look free on paper often regress task quality. +- **Net cost / latency arrows that don't match the rows.** If the + plan has 1 upgrade and 3 downgrades, the aggregate "net cost" + arrow can still go up depending on call frequency — don't just + count rows. State the assumption ("assumes uniform per-turn call + frequency across agents") if you can't measure it. + +## Cross-skill routing + +- **Telling the user to run `scan-agentic-app-perf` for findings + this skill should produce.** Model-mismatch findings (MA1-MA4 in + scan-agentic-app-perf) overlap with this skill's recommendations. + When invoked directly, produce the plan; don't punt to the scan + skill except when the inventory step fails (no agents detected + at all). + +## Role-model matrix maintenance + +- **Recommending a deprecated model id.** The matrix in + `references/role-model-matrix.md` must be checked against current + provider availability before each release. `gpt-3.5-turbo` and + `gpt-4-turbo-preview` are common stale recommendations — prefer + `gpt-4o-mini` and `gpt-4o` respectively for the same role tiers. +- **Recommending models the user's deployment doesn't have.** When + the AppHost's Foundry deployments are `chat` (= gpt-4o-mini), the + plan can recommend `gpt-4o` but must flag "requires adding a new + deployment in AppHost — see `setup-maf-evals` for the pattern". diff --git a/tests/dotnet-ai/agentic-perf-reviewer/eval.yaml b/tests/dotnet-ai/agentic-perf-reviewer/eval.yaml new file mode 100644 index 0000000000..647e1dc9ce --- /dev/null +++ b/tests/dotnet-ai/agentic-perf-reviewer/eval.yaml @@ -0,0 +1,105 @@ +name: agentic-perf-reviewer +required_agents: + - agentic-perf-reviewer + +# Routing-discrimination scenarios for the agent. +# These exercise whether the agent description matches the right prompts +# AND correctly punts non-agentic .NET perf questions to other agents. +# Static / no-fixture tests — the assertions are over which agent the +# host's description-matcher routes the prompt to. + +scenarios: + # ─── Should route TO agentic-perf-reviewer ───────────────────────── + + - name: prompt-mentions-agentic-app-and-slowness + prompt: | + My agentic .NET app feels slow during multi-agent handoffs. + Can you review it? + assertions: + - type: agent_invoked + name: agentic-perf-reviewer + + - name: prompt-mentions-aspire-foundry-perf-review + prompt: | + Review the perf of my Aspire + Foundry agent project at + ./MyApp for cost and latency issues. + assertions: + - type: agent_invoked + name: agentic-perf-reviewer + + - name: prompt-mentions-topology-and-model-selection + prompt: | + Audit my Microsoft Agent Framework topology and per-agent + model selection — I think we have too many handoffs. + assertions: + - type: agent_invoked + name: agentic-perf-reviewer + + - name: prompt-after-non-trivial-topology-change + prompt: | + I just added two new agents and an LLM-routed handoff edge. + Anything we should worry about before merging? + assertions: + - type: agent_invoked + name: agentic-perf-reviewer + + # ─── Should NOT route to agentic-perf-reviewer ───────────────────── + + - name: plain-dotnet-allocation-question-routes-to-perf-agent + prompt: | + My .NET service has high LOH allocations from string concatenation + in a hot loop. How do I fix? + assertions: + - type: agent_invoked + name: optimizing-dotnet-performance + # explicitly NOT agentic-perf-reviewer per the description's + # "Do NOT use for non-agentic .NET performance reviews" carve-out + + - name: linq-hot-path-question-routes-to-perf-agent + prompt: | + Replace this LINQ-heavy hot path with something faster. + assertions: + - type: agent_invoked + name: optimizing-dotnet-performance + + - name: async-anti-pattern-question-routes-to-perf-agent + prompt: | + I'm seeing thread-pool starvation. Where are my sync-over-async bugs? + assertions: + - type: agent_invoked + name: optimizing-dotnet-performance + + - name: non-dotnet-project-no-invocation + prompt: | + Review my Python LangChain agent app for perf issues. + assertions: + - type: agent_invoked + name: none + # the description scopes to .NET MAF + Aspire + Foundry; + # Python LangChain should not match. + + # ─── Should route to a child skill, NOT the agent ────────────────── + + - name: install-perf-rules-directly-routes-to-skill + prompt: | + Install the agentic perf rules into my project's copilot instructions. + assertions: + - type: skill_invoked + name: configure-agentic-perf-rules + # direct rules-install requests should hit the skill, not the + # umbrella agent. The agent's description says "review ... + + # orchestrates", but pure install requests skip the review pass. + + - name: pick-models-directly-routes-to-skill + prompt: | + Which model should each agent in my workflow use? + assertions: + - type: skill_invoked + name: select-agent-models + + - name: wire-evals-directly-routes-to-skill + prompt: | + Set up evals for my agent app. + assertions: + - type: skill_invoked + name: setup-maf-evals From 676963986fba7cba96e426fddccc24acdfd9f771 Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Mon, 22 Jun 2026 14:34:55 -0700 Subject: [PATCH 12/18] docs(scan-agentic-app-perf): escape placeholder tokens in checks references Wrap , , , , , , , , , , placeholder identifiers in backticks across the 5 *-checks.md files. These had been parsed as raw inline HTML elements by markdownlint (MD033/no-inline-html), producing 17 pre-existing errors. The intent was always literal placeholders in the 'Next:' action templates, so backticks are the correct fix. Verified: markdownlint-cli2 on all 5 changed files now reports 0 errors. Skill validator still passes (9 skills + 1 agent + 1 plugin). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../references/message-history-checks.md | 4 ++-- .../references/model-assignment-checks.md | 2 +- .../references/parallelism-checks.md | 4 ++-- .../references/tool-inventory-checks.md | 8 ++++---- .../scan-agentic-app-perf/references/topology-checks.md | 8 ++++---- 5 files changed, 13 insertions(+), 13 deletions(-) diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/message-history-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/message-history-checks.md index 0d858379cc..82cb0ff1a5 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/message-history-checks.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/message-history-checks.md @@ -15,7 +15,7 @@ token cost. With 4 agents and a 6K-token history, you spend 24K input tokens per turn doing nothing. **Next:** "Pass only the last user message and a one-paragraph summary to -. Use `IChatHistoryReducer` or a manual slice." +``. Use `IChatHistoryReducer` or a manual slice." ### MH2. No history cap (warn) @@ -35,5 +35,5 @@ validator, tool router) is given full chat history. **Why:** deterministic steps do not need conversational context. Their prompt cost should be near-constant. -**Next:** "Pass only the immediate input artifact to ; drop the +**Next:** "Pass only the immediate input artifact to ``; drop the chat history." diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/model-assignment-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/model-assignment-checks.md index 5b755cac64..89c5101be8 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/model-assignment-checks.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/model-assignment-checks.md @@ -29,7 +29,7 @@ is using a frontier reasoning model. **Why:** the marginal quality is near-zero; you are paying for unused capability and per-call latency. -**Next:** "Downgrade to a small model. Validate via +**Next:** "Downgrade `` to a small model. Validate via `setup-maf-evals` quality mode." **Ref:** `skill:select-agent-models` diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/parallelism-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/parallelism-checks.md index ab41ca8059..a571d9993d 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/parallelism-checks.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/parallelism-checks.md @@ -32,7 +32,7 @@ input does not depend on A's output. **Why:** the second call could start as soon as the inputs are known. -**Next:** "Run and with `Task.WhenAll`. Rejoin in the parent +**Next:** "Run `` and `` with `Task.WhenAll`. Rejoin in the parent agent for the consolidation step." ### P3. Tool fan-out behind a single tool wrapper (info) @@ -44,5 +44,5 @@ APIs sequentially. that is internally serial is the hardest kind of latency to find from the outside. -**Next:** "Parallelize the inner calls in ; document the +**Next:** "Parallelize the inner calls in ``; document the expected bound in the tool description so the agent can plan around it." diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/tool-inventory-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/tool-inventory-checks.md index c94962f9fb..dc281f6d2a 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/tool-inventory-checks.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/tool-inventory-checks.md @@ -12,7 +12,7 @@ Detect bloat and redundancy in the per-agent tool list. **Why:** tool descriptions are sent in the system prompt every turn. 15 tools at ~80 tokens each is 1.2K tokens of overhead before the user message. -**Next:** "Split into two agents by domain, or move rarely-used +**Next:** "Split `` into two agents by domain, or move rarely-used tools behind a single 'lookup' tool that takes a category argument." ### TI2. Duplicate tool functionality across agents (warn) @@ -23,7 +23,7 @@ description or near-identical signatures. **Why:** duplication forces the router LLM to disambiguate every turn and inflates aggregate prompt size. -**Next:** "Consolidate and into a single shared tool +**Next:** "Consolidate `` and `` into a single shared tool exposed by both agents." ### TI3. Dead tools (info) @@ -35,7 +35,7 @@ prompt or instructions. **Why:** every registered tool costs prompt tokens whether it gets called or not. -**Next:** "Remove from 's tool list." +**Next:** "Remove `` from ``'s tool list." ### TI4. Tool description > 200 chars (warn) @@ -45,5 +45,5 @@ longer than 200 characters. **Why:** long descriptions multiply across agents that import the tool. Most tools can be described in one sentence. -**Next:** "Trim 's description from chars to ≤ 200; move the +**Next:** "Trim ``'s description from `` chars to ≤ 200; move the detailed contract into XML docs on the parameters." diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/topology-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/topology-checks.md index d1ca960dbb..6a16cac90e 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/topology-checks.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/topology-checks.md @@ -12,7 +12,7 @@ in the AppHost and agent service projects. **Why:** more agents = more LLM hops per turn. Every additional agent that can be routed to costs at least one extra round trip. -**Next:** "Collapse into a single agent with two tool calls instead +**Next:** "Collapse `` into a single agent with two tool calls instead of two agents." **Ref:** `skill:configure-agentic-perf-rules` if the project has no rules @@ -29,7 +29,7 @@ chat completion. next agent before the work even starts is the most common cause of "why is my agent so slow". -**Next:** "Replace the LLM router between and with a deterministic +**Next:** "Replace the LLM router between `` and `` with a deterministic intent classifier or a tool call on the source agent." ### T3. Cycles in the agent graph (critical) @@ -39,7 +39,7 @@ intent classifier or a tool call on the source agent." **Why:** cycles risk infinite loops if the loop-break condition is LLM-judged. Even with a turn cap, a cycle burns budget on retries. -**Next:** "Break the cycle → → by making 's exit condition +**Next:** "Break the cycle `` → `` → `` by making ``'s exit condition deterministic." ### T4. Single-leaf graph with > 2 hops (warn) @@ -51,7 +51,7 @@ agents to reach it. could be one tool call. **Next:** "Move the routing logic into a tool on the entry agent and call - directly." +`` directly." ## Out of scope here From 5dffeffbfcdb35a76eee09222b488d349c14e8fe Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Mon, 22 Jun 2026 16:26:44 -0700 Subject: [PATCH 13/18] feat(select-agent-models): add greenfield 'plan' sub-mode MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Surfaced as a skill-gap during the code-review-buddy dogfood (leslierichardson95/code-review-buddy Phase 2.5): users designing a new agentic app had no skill-driven way to consult the role-model matrix before writing any code — they had to read references/role-model-matrix.md directly. Plan mode is source-free: - Triggers on 'design my topology', 'planning a new app', 'what models should I use, haven't written code yet'. - Asks for { agents, quality_priority, provider_constraint }. - Single-agent plans are valid (recommend mode's <2 abort is bypassed). - Output adds a deployment-shape section (alias -> model id, who uses it) + a verify-checklist driving a later recommend-mode pass for prediction-held confirmation. Files written to a separate model-plan-design-.md series so the later recommend run can sit next to it. Skill description / WHEN / NOT-WHEN updated. references/plan-template.md documents the new layout + relaxed empty-plan contract. Validated against the existing dogfood: code-review-buddy/.copilot/perf-reports/model-plan-design.md was hand- written following exactly this layout in Phase 2.5; verify-mode (model-plan-verify.md) confirmed every prediction held. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../skills/select-agent-models/SKILL.md | 74 +++++++++++++++++-- .../references/plan-template.md | 54 +++++++++++++- 2 files changed, 120 insertions(+), 8 deletions(-) diff --git a/plugins/dotnet-ai/skills/select-agent-models/SKILL.md b/plugins/dotnet-ai/skills/select-agent-models/SKILL.md index 4539836160..719cd63498 100644 --- a/plugins/dotnet-ai/skills/select-agent-models/SKILL.md +++ b/plugins/dotnet-ai/skills/select-agent-models/SKILL.md @@ -1,7 +1,7 @@ --- name: select-agent-models description: | - Recommend a per-agent model assignment for a .NET agentic application (Microsoft Agent Framework + Aspire + Foundry). Reads the existing topology, classifies each agent (router, planner, decomposer, worker, validator, formatter, summarizer), then maps each role to a model from a curated role-model matrix balancing latency, quality, and per-call cost. Two modes: read-only "recommend" (default, writes a plan to .copilot/perf-reports/model-plan-.md) and "apply" (diff-preview-and-confirm, edits AppHost connection strings and per-agent IChatClient registrations). Never applies without explicit confirmation. WHEN: user asks "which model should each agent use", "audit my model selection", "everyone defaults to gpt-4o-mini", or has just received an MA finding from scan-agentic-app-perf. NOT-WHEN: user is comparing providers, tuning prompts, or has only one agent (run scan-agentic-app-perf first). + Recommend a per-agent model assignment for a .NET agentic application (Microsoft Agent Framework + Aspire + Foundry). Reads the existing topology, classifies each agent (router, planner, decomposer, worker, validator, formatter, summarizer), then maps each role to a model from a curated role-model matrix balancing latency, quality, and per-call cost. Three modes: read-only "recommend" (default, scans existing source and writes a plan to .copilot/perf-reports/model-plan-.md), "plan" (greenfield/design-time — no source scan; user declares intended roles, skill emits a plan + recommended Foundry deployment shape), and "apply" (diff-preview-and-confirm, edits AppHost connection strings and per-agent IChatClient registrations). Never applies without explicit confirmation. WHEN: user asks "which model should each agent use", "audit my model selection", "everyone defaults to gpt-4o-mini", "I'm planning a new agentic app — what models should I use", "design my topology before I write code", or has just received an MA finding from scan-agentic-app-perf. NOT-WHEN: user is comparing providers, tuning prompts, or has only one agent (run scan-agentic-app-perf first). --- # select-agent-models @@ -10,7 +10,19 @@ Recommend per-agent model assignments based on each agent's role. ## Workflow -### 1. Inventory agents and current models +### 0. Pick the mode + +| Mode | When to use | Source scan? | Edits files? | +|-------------|------------------------------------------------------------------------|--------------|--------------| +| `recommend` | App already exists; you want a per-agent recommendation (default) | yes | no | +| `plan` | Greenfield — user is **about to** build an agentic app, no code yet | no | no | +| `apply` | After `recommend`, user explicitly says "apply" / "make the changes" | yes | yes (with confirmation) | + +If the user says "design", "planning", "haven't written code yet", +"about to build", or "what should I use" with no source repo in scope, +default to `plan` mode (see step 1a). Otherwise default to `recommend`. + +### 1. Inventory agents and current models (recommend mode) For each agent in the project, record: @@ -20,8 +32,29 @@ For each agent in the project, record: `IChatClient` builder) - Estimated per-turn input tokens (system prompt + history + tool descs) -If fewer than 2 agents are detected, abort. Single-agent apps do not benefit -from this skill. +If fewer than 2 agents are detected, abort recommend mode. Single-agent +apps do not benefit from this skill. If the user truly has only one +agent planned, route them to `plan` mode instead. + +### 1a. Collect intended topology (plan mode) + +Plan mode is **read-only** and does not scan source. Ask the user for: + +- **Agents** — a list of `{ name, intended_role, brief_purpose }`. + `intended_role` must be one of the seven roles from step 2; if the + user is unsure, walk them through the role list and pick together. +- **Quality priority** — `cost` (default) | `latency` | `quality`. + Biases tie-breaks between matrix-primary and acceptable-alternatives. +- **Provider constraint** — `foundry` (default) | `azure-openai` | + `openai` | `any`. Filters out matrix entries that can't be hosted on + the declared provider. + +Plan mode requires at least **1** agent (single-agent plans are valid +here — the matrix still gives useful guidance for a greenfield single +agent). Skip step 1's "abort if < 2" guard in this mode. + +Continue to step 2 with the user-declared roles instead of classifying +from source. ### 2. Classify each agent's role @@ -74,13 +107,41 @@ Aggregate notes: - Net latency change estimate (qualitative: ↓ / ↔ / ↑) - Risks to validate (e.g. "downgrading requires a quality eval first") +### 4a. Plan-mode extras — deployment shape + +In `plan` mode only, after step 4 also produce a **deployment-shape** +recommendation derived from the per-agent rows: + +- Group agents by `recommended_model` and emit one Foundry / Azure + OpenAI deployment alias per distinct model id. +- Use stable alias names (e.g. `chat`, `chat-high`, `chat-reasoning`) + not the model id directly, so the AppHost stays decoupled. +- For each alias, list which agents will resolve it. + +Also emit a **verify-checklist** the user runs after wiring the code: + +1. Re-run `select-agent-models` in `recommend` mode against the built app. +2. Confirm `delta: same` for every agent — that means the plan held. +3. Any `delta: upgrade|downgrade` means a role was classified + differently from source than the user declared at plan time — + reconcile before continuing. + ### 5. Write the recommendation file -Write to: +In `recommend` mode write to: - `.copilot/perf-reports/model-plan-.md` - `.copilot/perf-reports/latest-model-plan.md` +In `plan` mode write to: + +- `.copilot/perf-reports/model-plan-design-.md` +- `.copilot/perf-reports/latest-model-plan-design.md` + +The two file series stay separate so a later `recommend` run can sit +next to the original design-time plan and the verify-checklist can +compare them. + Layout in `references/plan-template.md`. Surface in chat: per-agent row plus the aggregate notes plus the file path. @@ -177,4 +238,5 @@ After apply mode: `IChatClient`s in the AppHost with distinct model ids. - `references/agent-resolution-template.md` — per-agent service registration patterns. -- `references/plan-template.md` — exact Markdown layout for the plan file. +- `references/plan-template.md` — exact Markdown layout for the plan file + (recommend-mode + plan-mode extras). diff --git a/plugins/dotnet-ai/skills/select-agent-models/references/plan-template.md b/plugins/dotnet-ai/skills/select-agent-models/references/plan-template.md index fa63694638..e2791745d2 100644 --- a/plugins/dotnet-ai/skills/select-agent-models/references/plan-template.md +++ b/plugins/dotnet-ai/skills/select-agent-models/references/plan-template.md @@ -47,7 +47,57 @@ Diff: - If quality regresses, revert the affected agent only via this skill. ``` +## Plan mode (greenfield / design-time) layout + +Written to `.copilot/perf-reports/model-plan-design-.md` and +`latest-model-plan-design.md`. The inventory section is replaced with +the user-declared topology, and a deployment-shape + verify-checklist +section is appended. + +```markdown +# Model selection plan (design) — {{ project_name }} + +Run: {{ utc_timestamp }} +Mode: plan (greenfield, no source scan) +Quality priority: {{ cost | latency | quality }} +Provider constraint: {{ foundry | azure-openai | openai | any }} + +## Declared topology + +| Agent | Intended role | Purpose (user-declared) | +|--------------------|---------------|--------------------------| +| diff_summarizer | worker | Summarize a git diff for downstream review | +| style_critic | worker | Markdown code review over the summary | + +## Per-agent recommendations + +| Agent | Role | Recommended model | Rationale | +|--------------------|--------|-------------------|--------------------------------------------------| +| diff_summarizer | worker | gpt-4o-mini | Matrix-primary for worker; latency dominates | +| style_critic | worker | gpt-4o-mini | Matrix-primary for worker; candidate-upgrade if quality bar missed | + +## Deployment shape + +| Alias | Model id | Used by | +|--------|--------------|----------------------------------| +| chat | gpt-4o-mini | diff_summarizer, style_critic | + +Use stable alias names (`chat`, `chat-high`, `chat-reasoning`) in +AppHost — never the model id directly. + +## Verify checklist (run after wiring) + +- [ ] Re-run `select-agent-models` in **recommend** mode against the built app. +- [ ] Confirm every agent shows `Δ: same`. +- [ ] Any `upgrade`/`downgrade` means the source-classified role + differs from what was declared here. Reconcile before continuing. +``` + ## Empty-plan contract -If the inventory finds < 2 agents, the skill aborts and does not write -a plan file. The chat output explains why. +In `recommend` mode, if the inventory finds < 2 agents, the skill aborts +and does not write a plan file. The chat output explains why and routes +the user to `plan` mode instead. + +In `plan` mode there is no such guard — single-agent design plans are +valid. From ecce98a064ab508600c067a995422031ecf500e4 Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Wed, 24 Jun 2026 11:22:12 -0700 Subject: [PATCH 14/18] feat(perf-skills): retire select-agent-models; fold into perf-rules MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Resets scope after dogfood feedback. Original motivation for select-agent-models was 'Copilot always defaults to gpt-4o when scaffolding agents.' The skill grew well past that — recommend/plan/ apply modes, role-model matrix doc, registry-currency questions, greenfield plan-mode follow-on, etc. — and Foundry's model-router covers the per-request piece better than the skill ever could. Net change: minimum viable fix lives where it always should have — in rule #3 of the configure-agentic-perf-rules managed block that Copilot already reads on every turn. configure-agentic-perf-rules (bumped to v0.2.0): - Rule #3 in managed-block-template.md is now self-contained and prescriptive: a role-pick table covering router/validator/formatter/ worker -> small-fast; planner -> reasoning-class; creative -> frontier. Calls out Foundry model-router as the recommended worker pick when prompt length varies. Says 'stop and ask if unsure; do not silently default to gpt-4o.' - rule-rationales.md rule #3 mirrors the new table with why-it-matters + when-frontier-is-justified + 'state the why in code' guidance. - Removed all references to select-agent-models from NOT-WHEN / companion-skills sections. scan-agentic-app-perf: - MA1/MA2/MA3 'Next' actions now point at rule #3 of the managed block instead of select-agent-models. 'Ref:' fields updated. - check-id-glossary.md, common-pitfalls.md, report-template.md, SKILL.md description + follow-up offer updated to drop the skill reference. setup-maf-evals: - Description NOT-WHEN no longer mentions select-agent-models. - Follow-up recommendation after telemetry/quality runs now references 'a model swap per rule #3' instead of the deleted skill. - aspire-dashboard-panel.md drops the cross-reference. agentic-perf-reviewer.agent.md: - Description + Skills-used drop select-agent-models. - Pass 2 model-finding route now points at configure-agentic-perf-rules. - Pass 3 follow-up letters condensed from 4 to 3 (A=rules, B=evals, C=stop). - Single-agent-app pitfall reworded: rule #3 covers single agents too, no abort-to-skip needed. DELETED: plugins/dotnet-ai/skills/select-agent-models/ (6 files). The audit case (scan-agentic-app-perf MA1-MA4) still covers existing apps. The greenfield case is the rule itself, sitting in copilot- instructions.md where Copilot reads it on every scaffold turn — which is the actual problem we set out to solve. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../agents/agentic-perf-reviewer.agent.md | 43 ++-- .../configure-agentic-perf-rules/SKILL.md | 8 +- .../references/managed-block-template.md | 18 +- .../references/rule-rationales.md | 36 ++- .../skills/scan-agentic-app-perf/SKILL.md | 6 +- .../references/check-id-glossary.md | 2 +- .../references/common-pitfalls.md | 2 +- .../references/model-assignment-checks.md | 24 +- .../references/report-template.md | 2 +- .../skills/select-agent-models/SKILL.md | 242 ------------------ .../references/agent-resolution-template.md | 63 ----- .../apphost-multi-client-template.md | 74 ------ .../references/common-pitfalls.md | 86 ------- .../references/plan-template.md | 103 -------- .../references/role-model-matrix.md | 95 ------- .../dotnet-ai/skills/setup-maf-evals/SKILL.md | 6 +- .../references/aspire-dashboard-panel.md | 5 +- 17 files changed, 89 insertions(+), 726 deletions(-) delete mode 100644 plugins/dotnet-ai/skills/select-agent-models/SKILL.md delete mode 100644 plugins/dotnet-ai/skills/select-agent-models/references/agent-resolution-template.md delete mode 100644 plugins/dotnet-ai/skills/select-agent-models/references/apphost-multi-client-template.md delete mode 100644 plugins/dotnet-ai/skills/select-agent-models/references/common-pitfalls.md delete mode 100644 plugins/dotnet-ai/skills/select-agent-models/references/plan-template.md delete mode 100644 plugins/dotnet-ai/skills/select-agent-models/references/role-model-matrix.md diff --git a/plugins/dotnet-ai/agents/agentic-perf-reviewer.agent.md b/plugins/dotnet-ai/agents/agentic-perf-reviewer.agent.md index 78c12cf3f6..b5633494ee 100644 --- a/plugins/dotnet-ai/agents/agentic-perf-reviewer.agent.md +++ b/plugins/dotnet-ai/agents/agentic-perf-reviewer.agent.md @@ -1,5 +1,5 @@ --- -description: "Reviews .NET agentic applications (Microsoft Agent Framework + Aspire + Foundry) for performance, cost, and reliability issues across topology, tools, message history, prompts, parallelism, OTel coverage, and per-agent model selection. Orchestrates scan-agentic-app-perf, select-agent-models, setup-maf-evals, and configure-agentic-perf-rules to produce a single end-to-end review with actionable recommendations. Use when reviewing an MAF agentic app for perf or cost, when an agent app feels slow, or after non-trivial topology changes. Do NOT use for non-agentic .NET performance reviews (hot-path optimization, allocations, LINQ, async, serialization, general code perf) — use optimizing-dotnet-performance instead." +description: "Reviews .NET agentic applications (Microsoft Agent Framework + Aspire + Foundry) for performance, cost, and reliability issues across topology, tools, message history, prompts, parallelism, OTel coverage, and per-agent model selection. Orchestrates scan-agentic-app-perf, setup-maf-evals, and configure-agentic-perf-rules to produce a single end-to-end review with actionable recommendations. Use when reviewing an MAF agentic app for perf or cost, when an agent app feels slow, or after non-trivial topology changes. Do NOT use for non-agentic .NET performance reviews (hot-path optimization, allocations, LINQ, async, serialization, general code perf) — use optimizing-dotnet-performance instead." name: agentic-perf-reviewer tools: ['read', 'search', 'task', 'skill', 'ask_user'] license: MIT @@ -53,8 +53,11 @@ triage. Do not ask whether to proceed. prefix encodes the category — `T*` topology, `TI*` tool inventory, `MH*` message history, `PW*` prompt weight, `P*` parallelism, `O*` OTel coverage, `MA*` model assignment. Routing rules: - - Any `MA*` finding (model assignment) → suggest loading - `select-agent-models` in recommend mode. + - Any `MA*` finding (model assignment) → suggest installing/updating + `configure-agentic-perf-rules` so rule #3 (role-aware model + selection) steers future agent code. If the managed block already + exists, point the user at rule #3 in the rendered + `.github/copilot-instructions.md`. - Any `O*` finding (OTel coverage) → suggest loading `setup-maf-evals` so telemetry/cost are surfaced going forward. - If the project has no `.github/copilot-instructions.md` managed @@ -81,11 +84,10 @@ After Pass 2, produce a single prioritized action list: skills now. Render only the lettered options whose target skill is actually referenced in the synthesis, e.g.: > Want me to run any of these now? - > - **A.** `select-agent-models` (recommend mode) for the model - > findings. + > - **A.** `configure-agentic-perf-rules` to install/update the + > always-on rules (rule #3 covers model selection). > - **B.** `setup-maf-evals` to capture token/quality numbers. - > - **C.** `configure-agentic-perf-rules` to install always-on rules. - > - **D.** No — leave the report and stop. + > - **C.** No — leave the report and stop. If the user picks a letter, hand off to that skill with the audit report path as context. The invoked skill still owns its own diff-and-confirm flow — do not pre-confirm on the user's behalf @@ -96,7 +98,9 @@ After Pass 2, produce a single prioritized action list: - **Do not edit source.** This agent has no `edit` tool. If a fix requires file modifications, route to a skill that owns the diff-and-confirm flow. -- Do not pick models without running `select-agent-models`. +- Do not pick specific model ids without consulting rule #3 of the + managed block in `.github/copilot-instructions.md` (installed by + `configure-agentic-perf-rules`). - Do not recommend a model downgrade without recommending a `setup-maf-evals` quality follow-up. - Cite findings by `check_id` and `file:line`; do not summarize the @@ -104,9 +108,9 @@ After Pass 2, produce a single prioritized action list: - **Apply-mode chaining:** if the user says something like "apply the fixes" in the same turn as invoking this agent, treat that as *intent* but not as *confirmation*. The invoked skill (e.g. - `select-agent-models` apply mode) must still present its own diff - and obtain its own confirmation before any write. Do not pre-confirm - on the user's behalf. + `configure-agentic-perf-rules` apply mode) must still present its + own diff and obtain its own confirmation before any write. Do not + pre-confirm on the user's behalf. - Do not apply this agent to non-agentic .NET apps. If detection in Pass 1 fails, say so and stop. @@ -123,9 +127,8 @@ Keep reports concise and actionable. ## Skills used - `scan-agentic-app-perf` — read-only audit, the workhorse of Pass 2. -- `select-agent-models` — per-agent model recommendations. - `setup-maf-evals` — telemetry / quality / compare harness. -- `configure-agentic-perf-rules` — install always-on rules. +- `configure-agentic-perf-rules` — install always-on rules (rule #3 covers role-aware model selection). ## Common pitfalls @@ -153,8 +156,8 @@ target apps used during dogfooding. re-read each finding's `file:line`, then write the synthesis. - **Pre-confirming a fix on the user's behalf.** When the user says "audit and apply the fixes" in one turn, treat it as INTENT but - not as CONFIRMATION. The invoked skill (e.g. `select-agent-models` - apply mode, `configure-agentic-perf-rules`) must present its OWN + not as CONFIRMATION. The invoked skill (e.g. + `configure-agentic-perf-rules` apply mode) must present its OWN diff and obtain its OWN confirmation before any write. Pre- confirming with "since you said apply, I'll go ahead" is a real source of regressions. @@ -168,12 +171,10 @@ target apps used during dogfooding. "this will save 40% latency" without a `setup-maf-evals` compare report you can cite. Replace with "should lower per-turn token cost" or "may reduce critical-path latency". -- **Single-agent apps reaching Pass 2.** Pass 2 will route to - `select-agent-models`, which then aborts (single-agent apps don't - benefit from per-agent model selection). When Pass 1 detects only - 1 agent, skip the `select-agent-models` route in Pass 3's - follow-up offer — keep only the routes that actually apply - (typically `setup-maf-evals` + `configure-agentic-perf-rules`). +- **Single-agent apps.** Model-selection findings (MA*) still apply + to single-agent apps — the `configure-agentic-perf-rules` rule #3 + table covers single agents too. There's no separate "abort if <2 + agents" gate to skip in Pass 3. - **Forgetting `configure-agentic-perf-rules` when no managed block exists.** Pass 2 step 3 mandates this check. If the project has no `.github/copilot-instructions.md` managed block, list it diff --git a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/SKILL.md b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/SKILL.md index 2247e5f4a2..7015504944 100644 --- a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/SKILL.md +++ b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/SKILL.md @@ -1,6 +1,6 @@ --- name: configure-agentic-perf-rules -version: 0.1.0 +version: 0.2.0 description: > Installs or updates an always-on rules block in a .NET agentic app that makes coding agents volunteer perf and cost concerns by default — agent count, handoff edges, @@ -43,7 +43,6 @@ clobbering user-edited threshold values. - The user wants the agent to actually audit existing code right now — use `scan-agentic-app-perf` instead. This skill only installs guidance. - The user wants to measure tokens, latency, or quality scores — use `setup-maf-evals`. -- The user wants to pick or change per-agent model assignments — use `select-agent-models`. - Generic prompt-engineering or non-perf coding-agent rules (keep those in the user's own instructions section, outside the managed block). @@ -160,7 +159,8 @@ Each rule is in the form **"Before X, justify Y."** Categories, in order: deterministic edge or a conditional `WorkflowBuilder` branch will not work. Default ceiling: 2 LLM-routed edges traversed per user turn. 3. **Model selection.** Before defaulting to a frontier model (e.g. `gpt-4o`), name the - agent's role and pick from the role→model matrix in the `select-agent-models` skill. + agent's role and pick from the role table inside rule #3 of the managed block. + Routers/validators/formatters/workers → small-fast; planners → reasoning-class. 4. **Message-history strategy.** Before sending the full conversation history to an agent, state the bound — turn count, token cap, summarization point, or retrieval strategy. Default warning when unbounded full-history is used in a multi-turn workflow. @@ -238,4 +238,4 @@ If `AGENTS.md` was updated, also confirm the stub line is present exactly once. - `references/threshold-defaults.md` — default numeric values and the rationale for each. - `references/rule-rationales.md` — long-form prose for each of the six rule categories, with examples and counter-examples. -- Companion skills: `scan-agentic-app-perf`, `select-agent-models`, `setup-maf-evals`. +- Companion skills: `scan-agentic-app-perf`, `setup-maf-evals`. diff --git a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/managed-block-template.md b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/managed-block-template.md index 568d302d4e..e8ff121e7d 100644 --- a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/managed-block-template.md +++ b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/managed-block-template.md @@ -46,10 +46,20 @@ conditional `WorkflowBuilder` branch will not work. Default ceiling: ### 3. Model selection -Before defaulting to a frontier model (e.g. `gpt-4o`), name the agent's role and pick -from the role→model matrix in the `select-agent-models` skill. Routers, classifiers, -and summarizers usually want a smaller/faster model; reasoning steps may want a -reasoning-class model. +Before defaulting to a frontier model like `gpt-4o`, name the agent's role and pick +from the table below. Routers, validators, formatters, and workers almost never need +a frontier model; defaulting to one is the largest single source of unnecessary spend. + +| Role | Pick | +|-------------------------------------|-----------------------------------------------------------------------------------| +| router / validator / formatter | small-fast model (e.g. `gpt-4o-mini` or current cheap-fast in your Foundry catalog) | +| worker / summarizer / extraction | small-fast model, **or** Foundry `model-router` deployment if prompt length varies | +| planner / decomposer / open reasoning | reasoning-class model (e.g. `o4-mini` or current reasoning model) — state *why* in a code comment | +| creative / nuanced generation | frontier (e.g. `gpt-4o`) — state *why* in a code comment | + +If unsure which role applies, **stop and ask the user** — do not default to `gpt-4o`. +Specific model ids age fast; check your Foundry catalog for the current cheap-fast, +reasoning-class, and frontier ids before pinning. ### 4. Message-history strategy diff --git a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/rule-rationales.md b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/rule-rationales.md index 3832820c5f..8495fb1734 100644 --- a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/rule-rationales.md +++ b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/rule-rationales.md @@ -62,27 +62,39 @@ runs, always go back to interviewer"), use a deterministic edge. ## 3. Model selection **Rule.** Before defaulting to a frontier model (e.g. `gpt-4o`), name the agent's role -and pick from the role→model matrix in the `select-agent-models` skill. +and pick from the table below. If unsure which role applies, **stop and ask the user** +— do not silently default to `gpt-4o`. **Why it matters.** A frontier model on a router/triage agent costs roughly 10x more per token than `gpt-4o-mini` and is often *worse* at the routing job (frontier models are tuned for nuanced generation, not cheap classification). The default-everything-to- gpt-4o pattern is the largest single source of unnecessary spend in agentic apps. -**Quick role mapping (full matrix in `select-agent-models`):** +**Role → model class:** -| Role | Recommended class | -|------|-------------------| -| Router / triage / "is this done?" | small-fast (gpt-4o-mini, gpt-5-mini, phi-4) | -| Classifier / scorer with structured JSON | small + JSON mode + low temp | -| Summarizer / extraction | small with high context | -| Open-ended reasoning / planning | reasoning class (o1, o3-mini) | -| Tool-heavy specialist | mid-tier with strong function-calling fidelity | -| Creative generation / nuanced writing | frontier (gpt-4o) | +| Role | Pick | Why | +|-------------------------------------|-------------------------------------------------------|--------------------------------------------------------| +| Router / triage / "is this done?" | small-fast (`gpt-4o-mini`, current cheap-fast id) | Classification, not generation; latency dominates | +| Validator / scorer / structured JSON| small-fast + JSON mode + low temp | Deterministic output; cache-friendly | +| Formatter (Markdown / JSON shape) | small-fast, pinned | Output stability matters more than peak quality | +| Worker / summarizer / extraction | small-fast, **or** Foundry `model-router` if prompt length varies | Most calls happen here; latency dominates | +| Planner / decomposer / reasoning | reasoning-class (`o4-mini`, current reasoning id) | Output drives N downstream calls; quality matters most | +| Creative / nuanced generation | frontier (`gpt-4o`, current frontier id) | Genuinely needs frontier capability | **When frontier is justified.** The agent's job is genuinely creative or nuanced -generation, or it must follow complex instructions reliably. Routers and scorers -almost never fall in this bucket. +generation, or it must follow complex instructions reliably. Routers, validators, +formatters, and most workers almost never fall in this bucket. + +**Specific model ids age fast.** The table above uses `gpt-4o-mini`, `o4-mini`, and +`gpt-4o` as anchor examples. Before pinning, check your Foundry catalog +(https://learn.microsoft.com/azure/foundry/openai/concepts/models) for the current +cheap-fast, reasoning-class, and frontier ids. Foundry's `model-router` deployment is +the recommended pick whenever the prompt length or complexity genuinely varies per +request and you don't need cache stability (typical: worker tier). + +**State the why in code.** When you pick a frontier or reasoning model, leave a +one-line comment naming the role and why the cheaper tier wouldn't work. This makes +the choice auditable and the next person can challenge it. --- diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md index a88bc9f562..fc5a4fd44c 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md @@ -1,7 +1,7 @@ --- name: scan-agentic-app-perf description: | - Scan a .NET agentic application (Microsoft Agent Framework + Aspire + Foundry) for performance, cost, and reliability issues across seven check categories: topology, tool inventory, message-history strategy, prompt weight, parallelism, OTel coverage, and per-agent model assignment. Produces a Markdown report at .copilot/perf-reports/scan-.md (plus latest-scan.md) with severity-tagged findings (critical/warn/info), file:line citations, evidence, and concrete next actions that can route into select-agent-models, setup-maf-evals, or configure-agentic-perf-rules. WHEN: user asks "why is my agent slow", "scan my agentic app", "audit my agentic app", "find perf issues", "is my topology too complex", or has just modified an agent topology. NOT-WHEN: user wants to install always-on rules (use configure-agentic-perf-rules), pick models per role (use select-agent-models), or wire up evaluations (use setup-maf-evals); not for non-agentic .NET apps. Read-only — never edits source files. + Scan a .NET agentic application (Microsoft Agent Framework + Aspire + Foundry) for performance, cost, and reliability issues across seven check categories: topology, tool inventory, message-history strategy, prompt weight, parallelism, OTel coverage, and per-agent model assignment. Produces a Markdown report at .copilot/perf-reports/scan-.md (plus latest-scan.md) with severity-tagged findings (critical/warn/info), file:line citations, evidence, and concrete next actions that can route into configure-agentic-perf-rules or setup-maf-evals. WHEN: user asks "why is my agent slow", "scan my agentic app", "audit my agentic app", "find perf issues", "is my topology too complex", or has just modified an agent topology. NOT-WHEN: user wants to install always-on rules (use configure-agentic-perf-rules), or wire up evaluations (use setup-maf-evals); not for non-agentic .NET apps. Read-only — never edits source files. --- # scan-agentic-app-perf @@ -57,7 +57,7 @@ line: 1-based line number, or null evidence: 1-3 line code snippet or measurement (must be present in the cited file) why: one paragraph explaining the impact next: concrete action the developer can take next -ref: optional cross-skill route (e.g. "skill:select-agent-models") +ref: optional cross-skill route (e.g. "skill:configure-agentic-perf-rules") ``` The `check_id` prefix encodes the category — there is no separate @@ -130,7 +130,7 @@ diff-and-confirm flow. 1. Aggregate the unique `ref:` values across all findings. 2. If the set is non-empty, print one prompt of the form: > Want me to follow up on any of these? - > - **A.** Run `select-agent-models` (recommend mode) for the `MA*` findings. + > - **A.** Install/update perf rules via `configure-agentic-perf-rules` to enforce role-aware model selection on future code. > - **B.** Run `setup-maf-evals` to capture token/quality numbers. > - **C.** Run `configure-agentic-perf-rules` to install always-on rules. > - **D.** No — just leave the report. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-id-glossary.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-id-glossary.md index 9db38254f9..fcab4b6c17 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-id-glossary.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-id-glossary.md @@ -49,7 +49,7 @@ report reader can decode the codes without opening the skill repo. | `ref:` value | Skill to run next | |-----------------------------------|----------------------------------| -| `skill:select-agent-models` | Per-agent model recommendations | +| `skill:configure-agentic-perf-rules` | Install/update the always-on perf rules (role-aware model selection lives in rule #3) | | `skill:setup-maf-evals` | Wire eval reports + telemetry | | `skill:configure-agentic-perf-rules` | Install always-on rules block | diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/common-pitfalls.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/common-pitfalls.md index dc7f7877db..19936f8cd0 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/common-pitfalls.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/common-pitfalls.md @@ -78,7 +78,7 @@ dogfooding (interview-coach v1/v2, ELI5Agent, behavioral-interview-coach). said "audit and fix it", this skill's job ends at the routing prompt. The follow-up skill is responsible for its own diff-and-confirm flow. Never call into `configure-agentic-perf-rules` - or `select-agent-models` (apply mode) without the user explicitly + (apply mode) without the user explicitly picking that letter at the prompt. - **Listing routes for skills not actually referenced.** Step 6 says to render only letters whose target skill is named in some diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/model-assignment-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/model-assignment-checks.md index 89c5101be8..ffa8ab9526 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/model-assignment-checks.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/model-assignment-checks.md @@ -14,11 +14,13 @@ Detect single-model defaulting and role-model mismatch. needs. A single default usually overspends on cheap roles and underspends on hard roles. -**Next:** "Run `select-agent-models` to get a per-role recommendation. -Common improvement: use a small/fast model for the router and a -reasoning-strong model only for the planner." +**Next:** "See rule #3 in `.github/copilot-instructions.md` (managed by +`configure-agentic-perf-rules`) — most apps want a small-fast model +for routers/validators/workers and a reasoning-class model only for +planners. If both agents are workers on the same cheap model, this is +expected and can be downgraded to info." -**Ref:** `skill:select-agent-models` +**Ref:** `skill:configure-agentic-perf-rules` ### MA2. Reasoning-strong model on a deterministic agent (warn) @@ -29,10 +31,11 @@ is using a frontier reasoning model. **Why:** the marginal quality is near-zero; you are paying for unused capability and per-call latency. -**Next:** "Downgrade `` to a small model. Validate via -`setup-maf-evals` quality mode." +**Next:** "Downgrade `` to a small-fast model per rule #3 in +`.github/copilot-instructions.md`. Validate via `setup-maf-evals` +quality mode." -**Ref:** `skill:select-agent-models` +**Ref:** `skill:configure-agentic-perf-rules` ### MA3. Cheap model on a planner / decomposer (warn) @@ -42,10 +45,11 @@ on a small model while leaf workers are on a large one. **Why:** plan-quality drives every downstream call. A bad plan from a cheap planner makes the expensive workers run more turns. -**Next:** "Promote the planner to a reasoning-strong model; consider -demoting one or more workers." +**Next:** "Promote the planner to a reasoning-class model per rule #3 +in `.github/copilot-instructions.md`; consider demoting one or more +workers." -**Ref:** `skill:select-agent-models` +**Ref:** `skill:configure-agentic-perf-rules` ### MA4. Hard-coded model id outside config (info) diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/report-template.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/report-template.md index dff808e238..ef16900f5a 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/report-template.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/report-template.md @@ -42,7 +42,7 @@ Project: {{ relative_project_path }} ## Next steps -- If you want to fix the model assignments above, run `select-agent-models`. +- If you want to fix the model assignments above, see rule #3 in `.github/copilot-instructions.md` (managed by `configure-agentic-perf-rules`). - If you want to capture token/quality numbers before vs after, run `setup-maf-evals`. - If you do not yet have always-on rules to prevent regressions, run diff --git a/plugins/dotnet-ai/skills/select-agent-models/SKILL.md b/plugins/dotnet-ai/skills/select-agent-models/SKILL.md deleted file mode 100644 index 719cd63498..0000000000 --- a/plugins/dotnet-ai/skills/select-agent-models/SKILL.md +++ /dev/null @@ -1,242 +0,0 @@ ---- -name: select-agent-models -description: | - Recommend a per-agent model assignment for a .NET agentic application (Microsoft Agent Framework + Aspire + Foundry). Reads the existing topology, classifies each agent (router, planner, decomposer, worker, validator, formatter, summarizer), then maps each role to a model from a curated role-model matrix balancing latency, quality, and per-call cost. Three modes: read-only "recommend" (default, scans existing source and writes a plan to .copilot/perf-reports/model-plan-.md), "plan" (greenfield/design-time — no source scan; user declares intended roles, skill emits a plan + recommended Foundry deployment shape), and "apply" (diff-preview-and-confirm, edits AppHost connection strings and per-agent IChatClient registrations). Never applies without explicit confirmation. WHEN: user asks "which model should each agent use", "audit my model selection", "everyone defaults to gpt-4o-mini", "I'm planning a new agentic app — what models should I use", "design my topology before I write code", or has just received an MA finding from scan-agentic-app-perf. NOT-WHEN: user is comparing providers, tuning prompts, or has only one agent (run scan-agentic-app-perf first). ---- - -# select-agent-models - -Recommend per-agent model assignments based on each agent's role. - -## Workflow - -### 0. Pick the mode - -| Mode | When to use | Source scan? | Edits files? | -|-------------|------------------------------------------------------------------------|--------------|--------------| -| `recommend` | App already exists; you want a per-agent recommendation (default) | yes | no | -| `plan` | Greenfield — user is **about to** build an agentic app, no code yet | no | no | -| `apply` | After `recommend`, user explicitly says "apply" / "make the changes" | yes | yes (with confirmation) | - -If the user says "design", "planning", "haven't written code yet", -"about to build", or "what should I use" with no source repo in scope, -default to `plan` mode (see step 1a). Otherwise default to `recommend`. - -### 1. Inventory agents and current models (recommend mode) - -For each agent in the project, record: - -- Agent name -- A short role guess from instructions / tool list / handoff position -- Current model id (from AppHost connection string or per-agent - `IChatClient` builder) -- Estimated per-turn input tokens (system prompt + history + tool descs) - -If fewer than 2 agents are detected, abort recommend mode. Single-agent -apps do not benefit from this skill. If the user truly has only one -agent planned, route them to `plan` mode instead. - -### 1a. Collect intended topology (plan mode) - -Plan mode is **read-only** and does not scan source. Ask the user for: - -- **Agents** — a list of `{ name, intended_role, brief_purpose }`. - `intended_role` must be one of the seven roles from step 2; if the - user is unsure, walk them through the role list and pick together. -- **Quality priority** — `cost` (default) | `latency` | `quality`. - Biases tie-breaks between matrix-primary and acceptable-alternatives. -- **Provider constraint** — `foundry` (default) | `azure-openai` | - `openai` | `any`. Filters out matrix entries that can't be hosted on - the declared provider. - -Plan mode requires at least **1** agent (single-agent plans are valid -here — the matrix still gives useful guidance for a greenfield single -agent). Skip step 1's "abort if < 2" guard in this mode. - -Continue to step 2 with the user-declared roles instead of classifying -from source. - -### 2. Classify each agent's role - -Use `references/role-model-matrix.md`. Roles are: - -- **router** — picks the next agent or tool. Short prompt, deterministic - behaviour preferred. -- **planner** — decomposes the task. Reasoning-strong; output drives - every downstream call. -- **decomposer** — splits work into N parallel items. Reasoning-medium. -- **worker** — does the unit-of-work the planner described. Medium quality; - most calls happen here so latency dominates. -- **validator** — yes/no/score on a small input. Deterministic, small - output. -- **formatter** — renders structured output (JSON, Markdown). Deterministic, - small output. -- **summarizer** — compresses chat history. Medium reasoning, often run - hot. - -If an agent does not cleanly map to one role, mark its role as -`unclear` and recommend that the user review it. - -### 3. Look up recommended model per role - -`references/role-model-matrix.md` contains the canonical recommendation -table. The matrix has columns: - -- Role -- Recommended model (primary) -- Acceptable alternatives -- Avoid -- Rationale (latency / quality / cost trade) - -### 4. Build the plan - -For each agent, produce a row: - -```yaml -agent: -current_model: -role: -recommended_model: -delta: same | upgrade | downgrade -rationale: -``` - -Aggregate notes: - -- Net cost change estimate (qualitative: ↓ / ↔ / ↑) -- Net latency change estimate (qualitative: ↓ / ↔ / ↑) -- Risks to validate (e.g. "downgrading requires a quality eval first") - -### 4a. Plan-mode extras — deployment shape - -In `plan` mode only, after step 4 also produce a **deployment-shape** -recommendation derived from the per-agent rows: - -- Group agents by `recommended_model` and emit one Foundry / Azure - OpenAI deployment alias per distinct model id. -- Use stable alias names (e.g. `chat`, `chat-high`, `chat-reasoning`) - not the model id directly, so the AppHost stays decoupled. -- For each alias, list which agents will resolve it. - -Also emit a **verify-checklist** the user runs after wiring the code: - -1. Re-run `select-agent-models` in `recommend` mode against the built app. -2. Confirm `delta: same` for every agent — that means the plan held. -3. Any `delta: upgrade|downgrade` means a role was classified - differently from source than the user declared at plan time — - reconcile before continuing. - -### 5. Write the recommendation file - -In `recommend` mode write to: - -- `.copilot/perf-reports/model-plan-.md` -- `.copilot/perf-reports/latest-model-plan.md` - -In `plan` mode write to: - -- `.copilot/perf-reports/model-plan-design-.md` -- `.copilot/perf-reports/latest-model-plan-design.md` - -The two file series stay separate so a later `recommend` run can sit -next to the original design-time plan and the verify-checklist can -compare them. - -Layout in `references/plan-template.md`. - -Surface in chat: per-agent row plus the aggregate notes plus the file path. - -### 6. Apply mode (only if user explicitly asks "apply" / "make the -changes") - -Apply mode is **off by default**. To run it, the user must say "apply", -"make the changes", "switch to recommended models", or similar. - -**Confirmation contract** — the initial apply request only enables -preview mode. The actual write requires a *second* user response after -the diff is shown. Wording in the initial request like "apply and -confirm", "yes do it", or "proceed" is treated as *intent*, not as -*confirmation* — the agent must still show the diff and ask. - -Steps: - -1. **Provider resolution.** Detect whether the project uses public - OpenAI ids, Azure OpenAI deployment aliases, Foundry deployments, - or another provider: - - Check the AppHost connection string type (`AddAzureOpenAI` vs - `AddOpenAI` vs others). - - Check `appsettings.json` for keys ending in `-deployment-name`, - `-deployment`, or values that look like custom names (not the - OpenAI public id pattern). - - If Azure is detected, refuse to write a public OpenAI id like - `gpt-4o-mini` directly. Map the recommendation to an existing - deployment alias by: - a. enumerating deployment aliases from `appsettings.json` / - AppHost parameters, and - b. asking the user to pick which alias maps to each role, or - c. recommending creating a new deployment if no suitable alias - exists (and stopping apply mode in that case). -2. **Pre-write validation.** For every file that would be modified, - verify it parses (JSON / C# compiles via `dotnet build` dry-run on - the AppHost). If any target file is unparseable, abort apply mode - before any write. -3. **Diff preview.** Show a unified diff of every change you intend to - make: - - AppHost connection-string updates / parameter values - - Per-agent `IChatClient` registrations - - `appsettings.json` model-id keys -4. **Confirm.** Ask the user to confirm. Only on `yes` / explicit - confirmation, proceed. -5. **Atomic write.** Write all changes. If any write fails midway, - restore *all* touched files from their pre-write content. Do not - leave the project in a partially-applied state. -6. **Build.** Run `dotnet build` on the touched projects. - - On success: record the apply timestamp and per-agent old → new - mapping in the plan file. - - On failure: revert all changes from step 5, surface the build - output, and report `apply: failed (build)`. Do not declare - success on a failing build. -7. **Recommend follow-up.** After a successful apply, recommend running - `setup-maf-evals` quality mode to validate no quality regression. - -If the user says no or anything other than yes, discard the diff and -leave files untouched. - -## Validation - -After read-only mode: - -- A new file exists at `.copilot/perf-reports/model-plan-.md`. -- `latest-model-plan.md` exists and matches the timestamped file. -- The plan lists every detected agent with a row. - -After apply mode: - -- The diff was shown and confirmed before any file write. -- All touched projects build (`dotnet build` exit 0). -- The plan file records the apply timestamp and which agents were modified. - -## Common pitfalls - -- **Applying without confirmation.** The default is recommend-only. Do not - edit files without an explicit user confirmation in apply mode. -- **Recommending a downgrade with no quality check.** Always pair a - downgrade recommendation with a `setup-maf-evals` follow-up. -- **Inventing roles.** If an agent's purpose is unclear, say so. Do not - guess; the user must classify it. -- **Hard-coding model ids in code.** When applying, prefer - `appsettings.json` over inline string literals so the next swap is a - config change. -- **Ignoring provider differences.** This skill targets model *selection* - within an already-chosen provider. If the user wants to compare - providers, surface that as a follow-up question, not a recommendation. - -## References - -- `references/role-model-matrix.md` — role → recommended model table. -- `references/apphost-multi-client-template.md` — wiring multiple - `IChatClient`s in the AppHost with distinct model ids. -- `references/agent-resolution-template.md` — per-agent service - registration patterns. -- `references/plan-template.md` — exact Markdown layout for the plan file - (recommend-mode + plan-mode extras). diff --git a/plugins/dotnet-ai/skills/select-agent-models/references/agent-resolution-template.md b/plugins/dotnet-ai/skills/select-agent-models/references/agent-resolution-template.md deleted file mode 100644 index 14ec5c561c..0000000000 --- a/plugins/dotnet-ai/skills/select-agent-models/references/agent-resolution-template.md +++ /dev/null @@ -1,63 +0,0 @@ -# Per-agent service — resolving the right IChatClient - -How an individual agent service consumes the keyed or -configuration-bound `IChatClient` from the AppHost. - -## Pattern — typed `IOptions` - -```csharp -// agent service Program.cs -var builder = WebApplication.CreateBuilder(args); -builder.AddServiceDefaults(); - -builder.Services.Configure(builder.Configuration); - -builder.Services.AddSingleton(sp => -{ - var opts = sp.GetRequiredService>().Value; - return new ChatClient(model: opts.Model, apiKey: opts.ApiKey); -}); - -builder.Services.AddSingleton(sp => new ChatClientAgent( - chatClient: sp.GetRequiredService(), - instructions: SystemPrompts.Router)); -``` - -```csharp -public sealed class AgentModelOptions -{ - public string Model { get; set; } = ""; - public string ApiKey { get; set; } = ""; -} -``` - -`appsettings.json`: - -```json -{ - "Model": "gpt-4o-mini", - "ApiKey": "..." -} -``` - -## Pattern — keyed resolution (multiple clients in one process) - -```csharp -public sealed class WorkerService -{ - private readonly IChatClient _client; - public WorkerService([FromKeyedServices("worker")] IChatClient client) - { - _client = client; - } -} -``` - -## What apply mode does NOT do - -- Rewrite agent classes. -- Change agent instructions. -- Switch DI lifetimes (Singleton vs Scoped). -- Move from Pattern B to Pattern A or vice versa. - -It only updates the model id values themselves. diff --git a/plugins/dotnet-ai/skills/select-agent-models/references/apphost-multi-client-template.md b/plugins/dotnet-ai/skills/select-agent-models/references/apphost-multi-client-template.md deleted file mode 100644 index f9fc90b46b..0000000000 --- a/plugins/dotnet-ai/skills/select-agent-models/references/apphost-multi-client-template.md +++ /dev/null @@ -1,74 +0,0 @@ -# AppHost — multi-client wiring template - -Show how to register multiple `IChatClient`s in the AppHost, each tied -to a distinct model deployment, so per-agent services can resolve the -client matching their role. - -## Pattern A — Aspire AppHost with named connection strings - -```csharp -// AppHost/Program.cs -var builder = DistributedApplication.CreateBuilder(args); - -var openai = builder.AddConnectionString("openai"); - -var routerModel = builder.AddParameter("router-model", secret: false); // e.g. "gpt-4o-mini" -var plannerModel = builder.AddParameter("planner-model", secret: false); // e.g. "o4-mini" -var workerModel = builder.AddParameter("worker-model", secret: false); // e.g. "gpt-4o-mini" - -builder.AddProject("router") - .WithReference(openai) - .WithEnvironment("Model", routerModel); - -builder.AddProject("planner") - .WithReference(openai) - .WithEnvironment("Model", plannerModel); - -builder.AddProject("worker") - .WithReference(openai) - .WithEnvironment("Model", workerModel); - -builder.Build().Run(); -``` - -`appsettings.json` in AppHost: - -```json -{ - "Parameters": { - "router-model": "gpt-4o-mini", - "planner-model": "o4-mini", - "worker-model": "gpt-4o-mini" - } -} -``` - -## Pattern B — single service with multiple clients - -```csharp -// for monolith services that host multiple agents in-process -builder.Services.AddKeyedSingleton("router", (sp, _) => - new ChatClient(model: "gpt-4o-mini", apiKey: cfg["OpenAI:Key"])); - -builder.Services.AddKeyedSingleton("planner", (sp, _) => - new ChatClient(model: "o4-mini", apiKey: cfg["OpenAI:Key"])); - -builder.Services.AddKeyedSingleton("worker", (sp, _) => - new ChatClient(model: "gpt-4o-mini", apiKey: cfg["OpenAI:Key"])); -``` - -Then resolve with `[FromKeyedServices("router")] IChatClient router`. - -## What apply mode edits - -Apply mode of `select-agent-models` updates: - -1. The model parameter in AppHost (Pattern A) or the keyed registration - (Pattern B). -2. The matching `appsettings.json` value. -3. Nothing else. It does not change the agent class itself or the - instructions. - -If the project does not yet follow either pattern, the skill recommends -the migration in the plan file but does not perform it in apply mode — -that is a structural change beyond model selection. diff --git a/plugins/dotnet-ai/skills/select-agent-models/references/common-pitfalls.md b/plugins/dotnet-ai/skills/select-agent-models/references/common-pitfalls.md deleted file mode 100644 index 40ab8c27ac..0000000000 --- a/plugins/dotnet-ai/skills/select-agent-models/references/common-pitfalls.md +++ /dev/null @@ -1,86 +0,0 @@ -# Common pitfalls - -Real-world failure modes for `select-agent-models` — observed during -dogfooding (interview-coach v2, behavioral coach, ELI5Agent abort path). - -## Applicability - -- **Running on single-agent apps.** Step 1 requires ≥2 agents. - Single-agent apps trivially don't benefit — there's no role - diversity to optimize across. Abort cleanly with a chat message - ("ELI5Agent has 1 agent — `select-agent-models` does not apply. - Use `scan-agentic-app-perf` for a single-agent check"). Do NOT - write a model-plan file. -- **Running on apps without distinct model literals.** If every - agent already resolves the same `IChatClient` from DI (Aspire's - default), there's no per-agent override surface to recommend - against. Still produce a plan — but mark the apply mode as - "requires opting into per-agent `AddChatClient()` first". - -## Role classification - -- **Forcing a role on every agent.** When an agent's instructions - and tool list don't cleanly map to one of the seven roles, mark - the role as `unclear` and explicitly ask the user. Do not silently - default to "worker" — that produces wrong recommendations the user - doesn't realize are wrong. -- **Classifying by agent name alone.** Names lie. `"PlannerAgent"` - might actually be a router; `"FastWorker"` might be doing planning. - Classify by: - 1. The system prompt's verbs (decompose / pick / validate / format / summarize), - 2. The tool list (no tools → not a worker), - 3. The handoff position (root agent → likely router or planner). -- **Treating multi-role agents as one.** If an agent is doing both - planning and validation, surface it as "role: unclear — appears - to mix planning and validation; consider splitting". Don't pick - one role and ignore the other. - -## Apply mode - -- **Applying without a diff preview.** "Apply" mode MUST show a - unified diff of the changes (AppHost connection-string updates + - per-agent `IChatClient` registration changes) AND wait for explicit - user confirmation before any write. The diff is the user's last - chance to catch a wrong role classification before it ships. -- **Editing files outside `*.AppHost/` and `*.Agent*/`.** The only - files this skill edits in apply mode are AppHost code (deployment - declarations + connection strings) and agent service code (the - `AddChatClient` call site). `appsettings.json` model overrides are - an advanced opt-in, not a default write target. -- **Skipping the rollback hint.** After a successful apply, the chat - output must include "Revert with `git checkout -- `" - naming the exact files touched. Apply mode is a destructive - operation; the user needs to know the undo path. - -## Plan output - -- **Recommending downgrades without an eval gate.** Any `delta: - downgrade` row should have a `risks` line that explicitly says - "validate via `setup-maf-evals` quality mode before shipping". - Downgrades that look free on paper often regress task quality. -- **Net cost / latency arrows that don't match the rows.** If the - plan has 1 upgrade and 3 downgrades, the aggregate "net cost" - arrow can still go up depending on call frequency — don't just - count rows. State the assumption ("assumes uniform per-turn call - frequency across agents") if you can't measure it. - -## Cross-skill routing - -- **Telling the user to run `scan-agentic-app-perf` for findings - this skill should produce.** Model-mismatch findings (MA1-MA4 in - scan-agentic-app-perf) overlap with this skill's recommendations. - When invoked directly, produce the plan; don't punt to the scan - skill except when the inventory step fails (no agents detected - at all). - -## Role-model matrix maintenance - -- **Recommending a deprecated model id.** The matrix in - `references/role-model-matrix.md` must be checked against current - provider availability before each release. `gpt-3.5-turbo` and - `gpt-4-turbo-preview` are common stale recommendations — prefer - `gpt-4o-mini` and `gpt-4o` respectively for the same role tiers. -- **Recommending models the user's deployment doesn't have.** When - the AppHost's Foundry deployments are `chat` (= gpt-4o-mini), the - plan can recommend `gpt-4o` but must flag "requires adding a new - deployment in AppHost — see `setup-maf-evals` for the pattern". diff --git a/plugins/dotnet-ai/skills/select-agent-models/references/plan-template.md b/plugins/dotnet-ai/skills/select-agent-models/references/plan-template.md deleted file mode 100644 index e2791745d2..0000000000 --- a/plugins/dotnet-ai/skills/select-agent-models/references/plan-template.md +++ /dev/null @@ -1,103 +0,0 @@ -# Plan template - -The exact Markdown layout written to -`.copilot/perf-reports/model-plan-.md` and -`latest-model-plan.md`. - -```markdown -# Model selection plan — {{ project_name }} - -Run: {{ utc_timestamp }} -Project: {{ relative_project_path }} -Mode: recommend | apply ({{ confirmed_at | "n/a" }}) - -## Per-agent recommendations - -| Agent | Role | Current model | Recommended model | Δ | Rationale | -|------------|------------|------------------|-------------------|------------|--------------------------------------------| -| router | router | gpt-4o | gpt-4o-mini | downgrade | One-shot classification; latency dominates | -| planner | planner | gpt-4o-mini | o4-mini | upgrade | Plan quality drives N downstream calls | -| worker | worker | gpt-4o | gpt-4o-mini | downgrade | Most calls; latency dominates | - -## Aggregate notes - -- **Cost:** ↓ (downgrades on router and worker outweigh planner upgrade) -- **Latency:** ↓ (router and worker shrink; planner runs once per turn) -- **Quality risk:** validate planner upgrade and worker downgrade with - `setup-maf-evals` quality mode before promoting. - -## Apply preview (only present in apply mode) - -Files to be modified: - -- `MyApp.AppHost/appsettings.json` -- `MyApp.AppHost/Program.cs` (parameter declarations only) - -Diff: - -```diff -- "worker-model": "gpt-4o", -+ "worker-model": "gpt-4o-mini", -``` - -## Next steps - -- Run `setup-maf-evals` quality mode against the new assignments. -- Re-run `scan-agentic-app-perf` after evals confirm parity. -- If quality regresses, revert the affected agent only via this skill. -``` - -## Plan mode (greenfield / design-time) layout - -Written to `.copilot/perf-reports/model-plan-design-.md` and -`latest-model-plan-design.md`. The inventory section is replaced with -the user-declared topology, and a deployment-shape + verify-checklist -section is appended. - -```markdown -# Model selection plan (design) — {{ project_name }} - -Run: {{ utc_timestamp }} -Mode: plan (greenfield, no source scan) -Quality priority: {{ cost | latency | quality }} -Provider constraint: {{ foundry | azure-openai | openai | any }} - -## Declared topology - -| Agent | Intended role | Purpose (user-declared) | -|--------------------|---------------|--------------------------| -| diff_summarizer | worker | Summarize a git diff for downstream review | -| style_critic | worker | Markdown code review over the summary | - -## Per-agent recommendations - -| Agent | Role | Recommended model | Rationale | -|--------------------|--------|-------------------|--------------------------------------------------| -| diff_summarizer | worker | gpt-4o-mini | Matrix-primary for worker; latency dominates | -| style_critic | worker | gpt-4o-mini | Matrix-primary for worker; candidate-upgrade if quality bar missed | - -## Deployment shape - -| Alias | Model id | Used by | -|--------|--------------|----------------------------------| -| chat | gpt-4o-mini | diff_summarizer, style_critic | - -Use stable alias names (`chat`, `chat-high`, `chat-reasoning`) in -AppHost — never the model id directly. - -## Verify checklist (run after wiring) - -- [ ] Re-run `select-agent-models` in **recommend** mode against the built app. -- [ ] Confirm every agent shows `Δ: same`. -- [ ] Any `upgrade`/`downgrade` means the source-classified role - differs from what was declared here. Reconcile before continuing. -``` - -## Empty-plan contract - -In `recommend` mode, if the inventory finds < 2 agents, the skill aborts -and does not write a plan file. The chat output explains why and routes -the user to `plan` mode instead. - -In `plan` mode there is no such guard — single-agent design plans are -valid. diff --git a/plugins/dotnet-ai/skills/select-agent-models/references/role-model-matrix.md b/plugins/dotnet-ai/skills/select-agent-models/references/role-model-matrix.md deleted file mode 100644 index 37ab14423f..0000000000 --- a/plugins/dotnet-ai/skills/select-agent-models/references/role-model-matrix.md +++ /dev/null @@ -1,95 +0,0 @@ -# Role → model matrix - -Recommendations are model-family-neutral where possible, with concrete -defaults for OpenAI/Foundry. Names below are illustrative; substitute the -deployment the user actually has access to. - -## Matrix - -| Role | Recommended (primary) | Acceptable alternatives | Avoid | Rationale (latency / quality / cost) | -|-------------|-----------------------|------------------------------------|--------------------------------|------------------------------------------------------------------------------------------------------| -| router | gpt-4o-mini | o4-mini, gpt-4.1-mini | frontier reasoning models | One-shot classification. Latency dominates; a small fast model is correct. Quality differential is negligible for ≤ 5-way routes. | -| planner | o4-mini (reasoning) | gpt-4o, o3-mini | gpt-3.5-turbo, gpt-4o-mini | Plan quality drives N downstream calls. Reasoning model pays back in fewer worker turns. | -| decomposer | o4-mini | gpt-4o, gpt-4.1 | small chat-only models | Similar to planner, but typically smaller output. Reasoning-medium is enough. | -| worker | gpt-4o-mini | gpt-4.1-mini, gpt-4o | frontier models for bulk work | Most calls happen here; latency dominates. Bumping every worker to a frontier model is the most common cost mistake. | -| validator | gpt-4o-mini | gpt-4.1-mini | reasoning models | Yes/no/score; small input, small output, deterministic. A small model with a tight rubric beats a large one with a fuzzy prompt. | -| formatter | gpt-4o-mini | gpt-4.1-mini | reasoning models | Structured-output transformation. Quality plateau is hit quickly. | -| summarizer | gpt-4o-mini | gpt-4.1-mini, gpt-4o | reasoning models for hot loops | Runs every turn (or near it). Latency and cost matter more than peak quality. | - -## Provider notes - -- **Public OpenAI:** model id is the canonical OpenAI name - (`gpt-4o-mini`, `o4-mini`, etc.). Apply mode writes the id directly. -- **Foundry / Azure OpenAI:** the model id stored in `appsettings.json` - is the **deployment alias** (e.g. `my-prod-mini`), not the OpenAI - public id. Apply mode must NOT write a public id like `gpt-4o-mini` - into an Azure project's `appsettings.json` — that will break at - runtime. Instead: - - Recommend the model *family* in the plan (e.g. "use a - small/fast model for the router"). - - In apply mode, ask the user which deployment alias maps to each - recommended role, or recommend creating a new deployment. - - If no suitable deployment exists, the plan's `delta` should be - `unmapped` and apply mode must abort for that agent. -- **Anthropic / Bedrock:** map "reasoning-strong" → Claude Sonnet, - "small/fast" → Claude Haiku. -- **Local / Ollama:** map "small/fast" → Llama 3.2 / Phi-3, "reasoning" - → Llama 3.3 70B or DeepSeek-R1; expect higher latency than hosted - reasoning models and re-eval quality. - -## Router sub-types - -The default `router → gpt-4o-mini` recommendation assumes a *simple -classifier*. Promote to a stronger model when **any** of these apply: - -- The router generates **tool-call arguments** (not just selects a - destination). -- The router performs **schema validation** or policy checks on user - input. -- The router chooses among **more than 5 destinations** with - overlapping descriptions. -- A misroute is **expensive** (e.g. routes to a long-running workflow). - -In those cases, recommend `gpt-4o` / `o4-mini` instead and note the -upgrade in the plan's rationale. - -## Planner: `o4-mini` vs `gpt-4o` - -Recommend a reasoning model (`o4-mini`, `o3-mini`) when: - -- The plan has multi-step dependencies between worker outputs. -- The user task is open-ended and the planner must choose what to do - before how. - -Recommend `gpt-4o` / `gpt-4.1` when: - -- Latency dominates (interactive UX with strict p95 budget). -- The plan output is mostly structured (JSON shape known in advance). -- Planning depth is shallow (≤ 3 steps). - -## Multi-role agents - -When an agent fits more than one role, classify by the **highest- -consequence output downstream agents consume**: - -| Combination | Classify as | -|-----------------------------------|------------------------------| -| router + shallow input validation | router | -| router + tool-arg generation | reasoning router (see above) | -| planner + output formatting | planner | -| validator + scoring | validator | -| worker + summarizer | worker | - -If the role is genuinely unclear after applying these rules, mark as -`unclear` in the plan and ask the user to classify before apply mode. - -## When to deviate - -- **Strict-latency interactive UX (≤ 1.5s p95):** override planner to - `gpt-4o-mini` and accept a small quality hit. Validate with evals. -- **High-stakes single-shot (e.g. legal summarization):** override - worker to a frontier model for the critical step only; keep the rest - on small models. -- **Strict cost budget (≤ $X / 1K turns):** start every role at - small/fast, then upgrade only the role that fails the eval-quality - bar. diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md b/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md index e3da25f1e3..74d580f4e1 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/SKILL.md @@ -1,7 +1,7 @@ --- name: setup-maf-evals description: | - Scaffold an `.Evals.Tests` MSTest project alongside a .NET agentic app (MAF + Aspire + Foundry) wired to the GA `Microsoft.Extensions.AI.Evaluation.Reporting` pipeline. Three evaluator categories: **NLP** (deterministic BLEU/GLEU/F1, no API key), **Quality** (LLM-as-judge Relevance/Coherence/Fluency, etc.), **Safety** (Hate/Violence/SelfHarm/Sexual via Azure AI Foundry). Auto-installs the `aieval` dotnet tool, detects the app's `IChatClient` registration and generates a factory so `EVAL_USE_REAL_AGENT=1` works without manual wiring, and emits an HTML report at `.copilot/perf-reports/evals//report.html`. Optional GitHub Actions workflow runs the evals on every PR. WHEN: user asks "set up evals", "add evaluation harness", "measure my agent perf", "validate quality after a model change", "compare gpt-4o vs gpt-4o-mini", "add safety evaluators", "generate eval report". NOT-WHEN: one-shot audit (use scan-agentic-app-perf), install rules (configure-agentic-perf-rules), pick models (select-agent-models). + Scaffold an `.Evals.Tests` MSTest project alongside a .NET agentic app (MAF + Aspire + Foundry) wired to the GA `Microsoft.Extensions.AI.Evaluation.Reporting` pipeline. Three evaluator categories: **NLP** (deterministic BLEU/GLEU/F1, no API key), **Quality** (LLM-as-judge Relevance/Coherence/Fluency, etc.), **Safety** (Hate/Violence/SelfHarm/Sexual via Azure AI Foundry). Auto-installs the `aieval` dotnet tool, detects the app's `IChatClient` registration and generates a factory so `EVAL_USE_REAL_AGENT=1` works without manual wiring, and emits an HTML report at `.copilot/perf-reports/evals//report.html`. Optional GitHub Actions workflow runs the evals on every PR. WHEN: user asks "set up evals", "add evaluation harness", "measure my agent perf", "validate quality after a model change", "compare gpt-4o vs gpt-4o-mini", "add safety evaluators", "generate eval report". NOT-WHEN: one-shot audit (use scan-agentic-app-perf), install rules (configure-agentic-perf-rules). --- # setup-maf-evals @@ -214,8 +214,8 @@ zero-config sanity check and Safety/Compare as additions. 6. **CLI invocations.** `dotnet test`, `dotnet tool run aieval report`, and the IChatClient detection result so the user knows what was auto-wired. -7. **Follow-up recommendation.** "Re-run after applying a - `select-agent-models` recommendation to confirm no quality +7. **Follow-up recommendation.** "Re-run after swapping a model per + rule #3 of `configure-agentic-perf-rules` to confirm no quality regression." 8. **Cache payoff.** "First `dotnet test` populates `_store/cache/` and takes the full ~60s. Every subsequent run against unchanged inputs diff --git a/plugins/dotnet-ai/skills/setup-maf-evals/references/aspire-dashboard-panel.md b/plugins/dotnet-ai/skills/setup-maf-evals/references/aspire-dashboard-panel.md index e899d633de..943e43a871 100644 --- a/plugins/dotnet-ai/skills/setup-maf-evals/references/aspire-dashboard-panel.md +++ b/plugins/dotnet-ai/skills/setup-maf-evals/references/aspire-dashboard-panel.md @@ -56,9 +56,8 @@ setInterval(tick, 2000); tick(); - The panel reads only the latest `telemetry.json`; it does not retain history across runs. - If the AppHost project does not already enable static files, the - skill adds `app.UseStaticFiles()` (in apply mode only, with the - same diff-preview-and-confirm flow as `select-agent-models` apply - mode). + skill adds `app.UseStaticFiles()` (in apply mode only, with a + standard diff-preview-and-confirm flow). ## Future v2 From 76cc8a5fe42ac415a5989cf28a3fe31832eeacb1 Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Wed, 24 Jun 2026 12:54:13 -0700 Subject: [PATCH 15/18] feat(perf-skills): retire agentic-perf-reviewer agent Same simplification rationale as the select-agent-models retirement. After moving role-aware model selection into rule #3 of configure-agentic-perf-rules, the agent's orchestration logic collapsed to: - run scan-agentic-app-perf (Pass 2) - route MA* findings to configure-agentic-perf-rules - route O* findings to setup-maf-evals That's logic Copilot already does organically from the skills' descriptions and the scan's existing 'ref:' fields + step-6 routing offer. The agent was adding ~70% ceremony for ~30% real value. Net change: - DELETED agents/agentic-perf-reviewer.agent.md (~180 lines) - DELETED agents/ directory (now empty) - Removed 'agents' key from plugin.json and .codex-plugin/plugin.json - The one unique guard worth keeping ('don't recommend a model downgrade without an evals quality follow-up') migrated into scan-agentic-app-perf SKILL.md common pitfalls. - The other 'don't pre-confirm apply-mode writes on user's behalf' guard already lives in scan-agentic-app-perf step 6.4. Users now invoke the skills directly: - 'scan my agentic app' -> scan-agentic-app-perf (which offers its own A/B/C follow-up menu) - 'install perf rules' -> configure-agentic-perf-rules - 'wire up evals' -> setup-maf-evals Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- plugins/dotnet-ai/.codex-plugin/plugin.json | 3 +- .../agents/agentic-perf-reviewer.agent.md | 183 ------------------ plugins/dotnet-ai/plugin.json | 3 +- .../skills/scan-agentic-app-perf/SKILL.md | 6 + 4 files changed, 8 insertions(+), 187 deletions(-) delete mode 100644 plugins/dotnet-ai/agents/agentic-perf-reviewer.agent.md diff --git a/plugins/dotnet-ai/.codex-plugin/plugin.json b/plugins/dotnet-ai/.codex-plugin/plugin.json index 4981aba703..20fbc8bec8 100644 --- a/plugins/dotnet-ai/.codex-plugin/plugin.json +++ b/plugins/dotnet-ai/.codex-plugin/plugin.json @@ -2,6 +2,5 @@ "name": "dotnet-ai", "version": "0.1.0", "description": "AI and ML skills for .NET: technology selection, LLM integration, agentic workflows, RAG pipelines, MCP, and classic ML with ML.NET.", - "skills": ["./skills/"], - "agents": ["./agents/agentic-perf-reviewer.agent.md"] + "skills": ["./skills/"] } diff --git a/plugins/dotnet-ai/agents/agentic-perf-reviewer.agent.md b/plugins/dotnet-ai/agents/agentic-perf-reviewer.agent.md deleted file mode 100644 index b5633494ee..0000000000 --- a/plugins/dotnet-ai/agents/agentic-perf-reviewer.agent.md +++ /dev/null @@ -1,183 +0,0 @@ ---- -description: "Reviews .NET agentic applications (Microsoft Agent Framework + Aspire + Foundry) for performance, cost, and reliability issues across topology, tools, message history, prompts, parallelism, OTel coverage, and per-agent model selection. Orchestrates scan-agentic-app-perf, setup-maf-evals, and configure-agentic-perf-rules to produce a single end-to-end review with actionable recommendations. Use when reviewing an MAF agentic app for perf or cost, when an agent app feels slow, or after non-trivial topology changes. Do NOT use for non-agentic .NET performance reviews (hot-path optimization, allocations, LINQ, async, serialization, general code perf) — use optimizing-dotnet-performance instead." -name: agentic-perf-reviewer -tools: ['read', 'search', 'task', 'skill', 'ask_user'] -license: MIT ---- - -# agentic-perf-reviewer - -You are an architect for .NET agentic applications. Help developers find -and fix the perf, cost, and reliability issues that Copilot routinely -overlooks: agent sprawl, single-model defaulting, full-history sharing, -prompt bloat, missing parallelism, and missing telemetry. - -## Three-Pass Review - -Every review uses three passes. All are mandatory unless the user has -explicitly asked for "quick triage only", in which case stop after -Pass 1 and recommend running the full audit. - -### Pass 1: Direct Read (No Skills) - -Analyze the project using your own knowledge. Do not load skills. - -1. Detect the agentic app: - - Look for `*.AppHost.csproj` files first. - - If none, look for project references to `Microsoft.Agents.AI`, - `Microsoft.Extensions.AI`, `ChatClientAgent`, `IChatClient` builders, - or Foundry agent config. - - If the user named a specific project path, use that even if no - AppHost is present. - - If neither AppHost nor agent signals are present and the user did - not name a target, ask one clarifying question, then stop if the - user cannot identify the agentic app. -2. Inventory: AppHost, agent service projects, per-agent models. -3. Identify the agent topology (count, handoff edges, cycles). -4. Identify the obvious performance smells (one-model defaulting, - full-history sharing, oversized system prompts, sequential awaits). -5. Provide a one-paragraph initial impression. Use **qualitative** - language only — do not produce numeric latency / cost / quality - estimates without telemetry, benchmark, or eval evidence. - -Label this section **"Pass 1: Initial Review"**. - -### Pass 2: Skill-Based Deep Audit - -**Always execute after Pass 1** unless the user asked for quick -triage. Do not ask whether to proceed. - -1. Load **scan-agentic-app-perf** and run it. Capture the report - path at `.copilot/perf-reports/scan-.md`. -2. Read the report file. For each finding, look at the `check_id`. The - prefix encodes the category — `T*` topology, `TI*` tool inventory, - `MH*` message history, `PW*` prompt weight, `P*` parallelism, `O*` - OTel coverage, `MA*` model assignment. Routing rules: - - Any `MA*` finding (model assignment) → suggest installing/updating - `configure-agentic-perf-rules` so rule #3 (role-aware model - selection) steers future agent code. If the managed block already - exists, point the user at rule #3 in the rendered - `.github/copilot-instructions.md`. - - Any `O*` finding (OTel coverage) → suggest loading - `setup-maf-evals` so telemetry/cost are surfaced going forward. - - If the project has no `.github/copilot-instructions.md` managed - block from `configure-agentic-perf-rules`, suggest installing it - so future sessions volunteer perf concerns by default. -3. Cite findings by `check_id` and `file:line` from the report. Do - not summarize from memory. - -Label this section **"Pass 2: Deep Audit"**. - -### Pass 3: Synthesis - -After Pass 2, produce a single prioritized action list: - -1. The 3 highest-impact changes the user should make first. -2. For each, the skill that performs it (or "manual fix"). -3. The expected effect — qualitative only (e.g. "lower per-turn - token cost", "shorter critical-path latency"). Use numeric - estimates only if `setup-maf-evals` has already produced a report - you can cite. -4. The risk and how to validate (almost always: run setup-maf-evals). -5. **Offer the follow-ups.** Once the action list is on screen, ask - the user **once** whether they want to invoke any of the routed - skills now. Render only the lettered options whose target skill is - actually referenced in the synthesis, e.g.: - > Want me to run any of these now? - > - **A.** `configure-agentic-perf-rules` to install/update the - > always-on rules (rule #3 covers model selection). - > - **B.** `setup-maf-evals` to capture token/quality numbers. - > - **C.** No — leave the report and stop. - If the user picks a letter, hand off to that skill with the audit - report path as context. The invoked skill still owns its own - diff-and-confirm flow — do not pre-confirm on the user's behalf - (see Boundaries). - -## Boundaries - -- **Do not edit source.** This agent has no `edit` tool. If a fix - requires file modifications, route to a skill that owns the - diff-and-confirm flow. -- Do not pick specific model ids without consulting rule #3 of the - managed block in `.github/copilot-instructions.md` (installed by - `configure-agentic-perf-rules`). -- Do not recommend a model downgrade without recommending a - `setup-maf-evals` quality follow-up. -- Cite findings by `check_id` and `file:line`; do not summarize the - audit report from memory. -- **Apply-mode chaining:** if the user says something like "apply the - fixes" in the same turn as invoking this agent, treat that as - *intent* but not as *confirmation*. The invoked skill (e.g. - `configure-agentic-perf-rules` apply mode) must still present its - own diff and obtain its own confirmation before any write. Do not - pre-confirm on the user's behalf. -- Do not apply this agent to non-agentic .NET apps. If detection in - Pass 1 fails, say so and stop. - -## Output Format - -Keep reports concise and actionable. - -1. **Pass 1: Initial Review** — paragraph + 3-5 bullets. -2. **Pass 2: Deep Audit** — top critical / warn findings cited by - `check_id` and `file:line` with the report path. -3. **Pass 3: Synthesis** — numbered action list with skill routes. -4. **Next steps** — exact commands or skill names to run. - -## Skills used - -- `scan-agentic-app-perf` — read-only audit, the workhorse of Pass 2. -- `setup-maf-evals` — telemetry / quality / compare harness. -- `configure-agentic-perf-rules` — install always-on rules (rule #3 covers role-aware model selection). - -## Common pitfalls - -Real-world failure modes for this agent — observed across the four -target apps used during dogfooding. - -- **Routing to this agent when the project is non-agentic.** The - description string explicitly carves out plain .NET perf reviews - (allocations, async, LINQ, serialization, hot-path optimization) → - `optimizing-dotnet-performance`. If Pass 1 detection finds no - AppHost AND no `Microsoft.Agents.AI` reference AND no - `IChatClient` builders, abort cleanly. Do not "audit" a plain - ASP.NET API or a console app — surface the wrong-agent message - and recommend the .NET perf agent instead. -- **Skipping Pass 2 because Pass 1 looks "fine".** Even when Pass 1 - finds nothing alarming, Pass 2 (running `scan-agentic-app-perf`) - is mandatory. The scan has 24 checks the human eye misses (e.g. - T3 cycles in `WorkflowBuilder`, MH1 full-history sharing buried - in `WithChatOptions`). Quick-triage exit is only when the user - explicitly says "quick triage only". -- **Citing findings from memory.** Always re-read - `.copilot/perf-reports/scan-.md` between Pass 2 and Pass 3. - Token-stream context drift will mangle line numbers and code - snippets if you cite from working memory. Open the file path, - re-read each finding's `file:line`, then write the synthesis. -- **Pre-confirming a fix on the user's behalf.** When the user says - "audit and apply the fixes" in one turn, treat it as INTENT but - not as CONFIRMATION. The invoked skill (e.g. - `configure-agentic-perf-rules` apply mode) must present its OWN - diff and obtain its OWN confirmation before any write. Pre- - confirming with "since you said apply, I'll go ahead" is a real - source of regressions. -- **Recommending a model downgrade without an eval gate.** Any Pass - 3 action that says "downgrade Agent X from gpt-4o to gpt-4o-mini" - must be paired with "validate via `setup-maf-evals` quality mode - before shipping". Apparent free wins on cost frequently regress - quality on edge cases. -- **Producing numeric estimates without evidence.** Pass 1's - qualitative-only rule applies to the entire agent. Never write - "this will save 40% latency" without a `setup-maf-evals` compare - report you can cite. Replace with "should lower per-turn token - cost" or "may reduce critical-path latency". -- **Single-agent apps.** Model-selection findings (MA*) still apply - to single-agent apps — the `configure-agentic-perf-rules` rule #3 - table covers single agents too. There's no separate "abort if <2 - agents" gate to skip in Pass 3. -- **Forgetting `configure-agentic-perf-rules` when no managed - block exists.** Pass 2 step 3 mandates this check. If the project - has no `.github/copilot-instructions.md` managed block, list it - as a Pass 3 follow-up regardless of other findings — installing - rules is the only durable way to prevent future regressions - between reviews. diff --git a/plugins/dotnet-ai/plugin.json b/plugins/dotnet-ai/plugin.json index 4981aba703..20fbc8bec8 100644 --- a/plugins/dotnet-ai/plugin.json +++ b/plugins/dotnet-ai/plugin.json @@ -2,6 +2,5 @@ "name": "dotnet-ai", "version": "0.1.0", "description": "AI and ML skills for .NET: technology selection, LLM integration, agentic workflows, RAG pipelines, MCP, and classic ML with ML.NET.", - "skills": ["./skills/"], - "agents": ["./agents/agentic-perf-reviewer.agent.md"] + "skills": ["./skills/"] } diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md index fc5a4fd44c..c1e00e116d 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md @@ -180,6 +180,12 @@ After running: `configure-agentic-perf-rules` instead. - **Running on non-agentic apps.** If no agent registrations are found, abort cleanly. Do not invent an audit for a plain web API. +- **Recommending a model downgrade without an eval gate.** Any MA* + finding that says "downgrade Agent X from gpt-4o to gpt-4o-mini" + must be paired in the `next:` field with "validate via + `setup-maf-evals` quality mode before shipping". Apparent free wins + on cost frequently regress quality on edge cases — the eval gate + protects against that. ## References From a38f3f83b671f80edb81154845e1b54e45adfeba Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Wed, 24 Jun 2026 13:21:05 -0700 Subject: [PATCH 16/18] refactor(scan-agentic-app-perf): replace cryptic codes with category slugs; prune arbitrary thresholds MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two related cleanups based on dogfood UX feedback. 1. Cryptic codes -> human-readable slugs. Findings used to be tagged with MA1, TI4, PW3, etc. — codes that were stable + greppable but required readers to flip to a glossary file every time. The prefix-routing case (e.g. 'MA* -> select- agent-models') that justified the convention is gone now that select-agent-models and the orchestrator agent are retired. Schema: check_id: T1 | T2 | TI1 | ... | MA4 becomes: check: . where category is one of topology / tools / history / prompt / parallel / otel / model and the descriptor names the actual issue. Examples: MA1 -> model.same-default MH1 -> history.full-share TI4 -> tools.description-too-long O1 -> otel.missing-sdk No glossary lookup needed — the slug self-describes. check-id-glossary.md -> check-glossary.md with a new slug -> description table. 2. Pruned arbitrary-threshold checks. Five checks were taste calls dressed up as audit findings; they fired on legitimate designs as often as on real bloat: - T1 'agent count > 3' (DELETED) - T2 'LLM handoff edges per turn > 2' (DELETED) - TI1 'tools per agent > 8' (DELETED) - PW2 'few-shot examples > 3' (DELETED) - P1 'sequential awaits over independent inputs' (DELETED; general .NET concurrency anti-pattern, belongs to optimizing-dotnet-performance) Each deprecated check leaves a 'What used to live here' breadcrumb in its category file so users grep-finding the old name see why it was removed and where the underlying concern is now addressed. Net: 24 checks -> 19 checks, no schema gymnastics, no glossary round-trips for the reader. Files touched: - references/{topology,tool-inventory,message-history,prompt-weight,parallelism,otel-coverage,model-assignment}-checks.md rewritten - references/check-id-glossary.md -> check-glossary.md (rename + rewrite) - references/report-template.md schema update - references/common-pitfalls.md per-check sharpening updated for new slugs - SKILL.md finding schema + sort key + references list updated Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../skills/scan-agentic-app-perf/SKILL.md | 45 +++++++------ .../references/check-glossary.md | 61 +++++++++++++++++ .../references/check-id-glossary.md | 65 ------------------- .../references/common-pitfalls.md | 42 ++++++------ .../references/message-history-checks.md | 30 ++++----- .../references/model-assignment-checks.md | 17 +++-- .../references/otel-coverage-checks.md | 12 ++-- .../references/parallelism-checks.md | 47 +++++--------- .../references/prompt-weight-checks.md | 41 ++++++------ .../references/report-template.md | 6 +- .../references/tool-inventory-checks.md | 55 ++++++++-------- .../references/topology-checks.md | 53 +++++---------- 12 files changed, 219 insertions(+), 255 deletions(-) create mode 100644 plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-glossary.md delete mode 100644 plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-id-glossary.md diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md index c1e00e116d..4e548a0807 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md @@ -50,7 +50,8 @@ Every finding is a dict with these fields: ```yaml severity: critical | warn | info -check_id: T1 | T2 | T3 | T4 | TI1 | TI2 | TI3 | TI4 | MH1 | MH2 | MH3 | PW1 | PW2 | PW3 | P1 | P2 | P3 | O1 | O2 | O3 | O4 | MA1 | MA2 | MA3 | MA4 +check: one of the slugs listed in references/check-glossary.md + (e.g. model.same-default, history.full-share, otel.missing-sdk) title: short imperative phrase file: path relative to repo root line: 1-based line number, or null @@ -60,19 +61,22 @@ next: concrete action the developer can take next ref: optional cross-skill route (e.g. "skill:configure-agentic-perf-rules") ``` -The `check_id` prefix encodes the category — there is no separate -`category` field. Prefix glossary (also rendered at the top of the +The `check` field is a dotted slug: `.`. The +prefix before the dot encodes the category — there is no separate +`category` field. Category card (also rendered at the top of the report's `## Findings` section): -| Prefix | Category | Reference | -|--------|-------------------------|----------------------------------------| -| `T` | topology | `references/topology-checks.md` | -| `TI` | tool inventory | `references/tool-inventory-checks.md` | -| `MH` | message history | `references/message-history-checks.md` | -| `PW` | prompt weight | `references/prompt-weight-checks.md` | -| `P` | parallelism | `references/parallelism-checks.md` | -| `O` | OTel coverage | `references/otel-coverage-checks.md` | -| `MA` | model assignment | `references/model-assignment-checks.md`| +| Category | Coverage | Reference | +|------------|-------------------------|----------------------------------------| +| `topology` | agent graph shape | `references/topology-checks.md` | +| `tools` | per-agent tool list | `references/tool-inventory-checks.md` | +| `history` | chat history strategy | `references/message-history-checks.md` | +| `prompt` | system prompt size/reuse| `references/prompt-weight-checks.md` | +| `parallel` | concurrent invocations | `references/parallelism-checks.md` | +| `otel` | instrumentation | `references/otel-coverage-checks.md` | +| `model` | per-agent model id | `references/model-assignment-checks.md`| + +See `references/check-glossary.md` for the full slug→description table. **Evidence gate** — before adding a finding to the report, re-open the cited file and confirm the `evidence` snippet is present at or near @@ -89,16 +93,17 @@ Severity rules: ### 4. Aggregate and write the report -Sort findings by severity (critical → warn → info), then by `check_id` -(stable lexical order so `T1` < `T2` < `TI1` < `MH1` < ...). Write to: +Sort findings by severity (critical → warn → info), then by `check` +slug (stable lexical order so `history.*` < `model.*` < `otel.*` < +`parallel.*` < `prompt.*` < `tools.*` < `topology.*`). Write to: - `.copilot/perf-reports/scan-.md` (timestamped, kept) - `.copilot/perf-reports/latest-scan.md` (overwritten each run) -- `.copilot/perf-reports/check-id-glossary.md` (overwritten each run) - — a one-line-per-code reference card so first-time readers of a - report can decode `T1`/`TI3`/`MA4` without opening the skill repo. +- `.copilot/perf-reports/check-glossary.md` (overwritten each run) + — a one-line-per-check reference card so first-time readers of a + report can see the full check catalog without opening the skill repo. Source: copy the "Reference card" section verbatim from - `references/check-id-glossary.md`. + `references/check-glossary.md`. See `references/report-template.md` for the exact layout. @@ -114,7 +119,7 @@ it from this skill. Print: 1. Total counts (critical / warn / info). -2. The first up to 3 critical findings with title + check_id + file:line + next action. +2. The first up to 3 critical findings with title + `check` slug + file:line + next action. 3. The full report path. 4. If any findings have a `ref:` field, list the suggested follow-up skills. @@ -196,5 +201,5 @@ After running: - `references/parallelism-checks.md` — sequential calls that could fan out. - `references/otel-coverage-checks.md` — Aspire dashboard, token/cost telemetry. - `references/model-assignment-checks.md` — single-model defaulting, role mismatch. -- `references/check-id-glossary.md` — the reference card written alongside each report. +- `references/check-glossary.md` — the reference card written alongside each report. - `references/report-template.md` — exact Markdown layout for the report. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-glossary.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-glossary.md new file mode 100644 index 0000000000..db17754b2a --- /dev/null +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-glossary.md @@ -0,0 +1,61 @@ +# Check glossary + +This file is the source of truth for the human-readable check card +that `scan-agentic-app-perf` writes alongside every report. + +When the skill runs, it copies the "Reference card" section verbatim +to `.copilot/perf-reports/check-glossary.md` so a first-time report +reader can see the full check catalog without opening the skill repo. + +## Reference card + +> **Categories:** `topology` (agent graph shape) · `tools` (per-agent +> tool list) · `history` (chat history strategy) · `prompt` (system +> prompt size + reuse) · `parallel` (concurrent agent invocations) · +> `otel` (instrumentation coverage) · `model` (per-agent model +> selection). +> +> **Severity:** `critical` = likely to break a flow or blow the +> budget · `warn` = measurable cost/perf regression · `info` = +> observation only. + +| Check | Sev | What it catches | +|------------------------------------|----------|-----------------| +| `topology.cycle` | critical | Directed cycle in the agent handoff graph | +| `topology.deep-single-leaf` | warn | 3+ hops to reach a single terminal agent | +| `tools.duplicate` | warn | Same tool description on >1 agent | +| `tools.dead` | info | Tool registered but never referenced | +| `tools.description-too-long` | warn | `[Description]` attribute > 200 chars | +| `history.full-share` | critical | Entire chat history passed to a downstream agent | +| `history.unbounded` | warn | No `MaxMessages` / reducer / summarizer wired | +| `history.through-deterministic` | warn | Full history given to a deterministic agent | +| `prompt.oversized` | warn | System prompt > 2K tokens (critical at > 4K) | +| `prompt.duplicate-preamble` | warn | Same > 100-token block shared across agents | +| `parallel.independent-handoffs` | warn | Sequential `await`s on agents that don't share data | +| `parallel.hidden-tool-fanout` | info | Tool internally serializes 3+ API calls | +| `otel.missing-sdk` | critical | No `AddOpenTelemetry` / Aspire dashboard wiring | +| `otel.no-aspire-dashboard` | warn | Aspire dashboard not declared | +| `otel.no-token-cost` | warn | No `gen_ai.usage.*` tags surfaced | +| `otel.no-per-agent-source` | info | All agents share one `ActivitySource` | +| `model.same-default` | warn | All agents use the same model id | +| `model.reasoning-on-deterministic` | warn | Frontier model on a formatter / validator | +| `model.cheap-on-planner` | warn | Small model on the planner, larger on workers | +| `model.hardcoded` | info | Model id literal in agent service `.cs` | + +## Cross-skill routes embedded in `ref:` fields + +| `ref:` value | Skill to run next | +|--------------------------------------|------------------------------------------------------------| +| `skill:configure-agentic-perf-rules` | Install/update the always-on perf rules (rule #3 covers role-aware model selection) | +| `skill:setup-maf-evals` | Wire eval reports + telemetry | + +## Notes for skill implementation + +- When generating a report, write a copy of the "Reference card" section + (the table and the legend immediately above it) to + `.copilot/perf-reports/check-glossary.md`. Overwrite each run; the + content is static within a skill version and does not depend on findings. +- The glossary file is per-repo (one file regardless of how many runs); + the timestamped `scan-.md` reports link to it relatively. +- Keep this table in lockstep with the per-category reference files. + Adding a new check in any `*-checks.md` file REQUIRES a row here. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-id-glossary.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-id-glossary.md deleted file mode 100644 index fcab4b6c17..0000000000 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-id-glossary.md +++ /dev/null @@ -1,65 +0,0 @@ -# Check-ID glossary - -This file is the source of truth for the check-ID reference card that -`scan-agentic-app-perf` writes alongside every report (parallel to the -metrics-glossary pattern in `setup-maf-evals`). - -When the skill runs, it copies this file's "Reference card" section -verbatim to `.copilot/perf-reports/check-id-glossary.md` so a first-time -report reader can decode the codes without opening the skill repo. - -## Reference card - -> **Prefixes:** `T*` topology · `TI*` tool inventory · `MH*` message -> history · `PW*` prompt weight · `P*` parallelism · `O*` OTel · `MA*` -> model assignment. -> -> **Severity:** `critical` = likely to break a flow or blow the budget · -> `warn` = measurable cost/perf regression · `info` = observation only. - -| ID | Sev | Title | Full check | -|-----|----------|-------------------------------------------------------------|------------| -| T1 | warn | Agent count > 3 | `topology-checks.md#t1` | -| T2 | warn | LLM-routed handoff edges per turn > 2 | `topology-checks.md#t2` | -| T3 | critical | Cycles in the agent graph | `topology-checks.md#t3` | -| T4 | warn | Single-leaf graph with > 2 hops | `topology-checks.md#t4` | -| TI1 | warn | Tools per agent > 8 | `tool-inventory-checks.md#ti1` | -| TI2 | warn | Duplicate tool functionality across agents | `tool-inventory-checks.md#ti2` | -| TI3 | info | Dead tools (declared, never invoked) | `tool-inventory-checks.md#ti3` | -| TI4 | warn | Tool description > 200 chars | `tool-inventory-checks.md#ti4` | -| MH1 | critical | Full chat history shared with every agent | `message-history-checks.md#mh1` | -| MH2 | warn | No history cap (unbounded growth) | `message-history-checks.md#mh2` | -| MH3 | warn | History passed through deterministic agents | `message-history-checks.md#mh3` | -| PW1 | warn | System prompt > 2K tokens | `prompt-weight-checks.md#pw1` | -| PW2 | warn | Few-shot examples in prompt > 3 | `prompt-weight-checks.md#pw2` | -| PW3 | warn | Identical preamble duplicated across agents | `prompt-weight-checks.md#pw3` | -| P1 | warn | Sequential awaits over independent inputs | `parallelism-checks.md#p1` | -| P2 | warn | Sequential agent handoffs that don't share context | `parallelism-checks.md#p2` | -| P3 | info | Tool fan-out behind a single tool wrapper | `parallelism-checks.md#p3` | -| O1 | critical | No `AddOpenTelemetry` call | `otel-coverage-checks.md#o1` | -| O2 | warn | No Aspire dashboard reference | `otel-coverage-checks.md#o2` | -| O3 | warn | Token / cost surfacing missing | `otel-coverage-checks.md#o3` | -| O4 | info | Per-agent activity source missing | `otel-coverage-checks.md#o4` | -| MA1 | warn | All agents on the same model | `model-assignment-checks.md#ma1` | -| MA2 | warn | Reasoning-strong model on a deterministic agent | `model-assignment-checks.md#ma2` | -| MA3 | warn | Cheap model on a planner / decomposer | `model-assignment-checks.md#ma3` | -| MA4 | info | Hard-coded model id outside config | `model-assignment-checks.md#ma4` | - -## Cross-skill routes embedded in `ref:` fields - -| `ref:` value | Skill to run next | -|-----------------------------------|----------------------------------| -| `skill:configure-agentic-perf-rules` | Install/update the always-on perf rules (role-aware model selection lives in rule #3) | -| `skill:setup-maf-evals` | Wire eval reports + telemetry | -| `skill:configure-agentic-perf-rules` | Install always-on rules block | - -## Notes for skill implementation - -- When generating a report, write a copy of the "Reference card" section - (the table and the prefix/severity legend immediately above it) to - `.copilot/perf-reports/check-id-glossary.md`. Overwrite each run; the - content is static within a skill version and does not depend on findings. -- The glossary file is per-repo (one file regardless of how many runs); - the timestamped `scan-.md` reports link to it relatively. -- Keep this table in lockstep with the per-category reference files. Adding - a new check ID in `topology-checks.md` REQUIRES a row here. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/common-pitfalls.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/common-pitfalls.md index 19936f8cd0..b4bfc0ef50 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/common-pitfalls.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/common-pitfalls.md @@ -7,7 +7,7 @@ dogfooding (interview-coach v1/v2, ELI5Agent, behavioral-interview-coach). - **Editing source code.** This skill is read-only. The ONLY write paths it owns are `.copilot/perf-reports/scan-.md`, - `latest-scan.md`, and `check-id-glossary.md`. Never touch + `latest-scan.md`, and `check-glossary.md`. Never touch `.gitignore`, source files, config files, AppHost, or `*.csproj`. If a check tempts you to fix the issue inline, stop and add it as a finding instead — the user opts into the fix through @@ -33,44 +33,40 @@ dogfooding (interview-coach v1/v2, ELI5Agent, behavioral-interview-coach). Re-read the cited region right before writing each finding, not before the entire batch. - **Absence-of-X findings without searched-pattern evidence.** When - a finding fires because something is *missing* (e.g. O3 "no token + a finding fires because something is *missing* (e.g. `otel.no-token-cost` "no token surfacing"), the `evidence` field MUST list the exact files and patterns that were searched — not a snippet. This is what lets the user reproduce the negative result. ## False positives we keep seeing -- **MA4 (hard-coded model id) firing on AppHost code.** MA4 is about +- **`model.hardcoded` firing on AppHost code.** This check is about model ids living in *agent service* files (`*.Agent/Program.cs` etc.) where swapping requires a code change. Model ids declared in the AppHost via `foundry.AddDeployment("chat", FoundryModel.OpenAI.Gpt4oMini)` are the **canonical Aspire-native place** — that's not a defect. - When you encounter this pattern, either suppress MA4 entirely or - downgrade to `info` with a "no action required" `next:`. -- **MA1 (single-model) on single-agent apps.** MA1 explicitly assumes - ≥2 agents (different roles, different needs). Do not fire on + When you encounter this pattern, either suppress `model.hardcoded` + entirely or downgrade to `info` with a "no action required" `next:`. +- **`model.same-default` on single-agent apps.** The check explicitly + assumes ≥2 agents (different roles, different needs). Do not fire on 1-agent apps; the check is trivially "satisfied" with one model. -- **O3 (no token surfacing) when `Microsoft.Extensions.AI` activity +- **`otel.no-token-cost` when `Microsoft.Extensions.AI` activity source is registered.** MEAI emits `gen_ai.usage.*` activity tags automatically; if `AddSource("Microsoft.Extensions.AI")` is wired in - OTel and an OTLP exporter is configured, O3 should NOT fire. The - check is for codebases that strip the source or wrap MEAI behind - custom infrastructure that loses the tags. + OTel and an OTLP exporter is configured, this check should NOT + fire. The check is for codebases that strip the source or wrap MEAI + behind custom infrastructure that loses the tags. ## Per-check sharpening -- **T2 (handoff edges per turn).** Count statically-resolvable edges - only. Don't try to simulate dynamic LLM-routed handoff at scan - time; the count is "edges declared in `CreateHandoffBuilderWith`", - not "edges executed". -- **PW1 (system prompt > 2K tokens).** Use a rough token estimator - (chars/4) or `cl100k_base` if available. Never claim an exact - token count without naming the encoder you used. -- **TI2 (duplicate tool functionality).** Compare tool *descriptions* - (the `[Description("...")]` attribute), not method names. Two tools - with the same method name but different descriptions are usually - fine; two tools with different names and a copy-pasted description - are usually a refactor candidate. +- **`prompt.oversized`.** Use a rough token estimator (chars/4) or + `cl100k_base` if available. Never claim an exact token count without + naming the encoder you used. +- **`tools.duplicate`.** Compare tool *descriptions* (the + `[Description("...")]` attribute), not method names. Two tools with + the same method name but different descriptions are usually fine; + two tools with different names and a copy-pasted description are + usually a refactor candidate. ## Routing offer (step 6) diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/message-history-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/message-history-checks.md index 82cb0ff1a5..db98a2b686 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/message-history-checks.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/message-history-checks.md @@ -4,30 +4,30 @@ Detect strategies that pass too much history to too many agents. ## Checks -### MH1. Full chat history shared with every agent (critical) +### `history.full-share` (critical) -**Detect:** code that passes the entire `IList` (or `History`) -from the entry agent to a downstream agent without filtering, slicing, or -summarizing. +**Detect:** code that passes the entire `IList` (or +`History`) from the entry agent to a downstream agent without +filtering, slicing, or summarizing. -**Why:** every additional agent that sees the full history pays the input -token cost. With 4 agents and a 6K-token history, you spend 24K input -tokens per turn doing nothing. +**Why:** every additional agent that sees the full history pays the +input token cost. With 4 agents and a 6K-token history, you spend 24K +input tokens per turn doing nothing. -**Next:** "Pass only the last user message and a one-paragraph summary to -``. Use `IChatHistoryReducer` or a manual slice." +**Next:** "Pass only the last user message and a one-paragraph summary +to ``. Use `IChatHistoryReducer` or a manual slice." -### MH2. No history cap (warn) +### `history.unbounded` (warn) -**Detect:** no usage of `MaxMessages`, `IChatHistoryReducer`, summarization -tool, or sliding-window code anywhere in agent setup. +**Detect:** no usage of `MaxMessages`, `IChatHistoryReducer`, +summarization tool, or sliding-window code anywhere in agent setup. **Why:** unbounded history = monotonically growing per-turn cost. **Next:** "Wire a `ChatHistoryReducer` with `MaxMessages = 20` or summarize-and-replace at the agent level." -### MH3. History passed through deterministic agents (warn) +### `history.through-deterministic` (warn) **Detect:** an agent whose role is purely deterministic (formatter, validator, tool router) is given full chat history. @@ -35,5 +35,5 @@ validator, tool router) is given full chat history. **Why:** deterministic steps do not need conversational context. Their prompt cost should be near-constant. -**Next:** "Pass only the immediate input artifact to ``; drop the -chat history." +**Next:** "Pass only the immediate input artifact to ``; drop +the chat history." diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/model-assignment-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/model-assignment-checks.md index ffa8ab9526..dbcdfe5577 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/model-assignment-checks.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/model-assignment-checks.md @@ -4,11 +4,12 @@ Detect single-model defaulting and role-model mismatch. ## Checks -### MA1. All agents on the same model (warn) +### `model.same-default` (warn) **Detect:** every agent is constructed with the same model id (e.g. `gpt-4o-mini`). Look at the AppHost connection string or the -`IChatClient` builder per agent service. +`IChatClient` builder per agent service. Requires ≥2 agents — skip on +single-agent apps. **Why:** different agent roles have different latency and quality needs. A single default usually overspends on cheap roles and @@ -22,7 +23,7 @@ expected and can be downgraded to info." **Ref:** `skill:configure-agentic-perf-rules` -### MA2. Reasoning-strong model on a deterministic agent (warn) +### `model.reasoning-on-deterministic` (warn) **Detect:** an agent whose prompt and tool list indicate a deterministic role (formatter, validator, classifier with ≤3 outputs) @@ -37,7 +38,7 @@ quality mode." **Ref:** `skill:configure-agentic-perf-rules` -### MA3. Cheap model on a planner / decomposer (warn) +### `model.cheap-on-planner` (warn) **Detect:** the agent that decides the plan or decomposes the task is on a small model while leaf workers are on a large one. @@ -51,13 +52,15 @@ workers." **Ref:** `skill:configure-agentic-perf-rules` -### MA4. Hard-coded model id outside config (info) +### `model.hardcoded` (info) **Detect:** model id literal (e.g. `"gpt-4o-mini"`) appears inside an agent service `.cs` file rather than `appsettings.json` or AppHost -parameters. +parameters. AppHost code that declares model ids via +`foundry.AddDeployment(...)` is the canonical Aspire pattern and does +NOT trigger this check. **Why:** swapping models for an A/B becomes a code change. **Next:** "Move model ids into `appsettings.json` and bind them via -`IOptions<...>`." +`IOptions<...>`, or declare them in the AppHost as Aspire deployments." diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/otel-coverage-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/otel-coverage-checks.md index e4278212b7..d25687e6e8 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/otel-coverage-checks.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/otel-coverage-checks.md @@ -4,20 +4,20 @@ Detect missing instrumentation that makes perf invisible at dev time. ## Checks -### O1. No `AddOpenTelemetry` call (critical) +### `otel.missing-sdk` (critical) **Detect:** the AppHost or service projects do not call `builder.Services.AddOpenTelemetry()` and do not import `OpenTelemetry.Exporter.OpenTelemetryProtocol` or `Aspire.Hosting.Dashboard`. -**Why:** without OTel, you cannot see per-call latency, token counts, or -spans. Every other check in this audit becomes a guess. +**Why:** without OTel, you cannot see per-call latency, token counts, +or spans. Every other check in this audit becomes a guess. **Next:** "Add `builder.AddServiceDefaults()` (Aspire) or wire OTel manually with HTTP + Activity sources for `Microsoft.Extensions.AI`." -### O2. No Aspire dashboard reference (warn) +### `otel.no-aspire-dashboard` (warn) **Detect:** the AppHost does not declare the dashboard, or the `appsettings.json` lacks a `Dashboard:OtlpEndpointUrl`. @@ -28,7 +28,7 @@ during local dev. **Next:** "Run with `dotnet run --project ` and ensure the dashboard URL is logged. If not, install `Aspire.Hosting.Dashboard`." -### O3. Token / cost surfacing missing (warn) +### `otel.no-token-cost` (warn) **Detect:** no log, no meter, no tag for `gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens` anywhere in the codebase. @@ -43,7 +43,7 @@ automatically. Confirm the OTel exporter forwards them, or run **Ref:** `skill:setup-maf-evals` -### O4. Per-agent activity source missing (info) +### `otel.no-per-agent-source` (info) **Detect:** all agents share a single activity source name; no way to filter the dashboard by agent. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/parallelism-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/parallelism-checks.md index a571d9993d..e27f78e3e9 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/parallelism-checks.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/parallelism-checks.md @@ -1,48 +1,37 @@ # Parallelism checks -Detect sequential awaits that could run concurrently. +Detect sequential agent invocations that could run concurrently. ## Checks -### P1. Sequential awaits over independent inputs (warn) - -**Detect:** a `foreach` / `for` loop that awaits an LLM call or tool call -each iteration where the iteration values are independent of each other. - -**Pattern:** - -```csharp -foreach (var item in items) -{ - results.Add(await agent.RunAsync(item)); // sequential -} -``` - -**Why:** N items × per-call latency. With N=5 and 2s/call, that's 10s -that could be 2s under `Task.WhenAll`. - -**Next:** "Replace the loop with `await Task.WhenAll(items.Select(i => -agent.RunAsync(i)))`. Watch for shared mutable state inside the agent." - -### P2. Sequential agent handoffs that don't share context (warn) +### `parallel.independent-handoffs` (warn) **Detect:** two consecutive `await downstreamA.RunAsync(...)` and `await downstreamB.RunAsync(...)` calls in the same method where B's input does not depend on A's output. **Why:** the second call could start as soon as the inputs are known. +Each sequential LLM hop adds full per-call latency. -**Next:** "Run `` and `` with `Task.WhenAll`. Rejoin in the parent -agent for the consolidation step." +**Next:** "Run `` and `` with `Task.WhenAll`. Rejoin in the +parent agent for the consolidation step." -### P3. Tool fan-out behind a single tool wrapper (info) +### `parallel.hidden-tool-fanout` (info) **Detect:** a tool method that internally loops and calls 3+ external APIs sequentially. -**Why:** tools hide their own latency from the agent. A single slow tool -that is internally serial is the hardest kind of latency to find from -the outside. +**Why:** tools hide their own latency from the agent. A single slow +tool that is internally serial is the hardest kind of latency to find +from the outside. **Next:** "Parallelize the inner calls in ``; document the -expected bound in the tool description so the agent can plan around it." +expected bound in the tool description so the agent can plan around +it." + +## What used to live here + +`parallel.sequential-awaits` (was `P1`, a generic `foreach (var x in +xs) await ...` pattern) was removed — that's a general .NET concurrency +anti-pattern, not specific to agents. For that class of finding run +`optimizing-dotnet-performance` instead. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/prompt-weight-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/prompt-weight-checks.md index e7a086ef4e..7e76736b0c 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/prompt-weight-checks.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/prompt-weight-checks.md @@ -4,40 +4,37 @@ Detect oversized system prompts and per-agent prompt cost. ## Checks -### PW1. System prompt > 2K tokens (warn) / > 4K tokens (critical) +### `prompt.oversized` (warn at >2K tokens / critical at >4K) -**Detect:** count tokens (or chars / 4 as approximation) in each agent's -`Instructions` or system prompt string. +**Detect:** count tokens (or chars / 4 as approximation) in each +agent's `Instructions` or system prompt string. -**Why:** every turn pays this cost. A 4K-token system prompt at $0.005/1K -input tokens × 1000 turns/day = $20/day per agent on prompt overhead alone. +**Why:** every turn pays this cost. A 4K-token system prompt at +$0.005/1K input tokens × 1000 turns/day = $20/day per agent on prompt +overhead alone. -**Next:** "Move static rules into a tool the agent can call when needed, -or split the prompt into a short policy section and a separate few-shot -example doc retrieved on demand." +**Next:** "Move static rules into a tool the agent can call when +needed, or split the prompt into a short policy section and a separate +few-shot example doc retrieved on demand." -### PW2. Few-shot examples in prompt > 3 (warn) - -**Detect:** count "Example:", "User:", "Assistant:" turn pairs in the -prompt string. - -**Why:** few-shot examples scale linearly with token cost. After 3 -examples the marginal accuracy gain is usually < 1%. - -**Next:** "Keep the 2 strongest examples; move the rest behind a -`getExample(category)` tool." - -### PW3. Identical preamble duplicated across agents (warn) +### `prompt.duplicate-preamble` (warn) **Detect:** two or more agents share the same > 100-token block at the start or end of their system prompts. **Why:** the same tokens are billed N times per turn (once per agent). -**Next:** "Lift the shared block into a single deterministic preprocessor -or attach it as a tool result rather than a system prompt repeat." +**Next:** "Lift the shared block into a single deterministic +preprocessor or attach it as a tool result rather than a system prompt +repeat." ## Token estimation If a real tokenizer is not available, approximate as `chars / 4`. Mark findings using approximation as `(estimated)` in the evidence. + +## What used to live here + +`prompt.too-many-fewshots` (was `PW2`, fired at >3 few-shot examples) +was a taste-based threshold. Few-shot count alone is not a reliable +signal; `prompt.oversized` already captures the cost dimension. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/report-template.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/report-template.md index ef16900f5a..2bb7ff6858 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/report-template.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/report-template.md @@ -26,9 +26,9 @@ Project: {{ relative_project_path }} ## Findings -> **Check ID prefixes:** `T*` topology · `TI*` tool inventory · `MH*` message history · `PW*` prompt weight · `P*` parallelism · `O*` OTel · `MA*` model assignment. +> **Categories:** `topology` · `tools` · `history` · `prompt` · `parallel` · `otel` · `model`. See `check-glossary.md` for the full slug→description table. -### [critical] [{{ check_id }}] {{ title }} +### [critical] [`{{ check }}`] {{ title }} - **File:** `{{ file }}:{{ line }}` - **Evidence:** ```csharp @@ -38,7 +38,7 @@ Project: {{ relative_project_path }} - **Next:** {{ action }} - **Cross-ref:** {{ skill: ... | omit if none }} -(... repeat per finding, ordered: critical → warn → info, then by `check_id` ...) +(... repeat per finding, ordered: critical → warn → info, then by `check` slug ...) ## Next steps diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/tool-inventory-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/tool-inventory-checks.md index dc281f6d2a..180c4abeee 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/tool-inventory-checks.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/tool-inventory-checks.md @@ -4,46 +4,43 @@ Detect bloat and redundancy in the per-agent tool list. ## Checks -### TI1. Tools per agent > 8 (warn) / > 15 (critical) - -**Detect:** count tools registered on each agent (`AIFunctionFactory.Create`, -`[Description]`-attributed methods passed to `tools:`, MCP tool imports). - -**Why:** tool descriptions are sent in the system prompt every turn. 15 -tools at ~80 tokens each is 1.2K tokens of overhead before the user message. - -**Next:** "Split `` into two agents by domain, or move rarely-used -tools behind a single 'lookup' tool that takes a category argument." - -### TI2. Duplicate tool functionality across agents (warn) +### `tools.duplicate` (warn) **Detect:** two or more tools across different agents with the same description or near-identical signatures. -**Why:** duplication forces the router LLM to disambiguate every turn and -inflates aggregate prompt size. +**Why:** duplication forces the router LLM to disambiguate every turn +and inflates aggregate prompt size. -**Next:** "Consolidate `` and `` into a single shared tool -exposed by both agents." +**Next:** "Consolidate `` and `` into a single shared +tool exposed by both agents." -### TI3. Dead tools (info) +### `tools.dead` (info) -**Detect:** tools registered but never invoked in any code path or call -trace. Static check: tool name does not appear in any agent's system -prompt or instructions. +**Detect:** tools registered but never invoked in any code path or +call trace. Static check: tool name does not appear in any agent's +system prompt or instructions. -**Why:** every registered tool costs prompt tokens whether it gets called -or not. +**Why:** every registered tool costs prompt tokens whether it gets +called or not. **Next:** "Remove `` from ``'s tool list." -### TI4. Tool description > 200 chars (warn) +### `tools.description-too-long` (warn) + +**Detect:** a tool's `[Description]` attribute or `description:` field +is longer than 200 characters. Measure after string concatenation — +multi-line `+` concatenations count as one description. + +**Why:** long descriptions multiply across agents that import the +tool. Most tools can be described in one sentence. -**Detect:** a tool's `[Description]` attribute or `description:` field is -longer than 200 characters. +**Next:** "Trim ``'s description from `` chars to ≤ 200; move +the detailed contract into XML docs on the parameters." -**Why:** long descriptions multiply across agents that import the tool. -Most tools can be described in one sentence. +## What used to live here -**Next:** "Trim ``'s description from `` chars to ≤ 200; move the -detailed contract into XML docs on the parameters." +`tools.too-many-per-agent` (was `TI1`) was a taste-based threshold +(>8 tools per agent) that fired on legitimate designs. The real +question — "is this tool earning its prompt-token weight" — is better +served by `tools.dead` and `tools.duplicate`. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/topology-checks.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/topology-checks.md index 6a16cac90e..7f07e4e5fb 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/topology-checks.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/topology-checks.md @@ -4,56 +4,37 @@ Detect structural issues in the agent graph that drive latency or runaway loops. ## Checks -### T1. Agent count > 3 (warn) / > 6 (critical) - -**Detect:** count distinct `ChatClientAgent` / `AddAgent(...)` registrations -in the AppHost and agent service projects. - -**Why:** more agents = more LLM hops per turn. Every additional agent that -can be routed to costs at least one extra round trip. - -**Next:** "Collapse `` into a single agent with two tool calls instead -of two agents." - -**Ref:** `skill:configure-agentic-perf-rules` if the project has no rules -file installed yet. - -### T2. LLM-routed handoff edges per turn > 2 (warn) / > 4 (critical) - -**Detect:** edges where the *destination* agent is selected by an LLM (not -deterministic code). Look for `Handoff` builders, `RoutingAgent`, or -`switch`/`if` blocks that select an agent based on a string returned by a -chat completion. - -**Why:** LLM-routed edges multiply tail latency. Two LLM hops to pick the -next agent before the work even starts is the most common cause of "why -is my agent so slow". - -**Next:** "Replace the LLM router between `` and `` with a deterministic -intent classifier or a tool call on the source agent." - -### T3. Cycles in the agent graph (critical) +### `topology.cycle` (critical) **Detect:** any directed cycle in the static handoff graph. **Why:** cycles risk infinite loops if the loop-break condition is LLM-judged. Even with a turn cap, a cycle burns budget on retries. -**Next:** "Break the cycle `` → `` → `` by making ``'s exit condition -deterministic." +**Next:** "Break the cycle `` → `` → `` by making ``'s exit +condition deterministic." -### T4. Single-leaf graph with > 2 hops (warn) +### `topology.deep-single-leaf` (warn) **Detect:** graph that always ends at one agent but routes through 3+ agents to reach it. -**Why:** the intermediate hops are usually classification or routing that -could be one tool call. +**Why:** the intermediate hops are usually classification or routing +that could be one tool call. -**Next:** "Move the routing logic into a tool on the entry agent and call -`` directly." +**Next:** "Move the routing logic into a tool on the entry agent and +call `` directly." ## Out of scope here - Tool counts → see `tool-inventory-checks.md`. - Per-agent model selection → see `model-assignment-checks.md`. + +## What used to live here + +`topology.agent-count` (was `T1`) and `topology.handoff-fanout` (was +`T2`) were removed in v0.2 — both were taste-based thresholds (>3 +agents, >2 handoff edges) that fired on legitimate designs as often +as on real bloat. If you want broad architectural feedback, run +`configure-agentic-perf-rules` so rule #1 (single-agent default) and +rule #2 (handoff justification) can guide the design at scaffold time. From b0488de7c33e443b9f7da190797b3f923a17ede9 Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Wed, 24 Jun 2026 13:42:12 -0700 Subject: [PATCH 17/18] refactor(scan-agentic-app-perf): collapse output to a single overwritten scan.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Dogfood feedback: three generated files per run (scan-.md + latest-scan.md + check-glossary.md) plus a new timestamped file every scan made even a clean repo's .copilot/perf-reports/ look like a hoarder's basement after a week. The slug rename made the glossary file redundant anyway — 'history.full-share' is self-describing in a way that 'MH1' never was. Changes: - Single output file: .copilot/perf-reports/scan.md, overwritten on every run. No timestamped copies, no latest-* mirrors, no per-run glossary file. - Category legend is inlined into the report (the blockquote at the top of '## Findings') so readers see what topology/history/model/ etc. mean without leaving the file. - references/check-glossary.md remains as a dev-facing catalog (for skill maintainers adding new checks); it is explicitly NOT copied into user repos. - references/report-template.md updated with the new filename + the inlined legend block. - Validation contract simplified: one file exists, has a '## Findings' section, counts match chat. - Multi-AppHost case: per-host filename is now scan-.md (still overwritten each run, no timestamp suffix). Git is the history mechanism if anyone wants one — the skill no longer reinvents version control by hand. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../skills/scan-agentic-app-perf/SKILL.md | 40 +++++++++---------- .../references/check-glossary.md | 38 +++++------------- .../references/common-pitfalls.md | 32 +++++++-------- .../references/report-template.md | 20 +++++++--- 4 files changed, 58 insertions(+), 72 deletions(-) diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md index 4e548a0807..28c30589bb 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/SKILL.md @@ -95,24 +95,23 @@ Severity rules: Sort findings by severity (critical → warn → info), then by `check` slug (stable lexical order so `history.*` < `model.*` < `otel.*` < -`parallel.*` < `prompt.*` < `tools.*` < `topology.*`). Write to: +`parallel.*` < `prompt.*` < `tools.*` < `topology.*`). -- `.copilot/perf-reports/scan-.md` (timestamped, kept) -- `.copilot/perf-reports/latest-scan.md` (overwritten each run) -- `.copilot/perf-reports/check-glossary.md` (overwritten each run) - — a one-line-per-check reference card so first-time readers of a - report can see the full check catalog without opening the skill repo. - Source: copy the "Reference card" section verbatim from - `references/check-glossary.md`. +Write a **single file**: `.copilot/perf-reports/scan.md`, overwritten +on every run. Git provides whatever history you want; the skill does +not maintain timestamped copies or `latest-*` mirrors. -See `references/report-template.md` for the exact layout. +The category legend (one-liners describing what each prefix covers) is +inlined into the report itself — see `references/report-template.md`. +There is no separate glossary file. The slug encodes the check (e.g. +`history.full-share`, `topology.cycle`); readers don't need a lookup +table. Create `.copilot/perf-reports/` if it does not exist. This skill never touches `.gitignore`. If the user wants the report -folder ignored, recommend in chat that they add -`.copilot/perf-reports/` to their `.gitignore` themselves; do not edit -it from this skill. +ignored, recommend in chat that they add `.copilot/perf-reports/` to +their `.gitignore` themselves; do not edit it from this skill. ### 5. Surface top findings in chat @@ -160,19 +159,18 @@ ends here. After running: -- A new file exists at `.copilot/perf-reports/scan-.md`. -- `latest-scan.md` exists in the same folder and matches the timestamped - file byte-for-byte. +- A file exists at `.copilot/perf-reports/scan.md` (overwritten if it + was there before). - The report has a `## Findings` section, even if empty (containing `_No findings._`). - The summary counts in the report match the chat output. ## Common pitfalls -- **Editing source code.** This skill is read-only and only writes to - `.copilot/perf-reports/`. Never edit `.gitignore`, source files, - config files, or anything else. If a check tempts you to fix the - issue inline, stop and add it as a finding instead. +- **Editing source code.** This skill is read-only. The ONLY write path + it owns is `.copilot/perf-reports/scan.md`. Never edit `.gitignore`, + source files, config files, or anything else. If a check tempts you + to fix the issue inline, stop and add it as a finding instead. - **Hallucinating findings.** Every finding must cite a real file and (where applicable) a real line. Before adding a finding to the report, re-open the cited file and verify the snippet exists at the @@ -201,5 +199,5 @@ After running: - `references/parallelism-checks.md` — sequential calls that could fan out. - `references/otel-coverage-checks.md` — Aspire dashboard, token/cost telemetry. - `references/model-assignment-checks.md` — single-model defaulting, role mismatch. -- `references/check-glossary.md` — the reference card written alongside each report. -- `references/report-template.md` — exact Markdown layout for the report. +- `references/check-glossary.md` — dev-facing catalog of all check slugs (NOT copied to user repos; for skill maintainers). +- `references/report-template.md` — exact Markdown layout for `scan.md`. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-glossary.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-glossary.md index db17754b2a..3faa29a706 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-glossary.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/check-glossary.md @@ -1,23 +1,14 @@ -# Check glossary +# Check glossary (dev-facing) -This file is the source of truth for the human-readable check card -that `scan-agentic-app-perf` writes alongside every report. +Canonical catalog of every `check` slug the skill can emit. This file +is for **skill maintainers** — adding or renaming a check requires a +row here. It is **not copied** into user repositories; the slugs are +self-describing in the report itself. -When the skill runs, it copies the "Reference card" section verbatim -to `.copilot/perf-reports/check-glossary.md` so a first-time report -reader can see the full check catalog without opening the skill repo. +If you change this table, also update the matching `*-checks.md` +reference file with the per-check detection logic. -## Reference card - -> **Categories:** `topology` (agent graph shape) · `tools` (per-agent -> tool list) · `history` (chat history strategy) · `prompt` (system -> prompt size + reuse) · `parallel` (concurrent agent invocations) · -> `otel` (instrumentation coverage) · `model` (per-agent model -> selection). -> -> **Severity:** `critical` = likely to break a flow or blow the -> budget · `warn` = measurable cost/perf regression · `info` = -> observation only. +## Catalog | Check | Sev | What it catches | |------------------------------------|----------|-----------------| @@ -42,20 +33,9 @@ reader can see the full check catalog without opening the skill repo. | `model.cheap-on-planner` | warn | Small model on the planner, larger on workers | | `model.hardcoded` | info | Model id literal in agent service `.cs` | -## Cross-skill routes embedded in `ref:` fields +## Cross-skill routes | `ref:` value | Skill to run next | |--------------------------------------|------------------------------------------------------------| | `skill:configure-agentic-perf-rules` | Install/update the always-on perf rules (rule #3 covers role-aware model selection) | | `skill:setup-maf-evals` | Wire eval reports + telemetry | - -## Notes for skill implementation - -- When generating a report, write a copy of the "Reference card" section - (the table and the legend immediately above it) to - `.copilot/perf-reports/check-glossary.md`. Overwrite each run; the - content is static within a skill version and does not depend on findings. -- The glossary file is per-repo (one file regardless of how many runs); - the timestamped `scan-.md` reports link to it relatively. -- Keep this table in lockstep with the per-category reference files. - Adding a new check in any `*-checks.md` file REQUIRES a row here. diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/common-pitfalls.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/common-pitfalls.md index b4bfc0ef50..9f9bd25cc6 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/common-pitfalls.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/common-pitfalls.md @@ -5,20 +5,19 @@ dogfooding (interview-coach v1/v2, ELI5Agent, behavioral-interview-coach). ## Output discipline -- **Editing source code.** This skill is read-only. The ONLY write - paths it owns are `.copilot/perf-reports/scan-.md`, - `latest-scan.md`, and `check-glossary.md`. Never touch - `.gitignore`, source files, config files, AppHost, or - `*.csproj`. If a check tempts you to fix the issue inline, stop and - add it as a finding instead — the user opts into the fix through - the step-6 routing prompt. +- **Editing source code.** This skill is read-only. The ONLY write path + it owns is `.copilot/perf-reports/scan.md`. Never touch `.gitignore`, + source files, config files, AppHost, or `*.csproj`. If a check tempts + you to fix the issue inline, stop and add it as a finding instead — + the user opts into the fix through the step-6 routing prompt. - **Surfacing more than 3 critical findings in chat.** Step 5 caps chat output at 3 critical findings + counts + report path. Anything - beyond that goes in the report file only. Pasting the entire - report into chat defeats the "single file you can re-open" UX. -- **Forgetting `latest-scan.md`.** The timestamped report is for - history; `latest-scan.md` is what tooling and humans will actually - open first. They must be byte-for-byte identical for the same run. + beyond that goes in the report file only. Pasting the entire report + into chat defeats the "single file you can re-open" UX. +- **Generating extra report files.** One file: `scan.md`. Do not + write timestamped copies, `latest-*` mirrors, glossary files, or + any other artifact. Git is the history mechanism if the user wants + one. ## Evidence integrity @@ -84,10 +83,11 @@ dogfooding (interview-coach v1/v2, ELI5Agent, behavioral-interview-coach). ## Inventory edge cases - **Multiple AppHost projects.** If a solution has more than one - `*.AppHost.csproj`, scan each one and emit one report per host - with the host name in the filename - (`scan--.md`). Do NOT merge — the topology, model set, - and OTel wiring belong to each host independently. + `*.AppHost.csproj`, scan each one and emit one report per host as + `.copilot/perf-reports/scan-.md` (e.g. `scan-WebApp.md`, + `scan-Worker.md`). Do NOT merge — the topology, model set, and + OTel wiring belong to each host independently. Each per-host file + is still overwritten on each scan; no timestamp suffixes. - **Agent registered via DI lambda without `AddAIAgent`.** Patterns like `services.AddSingleton(...)` or custom factories also count as agents. The detection in step 1 must include any path diff --git a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/report-template.md b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/report-template.md index 2bb7ff6858..818ceabdc7 100644 --- a/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/report-template.md +++ b/plugins/dotnet-ai/skills/scan-agentic-app-perf/references/report-template.md @@ -1,10 +1,11 @@ # Report template -The exact Markdown layout written to -`.copilot/perf-reports/scan-.md` and `latest-scan.md`. +The exact Markdown layout written to `.copilot/perf-reports/scan.md`. +This file is **overwritten on every run**. Git provides history; the +skill does not maintain timestamped copies or `latest-*` mirrors. ```markdown -# Agentic perf audit — {{ project_name }} +# Agentic perf scan — {{ project_name }} Run: {{ utc_timestamp }} Project: {{ relative_project_path }} @@ -26,7 +27,13 @@ Project: {{ relative_project_path }} ## Findings -> **Categories:** `topology` · `tools` · `history` · `prompt` · `parallel` · `otel` · `model`. See `check-glossary.md` for the full slug→description table. +> **Slug format:** `.` — categories are +> `topology` (graph shape) · `tools` (per-agent tool list) · +> `history` (chat history strategy) · `prompt` (system prompt +> size/reuse) · `parallel` (concurrent invocations) · `otel` +> (instrumentation) · `model` (per-agent model id). Severities: +> `critical` (likely to break a flow or blow the budget) · +> `warn` (measurable cost/perf regression) · `info` (observation). ### [critical] [`{{ check }}`] {{ title }} - **File:** `{{ file }}:{{ line }}` @@ -51,5 +58,6 @@ Project: {{ relative_project_path }} ## Empty-report contract -If there are zero findings, the `## Findings` section still appears with the -literal text `_No findings._`. The `## Summary` section shows zeros. +If there are zero findings, the `## Findings` section still appears +(after the legend blockquote) with the literal text `_No findings._`. +The `## Summary` section shows zeros. From b059f54c706ae833497c08405599b58376746772 Mon Sep 17 00:00:00 2001 From: Leslie Richardson Date: Wed, 24 Jun 2026 13:59:49 -0700 Subject: [PATCH 18/18] refactor(configure-agentic-perf-rules): drop arbitrary thresholds from rules #1/#2; bump to 0.3.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rules #1 (agent count) and #2 (handoff edges) shipped with numeric ceilings (agent_count_max: 3, llm_routed_edges_max_per_turn: 2) that tripped legitimate designs (specialist-handoff workflows, multi-hop deterministic chains with one LLM-routed intent decision) as often as they caught real bloat. The justify-the-add gates were always the actual mechanism; the numbers were illusory precision. Mirrors the same prune applied to scan-agentic-app-perf (a38f3f8) where the topology.agent-count and topology.handoff-fanout checks were removed for the same reason. Changes: - managed-block-template.md: drop both keys from the thresholds YAML block; reword rules #1/#2 to explicitly say 'no hard ceiling — justify the add'. - rule-rationales.md: drop the 'when you must add a 4th+' subsection; add a 'on thresholds' note explaining why the numeric ceilings were removed (so consumers reading rationales understand the v0.3.0 change). - threshold-defaults.md: drop the first 2 rows of the threshold table; add a deprecation note pointing to the rules. - SKILL.md: bump version 0.2.0 -> 0.3.0; update the threshold-preservation example from agent_count_max to per_turn_input_token_warn. - common-pitfalls.md: update strict-parse example from agent_count_max to per_turn_input_token_warn. - eval.yaml: drop agent_count_max literal assertion from fresh-install test; bump idempotency-test fixture from v0.1.0 -> v0.3.0; bump threshold-preservation fixture from v0.0.1 to drop deprecated keys from the input YAML; add new scenario validating v0.2.0 -> v0.3.0 upgrade drops deprecated keys with a chat warning. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../configure-agentic-perf-rules/SKILL.md | 4 +- .../references/common-pitfalls.md | 2 +- .../references/managed-block-template.md | 15 +++--- .../references/rule-rationales.md | 33 ++++++++----- .../references/threshold-defaults.md | 9 +++- .../configure-agentic-perf-rules/eval.yaml | 48 ++++++++++++++++--- 6 files changed, 79 insertions(+), 32 deletions(-) diff --git a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/SKILL.md b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/SKILL.md index 7015504944..e1ec6c5bf2 100644 --- a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/SKILL.md +++ b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/SKILL.md @@ -1,6 +1,6 @@ --- name: configure-agentic-perf-rules -version: 0.2.0 +version: 0.3.0 description: > Installs or updates an always-on rules block in a .NET agentic app that makes coding agents volunteer perf and cost concerns by default — agent count, handoff edges, @@ -112,7 +112,7 @@ The current skill version is the `version:` field at the top of this SKILL.md. If parsing fails, refuse to edit and ask the user to repair the YAML manually. 2. Construct the new defaults map `new_defaults` from `references/threshold-defaults.md`. 3. For each known key in `new_defaults`, override with the value from `prev_user` if - present and the value passes type validation (e.g. integer for `agent_count_max`). + present and the value passes type validation (e.g. integer for `per_turn_input_token_warn`). 4. Drop unknown keys from `prev_user` with a chat warning naming each dropped key. 5. The merged map becomes the new managed block's `thresholds:` content. diff --git a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/common-pitfalls.md b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/common-pitfalls.md index e4047ba743..b892442cb2 100644 --- a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/common-pitfalls.md +++ b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/common-pitfalls.md @@ -35,7 +35,7 @@ no-op re-run, ELI5Agent fresh install). a chat warning naming the dropped key. Silent drops break audit trails — the user needs to know their override no longer applies. - **Type-validating with `Convert.ToInt32` instead of strict parse.** - `agent_count_max: "three"` should fail validation, not coerce to + `per_turn_input_token_warn: "eight thousand"` should fail validation, not coerce to some default. Use strict numeric parsing; on failure, keep the default and warn. diff --git a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/managed-block-template.md b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/managed-block-template.md index e8ff121e7d..fe69aa65f2 100644 --- a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/managed-block-template.md +++ b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/managed-block-template.md @@ -18,8 +18,6 @@ interpreted as closing it. ```yaml # thresholds — edit values below to override per-project defaults thresholds: - agent_count_max: 3 - llm_routed_edges_max_per_turn: 2 per_turn_input_token_warn: 8000 per_turn_output_token_warn: 2000 baseline_token_increase_warn_pct: 20 @@ -33,16 +31,19 @@ form **"Before X, justify Y."** When you cannot justify, prefer the safer altern ### 1. Agent count Before adding a new agent to a workflow, justify why the new responsibility cannot be a -tool call on an existing agent. Default ceiling: **`agent_count_max`** agents per -workflow. If the workflow already has that many agents, do not add another without -explicit user direction. +tool call on an existing agent. Each additional agent multiplies the routing surface and +inflates per-turn token cost (system prompts + tool descriptions are paid per agent). +**There is no hard ceiling** — the answer "this needs a clearly different system prompt, +toolset, or output style" is a valid justification. If you cannot articulate one, prefer +adding a tool to an existing agent. ### 2. Handoff edges Before adding an LLM-routed handoff edge (e.g. via `AgentWorkflowBuilder.CreateHandoffBuilderWith`), justify why a deterministic edge or a -conditional `WorkflowBuilder` branch will not work. Default ceiling: -**`llm_routed_edges_max_per_turn`** LLM-routed edges traversed per user turn. +conditional `WorkflowBuilder` branch will not work. Every LLM-routed edge is an extra LLM +call before the user gets a response. Deterministic routing is faster and cheaper; reserve +LLM routing for decisions that genuinely require reading user intent. ### 3. Model selection diff --git a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/rule-rationales.md b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/rule-rationales.md index 8495fb1734..03a465fbc6 100644 --- a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/rule-rationales.md +++ b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/rule-rationales.md @@ -9,12 +9,13 @@ managed block. The managed block itself is intentionally terse; this file is the ## 1. Agent count **Rule.** Before adding a new agent to a workflow, justify why the new responsibility -cannot be a tool call on an existing agent. Default ceiling: 3 agents per workflow. +cannot be a tool call on an existing agent. **No hard ceiling** — the answer "this needs +a clearly different system prompt, toolset, or output style" is a valid justification. -**Why it matters.** Each additional agent multiplies the routing surface area: the -LLM has to decide whether *this* turn should go to *that* agent, and decision quality -falls off as choices grow. Real-world failure mode: a 5-specialist workflow where the -router constantly mis-handoffs because too many specialists have overlapping domains. +**Why it matters.** Each additional agent multiplies the routing surface area (the LLM +has to decide whether *this* turn should go to *that* agent) AND inflates per-turn +input-token cost: every agent's system prompt + tool descriptions are paid on every +turn the agent participates in. Decision quality degrades as choices grow. **When a new agent is justified.** The new responsibility involves a meaningfully different *system prompt*, *toolset*, or *output style* that would muddy the existing @@ -26,22 +27,23 @@ agent's instructions if folded in. Examples: whatever agent needs the data, not its own agent. - **No, just a tool:** A "Formatter" agent that reformats output. Again, a tool. -**When you must add a 4th+ agent.** Surface the trade-off to the user. Mention the -default ceiling, why it exists, and what mitigations apply (e.g. tighter routing -prompts, explicit `WorkflowBuilder` branches instead of free-form handoffs). +**On thresholds.** Earlier versions of this rule had an `agent_count_max: 3` numeric +ceiling. It was removed because legitimate designs (e.g. specialist-handoff +interview/coaching workflows with 4-5 well-scoped agents) tripped it as often as +genuine bloat did. The justify-the-add gate is the actual mechanism; a number was +illusory precision. --- ## 2. Handoff edges **Rule.** Before adding an LLM-routed handoff edge, justify why a deterministic edge or -a conditional `WorkflowBuilder` branch will not work. Default ceiling: 2 LLM-routed -edges traversed per user turn. +a conditional `WorkflowBuilder` branch will not work. **No hard ceiling.** **Why it matters.** Every LLM-routed edge is an additional LLM call before the user -gets a response. Two routed decisions per turn (e.g. "router → specialist", "specialist -→ done-or-continue") is the practical latency ceiling before users notice. -Free-form-everywhere graphs also amplify decision-quality variance. +gets a response. Deterministic routing — "after Coach runs, always return to +Interviewer" expressed as a `WorkflowBuilder` edge — costs zero extra latency and zero +extra tokens. Free-form LLM routing also amplifies decision-quality variance. **When LLM routing is justified.** The decision genuinely requires reading the user's intent — for example, "is this answer detailed enough to grade?" or "which specialist @@ -57,6 +59,11 @@ runs, always go back to interviewer"), use a deterministic edge. - Five specialists in a fully-connected handoff graph where every transition is LLM-routed. Symptom: Copilot constantly proposes new edges as the workflow grows. +**On thresholds.** Earlier versions of this rule had an +`llm_routed_edges_max_per_turn: 2` numeric ceiling. Same reason as rule #1: the gate +is the justification, not the number. A 4-hop deterministic chain with one LLM-routed +intent decision is fine; two LLM-routed edges per turn in a misdesigned graph is not. + --- ## 3. Model selection diff --git a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/threshold-defaults.md b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/threshold-defaults.md index 987d204188..d2ce7698d9 100644 --- a/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/threshold-defaults.md +++ b/plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/threshold-defaults.md @@ -6,13 +6,18 @@ simple two-agent workflows, or complex tool-heavy pipelines) should adjust. | Threshold | Default | Rationale | |-----------|---------|-----------| -| `agent_count_max` | `3` | Most workflows that need more than 3 agents are better served by tools-on-fewer-agents. Real-world frustration: 5+ specialist workflows with free-form LLM routing dramatically slow per-turn latency and confuse handoff decisions. | -| `llm_routed_edges_max_per_turn` | `2` | Each LLM-routed edge is an extra LLM call. Two routed decisions per user turn (e.g. "router → specialist", "specialist → done") is the practical ceiling before latency becomes user-visible. | | `per_turn_input_token_warn` | `8000` | Modern reasoning models can take 100K+, but most chat-class models start showing meaningful latency and cost above ~8K input tokens. Projects with retrieval/RAG legitimately exceed this — override locally. | | `per_turn_output_token_warn` | `2000` | Output tokens are usually 4-10x more expensive than input on a per-token basis. 2000 is a reasonable "are you sure?" threshold; long-form generation tasks should override. | | `baseline_token_increase_warn_pct` | `20` | A 20% increase per turn meaningfully changes monthly bills at scale. Small tweaks under 20% are noise; over 20% is worth surfacing. | | `unbounded_history_warn` | `true` | Default-on. Sending full history forever is the single most common token-bloat pattern. Disable only if the workflow has already implemented a windowing/summarization strategy and the warning is now noise. | +**Note on removed thresholds.** Earlier versions of this skill (≤v0.2.0) shipped +`agent_count_max: 3` and `llm_routed_edges_max_per_turn: 2`. Both were taste-call +ceilings that tripped legitimate designs (5-specialist handoff workflows, multi-hop +deterministic chains with one LLM-routed intent decision) as often as they caught real +bloat. The justify-the-add gates in rules #1 and #2 are the actual mechanism; the +numbers were illusory precision and were removed in v0.3.0. + ## Adjusting thresholds Users override defaults inside the managed block's `thresholds:` YAML map. The skill diff --git a/tests/dotnet-ai/configure-agentic-perf-rules/eval.yaml b/tests/dotnet-ai/configure-agentic-perf-rules/eval.yaml index cd99ffb476..2a23305c90 100644 --- a/tests/dotnet-ai/configure-agentic-perf-rules/eval.yaml +++ b/tests/dotnet-ai/configure-agentic-perf-rules/eval.yaml @@ -38,7 +38,7 @@ scenarios: value: "END: managed by configure-agentic-perf-rules" - type: "file_contains" path: ".github/copilot-instructions.md" - value: "agent_count_max" + value: "per_turn_input_token_warn" - type: "file_contains" path: ".github/copilot-instructions.md" value: "Agent count" @@ -113,14 +113,12 @@ scenarios: content: | # Project notes - + ## Agentic Performance Rules ```yaml thresholds: - agent_count_max: 3 - llm_routed_edges_max_per_turn: 2 per_turn_input_token_warn: 8000 per_turn_output_token_warn: 2000 baseline_token_increase_warn_pct: 20 @@ -133,7 +131,7 @@ scenarios: assertions: - type: "file_contains" path: ".github/copilot-instructions.md" - value: "v0.1.0" + value: "v0.3.0" - type: "exit_success" rubric: - "Detected the existing managed block at the current skill version" @@ -153,8 +151,6 @@ scenarios: ```yaml thresholds: - agent_count_max: 3 - llm_routed_edges_max_per_turn: 2 per_turn_input_token_warn: 25000 per_turn_output_token_warn: 2000 baseline_token_increase_warn_pct: 20 @@ -176,6 +172,44 @@ scenarios: - "Showed the user a diff or summary of what changed before applying" timeout: 360 + - name: "Update from v0.2.0 — drops deprecated agent_count_max and llm_routed_edges_max_per_turn with warning" + prompt: "Update the agentic-perf rules in this project to the latest version." + setup: + files: + - path: ".github/copilot-instructions.md" + content: | + + + ## Agentic Performance Rules + + ```yaml + thresholds: + agent_count_max: 5 + llm_routed_edges_max_per_turn: 3 + per_turn_input_token_warn: 12000 + per_turn_output_token_warn: 2000 + baseline_token_increase_warn_pct: 20 + unbounded_history_warn: true + ``` + + (older content) + + + assertions: + - type: "file_contains" + path: ".github/copilot-instructions.md" + value: "per_turn_input_token_warn: 12000" + - type: "exit_success" + rubric: + - "Detected the existing managed block at v0.2.0" + - "Replaced the block with the current-version template (v0.3.0+)" + - "Preserved per_turn_input_token_warn: 12000 (user override) — did NOT reset to default" + - "Dropped the deprecated `agent_count_max` key from the rendered YAML (no longer a threshold in current schema)" + - "Dropped the deprecated `llm_routed_edges_max_per_turn` key from the rendered YAML" + - "Emitted a chat warning naming each dropped key (`agent_count_max`, `llm_routed_edges_max_per_turn`) so the user has an audit trail" + - "Did NOT silently retain the dropped keys in the new managed block" + timeout: 360 + - name: "AGENTS.md stub when both instructions files exist" prompt: "Install the agentic-perf rules. The project uses both AGENTS.md and .github/copilot-instructions.md." setup: