setup-maf-evals: report generation, three evaluator tiers, MSTest project shape

plugins/dotnet-ai/.codex-plugin/plugin.json

@@ -2,6 +2,5 @@
   "name": "dotnet-ai",
   "version": "0.1.0",
   "description": "AI and ML skills for .NET: technology selection, LLM integration, agentic workflows, RAG pipelines, MCP, and classic ML with ML.NET.",
-  "skills": ["./skills/"],
-  "agents": ["./agents/agentic-perf-reviewer.agent.md"]
+  "skills": ["./skills/"]
 }

plugins/dotnet-ai/plugin.json

@@ -2,6 +2,5 @@
   "name": "dotnet-ai",
   "version": "0.1.0",
   "description": "AI and ML skills for .NET: technology selection, LLM integration, agentic workflows, RAG pipelines, MCP, and classic ML with ML.NET.",
-  "skills": ["./skills/"],
-  "agents": ["./agents/agentic-perf-reviewer.agent.md"]
+  "skills": ["./skills/"]
 }

plugins/dotnet-ai/skills/configure-agentic-perf-rules/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: configure-agentic-perf-rules
-version: 0.1.0
+version: 0.3.0
 description: >
   Installs or updates an always-on rules block in a .NET agentic app that makes coding
   agents volunteer perf and cost concerns by default — agent count, handoff edges,
@@ -43,7 +43,6 @@ clobbering user-edited threshold values.
 - The user wants the agent to actually audit existing code right now — use
   `scan-agentic-app-perf` instead. This skill only installs guidance.
 - The user wants to measure tokens, latency, or quality scores — use `setup-maf-evals`.
-- The user wants to pick or change per-agent model assignments — use `select-agent-models`.
 - Generic prompt-engineering or non-perf coding-agent rules (keep those in the user's own
   instructions section, outside the managed block).
 
@@ -113,7 +112,7 @@ The current skill version is the `version:` field at the top of this SKILL.md.
    If parsing fails, refuse to edit and ask the user to repair the YAML manually.
 2. Construct the new defaults map `new_defaults` from `references/threshold-defaults.md`.
 3. For each known key in `new_defaults`, override with the value from `prev_user` if
-   present and the value passes type validation (e.g. integer for `agent_count_max`).
+   present and the value passes type validation (e.g. integer for `per_turn_input_token_warn`).
 4. Drop unknown keys from `prev_user` with a chat warning naming each dropped key.
 5. The merged map becomes the new managed block's `thresholds:` content.
 
@@ -160,7 +159,8 @@ Each rule is in the form **"Before X, justify Y."** Categories, in order:
    deterministic edge or a conditional `WorkflowBuilder` branch will not work. Default
    ceiling: 2 LLM-routed edges traversed per user turn.
 3. **Model selection.** Before defaulting to a frontier model (e.g. `gpt-4o`), name the
-   agent's role and pick from the role→model matrix in the `select-agent-models` skill.
+   agent's role and pick from the role table inside rule #3 of the managed block.
+   Routers/validators/formatters/workers → small-fast; planners → reasoning-class.
 4. **Message-history strategy.** Before sending the full conversation history to an
    agent, state the bound — turn count, token cap, summarization point, or retrieval
    strategy. Default warning when unbounded full-history is used in a multi-turn workflow.
@@ -238,4 +238,4 @@ If `AGENTS.md` was updated, also confirm the stub line is present exactly once.
 - `references/threshold-defaults.md` — default numeric values and the rationale for each.
 - `references/rule-rationales.md` — long-form prose for each of the six rule categories,
   with examples and counter-examples.
-- Companion skills: `scan-agentic-app-perf`, `select-agent-models`, `setup-maf-evals`.
+- Companion skills: `scan-agentic-app-perf`, `setup-maf-evals`.

plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/common-pitfalls.md

@@ -0,0 +1,100 @@
+# Common pitfalls
+
+Real-world failure modes for `configure-agentic-perf-rules` — observed
+during dogfooding (interview-coach v2 first install, behavioral coach
+no-op re-run, ELI5Agent fresh install).
+
+## Sentinel parsing (FAIL CLOSED, always)
+
+- **Lenient sentinel matching.** The BEGIN and END regexes in step 2
+  are intentionally strict (anchored, full-line, exact version
+  pattern). Do not "fix up" a malformed sentinel by partial-matching
+  or by accepting trailing whitespace beyond what `\s*$` allows.
+  Auto-repair of corrupted sentinels is forbidden — the user might
+  have intentionally renamed the block while migrating, and silent
+  repair would clobber that.
+- **Refusing to edit on multiple BEGIN/END.** If the file contains
+  more than one BEGIN or END, abort with a chat message that names
+  every offending line number. Multiple managed blocks usually mean
+  a merge conflict went unresolved; appending a third would make it
+  worse.
+- **Out-of-order sentinels.** BEGIN must appear before its matching
+  END. If you find END before BEGIN, treat as malformed and abort —
+  do not swap them.
+
+## Threshold preservation
+
+- **Losing user-edited threshold values on update.** Step 2's
+  "Threshold preservation algorithm" is the contract: parse the
+  existing `thresholds:` map into `prev_user`, overlay the new
+  default map, override each known key from `prev_user`. Skipping
+  this step and writing the new defaults verbatim is the most
+  user-visible regression this skill can ship.
+- **Silently dropping unknown keys.** If `prev_user` has a key not
+  in `new_defaults` (e.g. a deprecated threshold), drop it AND emit
+  a chat warning naming the dropped key. Silent drops break audit
+  trails — the user needs to know their override no longer applies.
+- **Type-validating with `Convert.ToInt32` instead of strict parse.**
+  `per_turn_input_token_warn: "eight thousand"` should fail validation, not coerce to
+  some default. Use strict numeric parsing; on failure, keep the
+  default and warn.
+
+## Target-file selection
+
+- **Writing to `AGENTS.md` when `.github/copilot-instructions.md`
+  exists.** The order in step 2's target-file table is the spec:
+  `.github/copilot-instructions.md` is the primary destination;
+  `AGENTS.md` gets a stub pointer. The reverse only happens during
+  the explicit migration path (existing block in AGENTS.md, none in
+  copilot-instructions.md).
+- **Writing to both files.** "Both have a block" is the abort path —
+  the user must consolidate manually. Writing the rule prose into
+  two files would split the source of truth and let the two copies
+  drift on the next update.
+- **Creating `.github/` outside the project root.** If neither file
+  exists, create `.github/copilot-instructions.md` under the resolved
+  project root (the directory containing the `.sln`/`.slnx`/`*.AppHost.csproj`).
+  Never write to a parent directory or a sibling project.
+
+## Path safety
+
+- **Following symlinks out of the project root.** Step 2's "Path
+  safety" rule requires resolving the absolute path AND ensuring it
+  still starts with the project-root prefix after symlink resolution.
+  Some CI environments place repos under symlinks; a naive resolve
+  can end up writing to the symlink target outside the workspace.
+- **Accepting `..` in paths.** Reject any path containing `..`
+  segments before normalization, unless the post-normalization path
+  is still inside the project root. The simplest safe check:
+  `Path.GetFullPath(target).StartsWith(Path.GetFullPath(projectRoot))`.
+
+## Cross-tool stub on AGENTS.md
+
+- **Re-adding the stub on every run.** The stub is one line:
+  `> Agentic-perf rules for this project live in .github/copilot-instructions.md (managed by configure-agentic-perf-rules).`
+  Check whether that exact line is already present before appending;
+  re-running the skill should not grow the file by one line each time.
+- **Replacing user prose in AGENTS.md with the stub.** AGENTS.md
+  often contains real onboarding prose the user wrote. Append the
+  stub at the bottom if missing; never overwrite existing content.
+
+## Version handling
+
+- **Refusing to downgrade is correct.** If the file has a newer
+  version than this skill, abort cleanly with a chat message naming
+  both versions. Do not "merge" or "convert" — that's how data loss
+  happens.
+- **Comparing versions as strings.** "v0.1.10" sorts before "v0.1.2"
+  lexically. Parse into semver triples and compare numerically.
+
+## Idempotency
+
+- **No-op path must actually be a no-op.** When the block is present,
+  same version, and structurally valid, the skill must not touch the
+  file at all — not even to rewrite identical content. Tooling and
+  git status both rely on "no change on second run". Verify by
+  comparing the SHA256 hash before/after; they should match exactly.
+- **Tracking "already installed" silently.** Even on a no-op, the
+  chat output should say "configure-agentic-perf-rules v0.1.0 block
+  already current — no changes". Quiet no-ops make the user think
+  the skill didn't run.

plugins/dotnet-ai/skills/configure-agentic-perf-rules/references/managed-block-template.md

@@ -18,8 +18,6 @@ interpreted as closing it.
 ```yaml
 # thresholds — edit values below to override per-project defaults
 thresholds:
-  agent_count_max: 3
-  llm_routed_edges_max_per_turn: 2
   per_turn_input_token_warn: 8000
   per_turn_output_token_warn: 2000
   baseline_token_increase_warn_pct: 20
@@ -33,23 +31,36 @@ form **"Before X, justify Y."** When you cannot justify, prefer the safer altern
 ### 1. Agent count
 
 Before adding a new agent to a workflow, justify why the new responsibility cannot be a
-tool call on an existing agent. Default ceiling: **`agent_count_max`** agents per
-workflow. If the workflow already has that many agents, do not add another without
-explicit user direction.
+tool call on an existing agent. Each additional agent multiplies the routing surface and
+inflates per-turn token cost (system prompts + tool descriptions are paid per agent).
+**There is no hard ceiling** — the answer "this needs a clearly different system prompt,
+toolset, or output style" is a valid justification. If you cannot articulate one, prefer
+adding a tool to an existing agent.
 
 ### 2. Handoff edges
 
 Before adding an LLM-routed handoff edge (e.g. via
 `AgentWorkflowBuilder.CreateHandoffBuilderWith`), justify why a deterministic edge or a
-conditional `WorkflowBuilder` branch will not work. Default ceiling:
-**`llm_routed_edges_max_per_turn`** LLM-routed edges traversed per user turn.
+conditional `WorkflowBuilder` branch will not work. Every LLM-routed edge is an extra LLM
+call before the user gets a response. Deterministic routing is faster and cheaper; reserve
+LLM routing for decisions that genuinely require reading user intent.
 
 ### 3. Model selection
 
-Before defaulting to a frontier model (e.g. `gpt-4o`), name the agent's role and pick
-from the role→model matrix in the `select-agent-models` skill. Routers, classifiers,
-and summarizers usually want a smaller/faster model; reasoning steps may want a
-reasoning-class model.
+Before defaulting to a frontier model like `gpt-4o`, name the agent's role and pick
+from the table below. Routers, validators, formatters, and workers almost never need
+a frontier model; defaulting to one is the largest single source of unnecessary spend.
+
+| Role                                | Pick                                                                              |
+|-------------------------------------|-----------------------------------------------------------------------------------|
+| router / validator / formatter      | small-fast model (e.g. `gpt-4o-mini` or current cheap-fast in your Foundry catalog) |
+| worker / summarizer / extraction    | small-fast model, **or** Foundry `model-router` deployment if prompt length varies |
+| planner / decomposer / open reasoning | reasoning-class model (e.g. `o4-mini` or current reasoning model) — state *why* in a code comment |
+| creative / nuanced generation       | frontier (e.g. `gpt-4o`) — state *why* in a code comment                          |
+
+If unsure which role applies, **stop and ask the user** — do not default to `gpt-4o`.
+Specific model ids age fast; check your Foundry catalog for the current cheap-fast,
+reasoning-class, and frontier ids before pinning.
 
 ### 4. Message-history strategy