Capture the repo's hard-won skill-authoring lessons as guidance

.agents/skills/create-skill/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: create-skill
-description: Scaffolds new agent skills for the dotnet/skills repository. Use when creating a new skill, generating SKILL.md files, or setting up skill directory structures. Handles frontmatter generation, section templates, and validation guidance.
+description: Scaffolds new agent skills for the dotnet/skills repository. Use when creating a new skill, generating SKILL.md files, writing a skill description that the runtime will actually route to, or setting up skill directory structures. Handles frontmatter generation, section templates, and validation guidance. Do not use for fixing a skill that already fails its evaluation (use improve-skill-quality) or for writing eval.yaml (use create-skill-test).
 ---
 
 # Create Skill
@@ -17,6 +17,8 @@ This skill helps you scaffold new agent skills that conform to the Agent Skills
 ## When Not to Use
 
 - Modifying existing skills (edit directly instead)
+- Diagnosing or fixing a skill that fails its evaluation (use `improve-skill-quality`)
+- Writing the skill's `eval.yaml` (use `create-skill-test`)
 - Creating custom agents (use the agents/ directory pattern)
 
 ## Inputs
@@ -38,25 +40,66 @@ Ensure the name:
 - Does not contain consecutive hyphens
 - Is between 1-64 characters
 
-### Step 2: Create the skill directory
+### Step 2: Write the description — it is the router
 
-```
-skills/<skill-name>/
-└── SKILL.md
-```
-
-### Step 3: Generate SKILL.md with frontmatter
-
-Create the file with required YAML frontmatter:
+The `description` is the **only** text the runtime sees when deciding whether to load the skill.
+A perfect body behind a weak description never runs.
 
 ```yaml
 ---
 name: <skill-name>
-description: <description of what the skill does and when to use it>
+description: <what it does>. USE FOR: <symptoms, error codes, artifact names, quoted user requests>. DO NOT USE FOR: <nearby-but-wrong intents, with the skill that owns them>.
 ---
 ```
 
-### Step 4: Add body content sections
+- Lead with an action verb and use the user's own words: symptoms, error codes (`CS1501`,
+  `MSTEST0014`), artifact names (`.testsettings`, `binlog`), and requests phrased as a developer
+  would type them.
+- Partition against sibling skills on the **real discriminator**, not the topic. "Does the
+  abstraction already exist?" separates two skills; "testing" does not. Add the matching exclusion
+  to **both** siblings.
+- Claim the ambiguous words that would otherwise route to a sibling. If prompts say "review my
+  tests" and a sibling owns "review", say so explicitly.
+- Check every `DO NOT USE FOR` clause against the scenarios the skill exists to serve — an
+  exclusion like "already on v3" can lock out the post-upgrade fixes that are the skill's purpose.
+- Budget: 1,024 characters per description, and the whole plugin's rendered skill menu is also
+  budgeted. A helper skill users should never invoke directly can set
+  `disable-model-invocation: true` to free menu space while staying invocable by name.
+
+### Step 3: Write for delta over the baseline model
+
+Every skill is scored head-to-head against the same model with **no skill loaded**. Content the
+model already produces unaided is worth zero; content that makes it slower or more hedged is worth
+less than zero. See
+[improve-skill-quality/references/writing-for-baseline-delta.md](../improve-skill-quality/references/writing-for-baseline-delta.md)
+for the full evidence.
+
+| Do | Instead of |
+|----|------------|
+| Encode the decision the model would otherwise get wrong | Restating API signatures it already reproduces |
+| "When A, do B, never C, verify D" tables | Lists of plausible alternatives |
+| A concrete output contract (exact command, verdict line, findings table) | "Consider…", "you may want to…" |
+| Scale output structure to input size | A 12-section dashboard for an 8-test suite |
+| Stop-conditions that prevent over-applying | Acting before measuring, rewriting working code |
+| Instructing the agent to discover repo paths | Marking discoverable paths as required inputs |
+| Reporting restore/build/test failures truthfully | Claiming success after a failed command |
+| Verifying load-bearing API claims by compiling or probing | Trusting a source read |
+| Gating rare or expensive paths behind `references/` | One large SKILL.md carrying every path |
+
+Do not over-correct: a skilled answer shorter and less actionable than the baseline's still loses.
+
+### Step 4: Create the skill directory
+
+```
+skills/<skill-name>/
+└── SKILL.md
+```
+
+### Step 5: Generate SKILL.md with frontmatter
+
+Create the file with the frontmatter drafted in Step 2.
+
+### Step 6: Add body content sections
 
 Include these recommended sections:
 
@@ -68,7 +111,7 @@ Include these recommended sections:
 6. **Validation**: How to confirm the skill worked correctly
 7. **Common Pitfalls**: Known traps and how to avoid them
 
-### Step 5: Add optional directories (if needed)
+### Step 7: Add optional directories (if needed)
 
 ```
 skills/<skill-name>/
@@ -78,7 +121,7 @@ skills/<skill-name>/
 └── assets/        # Templates, images, data files
 ```
 
-### Step 6: Update CODEOWNERS
+### Step 8: Update CODEOWNERS
 
 Add entries in `.github/CODEOWNERS` for the new skill and its test directory:
 
@@ -89,12 +132,19 @@ Add entries in `.github/CODEOWNERS` for the new skill and its test directory:
 
 Match the owner pattern used by sibling skills in the same plugin.
 
-### Step 7: Validate the skill
+### Step 9: Validate the skill
 
 - Confirm frontmatter fields are valid
 - Ensure SKILL.md is under 500 lines
 - Check that file references use relative paths
 - Verify instructions are actionable and specific
+- Run `dotnet run --project eng/skill-validator/src/SkillValidator.csproj -- check --plugin ./plugins/<plugin>`
+
+### Step 10: Add the eval
+
+A skill without an `eval.yaml` has no evidence that it improves on the baseline. Use
+`create-skill-test` to add one in the same pull request, and size it for statistical power — an eval
+below five trials can never return a passing verdict.
 
 ## SKILL.md Template
 
@@ -161,6 +211,10 @@ After creating a skill, verify:
 - [ ] Validation section exists with observable success criteria
 - [ ] No secrets, tokens, or internal URLs included
 - [ ] `.github/CODEOWNERS` has entries for the new skill and its test directory
+- [ ] The description names concrete triggers and excludes the nearest sibling skills
+- [ ] Every section changes a decision the unskilled model would otherwise get wrong
+- [ ] The skill states when **not** to act, and what a truthful failure report looks like
+- [ ] An `eval.yaml` exists and clears the trial floor
 
 ## Common Pitfalls
 
@@ -173,9 +227,16 @@ After creating a skill, verify:
 | SKILL.md too long | Move detailed content to `references/` files |
 | Hardcoded environment assumptions | Document requirements in `compatibility` field |
 | Missing CODEOWNERS entry | Add entries for both `/plugins/<plugin>/skills/<skill-name>/` and `/tests/<plugin>/<skill-name>/` matching sibling skills' owner pattern |
+| Skill restates what the model already knows | Cut it; a skill is scored as a delta over the unskilled model |
+| Discoverable paths listed as required inputs | Tell the agent to discover them, or it will stop and ask the user |
+| Description partitioned by topic against a sibling | Partition on the real discriminator and exclude on both sides |
+| Exclusion clause blocks the skill's own use cases | Re-read every "do not use for" clause against real workflow phases |
+| Skill added without an eval | Add `eval.yaml` in the same PR; unevaluated skills carry no evidence |
 
 ## References
 
 - [Agent Skills Specification](https://agentskills.io/specification)
-- [Repository README](../../README.md)
-- [Contributing Guidelines](../../CONTRIBUTING.md)
+- [Repository README](../../../README.md)
+- [Contributing Guidelines](../../../CONTRIBUTING.md)
+- [create-skill-test](../create-skill-test/SKILL.md) — authoring the skill's `eval.yaml`
+- [improve-skill-quality](../improve-skill-quality/SKILL.md) — fixing a skill that loses to its baseline

.agents/skills/improve-skill-quality/SKILL.md

@@ -0,0 +1,187 @@
+---
+name: improve-skill-quality
+description: Diagnoses and fixes skills in the dotnet/skills repository that lose to their own baseline, fail to activate, time out, or return "no credible improvement". Use when an evaluation verdict is a regression or underpowered, when a skill regressed after a change, when /evaluate reports no results, or when deciding whether a weak skill should be strengthened or retired. Do not use for scaffolding a brand-new skill (use create-skill) or a brand-new eval (use create-skill-test).
+---
+
+# Improve Skill Quality
+
+Turn a failing or unconvincing evaluation into a targeted fix. The single most common mistake
+in this repo is rewriting skill prose in response to a verdict whose real cause was the eval,
+the fixtures, or the harness. Classify first, then fix.
+
+## When to Use
+
+- An evaluation verdict is a regression, underpowered, or "no credible improvement".
+- A skill wins in the isolated arm but not in the plugin arm, or is reported "not activated".
+- `/evaluate` reports "Evaluation ran but produced no results".
+- A skill scores well but costs too much (tokens, turns, wall time, plugin menu budget).
+- Deciding whether to strengthen or retire a persistently weak skill.
+
+## When Not to Use
+
+- Creating a new skill from scratch — use `create-skill`.
+- Creating a new `eval.yaml` from scratch — use `create-skill-test`.
+- Changing the harness itself (`eng/skill-validator`, `eng/vally-adapter`, `evaluation*.yml`).
+
+## Inputs
+
+| Input | Required | Description |
+|-------|----------|-------------|
+| Verdict evidence | Yes | The `/evaluate` PR comment, or `results.json` from the run artifacts |
+| Losing trial transcripts | Yes for content fixes | Baseline vs. skilled output plus the judge's stated reason |
+| W/T/L record and trial count | Yes | Distinguishes a real regression from an underpowered eval |
+| Activation status per arm | Yes | Isolated and plugin activation are different failures |
+
+## Workflow
+
+### Step 1: Get the evidence before forming a hypothesis
+
+Read [InvestigatingResults.md](../../../eng/vally-adapter/InvestigatingResults.md) for how to
+download artifacts and read `results.json`. Extract, per failing stimulus:
+
+- win / tie / loss record and total trials (`trials = stimuli × runs`)
+- activation status in the **isolated** and **plugin** arms, separately
+- the judge's verbatim reason on each losing trial
+- whether any trial errored, timed out, or produced empty output
+
+Do not proceed until you can quote a losing trial. "Every change is driven by the judge evidence
+from the losing trials, not by style preference" is the standard this repo holds itself to.
+
+### Step 2: Classify the failure
+
+Work down this table and stop at the first row that matches. Rows are ordered by how often the
+symptom has been misdiagnosed as a skill-content problem.
+
+| Symptom | Real cause class | Go to |
+|---------|------------------|-------|
+| No `results.json`, "produced no results", or the spec never loaded | Harness / spec-load | Step 3 |
+| Trials errored, timed out, or returned empty output | Reliability | Step 3 |
+| A fixture does not build, is untracked by git, or contradicts itself | Fixture | Step 4 |
+| Positive record (e.g. 16W/8T/1L) but the verdict is still not a pass | Statistical power | Step 5 |
+| Skilled arm equals baseline arm by construction | Eval design | Step 6 |
+| Activated and lost on quality, judge names a concrete defect | Skill content | Step 7 |
+| Activated in isolation, not in plugin | Activation / routing | Step 8 |
+| Not activated in either arm | Frontmatter description | Step 8 |
+| Wins but costs far more than baseline | Scope and cost | Step 7 |
+
+### Step 3: Rule out harness and reliability causes
+
+See [references/eval-triage.md](references/eval-triage.md) for the full catalogue. The recurring ones:
+
+- A spec declaring both `config:` and `defaults:` is rejected by vally, the job still exits 0, and
+  the PR comment blames "transient infrastructure". Merge them into one `defaults:` block.
+- An errored trial is not automatically a fixture problem — judge-side auth and `session.idle`
+  failures look identical from the verdict and need harness fixes, not SDK pins.
+- `expect_tools: [bash]` on an advisory question forces a restore or build and turns an answer into
+  a timeout with no quality gain.
+- Genuine code-generation stimuli need roughly 360s; a timeout yields empty output, which fails
+  every grader and hides the real quality signal.
+
+### Step 4: Verify the fixtures before touching the skill
+
+Run `python eng/eval-quality/check_eval_quality.py` — it blocks ten defect classes that each already
+cost a real result here. Then confirm by hand:
+
+- every buildable fixture actually builds, and every fixture actually reproduces the bug its
+  stimulus is named for;
+- every referenced fixture is in the git index (`git ls-files`), not merely on disk — `.gitignore`
+  has silently swallowed committed coverage fixtures;
+- coverage fixtures are self-consistent: declared `line-rate`, summary totals, and the `<line>`
+  elements must all report the same number, or the two arms legitimately read different truths.
+
+### Step 5: Check whether the eval could ever have passed
+
+The gate is an exact one-sided sign test over **discordant** (non-tie) trials.
+
+- Below 5 trials no record can pass, however good the skill.
+- At 5–7 trials only a clean sweep passes; one tie makes a pass arithmetically unreachable.
+- Tolerating a single loss needs 8 discordant trials.
+
+So a positive record with a failing verdict is a power problem, not a content problem. Fix it by
+adding **discriminating stimuli** (cross-task evidence) rather than raising `runs` (repetition
+only) — except where each stimulus drives an expensive pipeline. Record the reasoning in a comment
+above `defaults:`, as `tests/dotnet-test/grade-tests/eval.yaml` does.
+
+### Step 6: Check whether the two arms differ at all
+
+An eval that compares the skill against itself measures judge noise:
+
+- A dormancy guard (`expect_activation: false`) must **not** also set `constraints.reject_skills`.
+  That makes the skilled arm skill-free, i.e. identical to baseline. Across four evals the same
+  guard scored −0.4, +0.4, +0.4 and 0, twice costing a skill its pass.
+- A skill with `disable-model-invocation: true` cannot self-activate, so an eval graded on
+  activation compares two identical arms. Cover it through a consumer skill, or grade the answer
+  content instead (`tests/dotnet-test/filter-syntax/eval.yaml` is the one such eval here, and its
+  first real verdict is still outstanding).
+- A grader whose `config` is missing its required key enforces nothing, so the stimulus has one
+  fewer assertion than it appears to.
+
+### Step 7: Fix skill content against the losing trial
+
+Only now change the skill. Apply the patterns in
+[references/writing-for-baseline-delta.md](references/writing-for-baseline-delta.md); the ones that
+most often flip a loss:
+
+- Replace reference prose the model already knows with decisions it would otherwise get wrong.
+- Add stop-conditions so a strong skill does not over-apply — but do not over-correct into
+  answering more narrowly than the baseline did.
+- Scale output structure to input size; a dashboard for an 8-test suite loses to a direct answer.
+- Require truthful validation reporting; claiming "Build succeeded" after a failed restore is an
+  automatic loss.
+- Verify load-bearing API claims by compiling or probing, not by reading source.
+- For cost regressions, gate rare or expensive paths behind `references/` reads and size any
+  orchestration to the user's scope.
+
+### Step 8: Fix activation
+
+Activation failures are frontmatter and routing failures, not body failures. See
+[references/eval-triage.md](references/eval-triage.md). Summary:
+
+| Failure | Fix |
+|---------|-----|
+| Not activated in any arm | Put the user's own words in `description`: symptoms, error codes, artifact names, quoted requests |
+| A sibling skill wins the prompt | Claim the exact ambiguous words in `description`, and add matching exclusions on **both** siblings |
+| Model answers with no skill at all | Raise the stakes in the description, de-crowd the plugin menu, verify with the plugin arm |
+| Boundary excludes real scenarios | Re-read every "do not use for" clause against every eval prompt and real workflow phase |
+| Description at the 1,024-char ceiling | Cut restated body content, not trigger phrases; check the plugin menu budget too |
+
+### Step 9: Re-validate
+
+```bash
+dotnet run --project eng/skill-validator/src/SkillValidator.csproj -- check --plugin ./plugins/<plugin>
+python eng/eval-quality/check_eval_quality.py
+./eng/run-skill-evals.sh <plugin> <skill>
+```
+
+Then request the official run by submitting a PR review containing `/evaluate` (Files changed →
+Review changes), which binds the run to the reviewed commit. Before declaring a regression on the
+result, confirm the skill payload actually changed — reruns on byte-identical content have shifted
+7W/2T/2L to 4W/5T/2L.
+
+## Validation
+
+- [ ] A losing trial and the judge's stated reason are quoted in the PR description.
+- [ ] The failure was classified before any content was edited.
+- [ ] `check_eval_quality.py` and `skill-validator check` both pass.
+- [ ] Trial count clears the power bar for the observed tie rate, not just the floor of 5.
+- [ ] Isolated **and** plugin activation are both reported.
+- [ ] The PR body records root cause, fix, and validation so the lesson is reusable.
+
+## Common Pitfalls
+
+| Pitfall | Solution |
+|---------|----------|
+| Rewriting skill prose in response to an underpowered verdict | Underpowered means too few discordant trials; add discriminating stimuli instead |
+| Adding `defaults: runs:` to a spec that already has `config:` | Merge into a single `defaults:` block; vally rejects specs with both |
+| Padding `runs` to clear the trial floor | Five repeats of one stimulus measure one task; add stimuli |
+| Treating an errored trial as fixture nondeterminism | Read the stderr first; judge-side auth failures need harness fixes |
+| Fixing a "wrong" answer that the fixture actually made wrong | Check fixture self-consistency before blaming the response |
+| Strengthening a skill nobody uses and nothing passes | Weak eval signal plus thin telemetry is a valid retirement case |
+| Landing a fix without re-running | Verify the invoked payload contains the fix; judge noise is real |
+
+## References
+
+- [references/writing-for-baseline-delta.md](references/writing-for-baseline-delta.md) — content patterns that beat the unskilled model
+- [references/eval-triage.md](references/eval-triage.md) — symptom, cause and fix catalogue with PR citations
+- [eng/eval-quality/README.md](../../../eng/eval-quality/README.md) — the ten structural gate checks and why each exists
+- [eng/vally-adapter/InvestigatingResults.md](../../../eng/vally-adapter/InvestigatingResults.md) — downloading artifacts and reading `results.json`