feat: implement issue #1651 — [qa-lead S8] Fleet-wide: 11 of 12 eval sets fail case.schema.json and CI never notices

.github/workflows/lint.yml

@@ -254,7 +254,7 @@
     timeout-minutes: 5
     steps:
       - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
         # NOTE: persist-credentials is intentionally left at the default (true) —
         # vci_resolve_reusable needs `git fetch origin` to authenticate against this
         # private repo when resolving same-repo channel tags. Setting
         # persist-credentials: false would silently push every such caller into the
@@ -402,17 +402,16 @@
         run: python3 personas/validate-personas.py personas
 
   validate-eval-cases:
-    # Tree-wide per-case eval schema gate (#1645 AC #6). Per-case schema
-    # conformance was only ever checked in the validator's FILE mode, which
-    # nothing in CI invoked — so a non-conforming held-out set (qa-lead's) sat
-    # green for six weeks. This runs `validate-cases.py --schema-tree`, which
-    # validates every case in every split against evals/case.schema.json. It is
-    # deliberately SCOPED: SCHEMA_TREE_ALLOWLIST in the validator exempts the
-    # other 11 skills whose cases do not yet conform (that reconciliation is
-    # #1651's job), so the gate protects qa-lead now without failing every
-    # unrelated PR. When #1651 empties the allowlist, this gate covers the whole
-    # fleet unchanged. Fully HERMETIC (in-repo files + case.schema.json, no
-    # network). PR-triggered on the existing Lint workflow — no new cron.
+    # Tree-wide per-case eval schema gate (#1645 AC #6; fleet-wide since #1651).
+    # Per-case schema conformance was only ever checked in the validator's FILE
+    # mode, which nothing in CI invoked — so a non-conforming held-out set
+    # (qa-lead's) sat green for six weeks. This runs `validate-cases.py
+    # --schema-tree`, which validates every case in every split against its
+    # resolved schema: the per-skill evals/<skill>/case.schema.json when present,
+    # else the root evals/case.schema.json. #1651 removed the schema-tree allowlist,
+    # so the gate now covers the whole fleet with no exemptions. Fully HERMETIC
+    # (in-repo files + schemas, no network). PR-triggered on the existing Lint
+    # workflow — no new cron.
     runs-on: ubuntu-latest
     timeout-minutes: 5
     steps:

evals/README.md

@@ -26,13 +26,67 @@ evals/
   proposer must never tune a skill to these specific cases.
 
 `cases.jsonl` is [JSON Lines](https://jsonlines.org/): one JSON object per line.
-Blank lines are ignored. Every case **must** carry a non-empty string `id`. The
-rest of the case payload (e.g. `prompt`, `expected`, rubric fields) is owned by
-the scorer story (#583); this directory's validator is deliberately agnostic to
-it and only governs the split structure and `id` discipline.
-
-`example-skill/` is a template: copy its `dev/` and `holdout/` layout when adding
-a new skill's eval set.
+Blank lines are ignored. Every case **must** carry a non-empty string `id`.
+
+## The case-shape contract (per-skill schema, #1651)
+
+The case payload shape **is** governed — deliberately, per skill, not left
+agnostic. Two layers:
+
+- **The root schema, [`case.schema.json`](./case.schema.json), is the default
+  contract.** It fixes the shared *envelope* — `id` (kebab-case, unique), the
+  optional `description`/`tags`, and the pre-fetched `input` — and the triage
+  `{escalate, risk}` decision shape. `triage` and `qa-lead` are governed by it
+  directly.
+- **A skill whose decision shape legitimately differs ships its own
+  `evals/<skill>/case.schema.json`.** A persona that emits an ADR citation
+  (`solution-architect`), a risk tier (`devops-lead`/`security-lead`/`sre-lead`),
+  a readiness/decomposition/framing judgment (`dev-lead`/`scrum-master`/
+  `business-analyst`), or a structured deep-review verdict
+  (`deep-review`; `pr-review` carries both the triage and deep shapes) declares
+  that shape explicitly. Every per-skill schema keeps the root envelope (same
+  `id` pattern, `description`/`tags` constraints, `additionalProperties: false`
+  at the root **and** on `expected`) and names its `expected` fields explicitly —
+  it never merely widens `additionalProperties`, which would assert nothing.
+
+`validate-cases.py --schema-tree` resolves each skill against its own
+`case.schema.json` when present, else the root schema, and validates **every**
+case in **every** split. There is no allowlist: per-case conformance is enforced
+fleet-wide. This is the resolution of the earlier schema-vs-README contradiction,
+where the schema hardcoded triage's shape while this file claimed payload shape
+was ungoverned.
+
+`spec-drift` is a special case: it is an offline structured detector, not an
+advisory prompt, so its case carries a `(diff, acceptance_criteria, analysis,
+expected.verdict)` shape and is **exempt from the `input` pre-fetched-context
+contract** below. It is governed by its own `evals/spec-drift/case.schema.json`.
+
+`example-skill/` is a template: copy its `dev/` and `holdout/` layout — and, when
+a new skill's decision shape differs from the root, its `case.schema.json` — when
+adding a new skill's eval set.
+
+## Scorer mode and judge conventions (#1651 AC #5/#6)
+
+Every skill with a runnable scorer path declares its scorer mode explicitly in
+`evals/<skill>/scorer.json`, rather than silently inheriting the `deterministic`
+default (correct only for triage-shaped boolean decisions, wrong for prose
+skills). See [`scripts/evals/run-eval.sh`](../scripts/evals/run-eval.sh) for the
+`scorer.json` contract.
+
+- `triage` declares `deterministic` explicitly; every prose-output persona
+  declares `llm-judge`.
+- **Judge convention: per-skill.** A prose-output persona ships its own
+  `evals/<skill>/judge.md` (pointed at by `scorer.json`'s `judge_prompt`), because
+  the shared `evals/judge.md` is written for deep-review's structured
+  `decision`/`risk`/`key_findings` verdict and scores a prose advisory poorly.
+  `deep-review` legitimately points at the shared `evals/judge.md` because it
+  emits exactly that shape.
+- `spec-drift` is scored by its **own** offline tooling
+  (`evals/spec-drift/run-eval.sh`), not the shared `scripts/evals/run-eval.sh`
+  deterministic comparator (which compares `escalate`/`risk`, a shape spec-drift
+  does not use); it therefore carries no shared `scorer.json`.
+- `example-skill` has **no runnable scorer path** — it is a copy-me template and
+  is never scored.
 
 ## The no-overlap rule
 
@@ -45,13 +99,17 @@ test").
 `validate-cases.py` enforces this. Run it over the whole tree:
 
 ```bash
-python3 evals/validate-cases.py            # validates evals/ (this dir)
-python3 evals/validate-cases.py <root>     # validates an alternate root
+python3 evals/validate-cases.py                 # split hygiene over evals/ (this dir)
+python3 evals/validate-cases.py <root>          # split hygiene over an alternate root
+python3 evals/validate-cases.py --schema-tree evals  # + per-case schema conformance
 ```
 
 It fails (non-zero) on malformed JSONL, a missing/empty `id`, a duplicate `id`
 within a split, an `id` shared across splits, or a skill missing either split.
-The validator is exercised by `tests/test_validate_cases.bats` in CI.
+`--schema-tree` additionally validates every case against its resolved schema
+(per-skill `case.schema.json` when present, else the root). The validator is
+exercised by `tests/test_validate_cases.bats` and the `validate-eval-cases` job
+in `.github/workflows/lint.yml`.
 
 ## The advisory context contract (#1686)
 

evals/business-analyst/case.schema.json

@@ -0,0 +1,55 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "title": "Business-Analyst Eval Case",
+  "description": "One held-out eval case for the business-analyst advisory persona: an idea/opportunity input paired with its expected framing judgment and next step. Each line of evals/business-analyst/<split>/cases.jsonl is one object of this shape. The persona judges whether an idea is well framed and routes it to the right next activity, so the expected decision is analyst-shaped (framing / next / recommend) — NOT the triage {escalate, risk} shape of evals/case.schema.json. This is the trust anchor for the persona's skill-quality measurement and is CODEOWNER-gated separately from any scorer/proposer.",
+  "type": "object",
+  "additionalProperties": false,
+  "required": ["id", "input", "expected"],
+  "properties": {
+    "id": {
+      "type": "string",
+      "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$",
+      "minLength": 3,
+      "description": "Stable, unique kebab-case identifier for the case. Unique within a split and disjoint across dev/holdout (enforced by evals/validate-cases.py)."
+    },
+    "description": {
+      "type": "string",
+      "minLength": 5,
+      "description": "Human-readable note on what this case exercises and what grounds the expected framing judgment."
+    },
+    "tags": {
+      "type": "array",
+      "items": { "type": "string", "minLength": 2 },
+      "uniqueItems": true,
+      "description": "Optional labels for coverage tracking, e.g. the analyst activity a case exercises ('framing', 'market-research', 'brief', 'negative')."
+    },
+    "input": {
+      "type": "string",
+      "minLength": 1,
+      "description": "The pre-fetched review context exactly as the advisory consumes it: a self-contained description of the idea/opportunity under analysis."
+    },
+    "expected": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["framing", "next", "recommend"],
+      "description": "The fixed, reward-hack-resistant expected analyst judgment the persona must produce for this idea.",
+      "properties": {
+        "framing": {
+          "type": "string",
+          "enum": ["clear", "fuzzy"],
+          "description": "Whether the idea is framed around a real user problem ('clear') or is still solution-first / underspecified ('fuzzy')."
+        },
+        "next": {
+          "type": "string",
+          "enum": ["brainstorm", "market-research", "brief"],
+          "description": "The next analyst activity the idea should route to: reframe/brainstorm, market-research, or write a decision-grade brief."
+        },
+        "recommend": {
+          "type": "string",
+          "minLength": 1,
+          "description": "The concrete recommendation the advisory must land (what to do before the next stage)."
+        }
+      }
+    }
+  }
+}

evals/deep-review/case.schema.json

@@ -0,0 +1,72 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "title": "Deep-Review Eval Case",
+  "description": "One held-out eval case for the deep-review skill: a PR-context input paired with its expected deep-tier verdict. Each line of evals/deep-review/<split>/cases.jsonl is one object of this shape. Deep review emits a structured verdict — a decision, a risk classification, whether to escalate to the Opus security-audit tier, and grounded key findings — so the expected decision is verdict-shaped (decision / risk / escalate_to_opus / key_findings) — NOT the triage {escalate, risk} shape of evals/case.schema.json. This is the trust anchor for the skill's quality measurement and is CODEOWNER-gated separately from any scorer/proposer.",
+  "type": "object",
+  "additionalProperties": false,
+  "required": ["id", "input", "expected"],
+  "properties": {
+    "id": {
+      "type": "string",
+      "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$",
+      "minLength": 3,
+      "description": "Stable, unique kebab-case identifier for the case. Unique within a split and disjoint across dev/holdout (enforced by evals/validate-cases.py)."
+    },
+    "description": {
+      "type": "string",
+      "minLength": 5,
+      "description": "Human-readable note on what this case exercises and which shared.md/deep-review.md decision gate grounds the expected verdict."
+    },
+    "tags": {
+      "type": "array",
+      "items": { "type": "string", "minLength": 2 },
+      "uniqueItems": true,
+      "description": "Optional labels for coverage tracking, e.g. the high-risk trigger a case exercises ('auth-secrets', 'db-migration', 'security-anti-pattern', 'unresolved-threads')."
+    },
+    "input": {
+      "type": "string",
+      "minLength": 1,
+      "description": "The pre-fetched PR context exactly as deep-review consumes it: title, files changed, diff excerpt, gate signals, and the upstream triage result."
+    },
+    "expected": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["decision", "risk", "escalate_to_opus", "key_findings"],
+      "description": "The fixed, reward-hack-resistant expected deep-tier verdict the skill must produce for this PR.",
+      "properties": {
+        "decision": {
+          "type": "string",
+          "enum": ["approve", "escalate"],
+          "description": "Whether deep review approves the change or escalates it."
+        },
+        "risk": {
+          "type": "string",
+          "enum": ["LOW", "MEDIUM", "HIGH"],
+          "description": "Expected risk classification. Any HIGH signal must both escalate and escalate to the Opus security-audit tier."
+        },
+        "escalate_to_opus": {
+          "type": "boolean",
+          "description": "Whether the change must escalate to the Opus security-audit tier. HIGH-risk changes always do; a MEDIUM-risk gate failure escalates on deep review's own findings without the security-audit tier."
+        },
+        "key_findings": {
+          "type": "array",
+          "items": { "type": "string", "minLength": 1 },
+          "minItems": 1,
+          "uniqueItems": true,
+          "description": "The grounded findings the verdict rests on. At least one, each a non-empty statement tied to a decision gate or high-risk area."
+        }
+      },
+      "if": {
+        "properties": {
+          "risk": { "const": "HIGH" }
+        }
+      },
+      "then": {
+        "properties": {
+          "decision": { "const": "escalate" },
+          "escalate_to_opus": { "const": true }
+        }
+      }
+    }
+  }
+}

evals/dev-lead/case.schema.json

@@ -0,0 +1,50 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "title": "Dev-Lead Eval Case",
+  "description": "One held-out eval case for the dev-lead advisory persona: an issue/PR-readiness input paired with its expected readiness judgment. Each line of evals/dev-lead/<split>/cases.jsonl is one object of this shape. The persona judges whether work is ready to implement and lands a concrete recommendation, so the expected decision is readiness-shaped (readiness / recommend) — NOT the triage {escalate, risk} shape of evals/case.schema.json. This is the trust anchor for the persona's skill-quality measurement and is CODEOWNER-gated separately from any scorer/proposer.",
+  "type": "object",
+  "additionalProperties": false,
+  "required": ["id", "input", "expected"],
+  "properties": {
+    "id": {
+      "type": "string",
+      "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$",
+      "minLength": 3,
+      "description": "Stable, unique kebab-case identifier for the case. Unique within a split and disjoint across dev/holdout (enforced by evals/validate-cases.py)."
+    },
+    "description": {
+      "type": "string",
+      "minLength": 5,
+      "description": "Human-readable note on what this case exercises and what grounds the expected readiness judgment."
+    },
+    "tags": {
+      "type": "array",
+      "items": { "type": "string", "minLength": 2 },
+      "uniqueItems": true,
+      "description": "Optional labels for coverage tracking, e.g. the readiness concern a case exercises ('scoping', 'risk', 'readiness', 'negative')."
+    },
+    "input": {
+      "type": "string",
+      "minLength": 1,
+      "description": "The pre-fetched review context exactly as the advisory consumes it: a self-contained description of the issue or PR whose readiness is under review."
+    },
+    "expected": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["readiness", "recommend"],
+      "description": "The fixed, reward-hack-resistant expected readiness judgment the persona must produce for this work item.",
+      "properties": {
+        "readiness": {
+          "type": "string",
+          "enum": ["ready", "needs-scoping"],
+          "description": "Whether the work is ready to implement as-is ('ready') or must be scoped/specified first ('needs-scoping')."
+        },
+        "recommend": {
+          "type": "string",
+          "minLength": 1,
+          "description": "The concrete recommendation the advisory must land (what to do before or during implementation)."
+        }
+      }
+    }
+  }
+}

evals/devops-lead/case.schema.json

@@ -0,0 +1,64 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "title": "DevOps-Lead Eval Case",
+  "description": "One held-out eval case for the devops-lead advisory persona: an infrastructure/pipeline-review input paired with its expected risk-tiered advisory. Each line of evals/devops-lead/<split>/cases.jsonl is one object of this shape. The persona classifies a change's operational risk and lands a concrete recommendation, so the expected decision is risk-tiered (risk_tier / escalate / recommend) — NOT the triage {escalate, risk} shape of evals/case.schema.json. This is the trust anchor for the persona's skill-quality measurement and is CODEOWNER-gated separately from any scorer/proposer.",
+  "type": "object",
+  "additionalProperties": false,
+  "required": ["id", "input", "expected"],
+  "properties": {
+    "id": {
+      "type": "string",
+      "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$",
+      "minLength": 3,
+      "description": "Stable, unique kebab-case identifier for the case. Unique within a split and disjoint across dev/holdout (enforced by evals/validate-cases.py)."
+    },
+    "description": {
+      "type": "string",
+      "minLength": 5,
+      "description": "Human-readable note on what this case exercises and which devops principle grounds the expected decision."
+    },
+    "tags": {
+      "type": "array",
+      "items": { "type": "string", "minLength": 2 },
+      "uniqueItems": true,
+      "description": "Optional labels for coverage tracking, e.g. the anti-pattern a case exercises ('pipeline-bypass', 'environment-isolation', 'negative')."
+    },
+    "input": {
+      "type": "string",
+      "minLength": 1,
+      "description": "The pre-fetched review context exactly as the advisory consumes it: a self-contained description of the proposed infrastructure/pipeline change under review."
+    },
+    "expected": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["risk_tier", "escalate", "recommend"],
+      "description": "The fixed, reward-hack-resistant expected operational advisory the persona must produce for this change.",
+      "properties": {
+        "risk_tier": {
+          "type": "string",
+          "enum": ["LOW", "MEDIUM", "HIGH"],
+          "description": "Expected operational risk classification. Any HIGH signal must pair with escalate=true."
+        },
+        "escalate": {
+          "type": "boolean",
+          "description": "Whether the advisory must flag the change for escalation (true) or may say no escalation is warranted (false)."
+        },
+        "recommend": {
+          "type": "string",
+          "minLength": 1,
+          "description": "The operational recommendation the advisory must land, grounded in the devops principle the case exercises."
+        }
+      },
+      "if": {
+        "properties": {
+          "risk_tier": { "const": "HIGH" }
+        }
+      },
+      "then": {
+        "properties": {
+          "escalate": { "const": true }
+        }
+      }
+    }
+  }
+}