diff --git a/.github/workflows/eval-quality.yml b/.github/workflows/eval-quality.yml new file mode 100644 index 0000000000..016835550a --- /dev/null +++ b/.github/workflows/eval-quality.yml @@ -0,0 +1,63 @@ +name: eval-quality + +# Structural quality gate for evaluation specs and their fixtures. +# +# Each failing check corresponds to a defect that has already cost a real +# evaluation result on this repo — see eng/eval-quality/README.md. All of them +# are structural (file existence, git state, YAML keys), so the gate cannot +# fire spuriously on well-written prose. Judgement calls such as statistical +# power and orphaned fixtures are reported but never fail the build. + +on: + pull_request: + paths: + - "tests/**" + - "plugins/**" + - "eng/eval-quality/**" + - ".github/workflows/eval-quality.yml" + - ".gitignore" + push: + branches: [main] + # Kept in sync with the pull_request paths above. The gate reads plugins/* + # (skills with no eval) and .gitignore (fixtures excluded from the index), + # so omitting them here would let a direct push or a squash-merge that + # touches only those land on main without the gate ever running. + paths: + - "tests/**" + - "plugins/**" + - "eng/eval-quality/**" + - ".github/workflows/eval-quality.yml" + - ".gitignore" + workflow_dispatch: + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +permissions: + contents: read + +jobs: + check: + runs-on: ubuntu-latest + steps: + - name: Checkout repository + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v6 + with: + persist-credentials: false + + - name: Set up Python + uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5 + with: + python-version: "3.12" + + - name: Install PyYAML + run: python -m pip install --quiet pyyaml + + # The gate is only trustworthy if it has been shown to fire. This injects + # each defect into a scratch tree and asserts the gate rejects it. + - name: Self-test the gate + run: python eng/eval-quality/selftest_eval_quality.py + + - name: Check eval quality + run: python eng/eval-quality/check_eval_quality.py diff --git a/eng/eval-quality/README.md b/eng/eval-quality/README.md new file mode 100644 index 0000000000..2eb0af056d --- /dev/null +++ b/eng/eval-quality/README.md @@ -0,0 +1,198 @@ +# Eval quality gate + +`check_eval_quality.py` blocks defect classes that have each already cost a real +evaluation result on this repo. Every one of them was invisible to the existing +checks: the eval specs parsed, `skill-validator` passed, and the damage only +showed up as a skill mysteriously losing to its own baseline. + +Run it from the repository root: + +```bash +python eng/eval-quality/check_eval_quality.py # what CI runs +python eng/eval-quality/check_eval_quality.py --strict # also fail on warnings +python eng/eval-quality/selftest_eval_quality.py # prove the gate still fires +``` + +## Failing checks + +All six are **structural** — they inspect file existence, git state, declared +numbers, or YAML keys. None of them interprets prose, so they cannot fire +spuriously on a well-written eval. + +### 1. Referenced fixture missing on disk + +A stimulus points at a fixture path that does not exist. The scenario fails at +setup, which reads as a skill failure. + +### 2. Referenced fixture not tracked by git + +The fixture exists locally but is not in the index, so it will not exist on the +CI runner. + +This is the subtle one. `.gitignore` carries `coverage*.xml` (a sensible rule +for Coverlet output), which silently swallowed a committed Cobertura *fixture*. +`git add -A` reported success, the eval passed locally, and three scenarios +would have failed at setup in CI. Verifying against the working tree cannot +catch it — only the git index can. + +"In the index" means `git ls-files` alone. An earlier revision also unioned in +`git diff --cached --name-only`, which is worse than redundant: a fixture staged +for removal but left on disk appears there and would be counted back as tracked, +producing a false negative for exactly this bug class. The self-test commits +before mutating so that path is genuinely exercised — without the commit there +is no `HEAD`, `git diff --cached` errors out, and the defect stays hidden. + +### 3. Cobertura `line-rate` contradicts its own `` + +The `crap-score` skill documents both parse paths: + +> Parse the Cobertura XML to find each method's `line-rate` attribute … **If +> `line-rate` is not available at method level, compute it from the `` +> elements.** + +So when the two disagree, the baseline and skilled arms can legitimately read +*different coverage inputs* for the same method and compute different CRAP +scores. The comparison then measures which number the judge happened to treat +as authoritative rather than the skill. + +Observed live: a scenario lost −40% with the judge writing *"Response B made a +critical error by manually counting line hits (12/15 = 80%) instead of using +the XML's recorded line-rate of 0.55"*. The fixture was wrong, not the response. + +When fixing one of these, the **declared rate is normally the intent** — the +rubrics are written against it (which method is the risk hotspot) — so adjust +the `` data to match, then re-derive any rubric item that quotes a +coverage percentage, a CRAP score, or a "coverage needed" figure. + +### 4. Whole-file Cobertura totals contradict the file `line-rate` + +The same split-brain, one level up. A report also carries file-level summary +attributes, and those are a third way to read the same number: + +```xml + +``` + +`0.47` agreed with the per-method `` (22/47 = 0.468); `35/60` is 58.3%. +A skill reading the summary attributes and one recomputing from the payload +therefore disagreed by 11 points on the same fixture. Found in review on +`coverage-analysis/partial-coverage` after check 3 had already been applied — +the method-level check alone could not see it, because every individual method +was self-consistent. + +This compares two *declared* values, so it cannot fire on well-formed input. +Fix it by making the totals agree with both the declared rate and the summed +`` (here, `lines-covered="22" lines-valid="47"`) rather than only with +the rate — that leaves one number for every reader. The same applies to +`branches-covered`/`branches-valid` against `branch-rate`. + +### 5. Grader with a missing or empty required config + +A grader whose `config` is absent, null, or missing its required key +(`pattern`, `substring`, `command`, `path`) parses as valid YAML and **enforces +nothing**. The scenario looks like it has one more assertion than it really has. + +The failure mode is an indentation slip, usually from an edit: + +```yaml + - type: output-matches + config: # <- pattern belongs here + - type: output-matches # <- and ended up on the next list item + config: + pattern: \d+ call sites +``` + +Observed live on this repo: a grader-regex fix left the original +`- type: output-matches` / `config:` pair behind, producing a fourth grader with +`config: null` that shipped in a pushed commit. Neither YAML parsing nor a +bespoke regex validator caught it — the validator did +`(g.get("config") or {}).get("pattern")` and silently skipped the entry, so the +pattern count was identical before and after the fix. Only review caught it. + +### 6. Dormancy guard that also sets `reject_skills` + +A dormancy guard is a stimulus with `expect_activation: false`: an off-target +request where the skill should stay dormant rather than hijack the task. + +Adding `constraints.reject_skills: ["*"]` forces the skilled arm to run +skill-free — which makes it **identical to the baseline arm**. The head-to-head +score is then pure judge noise. Across four evals using this pattern the same +guard scored −0.4, +0.4, +0.4 and 0, and twice cost a skill its pass. + +The repo convention is `expect_activation: false` **alone** (see +`agent.test-quality-auditor`, `agent.test-migration`, +`system-text-json-net11`), so the skill is actually loaded and the guard +measures the real property. + +## Warnings (reported, never failing) + +### Aggregate `line-rate` vs the lines beneath it + +A file, package or class whose declared `line-rate` disagrees with the `` +elements underneath it. This is a warning rather than an error because a real +coverage report may legitimately summarise more than it enumerates, and because +the correct fix sometimes reaches into the scenario itself. + +Live example: `coverage-analysis/fixtures/plateau` declares 75% while its +`` imply 47%, and the scenario prompt says *"my coverage is stuck at +75%"*. It cannot simply be recomputed — `CalculateGpa` contributes 24 lines at +0% coverage and the rubric requires it to stay the 0% blocker, which caps the +achievable rate at 23/47 = 48.9%. Making the payload true would mean rewriting +the fixture and the prompt together, so the gate reports it and leaves the +judgement to a human. + +### Statistical power + +`dotnet-skills.experiment.yaml` sets `runs: 1`, so `n` is the scenario count and +one judge call decides each scenario. The pass gate is `mean > 0 ∧ ci_low > 0`, +i.e. + +``` +sqrt(n) × (mean / sd) > t(n-1) +``` + +| n | required mean/sd | +| ---: | ---: | +| 1 | undefined — a single trial decides | +| 2 | 3.04 | +| 3 | 1.84 | +| 4 | 1.39 | +| 6 | 1.00 | +| 8 | 0.82 | + +Consequences seen in practice: `coverage-analysis` **won 100% of its trials in +four consecutive runs and failed all four**; `migrate-static-to-wrapper` missed +by 0.4 of a percentage point at 4W/1T/0L. Neither is a content problem. + +Roughly half of the repo's skill evals sit at n ≤ 3. Raising `runs` is the +durable fix (`runs: 3` turns 3 scenarios into 9 trials and drops the required +ratio to 0.77) at a proportional increase in CI cost — a maintainer decision, +which is why this is a warning and not a failure. + +### Orphaned fixtures + +A fixture directory that is committed but that no stimulus references. Usually +means a scenario was planned and dropped, so the coverage it was built for is +being paid for in repo size but never exercised. Wiring these up is the cheapest +way to raise `n`. + +### Skills with no eval + +A skill that ships with `SKILL.md` but has no `tests///eval.yaml` +carries zero evidence of impact. + +### Dormancy guard without an anti-hijack rubric item + +Once `reject_skills` is removed the skill loads, so the judge scores the guard +against its rubric. If that rubric only says "wrote tests", the judge has +nothing to grade the real property with and falls back to comparing **output +volume** between two near-identical runs — which is exactly how a passing skill +regressed to a −40% loss on its own guard. + +Add an explicit criterion, e.g. *"Did not derail into a mutation analysis of +code the user never asked about"*, plus one instructing the judge not to reward +raw test count. + +This check is a warning rather than an error because detecting it requires +phrase matching over free text and will always have false positives — a gate +that blocks a PR spuriously is a gate the team switches off. diff --git a/eng/eval-quality/check_eval_quality.py b/eng/eval-quality/check_eval_quality.py new file mode 100644 index 0000000000..031542a7f9 --- /dev/null +++ b/eng/eval-quality/check_eval_quality.py @@ -0,0 +1,327 @@ +#!/usr/bin/env python3 +"""Eval quality gate. + +Codifies defect classes that have each cost a real evaluation result, so they +cannot silently recur in any plugin. + +FAILS on unambiguous bugs: + 1. A stimulus references a fixture that is missing on disk. + 2. A stimulus references a fixture that exists but is NOT tracked by git. + `.gitignore` once silently swallowed a Cobertura fixture: the scenarios + passed locally and would have failed at setup in CI. + 3. A Cobertura fixture whose declared `line-rate` contradicts its own + `` data. The crap-score skill documents both parse paths, so the + two arms of a comparison can legitimately read different inputs and the + eval measures the disagreement instead of the skill. + 4. A dormancy guard (`expect_activation: false`) that also sets + `reject_skills`. That forces the skilled arm skill-free, making it + identical to the baseline arm, so the score is judge noise. + +Every failing check above is structural — it inspects file existence, git +state, or YAML keys — so it cannot fire spuriously on well-written content. + +REPORTS (does not fail) pre-existing debt and judgement calls: statistical +power, orphaned fixtures, skills with no eval, and dormancy guards that appear +to lack an anti-hijack rubric item. That last one is deliberately a warning: +detecting "the rubric says the skill should stay dormant" needs phrase +matching, which will always have false positives, and a gate that blocks a PR +spuriously is a gate the team turns off. + +Usage: python eng/eval-quality/check_eval_quality.py [--strict] +""" +from __future__ import annotations + +import argparse +import glob +import math +import os +import subprocess +import sys +import xml.etree.ElementTree as ET + +try: + import yaml +except ImportError: # pragma: no cover + print("PyYAML is required: pip install pyyaml", file=sys.stderr) + raise SystemExit(2) + +T95 = {2: 4.303, 3: 3.182, 4: 2.776, 5: 2.571, 6: 2.447, 7: 2.365, 8: 2.306, + 9: 2.262, 10: 2.228, 11: 2.201, 12: 2.179, 13: 2.160, 14: 2.145, 15: 2.131} + +ANTI_HIJACK = ("derail", "did not attempt", "outside the scope", "out of scope", + "did not perform", "declined", "does not load", "does not reference", + "not load or reference", "none of its apis", "not needed here", + "did not apply", "stayed dormant", "without using the skill") + +# Grader types whose config carries a required key. A grader of one of these +# types with that key absent parses fine and enforces nothing. +GRADER_REQUIRED_KEY = { + "output-matches": "pattern", + "output-not-matches": "pattern", + "output-contains": "substring", + "output-not-contains": "substring", + "run-command": "command", + "file-exists": "path", +} + +errors: list[str] = [] +warnings: list[str] = [] + + +def git_tracked_files() -> set[str]: + # `git ls-files` reports the index, which already includes newly staged + # additions. Unioning in `git diff --cached --name-only` as well looked + # harmless but was actively wrong: a file staged for removal (`git rm + # --cached`, left on disk) shows up there and would be counted back as + # "tracked", the exact false negative the untracked-fixture check exists + # to catch. The self-test now commits before mutating, so this path is + # genuinely exercised. + try: + res = subprocess.run(["git", "ls-files"], capture_output=True, text=True, check=True) + except (subprocess.CalledProcessError, FileNotFoundError): + return set() + return set(res.stdout.splitlines()) + + +def files_under(path: str) -> list[str]: + if os.path.isfile(path): + return [path.replace(os.sep, "/")] + return [os.path.join(dp, f).replace(os.sep, "/") + for dp, _, fn in os.walk(path) for f in fn] + + +def check_fixtures(spec: str, doc: dict, tracked: set[str]) -> None: + base = os.path.dirname(spec) + for stim in doc.get("stimuli") or []: + for entry in (stim.get("environment") or {}).get("files") or []: + src = entry.get("src") + if not src: + continue + resolved = os.path.normpath(os.path.join(base, src)) + if not os.path.exists(resolved): + errors.append(f"{spec}: '{stim.get('name')}' references missing fixture {src}") + continue + untracked = [f for f in files_under(resolved) if f not in tracked] + if untracked: + errors.append( + f"{spec}: '{stim.get('name')}' references fixture files not tracked by git " + f"(they will not exist in CI): {untracked[:3]}") + + +def check_graders(spec: str, doc: dict) -> None: + """A grader whose config is missing its required key silently does nothing. + + The document still parses, so YAML validation is clean and the scenario + looks like it has one more assertion than it really enforces. Observed + live: an edit left `- type: output-matches` / `config:` with the pattern + attached to the next list item, producing a grader with `config: null` + that was invisible to both YAML parsing and a bespoke regex validator + (which did `(g.get("config") or {}).get("pattern")` and skipped it). + """ + for stim in doc.get("stimuli") or []: + for i, g in enumerate(stim.get("graders") or []): + if not isinstance(g, dict): + errors.append(f"{spec}: '{stim.get('name')}' grader[{i}] is not a mapping") + continue + need = GRADER_REQUIRED_KEY.get(g.get("type")) + if need is None: + continue # unknown or config-less grader type + cfg = g.get("config") + if not isinstance(cfg, dict): + errors.append( + f"{spec}: '{stim.get('name')}' grader[{i}] ({g.get('type')}) has no " + f"config; it silently enforces nothing. Check the indentation of the " + f"'{need}:' line.") + elif cfg.get(need) in (None, ""): + errors.append( + f"{spec}: '{stim.get('name')}' grader[{i}] ({g.get('type')}) is missing " + f"config.{need}; it silently enforces nothing") + + +def check_dormancy_guards(spec: str, doc: dict) -> None: + for stim in doc.get("stimuli") or []: + if stim.get("expect_activation") is not False: + continue + name = stim.get("name") + if (stim.get("constraints") or {}).get("reject_skills"): + errors.append( + f"{spec}: dormancy guard '{name}' also sets reject_skills; that makes the " + f"skilled arm identical to the baseline arm, so the score is judge noise") + rubric = " ".join(str(r) for r in (stim.get("rubric") or [])).lower() + if not any(p in rubric for p in ANTI_HIJACK): + # Warning, not an error: this is phrase matching over free text, so a + # legitimately-worded rubric can trip it. Blocking a PR on a heuristic + # is how gates get switched off. + warnings.append( + f"{spec}: dormancy guard '{name}' may lack an anti-hijack rubric item. Without " + f"one the judge scores it on output volume instead of on the skill staying " + f"dormant. Ignore if the rubric already asserts this in other words.") + + +def _payload(el) -> tuple[int, int]: + """(covered, total) implied by the elements beneath an element.""" + lines = list(el.iter("line")) + return sum(1 for ln in lines if int(ln.get("hits", "0")) > 0), len(lines) + + +def check_cobertura() -> None: + for path in sorted(glob.glob("tests/**/coverage*.xml", recursive=True)): + try: + tree = ET.parse(path) + except ET.ParseError as exc: + errors.append(f"{path}: not parseable as XML ({exc})") + continue + for cls in tree.iter("class"): + for m in cls.iter("method"): + covered, total = _payload(m) + if not total: + continue + actual = covered / total + declared = float(m.get("line-rate", "0")) + if abs(actual - declared) >= 0.011: + errors.append( + f"{path}: method '{m.get('name')}' declares line-rate={declared:.2f} but " + f"its imply {actual:.2f} ({covered}/{total}); a skill that " + f"recomputes from reads a different input than one that trusts " + f"the attribute") + + # The whole-file summary attributes are a third way to read the same + # number, and they were the ones that disagreed in practice. This is a + # comparison of two declared values, so it cannot fire spuriously. + root = tree.getroot() + for rate_attr, num, den, unit in ( + ("line-rate", "lines-covered", "lines-valid", "line"), + ("branch-rate", "branches-covered", "branches-valid", "branch"), + ): + if root.get(num) is None or root.get(den) is None or root.get(rate_attr) is None: + continue + valid = int(root.get(den)) + if valid <= 0: + continue + summary = int(root.get(num)) / valid + declared = float(root.get(rate_attr)) + if abs(summary - declared) >= 0.011: + errors.append( + f"{path}: file-level {rate_attr}={declared:.2f} but {num}/{den} = " + f"{root.get(num)}/{root.get(den)} = {summary:.2f}; the report states two " + f"different whole-file {unit} coverage numbers, so the arms disagree " + f"depending on which attribute a skill happens to read") + + # Aggregates vs the underlying payload. Reported rather than failed: + # a real report may legitimately summarise more than it enumerates, + # and forcing a rewrite of a scenario whose prompt quotes the declared + # figure is a bigger change than this check should compel. + for el, label in ( + [(tree.getroot(), "file")] + + [(p, f"package '{p.get('name')}'") for p in tree.iter("package")] + + [(c, f"class '{c.get('name')}'") for c in tree.iter("class")] + ): + covered, total = _payload(el) + declared = el.get("line-rate") + if not total or declared is None: + continue + if abs(covered / total - float(declared)) >= 0.011: + warnings.append( + f"{path}: {label} declares line-rate={float(declared):.2f} but the " + f" beneath it imply {covered / total:.2f} ({covered}/{total})") + + +def report_power(specs: list[str]) -> None: + thin = [] + for spec in specs: + with open(spec, encoding="utf-8") as fh: + doc = yaml.safe_load(fh) or {} + n = len(doc.get("stimuli") or []) + if n <= 3: + need = T95.get(n, 1.96) / math.sqrt(n) if n >= 2 else float("inf") + thin.append((n, need, spec)) + if not thin: + return + warnings.append( + f"{len(thin)} eval(s) have n<=3 scenarios. With runs=1 the pass gate needs " + f"mean/sd > t(n-1)/sqrt(n), so these can fail while winning every trial:") + for n, need, spec in sorted(thin): + need_s = "inf" if math.isinf(need) else f"{need:.2f}" + warnings.append(f" n={n} needs mean/sd > {need_s:>4} {spec}") + + +def report_orphans(specs: list[str]) -> None: + found = [] + for spec in specs: + fx = os.path.join(os.path.dirname(spec), "fixtures") + if not os.path.isdir(fx): + continue + with open(spec, encoding="utf-8") as fh: + raw = fh.read() + found += [f"{spec}: fixture '{n}' is committed but no stimulus references it" + for n in sorted(os.listdir(fx)) + if os.path.isdir(os.path.join(fx, n)) and n not in raw] + if found: + warnings.append(f"{len(found)} orphaned fixture(s) (committed but unused):") + warnings.extend(f" {f}" for f in found) + + +def report_uncovered() -> None: + missing = [] + for plugin_dir in sorted(glob.glob("plugins/*")): + plugin = os.path.basename(plugin_dir) + evals = {os.path.basename(os.path.dirname(f)) + for f in glob.glob(f"tests/{plugin}/*/eval.yaml")} + for skill_dir in sorted(glob.glob(f"{plugin_dir}/skills/*")): + skill = os.path.basename(skill_dir) + if os.path.isdir(skill_dir) and skill not in evals: + missing.append(f" {plugin}/{skill}") + if missing: + warnings.append(f"{len(missing)} skill(s) have no eval at all:") + warnings.extend(missing) + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--strict", action="store_true", help="treat warnings as failures") + args = ap.parse_args() + + specs = sorted(glob.glob("tests/*/*/eval.yaml")) + if not specs: + print("No eval specs found — run from the repository root.", file=sys.stderr) + return 2 + + tracked = git_tracked_files() + for spec in specs: + try: + with open(spec, encoding="utf-8") as fh: + doc = yaml.safe_load(fh) or {} + except yaml.YAMLError as exc: + errors.append(f"{spec}: YAML parse error: {exc}") + continue + check_fixtures(spec, doc, tracked) + check_graders(spec, doc) + check_dormancy_guards(spec, doc) + + check_cobertura() + report_power(specs) + report_orphans(specs) + report_uncovered() + + print(f"Eval quality gate — checked {len(specs)} eval spec(s).\n") + if warnings: + print("WARNINGS (reported, not failing):") + for w in warnings: + print(f" {w}") + print() + if errors: + print("ERRORS:") + for e in errors: + print(f" {e}") + print(f"\n{len(errors)} error(s). See eng/eval-quality/README.md for why each is a bug.") + return 1 + + print("No errors.") + if warnings and args.strict: + print("--strict: failing on warnings.") + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/eng/eval-quality/selftest_eval_quality.py b/eng/eval-quality/selftest_eval_quality.py new file mode 100644 index 0000000000..48e296a558 --- /dev/null +++ b/eng/eval-quality/selftest_eval_quality.py @@ -0,0 +1,179 @@ +"""Prove the eval quality gate catches each bug class it claims to. + +Injects each defect into a scratch copy of a real eval, runs the gate, and +asserts it fails; then restores and asserts it passes. Without this the gate +is just a script that has never been shown to fire. +""" +import os +import shutil +import subprocess +import sys +import tempfile + +REPO = os.getcwd() +GATE = os.path.join(REPO, "eng", "eval-quality", "check_eval_quality.py") + + +def run_gate(cwd): + r = subprocess.run([sys.executable, GATE], cwd=cwd, capture_output=True, text=True) + return r.returncode, r.stdout + r.stderr + + +def scratch(): + """A minimal repo-shaped tree the gate can scan.""" + d = tempfile.mkdtemp() + ev = os.path.join(d, "tests", "demo", "widget") + os.makedirs(os.path.join(ev, "fixtures", "sample")) + os.makedirs(os.path.join(d, "plugins", "demo", "skills", "widget")) + with open(os.path.join(ev, "fixtures", "sample", "Thing.cs"), "w") as f: + f.write("class Thing {}\n") + with open(os.path.join(ev, "eval.yaml"), "w") as f: + f.write( + "name: widget\n" + "stimuli:\n" + " - name: Does the thing\n" + " prompt: do it\n" + " environment:\n" + " files:\n" + " - src: fixtures/sample\n" + " dest: sample\n" + " rubric:\n" + " - Did the thing\n" + ) + # Make everything git-tracked so the tracked-files check is satisfied. + # The commit matters: without a HEAD, `git diff --cached` fails, which used + # to make the untracked-fixture case pass for the wrong reason and hid a + # false negative in git_tracked_files(). + subprocess.run(["git", "init", "-q"], cwd=d, check=True) + subprocess.run(["git", "config", "user.email", "selftest@example.invalid"], cwd=d, check=True) + subprocess.run(["git", "config", "user.name", "eval-quality self-test"], cwd=d, check=True) + subprocess.run(["git", "add", "-A"], cwd=d, check=True) + subprocess.run(["git", "commit", "-qm", "baseline"], cwd=d, check=True) + return d + + +def case(label, mutate, expect_fail): + d = scratch() + try: + mutate(d) + subprocess.run(["git", "add", "-A"], cwd=d, capture_output=True) + code, out = run_gate(d) + failed = code != 0 + ok = failed == expect_fail + want = "FAIL" if expect_fail else "PASS" + got = "FAIL" if failed else "PASS" + print(f" [{'OK ' if ok else 'BAD'}] {label:<52} expected={want} got={got}") + if not ok: + print(" " + out.strip().replace("\n", "\n ")[:900]) + return ok + finally: + shutil.rmtree(d, ignore_errors=True) + + +EV = lambda d: os.path.join(d, "tests", "demo", "widget", "eval.yaml") + + +def clean(d): + pass + + +def missing_fixture(d): + shutil.rmtree(os.path.join(d, "tests", "demo", "widget", "fixtures", "sample")) + + +def untracked_fixture(d): + # Present on disk but excluded from git — the .gitignore class of bug. + with open(os.path.join(d, ".gitignore"), "w") as f: + f.write("Thing.cs\n") + subprocess.run(["git", "rm", "--cached", "-q", + "tests/demo/widget/fixtures/sample/Thing.cs"], cwd=d, capture_output=True) + + +def bad_cobertura(d): + p = os.path.join(d, "tests", "demo", "widget", "fixtures", "sample", "coverage.cobertura.xml") + with open(p, "w") as f: + f.write( + '' + '' + '' # claims 90% + '' # actually 50% + "" + ) + + +def inconsistent_file_totals(d): + # Every method agrees with its own ; only the whole-file summary + # attributes disagree with the declared file line-rate. This is the shape + # that shipped in coverage-analysis/partial-coverage and that the + # method-level check alone could not see. + p = os.path.join(d, "tests", "demo", "widget", "fixtures", "sample", "coverage.cobertura.xml") + with open(p, "w") as f: + f.write( + '' + '' # 35/60 = 0.58 + '' + '' + '' + '' + "" + ) + + +def empty_grader_config(d): + # An edit that leaves `- type: output-matches` / `config:` with the pattern + # attached to the NEXT list item. The document still parses; the grader + # silently enforces nothing. + with open(EV(d), "a") as f: + f.write( + " graders:\n" + " - type: output-matches\n" + " config:\n" + " - type: output-matches\n" + " config:\n" + " pattern: Thing\n" + ) + + +def guard_with_reject_skills(d): + with open(EV(d), "a") as f: + f.write( + " - name: Decline off-target request\n" + " prompt: write me something else\n" + " expect_activation: false\n" + " rubric:\n" + " - Did not derail into widget analysis\n" + " constraints:\n" + " reject_skills:\n" + ' - "*"\n' + ) + + +def guard_ok(d): + with open(EV(d), "a") as f: + f.write( + " - name: Decline off-target request\n" + " prompt: write me something else\n" + " expect_activation: false\n" + " rubric:\n" + " - Did not derail into widget analysis\n" + ) + + +print("Eval quality gate — self-test\n") +results = [ + case("clean tree", clean, expect_fail=False), + case("fixture referenced but missing on disk", missing_fixture, expect_fail=True), + case("fixture present but NOT tracked by git", untracked_fixture, expect_fail=True), + case("Cobertura line-rate contradicts its ", bad_cobertura, expect_fail=True), + case("Cobertura file totals contradict file line-rate", inconsistent_file_totals, expect_fail=True), + case("grader with an empty config enforces nothing", empty_grader_config, expect_fail=True), + case("dormancy guard also sets reject_skills", guard_with_reject_skills, expect_fail=True), + case("well-formed dormancy guard", guard_ok, expect_fail=False), +] +print() +if all(results): + print(f"All {len(results)} self-tests passed: the gate fires on every bug class and stays " + f"quiet on well-formed input.") +else: + print("SELF-TEST FAILURE — the gate does not behave as documented.") +raise SystemExit(0 if all(results) else 1)