Tighten the eval gate: net-win floor, completion-regression hard gate, and power/independence fixes

[AbhitejJohn]

Context

PR #965 ("Fix the underpowered eval gate: score direction, not magnitude") replaced the old ciLow-based pass gate with an exact one-sided sign test over head-to-head trials, and strengthened the CI ratchet (check_power hard gate, check_floor_agreement() cross-language drift guard, check_allowlist_growth ref-resolution hardening). That was a clear net improvement and is merged.

This issue tracks the verdict-level gaps that #965 intentionally did not close — all in eng/vally-adapter/adapt.mjs, whose verdict logic is unchanged. Goal: tighten the gate so a genuinely-better change passes and a marginal or regressing one doesn't. See the follow-up discussion on #965 (issuecomment-5134235014).

The current verdict:

const passed = conclusive && !underpowered && direction === "better" && credible;

where underpowered = directions.length < MIN_CREDIBLE_TRIALS (counts all trials incl. ties), credible = signTestPValue(wins, losses) <= 0.05 (excludes ties), and MIN_CREDIBLE_TRIALS = 5.

---

1. Sparse wins pass on almost no evidence (net-win floor)

Proven. The underpowered floor counts all trials (ties included), but the sign test only sees discordant ones. So 5W / 95T / 0L has 100 counted trials (not underpowered), direction "better", and signTestPValue(5, 0) = 0.5^5 = 0.03125 ≤ 0.05 → passes. Five marginal wins and 95 "no meaningful difference" out of 100 clears the bar, because the 95 ties never enter the p-value.

Proposed fix: add a magnitude-free net-win floor, e.g. (wins − losses) / counted ≥ NET_WIN_MIN (counted incl. ties). 5W/95T/0L → 0.05 < 0.2 fails. Magnitude-independent, so it cannot revive the #952 variance reversal.

Open: threshold calibration — a 0.2 floor still lets 5W/20T/0L through at the boundary. Pick NET_WIN_MIN from desired operating characteristics, not intuition.

2. No hard gate on task-completion regression

Proven at the adapter. baselinePassed / treatmentPassed are carried into scenario output but never referenced in passed. So a change judged "slightly better" on style while breaking task completion on a scenario can still pass, as long as the win/loss record leans positive. The old C# skill-validator hard-failed on completion regression (Comparator.cs); that safety property did not carry across the C#→vally migration.

Proposed fix: magnitude-free Layer-0 hard gate — baselinePassed && !treatmentPassed on any scenario ⇒ fail. Consider aggregating completion at the scenario level (and/or requiring repeated regression) so a single flaky judge call doesn't hard-fail the build.

Inferred / needs confirmation: whether vally's upstream judge reliably turns a completion break into a "loss" signal (not visible in this repo).

3. Effect size ("by how much") is gone by design — do we want it fully gone?

Proven. The gate now reads only direction + frequency, never magnitude. Dropping magnitude out of the variance term was correct — that was #952's actual bug (ordinal scores feeding an sd/CI, where a more decisive win raised sd and lowered ciLow, flipping pass→fail).

Proposed (optional): a deterministic magnitude floor — mean(ordinal) ≥ E_MIN, no variance term. This is monotone (raising any trial's score can only move fail→pass, never pass→fail — cross-checked), so it recovers effect size without the #952 reversal. Ship report-only first: the ordinal scale (−1 / −0.4 / 0 / +0.4 / +1) is a policy weighting, not a true interval, so this is a utility gate, not a measurement. Prefer #1 (net-win floor) as the required practical-significance gate; treat #3 as opt-in.

4. Power / independence / rerun policy

• Correlated runs (inferred). defaults.runs on a single scenario feeds correlated pseudo-replicates into the sign test, inflating significance. Aggregating each scenario to one outcome before testing across scenarios fixes the unit of analysis — though outcomes still share prompt/skill/judge/model, so it tightens rather than fully restores independence. Decide the intended runs story.
• Rerun / p-hacking (inferred). If authors can rerun until green, the nominal p-value stops meaning what it says. Need a fixed trial plan or a stated rerun policy.
• Multiple comparisons (inferred). ~94 evals each gating at p ≤ 0.05 will produce false positives across the suite; consider FDR or family-level reporting.
• Regression symmetry. Define fail/regression criteria as the mirror of pass (sign test + net-loss floor for direction === "worse"), so a real regression is caught with the same rigor a pass requires.

---

Validation (required before shipping any floor)

Every threshold (NET_WIN_MIN, E_MIN, scenario floor, any FDR level) must be back-tested before merge:

• the Eval gate is underpowered: 27% of verdicts flip on byte-identical inputs (A/A test); 53% of evals run at n<=3 #952 A/A dataset (noise must still fail),
• the current ~94 real evals (no mass false-failure),
• historical pass/fail preservation, A/A false-positive rate, sensitivity to tie count and to few-scenario evals, and regression detection (not only pass detection).

Suggested sequencing

1. Net-win floor (Initial documentation and validation workflow #1) + completion hard gate (test: validating validation #2) — both magnitude-free, highest safety-per-line, back-test to calibrate.
2. Scenario-level aggregation + rerun policy (Add unit-testing agents & skill #4).
3. Optional deterministic magnitude floor (feat: Add WinForms Expert #3), report-only first.

Happy to prototype #1 and #2 against the backtest harness.

Analysis cross-checked with a second model family; claims marked proven are verified against the merged code, inferred items need confirmation or empirical calibration.