#187: Faster grade defaults — Haiku + per-provider fast models (plan)

.claude/rules/grade-layer.md

@@ -65,7 +65,7 @@ For 4 default criteria × ~12 artifacts per typical model = ~48 calls per `grade
 
 The cached prompt block is the rubric criterion list (constant per run); the dynamic block is the per-pair `<ARTIFACT>...</ARTIFACT>` envelope. Anthropic prompt-cache TTL defaults to `"1h"` for the grader (vs. drafter's `"5m"`).
 
-**Tolerant JSON extraction (issue #144).** `parse_grade_response` routes the response through `signalforge._common.json_payload.extract_json_payload` (after `_strip_code_fence`) so a judge that narrates a prose preamble before the `{` still parses. The judge model (`claude-sonnet-4-6`) does NOT support an assistant-turn prefill (API 400), so the parser is the only JSON-only guardrail. Same decode rule as the drafter — decode at the first structural char (`{` or `[`) only, return unchanged on failure — see `llm-drafter.md` § "Tolerant JSON extraction"; a no-JSON response still routes to `GradeOutputError(violation_type="json_parse")` and the conservative degrade.
+**Tolerant JSON extraction (issue #144).** `parse_grade_response` routes the response through `signalforge._common.json_payload.extract_json_payload` (after `_strip_code_fence`) so a judge that narrates a prose preamble before the `{` still parses. The Anthropic judge models (the `claude-sonnet-4-6` default, or the `claude-haiku-4-5` opt-in per #187) do NOT support an assistant-turn prefill (API 400), so the parser is the only JSON-only guardrail. Same decode rule as the drafter — decode at the first structural char (`{` or `[`) only, return unchanged on failure — see `llm-drafter.md` § "Tolerant JSON extraction"; a no-JSON response still routes to `GradeOutputError(violation_type="json_parse")` and the conservative degrade.
 
 ## Reproducibility hash fields on every GradeEvent (DEC-010, DEC-019)
 
@@ -132,6 +132,31 @@ Every `extra="ignore"` production model — `GradingResult`, `GradingReport`, `G
 
 The grade-stage block is `{ grade: { model, cache_ttl, max_output_tokens, max_retries_*, total_budget_seconds, min_pass_rate, min_mean_score, fail_on_below_threshold, rubric? } }`. Sibling top-level keys are reserved and silently ignored by the grade loader. `GradeConfig` uses `extra="forbid"`; `_GradeConfigFile` uses `extra="ignore"` at the top level. Mirrors the other layers' top-level-namespace pattern verbatim.
 
+## Locked defaults: per-provider fast model + 1024 output cap (DEC-026, #187)
+
+`GradeConfig`'s locked defaults (DEC-023..DEC-027) carry two #187 changes:
+
+- **`model` default is now a per-provider sentinel.** The field defaults to `None`; a `mode="before"` model-validator (`_resolve_model_default`) resolves the sentinel at config-load to the calling provider's default judge model from `signalforge.llm.providers.PROVIDER_DEFAULT_MODELS` — `anthropic` → `claude-sonnet-4-6`, `openai` → `gpt-4o-mini`, `gemini` → `gemini-2.5-flash`. **Anthropic stays on Sonnet:** the #187 calibration gate (`tests/research/187-haiku-calibration/`) found `claude-haiku-4-5` grades the rubric stricter than Sonnet (~77–82% concordance on a real `intuit_airflow`-drafted sample, below the 85% DEC-005 bar), so Haiku is an explicit opt-in (`grade.model: claude-haiku-4-5`), not the default. OpenAI/Gemini default to their fast judges (explicit operator choices, never calibrated). (Pre-#187 the default was the bare `claude-sonnet-4-6` literal regardless of provider; the per-provider resolver is the #187 improvement — `provider: openai`/`gemini` now Just Works without forcing an explicit model.) An explicit `model:` is honoured verbatim; after construction the field is always a concrete non-empty string, never `None`. The resolver runs `before` because `GradeConfig` is `frozen=True` and a `mode="after"` mutation would raise. A provider registered but absent from the default-model table with no explicit model fails loud at config-load (the compat validator requires an explicit `grade.model`) — a custom/plugin provider's model can't be guessed, and letting `None` reach the engine's `assert model is not None` would be a latent crash.
+- **`max_output_tokens` default raised 256 → 1024** so a verbose one-line `gemini-2.5-flash` grade JSON is not truncated (a truncation would surface as the wrong typed degrade — `GradeOutputError` instead of `GradeLLMError`). Still a cap, not a target; the expected JSON is ~150 tokens, so the larger ceiling costs nothing on the happy path.
+
+**Model↔provider compat validator (DEC-006 of #187).** A `mode="after"` validator (`_validate_model_provider_compat`) reads `signalforge.llm.providers.PROVIDER_SKU_PREFIXES` (`anthropic` → `claude-`, `openai` → `gpt-`, `gemini` → `gemini-`) and fails loud at config-load when `provider` is a known-prefix provider AND the resolved/explicit `model` carries a *different* known provider's SKU prefix (e.g. `provider: openai` with a `claude-` model). Two cases are deliberately left alone: a model whose prefix matches no known provider (forward-compat for future SKUs) and a registry-valid provider outside the prefix table (custom/plugin providers may use any model name). Both `PROVIDER_DEFAULT_MODELS` and `PROVIDER_SKU_PREFIXES` are the single source of truth — no hardcoded SKUs or prefixes in the grade config module. Every fast-model value is an exact key in `signalforge.llm.pricing.PRICES`, so the `--estimate` path never raises on the resolved default.
+
+## Reusable conventions distilled from #187
+
+Two patterns from the #187 sentinel-default work generalise beyond the grade layer. Reach for them whenever a config field's default depends on *another* field, or whenever a default is looked up in a table keyed by a registry-growable value.
+
+**Frozen-config "default from a sibling field" resolves in `@model_validator(mode="before")`, never `mode="after"`.** When a config field defaults based on another field on the same model (here `model` ← `provider`), the resolution MUST inject the computed value into the raw dict in a `mode="before"` validator — NOT mutate `self.<field>` in a `mode="after"` validator. The pipeline's config models are `frozen=True` (`extra="forbid"`), and a `mode="after"` `self.model = ...` raises (Pydantic forbids attribute assignment on a frozen instance). The before-validator runs ahead of field validation, so the injected value flows through the normal construction path and the field is concrete the moment the frozen instance exists. Copy-on-write the dict (`data = {**data, "model": resolved}`) so a caller-owned mapping is never mutated, and guard the input shape (`if not isinstance(data, dict): return data`) so an already-constructed instance passed to `model_validate` passes through untouched. This is the reusable convention for any future "this knob defaults from that knob" on a frozen `*Config` (e.g. a draft `cheap_model` ← `provider`, a prune `partition_filter` ← `scope`).
+
+**A default looked up in a table keyed by a registry-growable field must FAIL LOUD on a registered-but-absent key — never leak the sentinel.** This is the load-bearing #187 lesson. `model` defaults from `PROVIDER_DEFAULT_MODELS[provider]`, but `provider` is a *registry-validated `str`, not a `Literal`* (the provider registry is a plugin point designed to grow — see `llm-drafter.md`). So three population states exist for the key field, and each needs a distinct fate:
+
+1. **Unregistered provider** — the `provider` field-validator already raises `UnknownProviderError`. The before-validator declines to inject (uses `.get()`, not `[]`) so it never masks that with a `KeyError`.
+2. **Registered AND in the fast-model table** — the before-validator injects the fast model. Happy path.
+3. **Registered BUT absent from the fast-model table** (a custom/plugin provider — the registry-growth path) — the before-validator has no value to inject, so `model` reaches the `mode="after"` validator still `None`. **This case must fail loud**, requiring an explicit `grade.model`, rather than letting `None` flow downstream.
+
+State 3 is the trap. The #187 Quality-Gate review caught a real bug here: the original compat validator only checked SKU-prefix mismatches and silently returned for a non-prefix provider, so a registered-but-untabled provider left `model=None` — directly contradicting the engine's `assert config.model is not None` invariant (the engine, `GradeEvent.model`, and the cost-rollup's prefix dispatch all assert/depend on a concrete model). The fix makes `_validate_model_provider_compat` raise at config-load when `model is None`, keeping the "model is never `None` post-construction" invariant *genuinely* true rather than merely usually true.
+
+The general rule for any per-X default table whose key field comes from a growable registry: enumerate the three population states explicitly, and make the "registered-but-absent-from-the-table" state a loud config-load failure that names the remediation (set the field explicitly). A `None`/sentinel that survives construction because the table happened not to cover a key is exactly the silent-no-op failure mode that downstream `assert`/exact-match consumers turn into a confusing far-from-the-cause crash. `PROVIDER_DEFAULT_MODELS` / `PROVIDER_SKU_PREFIXES` live in `signalforge.llm.providers` and are the single source for per-provider fast models / SKU-prefix dispatch (shared with `cost/_rollup.py`, reusable by a future draft `--cheap`).
+
 ## Schema-version surfaces
 
 Two exported names ship but are not consumed. **Both re-verified still-reserved on 2026-05-22 (issue #62)** — the v0.1 designs each anticipated remain intact, so neither was promoted:

.claude/rules/llm-drafter.md

@@ -18,6 +18,7 @@ Every `# pyright: ignore[...]` and `# type: ignore[...]` comment for the Anthrop
 - **Neutral value objects:** `UsageMetrics` + the `ExceptionCategory` enum (`AUTH`, `RATE_LIMIT`, `SERVER_ERROR`, `CONNECTION`, `NO_RETRY`) keep the orchestrator off vendor-shaped dicts.
 - **Capability-gated behaviour (DEC-008):** `supports_prompt_caching=False` ⇒ no `cache_control` marker, no `extended-cache-ttl` beta header, 0 cache tokens, no dual-zero anomaly WARNING. `supports_token_count=False` ⇒ skip the pre-send count gate (no pre-send `LLMCacheTooLargeError`). Anthropic sets both `True`, so its emitted bytes/control flow are unchanged — the byte-identity gate (fixtures + prompt-cache snapshot + drift detectors) is the regression guard.
 - **`provider` config field (DEC-007):** `DraftConfig.provider` (`llm:` block) and `GradeConfig.provider` (`grade:` block), both registry-validated `str` defaulting to `"anthropic"` — **deliberately NOT a `Literal`** (a registry is a plugin point that grows; #136/#137 register a provider instead of editing a Literal in two configs). The validator raises `UnknownProviderError` (an `LLMError`, so Pydantic v2 does NOT wrap it into `ValidationError` — it propagates raw with the available-keys remediation).
+- **Provider→string mappings (#187):** `PROVIDER_DEFAULT_MODELS` and `PROVIDER_SKU_PREFIXES` also live in `signalforge.llm.providers` and are the single source of truth for two cross-cutting per-provider facts. `PROVIDER_DEFAULT_MODELS` (`anthropic` → `claude-sonnet-4-6`, `openai` → `gpt-4o-mini`, `gemini` → `gemini-2.5-flash`) is the **default judge SKU per provider** — every value is an exact key in `signalforge.llm.pricing.PRICES`. It drives the `GradeConfig.model` per-provider default (the `grade.model:`-unset sentinel resolves to it at config-load; see `grade-layer.md`). **Anthropic defaults to Sonnet, NOT its cheapest SKU** — the #187 calibration gate found `claude-haiku-4-5` grades stricter than Sonnet (below the 85% bar), so Haiku stays a grade opt-in; OpenAI/Gemini default to their fast judges (explicit operator choices). `PROVIDER_SKU_PREFIXES` (`claude-` / `gpt-` / `gemini-`) drives the cost-rollup's prefix dispatch AND the grade model↔provider compat validator. The cheap *draft* model is a separate concept: `DraftConfig.cheap_model` defaults to the bare `claude-haiku-4-5` SKU, which is NOT the grade default (Sonnet) — a future draft `--cheap` swap should resolve the cheap SKU from a dedicated source, not conflate it with `PROVIDER_DEFAULT_MODELS`.
 
 **Gate the cache marker on BOTH capability flags, not just `supports_prompt_caching` (#135 QG lesson).** `call_llm` sets `cache_marker_active = supports_prompt_caching AND supports_token_count`. The pre-send count gate is what enforces the sub-minimum drop + the 8000-token oversize cap; attaching a `cache_control` marker without that gate having run would send an *unvalidated* marker (a sub-minimum block silently no-ops the marker — paying the input premium with no discount; an oversize block bypasses `LLMCacheTooLargeError`). Anthropic is `True/True` so the default path is unaffected, but a future provider that caches yet has no token-count API (`True/False`) must degrade to no-caching rather than send an unguarded marker. A new provider's capability flags are load-bearing — set them honestly, and don't assume "supports caching" alone is sufficient to attach a marker.
 

.claude/rules/testing-signal.md

@@ -206,6 +206,25 @@ grep -c "rate limit" pytest-stderr.log
 
 **Markers that STAY serial — do NOT pass `-n` to these.** `cli_subprocess` and `wheel_smoke` invocations shell out to a single installed console-script / build a single wheel into shared `dist/`, so parallel workers would collide on the artefact. Run them as documented in `python-build.md` / `cli-layer.md`: `uv run pytest -m cli_subprocess --no-cov` and `uv run pytest -m wheel_smoke --no-cov`, sequential, no `-n`.
 
+## Gated calibration / concordance harness as a research-test pattern (issue #187)
+
+When a behaviour change swaps a default whose *correctness* depends on live-model behaviour (the #187 grade-default flip from Sonnet to Haiku), the empirical gate is a **calibration harness**: pin a baseline sample as committed data, re-grade it live with the new default, and assert a concordance threshold. The `tests/research/187-haiku-calibration/` precedent ships:
+
+- A **pinned baseline + substrate** as committed data (`sonnet_baseline_sample.json` — the prior model's per-`(artifact_id, criterion_id)` pass/fail verdicts; `_substrate.py` — the candidate/model builders). Per § "Engineered determinism for LLM-driven assertions", the *only* live variable is the new model's re-grade; everything it joins against is committed bytes, so the comparison is reproducible.
+- **Belt-and-suspenders gating, same as the e2e tests** (§ "Belt-and-suspenders gating"): `pytestmark = pytest.mark.anthropic` (deselected by the default `addopts` `-m 'not anthropic ...'`, so default CI never collects it) PLUS a runtime `_skip_reason()` → `pytest.skip(...)` when the live key is unset. CI can't run live API; the harness is a **maintainer-run decision gate** — the maintainer runs `pytest -m anthropic --no-cov tests/research/187-haiku-calibration/`, reads the printed breakdown, and transcribes the result into the writeup (`docs/research/187-haiku-calibration.md`).
+- The harness **prints the breakdown regardless of pass/fail** (the discordance list, the rates) so a sub-threshold run still surfaces *why* it fell short — the gate's value is the diagnostic, not just the boolean.
+
+When a future ticket swaps a default that only live behaviour can validate (a drafter model flip, a new provider's fast default, a rubric-text change), copy this shape: committed baseline + substrate, dual gating (marker + runtime skip), maintainer-run, printed diagnostics. Mirror the e2e-gated conventions above rather than inventing a parallel gate; reuse an existing live-API marker (`anthropic` / `openai` / `gemini`) rather than minting a new one.
+
+## Concordance-gate denominator hygiene (issue #187)
+
+A concordance/agreement metric over an LLM eval computes `agreements / comparable`. Two denominator traps, both pinned in `tests/research/187-haiku-calibration/test_haiku_calibration.py`:
+
+1. **Exclude degraded (`score=None`) pairs from the denominator — but do NOT let a degraded-dominated run masquerade as signal.** A degraded pair (`GradingResult.score is None` — the DEC-015 conservative degrade) could not be positively evaluated, so it is neither a concordance nor a discordance; folding it into the denominator would understate agreement. *However*, a run where degraded pairs DOMINATE has too small/biased a comparable set to trust — a high rate over a handful of survivors is not a real signal. The gate therefore asserts both `comparable >= 1` (something was actually compared) AND `comparable >= degraded` (the comparable set is the majority); a run that fails the second assertion fails loud with a remediation (raise `max_output_tokens` or investigate the degradations) rather than reporting a flattering percentage over noise.
+2. **Coverage assertion before the rate.** The harness asserts every `artifact_id` the engine grades is present in the committed baseline (`engine_ids - baseline_ids == set()`) BEFORE computing concordance, and never silently folds an unmatched verdict into the rate (`if key not in baseline: continue`). A silent coverage gap would otherwise inflate or deflate the denominator invisibly.
+
+Generalise this to any agreement/precision/recall gate computed over an LLM eval where some pairs can degrade to "not evaluated": exclude the un-evaluated pairs from the denominator, but require the evaluated subset to be the majority (or some explicit floor) so the metric is computed over enough signal to mean anything — and assert baseline coverage up-front so the denominator is the set you think it is.
+
 ## Reference
 
-`plans/super/1-project-scaffolding.md` — DEC-010. `plans/super/2-manifest-loader.md` — DEC-005, DEC-009, DEC-012, DEC-017. `plans/super/27-codecov-coverage.md` — DEC-001, DEC-004, DEC-009. `plans/super/10-e2e-bigquery-smoke.md` — DEC-001, DEC-002, DEC-004, DEC-008, DEC-010, DEC-022. `plans/super/157-e2e-cost-and-parallel.md` — DEC-001 … DEC-010 (parallel-safe e2e, `signalforge.llm.cost.rollup_audit_dir`, pricing-table-version parity gate). `tests/test_smoke.py`, `tests/manifest/`, `tests/fixtures/regenerate.sh`, `tests/cli/_e2e_helpers.py`, `tests/cli/test_e2e_bigquery_smoke.py`, `tests/test_contributing_e2e_enumeration_parity.py`, `src/signalforge/llm/cost/`.
+`plans/super/1-project-scaffolding.md` — DEC-010. `plans/super/2-manifest-loader.md` — DEC-005, DEC-009, DEC-012, DEC-017. `plans/super/27-codecov-coverage.md` — DEC-001, DEC-004, DEC-009. `plans/super/10-e2e-bigquery-smoke.md` — DEC-001, DEC-002, DEC-004, DEC-008, DEC-010, DEC-022. `plans/super/157-e2e-cost-and-parallel.md` — DEC-001 … DEC-010 (parallel-safe e2e, `signalforge.llm.cost.rollup_audit_dir`, pricing-table-version parity gate). `plans/super/187-fast-grade-defaults.md` — DEC-005 (gated calibration/concordance harness, maintainer-run decision gate). `tests/test_smoke.py`, `tests/manifest/`, `tests/fixtures/regenerate.sh`, `tests/cli/_e2e_helpers.py`, `tests/cli/test_e2e_bigquery_smoke.py`, `tests/test_contributing_e2e_enumeration_parity.py`, `tests/research/187-haiku-calibration/`, `src/signalforge/llm/cost/`.