#187: Faster grade defaults — Haiku + per-provider fast models (plan) - #193
Merged
Conversation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThis PR implements issue ChangesFast-grade defaults rollout
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related issues
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Comment |
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
wjduenow
marked this pull request as ready for review
June 2, 2026 16:04
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plans/super/187-fast-grade-defaults.md`:
- Line 74: The markdown table header line "| Provider | Cheapest known SKU |
input $/MTok | output $/MTok |" is breaking lint/rendering and there is a stray
literal "259" and inconsistent code-fence around the ASCII diagram; remove the
stray "259", ensure the table header is either a valid markdown table with a
separator row (e.g., add the --- row after the header) or convert the entire
section to a fenced code block, and make sure the ASCII diagram uses matching
```text opening and closing fences; apply the same cleanup to the related block
covering lines 251-259 so the table/code-block formatting is consistent and
lint-safe.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 3fe8f193-7b9a-4292-9a9d-7da2bf82c4da
📒 Files selected for processing (1)
plans/super/187-fast-grade-defaults.md
… claude-haiku-4-5 The dated id claude-haiku-4-5-20251001 is not a key in signalforge.llm.pricing.PRICES (which uses bare SKUs). Change the unused (declared-not-consumed) cheap_model default to the bare SKU so it matches pricing and the new grade default; update the DEC-017 reference comment and field docstring in lockstep. Lockstep consistency fix only — not wired into any code path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…XES constants Add two provider->string mapping constants to signalforge.llm.providers, keyed by the three registered provider names (anthropic/openai/gemini), and export both in __all__: - PROVIDER_FAST_MODELS — cheap/fast judge SKU per provider; every value is an exact key in signalforge.llm.pricing.PRICES (claude-haiku-4-5 / gpt-4o-mini / gemini-2.5-flash) so lookup()/--estimate never raise. - PROVIDER_SKU_PREFIXES — SKU-string prefix per provider. Refactor signalforge.llm.cost._rollup so its SKU-prefix dispatch derives _PROVIDER_PREFIXES by inverting PROVIDER_SKU_PREFIXES instead of duplicating the claude-/gpt-/gemini- literals. Classification behaviour is unchanged; existing rollup tests stay green and a new parametrised test pins the per-SKU classification plus the derived-from-source-of-truth invariant. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ver + compat validator + 1024 cap Make the grade-judge model default a per-provider fast model resolved at config-load, plus a model<->provider compat validator and a raised max_output_tokens cap. - model: str -> str | None = None sentinel. A @model_validator(mode="before") resolves the sentinel to PROVIDER_FAST_MODELS[provider] (frozen-safe; no mutation of the constructed instance). Unknown provider is NOT injected so the existing provider field-validator still raises UnknownProviderError rather than being masked. After construction model is always concrete. - _model_non_empty field-validator now passes None through cleanly (the only None-survival path is the unknown-provider case that raises downstream). - New @model_validator(mode="after") _validate_model_provider_compat rejects a SKU-prefix/provider mismatch (e.g. provider=openai + claude-* model), reusing PROVIDER_SKU_PREFIXES. Only the three known-prefix providers participate; custom/plugin providers (not in the table) may use any model name. - max_output_tokens default 256 -> 1024 (DEC-004) to avoid one-line gemini-2.5-flash grade-JSON truncation. It is a cap, not a target. - Module + class docstrings and the DEC-023..027 locked-defaults list updated. - tests/grade/test_config.py: updated the defaults regression test to the new resolved defaults (haiku, 1024) and added coverage for per-provider resolution, explicit-honour, compat reject/accept, whitespace guard, unknown-provider-not-masked, and the loader round-trips. - grade/engine.py + cli/_estimate.py: read-only assert narrows for the now str|None field (runtime-guaranteed concrete) to keep pyright green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…model resolver + 1024 cap
…ures for Haiku default The #187 US-002 grade default flip (anthropic resolves to claude-haiku-4-5, max_output_tokens 256 -> 1024) drifted two estimate fixtures pinned to the old claude-sonnet-4-6 grade default. Re-baseline in lockstep: - tests/fixtures/estimate/anthropic_byte_identity_golden.txt: regenerated. Only diff is the grade-model id (sonnet -> haiku) and the corresponding grade USD figures (per-criterion $0.0292 -> $0.0078, grade $0.1170 -> $0.0312, total $0.1814 -> $0.0956). Verified the draft section, token counts, call counts, artifact counts, and warehouse section are unchanged. - tests/cli/test_estimate_engine.py::test_estimate_total_llm_usd_matches_hand_calculation: the hand calc now keys the grade half on claude-haiku-4-5 pricing ($0.80/MTok input, $4/MTok output) instead of reusing the draft model's pricing. The grade output-token figure stays the fixed _GRADE_OUTPUT_TOKENS_PER_CALL (50) — the 256->1024 max_output_tokens bump is a response cap, not the estimate's per-call output projection. - tests/fixtures/grade/grade_event_v1.jsonl: cosmetic model id sonnet -> haiku for consistency with the new default (drift detector validates shape only; still green). No production source changed. Full suite green: 3329 passed, 8 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Build the maintainer-run empirical gate for the #187 Haiku grade default. - tests/research/187-haiku-calibration/_substrate.py: pinned Model + CandidateSchema (engineered-deterministic, spans the rubric calibration space) + curated Sonnet-baseline loader. Artifact-id set derived from the grade engine itself so it can't drift from the formatter. - sonnet_baseline_sample.json: curated 44-verdict Sonnet baseline (11 artifacts x 4 DEFAULT_RUBRIC criteria), NOT the un-committed #179 Phase-B dump; provenance documented. - test_haiku_calibration.py (@pytest.mark.anthropic + runtime skip on ANTHROPIC_API_KEY): re-grades the sample with the resolved Haiku default (claude-haiku-4-5, max_output_tokens=1024), joins to the baseline by (artifact_id, criterion_id), asserts >=85% per-criterion concordance, prints the breakdown for the writeup. - test_gemini_1024_no_truncation.py (@pytest.mark.gemini + runtime skip on SF_RUN_GEMINI/GOOGLE_API_KEY): grades a verbose artifact on gemini-2.5-flash @ 1024 tokens, asserts no score=None truncation degrade (verifies DEC-004). - docs/research/187-haiku-calibration.md: writeup mirroring #179's structure with a clearly-marked "Result (maintainer-filled)" TODO. Gating is belt-and-suspenders (existing anthropic/gemini markers excluded in addopts + runtime skip). No production default changed; src/ untouched; no new marker registered. Default `uv run pytest` stays fully green (3329 passed, gated tests deselected) at 97% coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…u grade default Update every non-code surface to agree with the #187 shipped behaviour: grade default model resolves per-provider (anthropic -> claude-haiku-4-5, openai -> gpt-4o-mini, gemini -> gemini-2.5-flash) via PROVIDER_FAST_MODELS; grade max_output_tokens default 256 -> 1024; model<->provider compat validation via PROVIDER_SKU_PREFIXES. - docs/grade-ops.md: model + max_output_tokens defaults; per-provider fast-default cost table (haiku/gpt-4o-mini/gemini-2.5-flash) with pricing.py figures; config example shows model auto-resolve. - docs/llm-providers-ops.md: PROVIDER_FAST_MODELS table + gemini-2.5-flash 1024-cap truncation note; default grader-model matrix row. - docs/draft-ops.md: cheap_model shown as bare SKU claude-haiku-4-5. - .claude/rules/grade-layer.md: locked-defaults section (per-provider sentinel resolution + compat validator). - .claude/rules/llm-drafter.md: PROVIDER_FAST_MODELS / PROVIDER_SKU_PREFIXES source note (reusable by a future draft --cheap). - CHANGELOG.md: Unreleased "Changed" entry. README/CLAUDE.md unchanged (neither enumerates the grade default model; the slim CLAUDE.md delegates API surfaces to CHANGELOG/docs by design). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Real bug (passes 1 & 2): GradeConfig.model could stay None for a registered-but-not-in-PROVIDER_FAST_MODELS provider (the custom/plugin growth path), contradicting the engine/_estimate `assert model is not None`. Now fails loud at config-load requiring an explicit grade.model, restoring the post-construction invariant. Updated the #135 neutrality test contract + the divergent-providers estimate test to supply an explicit model. Accuracy fixes (passes 3 & 4): - Soften the "gemini-2.5-flash JSON is not truncated at 1024" claim across config docstring, grade-ops.md, CHANGELOG — 1024 reduces but doesn't eliminate Gemini truncation at full-fixture scale (floors table wants 4096). - grade-ops.md: gpt-4o example reframed as explicit override (default resolves to gpt-4o-mini). - hand-calc estimate test: pin the resolved literal claude-haiku-4-5 so a resolver regression fails there instead of drifting into a tautology. - calibration gate: reject degraded-dominated runs (comparable >= degraded). - round-trip test docstring + example_config.yml comments corrected (fixture pins explicit overrides, not the new defaults). CodeRabbit: unavailable in this environment (skipped). Full validation green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…gistry-table-invariant + calibration-gate patterns Distil the durable, generalisable conventions surfaced by epic #187 into the rule files so future work benefits: - .claude/rules/grade-layer.md — new "Reusable conventions distilled from #187" subsection: (1) frozen-config default-from-sibling-field resolves in @model_validator(mode="before") not mode="after" (frozen=True forbids the after-mutation); (2) the load-bearing lesson — a default looked up in a table keyed by a registry-growable field must FAIL LOUD on a registered-but-absent key rather than leak a None/sentinel, citing the #187 QG bug as the cautionary example, generalised for any per-X default table. - .claude/rules/testing-signal.md — two additions cross-referencing the e2e-gated conventions: gated calibration/concordance harness as a research-test pattern (tests/research/187-haiku-calibration/ — pinned baseline + marker + runtime skip + maintainer-run decision gate); and concordance-gate denominator hygiene (exclude degraded score=None pairs but fail loud when degraded dominates: comparable >= degraded). Docs-only; no src/ or tests/ behaviour change. Full suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owner
Author
Implementation complete (Ralph run)All 8 beads of epic Shipped:
Quality Gate (4 review passes): found + fixed one real bug — a registered-but-untabled custom provider left
|
There was a problem hiding this comment.
🧹 Nitpick comments (2)
tests/research/187-haiku-calibration/test_haiku_calibration.py (1)
110-115: ⚡ Quick winBaseline already covers all (artifact_id, criterion_id) pairs; keep guard future-proof
The committed
sonnet_baseline_sample.jsoncontains all 44 unique(artifact_id, criterion_id)combinations (11 × 4). So the “denominator shrink” failure mode won’t happen with the current baseline data.Lines 110-115 still only assert artifact-id coverage, though; if the baseline ever drops a single pair in the future, the test won’t fail loudly. Add a pair-level coverage assertion (check
set(baseline)against the graded(artifact_id, criterion_id)keys) to keep the enforcement aligned with the documented intent.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/research/187-haiku-calibration/test_haiku_calibration.py` around lines 110 - 115, The test currently only asserts artifact-level coverage using engine_ids and baseline_ids; change it to assert pair-level coverage: build the set of graded pairs (artifact_id, criterion_id) that the engine will grade (instead of engine_ids) and compare that to set(baseline); compute graded_pairs = {...} from the same source you used to derive engine_ids (referencing expected_artifact_ids usage) and then do baseline_pairs = set(baseline); let missing = graded_pairs - baseline_pairs and assert not missing with a clear message so the test fails if any (artifact_id, criterion_id) pair is missing.src/signalforge/llm/providers.py (1)
359-378: ⚡ Quick winMake the exported provider tables actually immutable.
These are documented as read-only source-of-truth tables, but plain
dicts can still be mutated by any importer and silently change resolver / rollup behavior at runtime. Wrapping them in an immutable mapping would make the runtime contract match the docs.♻️ Proposed change
+from types import MappingProxyType -from typing import TYPE_CHECKING, Any, ClassVar +from typing import TYPE_CHECKING, Any, ClassVar, Mapping ... -PROVIDER_FAST_MODELS: dict[str, str] = { +PROVIDER_FAST_MODELS: Mapping[str, str] = MappingProxyType({ "anthropic": "claude-haiku-4-5", "openai": "gpt-4o-mini", "gemini": "gemini-2.5-flash", -} +}) ... -PROVIDER_SKU_PREFIXES: dict[str, str] = { +PROVIDER_SKU_PREFIXES: Mapping[str, str] = MappingProxyType({ "anthropic": "claude-", "openai": "gpt-", "gemini": "gemini-", -} +})🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/signalforge/llm/providers.py` around lines 359 - 378, PROVIDER_FAST_MODELS and PROVIDER_SKU_PREFIXES are declared as plain dicts but should be immutable; replace the mutable dicts with immutable mappings (e.g., wrap them in types.MappingProxyType) and update their type annotations to typing.Mapping[str, str] (or the concrete MappingProxyType if preferred), import types/typing as needed, and export the MappingProxyType instances so callers cannot mutate these tables at runtime.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@src/signalforge/llm/providers.py`:
- Around line 359-378: PROVIDER_FAST_MODELS and PROVIDER_SKU_PREFIXES are
declared as plain dicts but should be immutable; replace the mutable dicts with
immutable mappings (e.g., wrap them in types.MappingProxyType) and update their
type annotations to typing.Mapping[str, str] (or the concrete MappingProxyType
if preferred), import types/typing as needed, and export the MappingProxyType
instances so callers cannot mutate these tables at runtime.
In `@tests/research/187-haiku-calibration/test_haiku_calibration.py`:
- Around line 110-115: The test currently only asserts artifact-level coverage
using engine_ids and baseline_ids; change it to assert pair-level coverage:
build the set of graded pairs (artifact_id, criterion_id) that the engine will
grade (instead of engine_ids) and compare that to set(baseline); compute
graded_pairs = {...} from the same source you used to derive engine_ids
(referencing expected_artifact_ids usage) and then do baseline_pairs =
set(baseline); let missing = graded_pairs - baseline_pairs and assert not
missing with a clear message so the test fails if any (artifact_id,
criterion_id) pair is missing.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 9787ed31-cad5-4342-adb6-22718600f07a
📒 Files selected for processing (28)
.claude/rules/grade-layer.md.claude/rules/llm-drafter.md.claude/rules/testing-signal.mdCHANGELOG.mddocs/draft-ops.mddocs/grade-ops.mddocs/llm-providers-ops.mddocs/research/187-haiku-calibration.mdsrc/signalforge/cli/_estimate.pysrc/signalforge/draft/config.pysrc/signalforge/grade/config.pysrc/signalforge/grade/engine.pysrc/signalforge/llm/cost/_rollup.pysrc/signalforge/llm/providers.pytests/cli/test_estimate_engine.pytests/cli/test_generate_estimate.pytests/draft/test_config.pytests/fixtures/estimate/anthropic_byte_identity_golden.txttests/fixtures/grade/example_config.ymltests/fixtures/grade/grade_event_v1.jsonltests/grade/test_config.pytests/grade/test_provider_neutrality.pytests/llm/cost/test_rollup.pytests/llm/test_providers.pytests/research/187-haiku-calibration/_substrate.pytests/research/187-haiku-calibration/sonnet_baseline_sample.jsontests/research/187-haiku-calibration/test_gemini_1024_no_truncation.pytests/research/187-haiku-calibration/test_haiku_calibration.py
✅ Files skipped from review due to trivial changes (6)
- tests/research/187-haiku-calibration/sonnet_baseline_sample.json
- docs/draft-ops.md
- CHANGELOG.md
- .claude/rules/llm-drafter.md
- tests/fixtures/grade/example_config.yml
- docs/grade-ops.md
There was a problem hiding this comment.
Pull request overview
This PR implements the #187 plan to make grading faster/cheaper by switching the grade layer to per-provider “fast default” models (Haiku / 4o-mini / Gemini flash), raising the grade output cap to reduce Gemini truncation risk, and adding config-time provider↔model compatibility validation plus supporting docs/tests/fixtures.
Changes:
- Add
PROVIDER_FAST_MODELSandPROVIDER_SKU_PREFIXESas single sources of truth and refactor cost rollup to use them. - Update
GradeConfigto resolvemodel=Noneto a provider fast default at config-load, validate model/provider compatibility, and bumpmax_output_tokensdefault256 → 1024. - Update estimates/fixtures/docs and add maintainer-gated calibration harnesses under
tests/research/.
Reviewed changes
Copilot reviewed 29 out of 29 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
src/signalforge/llm/providers.py |
Adds exported per-provider fast model + SKU-prefix constants. |
src/signalforge/llm/cost/_rollup.py |
Derives prefix dispatch table from PROVIDER_SKU_PREFIXES to avoid duplicated literals. |
src/signalforge/grade/config.py |
Implements sentinel model resolution + provider/model compatibility validation; raises default output cap. |
src/signalforge/grade/engine.py |
Narrows config.model to a concrete string where required. |
src/signalforge/draft/config.py |
Aligns DraftConfig.cheap_model with priced bare SKU (claude-haiku-4-5). |
src/signalforge/cli/_estimate.py |
Updates estimate path to use resolved grade model (assert non-None) for token counting and pricing lookup. |
tests/llm/test_providers.py |
Adds unit tests pinning fast-model/prefix tables and export surface. |
tests/llm/cost/test_rollup.py |
Adds tests ensuring rollup provider classification remains identical post-refactor. |
tests/grade/test_config.py |
Updates default assertions and adds coverage for resolver + compatibility validator + new token cap default. |
tests/grade/test_provider_neutrality.py |
Updates plugin-provider behavior to require explicit model when no built-in default exists. |
tests/draft/test_config.py |
Updates default cheap_model assertion to bare SKU. |
tests/cli/test_generate_estimate.py |
Adjusts custom-provider estimate test to include explicit grade model. |
tests/cli/test_estimate_engine.py |
Updates expected USD math and explicitly pins resolved Haiku grade default in the test. |
tests/fixtures/grade/grade_event_v1.jsonl |
Updates fixture model field to new default (claude-haiku-4-5). |
tests/fixtures/grade/example_config.yml |
Clarifies explicit overrides vs auto-resolved defaults in the example config fixture. |
tests/fixtures/estimate/anthropic_byte_identity_golden.txt |
Updates estimate golden to reflect new grade default + cost changes. |
tests/research/187-haiku-calibration/_substrate.py |
Adds pinned calibration substrate builders + baseline loader. |
tests/research/187-haiku-calibration/sonnet_baseline_sample.json |
Adds committed baseline verdict sample for calibration gate. |
tests/research/187-haiku-calibration/test_haiku_calibration.py |
Adds maintainer-gated Haiku-vs-baseline concordance harness. |
tests/research/187-haiku-calibration/test_gemini_1024_no_truncation.py |
Adds maintainer-gated Gemini “1024 doesn’t truncate” check. |
docs/research/187-haiku-calibration.md |
Documents calibration method, substrate, and maintainer-run instructions/result placeholder. |
docs/llm-providers-ops.md |
Updates provider ops docs for per-provider grade defaults and token-cap note. |
docs/grade-ops.md |
Updates grade ops docs for new default model resolution and 1024 cap guidance. |
docs/draft-ops.md |
Updates draft ops docs to reflect bare cheap_model SKU. |
CHANGELOG.md |
Adds changelog entry describing per-provider grade defaults, cap bump, and compat validation. |
.claude/rules/testing-signal.md |
Adds documented pattern for maintainer-run concordance harnesses and denominator hygiene. |
.claude/rules/llm-drafter.md |
Documents new provider mapping constants and intended reuse pattern. |
.claude/rules/grade-layer.md |
Updates grade-layer rules for new defaults/resolution/compat validator patterns. |
plans/super/187-fast-grade-defaults.md |
Adds/updates the #187 “super plan” document describing decisions and story breakdown. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Replaces the hand-authored US-005 calibration baseline with a genuine one, per request: - capture_sonnet_baseline.py drafts artifacts for the real intuit_airflow model calendar_hour (schema-only, no warehouse), freezes them to real_candidate.json, and grades them with claude-sonnet-4-6. - _substrate.py: build_model() is now the real model (deterministic); build_candidate() loads the frozen drafted artifacts. - sonnet_baseline_sample.json: 80 live Sonnet verdicts (4 degraded excluded). RESULT — the gate FAILS: Haiku concordance 81.8% / 77.0% on two runs, below the 85% DEC-005 bar. Divergence is systematic (Haiku stricter than Sonnet, concentrated on no-redundant + clarity). Per DEC-005 this points to shipping Haiku as opt-in, NOT the default. Full analysis in docs/research/187-haiku-calibration.md § Result / Disposition. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owner
Author
|
| Run | Comparable | Agreement | Verdict |
|---|---|---|---|
| 1 | 77 | 81.8% | FAIL (<85%) |
| 2 | 74 | 77.0% | FAIL (<85%) |
Both runs miss the DEC-005 ≥85% bar, and the divergence is systematic, not noise: ~80% of discordances are sonnet=pass → haiku=fail — Haiku grades stricter than Sonnet, concentrated on no-redundant (8) and clarity (4). A Haiku default would flag more artifacts than the Sonnet baseline operators calibrate against.
Per DEC-005 (<85% → opt-in, not default), the recommendation is to NOT ship Haiku as the resolved Anthropic grade default — keep Sonnet the default and document grade.model: claude-haiku-4-5 as the opt-in fast mode. The resolver, compat validator, and 1024 cap (US-001..US-006) all stand; only the Anthropic default target would change. OpenAI/Gemini fast defaults are unaffected (explicit operator choices, never calibrated against Sonnet).
Full analysis + the three disposition options: docs/research/187-haiku-calibration.md § Result / Disposition. The gated test deliberately still asserts ≥85% (it fails on Haiku) so the signal isn't silently lost. This is a maintainer product decision.
The empirical calibration (real Sonnet baseline from intuit_airflow) found claude-haiku-4-5 grades stricter than Sonnet (~77-82% concordance, below the 85% DEC-005 bar). Per the decision rule, Haiku does NOT ship as the default. - Rename PROVIDER_FAST_MODELS -> PROVIDER_DEFAULT_MODELS (Sonnet isn't "fast"); anthropic -> claude-sonnet-4-6 (was claude-haiku-4-5). OpenAI/Gemini fast defaults unchanged (gpt-4o-mini / gemini-2.5-flash). - GradeConfig() now resolves to claude-sonnet-4-6; claude-haiku-4-5 is the documented opt-in (grade.model: claude-haiku-4-5). - Revert estimate goldens + grade_event fixture + hand-calc to Sonnet (estimate is independent of max_output_tokens, so origin/dev values apply). - Kept: the per-provider resolver, model<->provider compat validator, the 1024 max_output_tokens cap (still justified by the gemini-2.5-flash default). - Calibration gate now selects Haiku explicitly and still asserts >=85% (fails) as the durable record that Haiku is the stricter opt-in. - Docs/rules/CHANGELOG/writeup updated across all surfaces. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owner
Author
Decision implemented: Sonnet stays the grade default, Haiku is opt-inPer the calibration finding (Haiku ~77-82% concordance vs Sonnet, below the 85% bar), Option 1 is implemented:
ruff + pyright + 3329 passed. All docs/rules/CHANGELOG/writeup surfaces updated. |
gemini-2.5-flash graded a verbose artifact at the new max_output_tokens=1024 default with no score=None truncation degrade — DEC-004 holds at the single-artifact scale (full-fixture runs may still want 4096 per #158). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-defaults # Conflicts: # CHANGELOG.md
- test_haiku_calibration: lazy _substrate import inside the test (no import-time sys.path mutation when deselected) — Copilot. - docstrings: 'never collects' -> 'deselected'; 'Haiku default' -> 'Haiku opt-in' across the calibration tests — Copilot. - llm-providers-ops: drafter row now honest (no per-provider drafter default; switching provider requires llm.model) with a footnote — Copilot. - plan doc: ```text fence on the dependency graph + blank line before the SKU table (markdownlint) — CodeRabbit; Phase -> complete. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owner
Author
PR Review Summary (CodeRabbit + Copilot)Also resolved the Fixed
Addressed by subsequent work (valid when filed, since fixed)
No false positives, no deferrals — every thread is either fixed or addressed above. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Super plan for #187 — switch the default grader to a fast/cheap model, and (per the requested addendum) make the OpenAI and Gemini grade providers resolve to a fast default too.
Phase: detailing (awaiting approval)
Stories: 6 implementation + Quality Gate + Patterns & Memory
Decisions: DEC-001 … DEC-009
Headline decisions
GradeConfig.model→str | None = None, resolved at config-load fromPROVIDER_FAST_MODELS = {anthropic: claude-haiku-4-5, openai: gpt-4o-mini, gemini: gemini-2.5-flash}(all exact pricing keys). Explicitmodel:still honoured.@model_validator(mode="before")(notmode="after"—GradeConfigis frozen).max_output_tokensdefault 256 → 1024 so the new Gemini default doesn't truncate out of the box (supersedes Gemini MAX_TOKENS swallowed; live smoke flakes; per-provider full-pipeline e2e gap #155's 256 for the production default only).provider: openai, model: claude-…at config-load instead of at runtime; single prefix-map source shared withcost/_rollup.py.DraftConfig.cheap_modeldated id → bare SKUclaude-haiku-4-5(lockstep). Drafter stays on Sonnet.Plan document
See
plans/super/187-fast-grade-defaults.md.Next steps
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Configuration Changes
Validation Improvements
Documentation
Tests