chore: release 0.6.0 - #206
Merged
Merged
Conversation
chore: sync main into dev + begin 0.4.0.dev0
* Add super plan for #135: provider-neutral LLM seam Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Mark #135 plan published (PR #148) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Mark #135 plan devolved (beads epic bd_1-scaffolding-j2c) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-j2c.1: US-001 provider foundation (ExceptionCategory, UsageMetrics, LLMProvider ABC, registry) Adds the provider-neutral LLM seam foundation (DEC-001/002/003 of #135): - src/signalforge/llm/providers.py: ExceptionCategory (5-member enum), UsageMetrics (frozen pydantic value object, cache fields default 0), LLMProvider ABC, and the process-level registry (register_provider / provider_for). No vendor behaviour wired; registry starts empty. - UnknownProviderError(LLMError) in errors.py — lists available registered provider names, repr-safe message, default_remediation; registered at CLI exit-code tier 2 (looked-up-identifier-not-in-table input failure). - Exported the new public names from signalforge.llm.__all__; updated the documented-surface lists in test_public_api.py / test_schema.py and the errors count in test_errors.py. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-j2c.2: US-002 AnthropicProvider strategy + _anthropic_client rename + AST scan Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-j2c.3: US-003 generic call_llm orchestrator (rename, capability gating, strategy dispatch) Collapse call_anthropic into a provider-neutral call_llm orchestrator: - Resolve strategy via provider_for(provider); build client via strategy.make_client() when client is None (DEC-006). - Pre-send count gate now gated on strategy.supports_token_count; cache marker / beta header + dual-zero anomaly WARNING gated on supports_prompt_caching (DEC-008). Anthropic = both True ⇒ byte-identical. - Retry loop dispatches on strategy.classify_exception -> ExceptionCategory (DEC-001) instead of catching Anthropic SDK classes directly; per-class budgets, backoff math, WARNING shape, exhaustion raises preserved. - Response assembled via strategy.extract_text_blocks/extract_usage. - Neutral _LLMClientProtocol narrows the resolved client so no vendor-SDK type / type-checker suppression leaks into llm.client (DEC-012 confinement). Drop call_anthropic (name + __all__); add call_llm. Migrate draft_from_request and grade._grade_one call sites + all client/retry/public-api/schema tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-j2c.4: US-004 provider config field + stage threading + CLI client-construction migration DEC-006/007 of #135. Adds a registry-validated `provider: str = "anthropic"` field to DraftConfig and GradeConfig, each with a field_validator that delegates to providers.provider_for (raises UnknownProviderError listing available names on an unknown value — fails loud at config load). Threads provider=config.provider into call_llm from draft_from_request and grade._grade_one. Migrates the CLI generate path off the in-CLI _make_anthropic_client helper (removed): the real-run draft/grade stages now pass client=None so call_llm lazy-builds the client via the configured provider. The --estimate short-circuit (Anthropic-specific count_tokens) builds its concrete client via provider_for(draft_config.provider).make_client(). Migrates the 7 CLI monkeypatch tests: 5 batch/select tests drop the orphaned make_anthropic_client mock; the estimate test patches AnthropicProvider.make_client; test_lint unchanged (it patches the LLM shim, which still exists). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-j2c.5: US-005 no-cache fake provider neutrality proof (AC #2/#3) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-j2c.6: US-006 docs + surface parity (docs/ portion) Update operator-facing docs for the provider-neutral LLM seam (#135): - call_anthropic -> call_llm; llm._client -> llm._anthropic_client - document the provider config knob (llm.provider / grade.provider), default "anthropic", registry-validated, forward-looking plugin seam - note prompt caching + count_tokens are now provider capabilities (supports_prompt_caching / supports_token_count); Anthropic unchanged Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-j2c.6: US-006 .claude/rules seam-rename + provider-seam accuracy Orchestrator-only rule-file edits (workers can't write .claude/ in worktrees). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-j2c.7: Quality gate — fix bugs from code review - providers.build_count_tokens_kwargs: correct docstring (probe omits the cache_control marker because it doesn't affect the token count; the old claim of "matching the inline call_anthropic probe" was inaccurate). - call_llm: gate cache_marker_active on BOTH supports_prompt_caching AND supports_token_count, so a future caching=True/token_count=False provider degrades to no-caching rather than sending an unvalidated cache_control marker (no sub-minimum drop / no oversize cap). Anthropic is True/True so the default path is a no-op. (4-pass review latent-trap finding.) CodeRabbit: skill not available in this environment — skipped. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-j2c.8: Patterns & Memory — provider-seam convention + QG lesson - llm-drafter.md: capability-gating lesson (gate the cache marker on BOTH supports_prompt_caching AND supports_token_count — the #135 QG latent-trap finding) for future #136/#137 provider authors. - Added `provider` to the documented `llm:` config block. - Reference section now cites plans/super/135 + the no-cache neutrality proof. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #135: close Codecov patch gaps — count_tokens probe error mapping + classify fallthrough The provider-neutral refactor rewrote the count_tokens pre-send probe's error-mapping as new lines (auth/rate-limit/conn/5xx/other -> typed LLMError, no retry on the probe) that the existing retry tests (messages.create path) never exercised. Add 6 probe-failure tests + a missing-input_tokens case (client.py 320-348 now covered, 87% -> 96%). Also cover AnthropicProvider.classify_exception's defensive fallthrough for a non-5xx/non-4xx-non-auth APIStatusError (3xx) -> NO_RETRY (providers.py -> 100%). Remaining uncovered client.py helper branches (_extract_text_blocks / _extract_usage_field malformed-response raises) are pre-existing and not part of the #135 diff. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #135: address PR review (CodeRabbit/Copilot) - Stale-symbol docstrings: rename every `call_anthropic` :func: reference to `call_llm` across llm/{__init__,models,_anthropic_client,providers}.py, draft/schema.py, grade/{engine,config,errors}.py, cli/_estimate.py; reword the AnthropicProvider "preparatory refactor" note (US-003 already collapsed the duplication); rewrite the llm package docstring to the provider-neutral seam. - client.py: count_tokens NO_RETRY fallback message no longer claims a "status" (it also covers non-status exceptions) — now "failed with a non-retryable error". - generate.py --estimate: fail fast with CliInputError when draft.provider != grade.provider or the provider isn't 'anthropic' (the estimate engine drives a single Anthropic-cast count_tokens client across both configs). + 2 regression tests. - plan SD-1: drop the stale `Literal` wording (DEC-007 settled it as a registry-validated str). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(readme): refresh status, generalize config, redraft roadmap The v0.1 "Status:" line was four releases stale and BigQuery-only. Replace with a Supported-warehouses block covering BigQuery (v0.1) and Snowflake (v0.3 — incl. the deferred aggregate-only / column_stats limitation), generalize Configuration to dispatch on dbt profile type, and redraft the Roadmap into Shipped (v0.1/v0.2/v0.3 with release dates) + Planned (Gemini/OpenAI → installable skill → Airflow → GitHub Action → rubric customization → dbt Fusion). Drop the now-stale "prune-existing is dev-branch only" and "first four ship; prune-existing is v0.2 dev" callouts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(readme): add custom-business-logic callout, strip non-roadmap version markers - Add a **Custom business logic** paragraph alongside Supported warehouses, explaining `meta.signalforge.business_rules` → `custom_sql` drafting + the ownership-marker write semantics + always-pass drop behaviour. - Remove every version marker outside the Roadmap section: "since v0.1/v0.3" in the warehouses block, "*(v0.2)*" on the prune-existing bullet, the "v0.2 default" config comment, the "(v0.1)"/"(v0.3)" subsection headers, and the "pre-alpha … v0.1 milestone" line in Contributing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(readme): reorganize scope callouts; promote Supported warehouses to its own H2 The two bolded standalone paragraphs at the top of the README (Supported warehouses, Custom business logic) sat before the "Why this exists" pitch and front-loaded dense capability detail. The custom-business-logic block was also triple-covered (standalone block + "What it does" bullet + Quick-start worked example). - Drop both standalone bolded paragraphs from the top. - Promote Supported warehouses to a proper H2 between "How it works" and the "Live on PyPI" callout — brief, points to Configuration for per-warehouse setup; Snowflake's known limitation stays in its single home under Configuration > Snowflake. - Strengthen the custom-business-rule bullet in "What it does" with the "fifth test type" framing and a direct link to the worked example, so the feature stays prominent without a redundant top-of-readme banner. New flow: pitch → features → architecture → scope → quickstart → reference. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* #136: super plan for OpenAI grading provider Phase: detailing (awaiting approval). 13 DECs captured: Chat Completions API, no provider/model cross-validation, fully wire --estimate for OpenAI (new estimate_input_tokens ABC), default judge gpt-4o, scope both grade + draft, server-side response_format=json_object, four pricing SKUs (gpt-4o, -mini, 4.1, 4-turbo), live smoke covers grade + estimate, .messages adapter wraps chat.completions, AST Scan 3 extension, [openai] extra + lazy tiktoken, Anthropic estimate byte-identity floor. 9 stories: shim + AST confinement, OpenAIProvider + registration, FakeOpenAIClient + neutrality test, pricing entries, --estimate provider-aware counting, live gated smokes, docs, Quality Gate, Patterns & Memory. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #136: link plan to PR #152 (phase=published) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #136: apply 5 plan revisions from #137 cross-review Cross-review against #137 (Gemini grading) plan surfaced 5 substantive gaps in the OpenAI plan: 1. DEC-010 — fix wording: a NEW 9th AST scan (Scan 3 is Anthropic-specific), not an extension. Reuse _QualifiedNameCallFinder + three-pattern bypass regression test per testing-signal.md. 2. DEC-012 — fix pyproject.toml listing: three slots in lockstep ([project.optional-dependencies].openai + .dev + [dependency-groups].dev), mirroring Snowflake precedent per python-build.md. Plan previously named only two. 3. DEC-014 (new) — codify the _load_openai_exception_classes empty-tuple fallback on ImportError (mirrors AnthropicProvider). Includes refusal / content-filter symmetry note explaining why OpenAI needs no Gemini-style safety_filter typed-degrade DEC. 4. US-006 — add a third live test (draft_schema) honouring DEC-005's "scope both stages" commitment at live level too. DEC-008 updated to match (three tests, not two). 5. New "Open notes for implementation" section — pragmatic SDK-class-name verification, .messages adapter shape confirmation, tiktoken fallback table, response_format prompt-requirement check, Anthropic byte-identity snapshot capture protocol. Plus a new "Worker-writability routing" section codifying that US-009 (Patterns & Memory) is orchestrator-only because it edits .claude/rules/, mirroring #137's same routing. Plan grows from 319 → 343 lines (+40/-17). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #136: devolve plan into bd beads (epic bd_1-scaffolding-4tw + 9 tasks) Fills in the Beads manifest section. Epic bd_1-scaffolding-4tw with 9 child tasks (US-001 … US-009) created with dep edges per the plan's "Depends on:" lines: US-001 → none (READY) US-002 → US-001 US-003 → US-002 US-004 → none (parallel-safe) (READY) US-005 → US-002, US-004 US-006 → US-002, US-005 US-007 → US-001, US-002, US-005 US-008 → US-001..US-007 (QG) US-009 → US-008 (P&M; orchestrator-only) Cross-epic gate wired downstream: #137's sentinel bd_1-scaffolding-41a now DEPENDS ON US-009, so #137 US-007 stays mechanically blocked until this epic completes + sentinel closes after PR #152 merges to dev. Parallel-safe at the entry points: US-001 and US-004 edit disjoint files (shim/pyproject/AST vs pricing) — ralph-serialize-shared-registry does not apply. Run concurrently. bd ready (post-devolve) surfaces .1 + .4 of this epic + the two epics themselves + the unrelated Airflow integration. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-4tw.4: #136 US-004 — OpenAI pricing SKUs (gpt-4o + 3 siblings) Adds four OpenAI SKUs to _PRICES_MUTABLE per DEC-007: gpt-4o (default judge per DEC-004), gpt-4o-mini (budget), gpt-4.1 (newer flagship), gpt-4-turbo (back-compat). Cache fields = 0.0 (OpenAI has no equivalent cache discount). PRICE_TABLE_VERSION bumped to 2026-05-28. Anthropic SKUs unchanged (byte-identical) — Anthropic estimate byte-identity floor (DEC-013) preserved at the pricing layer. Tests assert: non-zero input/output rates + zero cache fields for all four; unknown model still raises EstimateUnknownModelError; Anthropic SKUs unchanged; version bump pinned. OpenAI per-MTok rates are calibration figures captured at PR-prep time pending operator verification against https://openai.com/api/pricing/; the figures are sanity-check baselines, not billing guarantees. Traces: DEC-003, DEC-004, DEC-007. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-4tw.1: #136 US-001 — _openai_client.py shim + [openai] extra + 9th AST scan Implements the per-vendor shim confining every openai SDK type ignore. Adds [openai] optional-dep extra (openai + tiktoken) in three-slot lockstep across [project.optional-dependencies].openai, .dev, and [dependency-groups].dev. Adds the new 9th AST scan in tests/test_audit_completeness.py reusing _QualifiedNameCallFinder (NOT an extension of Scan 3 — Scan 3 is Anthropic-specific) and a companion per-file confinement test mirroring Snowflake's. Bumps the docstring tally 8→9. DEC-014 empty-tuple ImportError fallback on _load_openai_exception_classes mirrors AnthropicProvider; tiktoken cl100k_base fallback for unknown model ids. Traces: DEC-001, DEC-009, DEC-010, DEC-012, DEC-014. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-4tw.2: #136 US-002 — OpenAIProvider + registration + config-validator coverage Implements OpenAIProvider(LLMProvider) with both capability flags False (no prompt caching, no pre-send token count) — mirrors FakeNoCacheProvider end-to-end shape. build_create_kwargs attaches response_format= {type: json_object} per DEC-006 (server-side JSON enforcement); no cache_control / extra_headers. extract_text_blocks reads response.choices[0].message.content; extract_usage maps usage.prompt_tokens + completion_tokens → UsageMetrics with cache fields 0. classify_exception covers all five ExceptionCategory branches via _load_openai_exception_classes from the US-001 shim. register_provider(OpenAIProvider()) at module scope so GradeConfig/ DraftConfig validators accept provider='openai'. Exported from signalforge.llm.__init__. Tests pin: each ABC method's contract; config validator acceptance for both stages; UnknownProviderError lists both anthropic and openai. Traces: DEC-001, DEC-005, DEC-006, DEC-009, DEC-011. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-4tw.3: #136 US-003 — FakeOpenAIClient + grade neutrality e2e Adds tests/llm/_fake_openai.py with FakeOpenAIClient + expect_messages_create API mirroring FakeAnthropicClient verbatim. Support dataclasses match the real chat-completions response shape (choices[0].message.content; usage with prompt_tokens + completion_tokens). Adds tests/grade/test_provider_neutrality_openai.py — the OpenAI analogue of the FakeNoCacheProvider neutrality proof. Asserts cache_*=0 in JSONL audit, blake2b-8 reproducibility hashes present, sidecar drift-detector round-trip, no dual-zero cache-anomaly WARNING in caplog, all expectations consumed. Traces: DEC-001, DEC-005, DEC-006, DEC-009, DEC-011. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-4tw.5: #136 US-005 — --estimate provider-aware token counting Adds LLMProvider.estimate_input_tokens(model, text, *, client=None) -> int as an abstract method on the ABC (DEC-003). AnthropicProvider impl delegates to the SDK's messages.count_tokens with a single user-message text envelope; OpenAIProvider impl delegates to _count_openai_tokens (tiktoken with cl100k_base fallback per DEC-012); FakeNoCacheProvider and _DummyProvider impls return trivial deterministic answers so the existing neutrality + registry tests still pass. Refactors cli/_estimate.py: _count_draft_tokens and the grader-side equivalent dispatch through provider_for(config.provider).estimate_input_tokens — no more hard-coded anthropic_client.messages.count_tokens calls. The engine signature relaxes anthropic_client to 'object | None' so the OpenAI path can pass None (tiktoken is local). Lifts the '--estimate currently supports only provider=anthropic' gate in cli/generate.py; the divergent-providers check stays (the engine still takes one optional client). Pins DEC-013 Anthropic byte-identity floor via tests/cli/test_estimate.py::test_estimate_anthropic_byte_identity_golden and tests/fixtures/estimate/anthropic_byte_identity_golden.txt (captured 2026-05-28 before the refactor; reproduced verbatim after). New companion tests prove the OpenAI path produces non-zero token counts + non-zero USD, ignores the threaded anthropic_client, and goes through _count_openai_tokens (patched-helper assertion). Traces: DEC-003, DEC-007, DEC-012, DEC-013. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-4tw.7: #136 US-007 — operator docs + CHANGELOG (worker-writable portion) Updates operator-facing documentation surfaces for OpenAI grading + drafting: - docs/grade-ops.md — OpenAI provider section (config snippet, OPENAI_API_KEY, no-cache caveat, live smoke gating link) - docs/draft-ops.md — equivalent for llm.provider: openai - docs/cost-estimate-ops.md — tiktoken note, [openai] extra install, four pricing SKUs - CHANGELOG.md — [Unreleased] Added entry for #136 - README.md — provider enumeration extended (if applicable) The .claude/rules/llm-drafter.md update is deferred to US-009 (Patterns & Memory) per the project's worker-writability rule (orchestrator-only edits under .claude/). Traces: DEC-001 through DEC-014. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-4tw.6: #136 US-006 — live gated smoke tests (openai marker + 3 tests) Adds @pytest.mark.openai marker + three gated live-API tests: - tests/grade/test_smoke_real_api_openai.py — grade_artifacts(provider=openai) - tests/draft/test_smoke_real_api_openai.py — draft_schema(provider=openai) (honors DEC-005 both-stages at live level) - tests/cli/test_e2e_estimate_openai.py — generate --estimate openai All three env-gated on SF_RUN_OPENAI=1 + OPENAI_API_KEY via belt-and- suspenders _skip_reason() helper per testing-signal.md. Marker registered in pyproject.toml + added to addopts exclusion so default pytest doesn't collect them. CONTRIBUTING.md documents the maintainer-only run pattern. Traces: DEC-001, DEC-004, DEC-005, DEC-008. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-4tw.8: #136 Quality Gate — fix 2 majors + 1 minor from code review x4 Code-review pass 3 surfaced two correctness MAJORS + pass 4 surfaced a doc MINOR; all fixed inline per /ralph-run Step 3b. Pass 1, 2, 4 (other surfaces) reported no correctness bugs. MAJOR 1 — EstimateUnknownModelError.default_remediation enumerated only the three Anthropic SKUs. An operator typo on an OpenAI SKU got pointed at Anthropic options (violates "errors carry remediation" in manifest-readers.md). Updated to enumerate all 7 SKUs (claude-sonnet-4-6, claude-opus-4-7, claude-haiku-4-5, gpt-4o, gpt-4o-mini, gpt-4.1, gpt-4-turbo) + refreshed the locked-text test that pinned the stale string. MAJOR 2 — AnthropicProvider.estimate_input_tokens dropped the pre-refactor `system=` kwarg, concatenating system+cached+dynamic into a single user-content string. Anthropic's server-side tokenizer counts the system block with its own envelope tokens; without the kwarg, real-API counts under-report by the system-envelope size. The fake-driven byte-identity test passed only because the fake returned canned input_tokens regardless of kwargs (so rendered-output identity held, but DEC-013's "byte-identity vs the pre-refactor Anthropic count_tokens call" spirit broke at the real-API level). Fix extends the ABC signature with a keyword-only `system: str = ""` parameter. AnthropicProvider passes it via `system=` kwarg when non-empty (matches pre-refactor real-API call shape). OpenAIProvider concatenates `system + text` before tiktoken (tiktoken has no system-envelope distinction; the total still counts every token). FakeNoCacheProvider concatenates for the word-count proxy. _DummyProvider gains the new kwarg for ABC parity. cli/_estimate.py callers (_count_draft_tokens, _count_grade_criterion_tokens) thread `system=` separately; grade-side preserves the pre-existing double-count of the rubric (passed as both system= AND in user content) to keep byte-identity with the pre-refactor shape. MINOR — docs/grade-ops.md:118 config-snippet comment said "only anthropic registered today", contradicting the same file's later "## OpenAI provider" section and docs/draft-ops.md's correct wording. Updated to mirror draft-ops's form. Validation: ruff/pyright/pytest all green (2498 passed, 62 deselected, 97.34% coverage); wheel_smoke green (2 passed). Note: CodeRabbit skill not available in this environment, skipped per Step 3b "(if available)". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-4tw.9: #136 Patterns & Memory — codify OpenAI seam conventions Orchestrator-only commit (workers can't write under .claude/ in worktrees per ralph-worker-claude-dir-perms.md). Updates .claude/rules/llm-drafter.md to capture three durable patterns from #136: 1. Provider-neutral seam — adds OpenAI as the second registered provider (name="openai", both capability flags False) and extends the LLMProvider ABC surface enumeration with the new estimate_input_tokens(model, text, *, system="", client=None) method (#136 US-005). 2. "OpenAI provider shape" subsection — codifies the .messages.create façade adapter pattern (SimpleNamespace delegating to chat.completions.create), the response_format={type:json_object} server-side JSON enforcement (with the cross-ref to issue #144's tolerant parser as fallback), and the tiktoken cl100k_base fallback for unknown model ids. This becomes the canonical precedent for #137 Gemini's response_mime_type=application/json equivalent. 3. estimate_input_tokens(*, system=...) — documents the load-bearing reason the system envelope is threaded separately (Anthropic's server-side tokenizer applies system-block envelope tokens; dropping the kwarg under-reports real-API counts silently because fake-driven byte-identity tests can't catch call-shape drift). Also pins the deliberate double-count of system_and_rubric in _count_grade_criterion_tokens as pre-existing behaviour that must be preserved. Bumps AST-scan tally section from "four" → "five" (adds the openai.OpenAI 9th-project-scan); flags #137 Gemini as the 10th. Updates the References block with #136 plan + new files. Memory entry: ~/.claude/projects/.../memory/fake-driven-byte-identity- blind-spot.md captures the QG lesson (fakes return canned values regardless of kwargs → call-shape drift slips past rendered-output snapshots → needs explicit kwargs-shape assertion OR live test). Indexed in MEMORY.md. Validation: all four canonical gates green (2498 passed, 97.34% coverage). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #136: address PR #152 review (6 actionable + meta) Copilot: - MAJOR _estimate.py:359 — OpenAI grade-side over-counted rubric. The first QG fix preserved a pre-existing Anthropic double-count of system_and_rubric (passed as both system= AND in user content), which triple-counted on OpenAI (system→system+text concat → rubric prefix in text). Corrected to match the runtime grader call: rubric in system= once, artifact envelope in user content. Anthropic real- API counts drop one rubric copy from the buggy pre-refactor bytes; OpenAI counts each input once. Fake-driven byte-identity golden still passes (canned token counts are call-shape-agnostic). CHANGELOG documents the calibration shift. - NIT _estimate.py:421 — renamed anthropic_client → client. Type was already object | None and forwarded to whichever provider strategy is active; old name implied Anthropic-only and would mislead a future #137 Gemini wiring. CLI in generate.py already passed None for non-Anthropic; rename surfaces the contract without behaviour change. - META plan/PR description — addressed by updating the PR body separately (not part of this commit). CodeRabbit: - CONTRIBUTING.md:64 — added snowflake to the gated-marker audit command + the corresponding SNOWFLAKE_* env vars. - tests/llm/_fake_provider.py:234 — boundary-word undercount fix in the FakeNoCacheProvider word-count proxy (f"{system} {text}" with a delimiter; was system+text which merged the last word of system with the first word of text under .split()). - tests/llm/test_openai_client_confinement.py:47 — glob("*.py") → rglob("*.py") so the confinement scan catches openai-mentioning ignore directives in any future nested signalforge/llm/ subpackage, not just top-level files. Path display becomes relative-to-_LLM_DIR. - tests/test_audit_completeness.py:1080 — closed the "call-before- import alias" bypass on _AttributeCallFinder by adding a two-pass visit_Module that pre-collects every alias module-wide before visiting any Call node. Mirrors the same fix already shipped on _QualifiedNameCallFinder (PR #69 / DEC-013). Pattern-4 regression test added. Validation: ruff/format/pyright/pytest all green (2498 passed, 97.34% coverage); wheel_smoke green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #136: close PR #152 codecov gaps (9 lines → 0) Codecov flagged 91.82% patch coverage with 9 lines missing across 3 files. Closed all 9 with 6 focused unit tests: src/signalforge/llm/_openai_client.py (was 85%, now 100%): - _OpenAIClientAdapter.__init__ + ._messages_create — no production caller exercises the adapter (orchestrator drives via FakeOpenAIClient which has its own .messages.create); added test_openai_client_adapter_messages_create_delegates_to_chat_completions that builds a SimpleNamespace raw client and pins the delegation + kwargs forwarding. - _count_openai_tokens cl100k_base fallback (DEC-012 unknown-model branch) — added test_count_openai_tokens_falls_back_to_cl100k_base_ for_unknown_model. src/signalforge/llm/providers.py (was 98%, now 100%): - AnthropicProvider.estimate_input_tokens no-system-kwarg arm (the else branch added in QG) — added test_anthropic_provider_estimate_ input_tokens_skips_system_kwarg_when_empty, which ALSO pins the load-bearing invariant that an empty system MUST be omitted from the SDK kwargs (otherwise the count carries a spurious system="" block). - LLMResponseFormatError raise on missing input_tokens — added test_anthropic_provider_estimate_input_tokens_raises_on_missing_ input_tokens. - OpenAIProvider.extract_text_blocks missing-message arm — added test_openai_provider_extract_text_blocks_missing_message_attr_ raises (distinct from the existing content=None and empty-choices cases — those exercise different arms). src/signalforge/cli/generate.py:846 (was 66%, now covered): - The non-Anthropic `client = None` branch in cmd_generate's --estimate short-circuit — exercised only by the @pytest.mark.openai live smoke before. Added test_generate_ estimate_openai_provider_passes_client_none_to_engine that drives cmd_generate with llm.provider: openai + grade.provider: openai in a tmp signalforge.yml, spies on AnthropicProvider.make_client to confirm it's NEVER called on the openai path, and captures the client kwarg into estimate(...) to pin client=None. The remaining 7 uncovered lines in generate.py are pre-PR and out of scope. Total: 2504 passed (was 2498), 97.47% coverage (was 97.34%), no new gated tests added (all unit tests, default-CI-included). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* #137: Gemini grading provider (plan) Super plan for #137 (Gemini grading provider) — depends on #135 (merged). 9 stories: shim + provider + extra + fake/unit + neutrality + live + docs + QG + P&M. Caching deferred (both capability flags False); safety-filter no-content surfaces as LLMResponseFormatError → grade GradeLLMError degrade. Both drafter and grader covered. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #137: extend plan after comparison with #136 OpenAI plan Closes the three gaps surfaced by the #136 comparison: - Add --estimate integration (DEC-016): GeminiProvider.estimate_input_tokens via native client.models.count_tokens (cleaner than #136's tiktoken path — Gemini has a first-party count endpoint). - Add 3 Gemini pricing SKUs to pricing.py (DEC-017): gemini-2.5-pro, gemini-2.5-flash, gemini-2.0-flash; bump PRICE_TABLE_VERSION. - Add server-side JSON enforcement (DEC-018): response_mime_type= application/json — mirrors #136 DEC-006. - Add CHANGELOG entry to docs story (US-009). - Add wheel_smoke to QG (new [gemini] extra changes packaging). - Sequence after #136 (DEC-019): inherit ABC extension + estimate refactor. New stories: US-006 (pricing, parallel-safe), US-007 (estimate impl, depends on #136). Old US-006/007 renumbered to US-008/009. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #137: apply 3 plan refinements from #136 cross-review Closes the smaller gaps surfaced by re-reviewing #137 against #136 after the #136 plan was tightened (520b36c): 1. DEC-009 — merge-order-conditional AST scan tally. Currently said "8 → 9" unconditionally, but DEC-019 specifies #137 ships AFTER #136. Realistic bump is 9 → 10 (since #136's openai.OpenAI scan bumps 8 → 9 first); added the fallback wording for the slipped-sequencing case. 2. Dedicated "Worker-writability routing" top-level section codifying that Patterns & Memory is orchestrator-only (.claude/rules/ writes). Was a one-line architecture-review row; now a durable section that mirrors #136's codification — future #138+ provider plans copy the pattern. 3. Open notes for implementation — two additions: - response_mime_type requires NO prompt keyword (contrast with OpenAI's response_format which fails without "json" in prompt). Prevents a future maintainer adding defensive prompt text. - pricing.lookup zero cache fields math validation (symmetric to #136 verification item). Major gaps from the prior cross-review (--estimate, pricing, dual-listing, JSON enforcement, CHANGELOG, three live tests, sequencing) were already addressed in b28c0b5. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #137: devolve plan to beads (phase=devolved) Epic: bd_1-scaffolding-txe Tasks: 11 (US-001 through US-009 + Quality Gate + Patterns & Memory) Dependencies: 19 edges per the plan's dependency graph Ready queue: US-001 (shim) + US-006 (pricing, parallel-safe) US-007 (--estimate integration) is blocked on US-002 + US-006 AND carries a cross-epic gate on #136 landing first per DEC-019 — that sequencing is documented in the task's description rather than wired as a bead dep since #136's beads live in a different epic. US-011 (Patterns & Memory) is flagged orchestrator-only because it edits .claude/rules/ which Ralph workers can't write under in worktrees (per the Worker-writability routing section of the plan). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #137: adjust plan now that #136 is being implemented first Folds devolve-time annotations (phase=devolved, beads epic bd_1-scaffolding-txe, per-task status + cross-epic blocker on US-007) together with five #136-driven adjustments: 1. AST-scan tally locked to 10th (not 9th) — #136 owns the 8→9 bump for openai.OpenAI; #137 takes 9→10 for genai.Client. Updated in DEC-009, US-001, P&M, and Open notes. 2. DEC-019 sharpened: #136 plan → #136 implementation; explicit rebase guidance (rebase on dev after #136 merges) rather than contingency wording. 3. DEC-005 gains a refusal/content-filter symmetry note cross-ref'ing #136 DEC-014 — explains why OpenAI deliberately ships no safety-filter typed-degrade and why Gemini deliberately does. 4. New "Worker-writability routing" section mirroring #136's, codifying that P&M is orchestrator-only because it edits .claude/rules/. 5. Open notes extended with two pragmatic items parallel to #136's: - google-genai count_tokens response field-name verification (.total_tokens vs .total_token_count) at US-007 implementation. - Anthropic byte-identity snapshot ownership: #136 owns it; #137 inherits it; the wiring is wrong if it moves the snapshot. Plan grows 762 → 825 lines (+70/-30). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-txe.6: #137 US-006 — add Gemini pricing SKUs (gemini-2.5-pro, gemini-2.5-flash, gemini-2.0-flash) Adds three Gemini SKUs to _PRICES_MUTABLE per DEC-017, each with positive input_per_mtok / output_per_mtok per Google's public Gemini API price page and cache fields = 0.0 (v0.3 ships Gemini without an Anthropic-equivalent prompt-cache discount). Bumps PRICE_TABLE_VERSION to "2026-05-27". Tests parametrise the three SKUs, pin Anthropic SKU byte-identity (additive-only), and assert lookup("gemini-unknown") still raises EstimateUnknownModelError. cli/_estimate.py reads only input_per_mtok and output_per_mtok in the USD math (lines 468, 511), so zero cache fields are safe — no divide-by-zero or NaN risk. The EstimateUnknownModelError.default_remediation in llm/errors.py still lists only the three Claude SKUs by name; per the story scope ("pricing-only; DO NOT touch any other file") this is left for a follow-up to broaden once the wider Gemini surface (provider class + --estimate integration in US-007) lands. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-txe.1: #137 US-001 — _gemini_client.py shim + new AST confinement scan Land the per-vendor Gemini SDK shim and the 9th AST audit-completeness scan that confines genai.Client(...) constructions to it. Mirrors the Anthropic shim shape verbatim — GeminiClientProtocol (with .messages façade for US-002 and .models for US-007), lazy-import factory, frozen _GeminiExceptionClasses dataclass with empty-tuple fallback per DEC-015 so the module imports cleanly without the [gemini] extra installed. The new scan extends _AttributeCallFinder with an optional parent_module parameter so the namespace-package shape `from google import genai; genai.Client(...)` is caught alongside `from google.genai import Client` and its alias variant — five planted-violation regression tests pin the three bypass patterns plus a negative case and an Anthropic-path-unchanged guard. A line-based confinement test (tests/llm/test_gemini_client_confinement.py) rejects any google.genai-mentioning `# type: ignore` outside the shim, mirroring tests/warehouse/test_snowflake_client_confinement.py. The existing Anthropic shim's regex-level confinement test narrowed from "any ignore" to "anthropic-mentioning ignore" so the new Gemini shim's own SDK ignores don't trip it. All four canonical validation steps pass; 2474 tests; coverage 97.33%. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-txe.3: #137 US-003 — pyproject.toml [gemini] extra + dev-group sync google-genai>=0.5,<1 listed in three pyproject slots in lockstep per DEC-010 (operator install, pip dev back-compat, uv dev group). uv.lock regenerated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-txe.2: #137 US-002 — GeminiProvider(LLMProvider) + registration GeminiProvider concrete strategy registered at module import; both capability flags False per DEC-003. build_create_kwargs sets response_mime_type= application/json (DEC-018) and maps system → system_instruction + single user turn (DEC-004). Safety-blocked / no-content responses → LLMResponseFormatError per DEC-005. Exception taxonomy per DEC-006 — verified against google-genai==0.8.0. Notes from verification against the installed SDK: - google.genai.errors.APIError (the shared parent of ClientError + ServerError) stores the HTTP code on .code (an int), not .status_code — the classifier reads .code. ClientError covers 4xx (incl. 401/403/429); ServerError covers 5xx. ServerError is checked BEFORE ClientError so the narrower 5xx bucket cannot be shadowed by a future shared parent class. - The SDK's models.generate_content(config=) accepts both GenerateContentConfig and the plain-dict GenerateContentConfigDict form. build_create_kwargs returns the plain-dict form so providers.py never imports google.genai.types at any scope — keeping test_gemini_client_confinement.py green and base-install module-import clean. - The .messages.create façade required by the orchestrator is wired in GeminiProvider.make_client via a small _GeminiClientAdapter / _GeminiMessagesAdapter pair (the shim returns the bare client; US-002 owns the adapter per the shim's docstring). The adapter forwards **kwargs straight to client.models.generate_content / count_tokens. - Connection-flavoured exceptions (httpx.ConnectError / httpx.TimeoutException) leak through the SDK on hard network failures; the classifier handles them with a lazy httpx import and an ImportError-safe fallback to NO_RETRY. Updates the pinned signalforge.llm __all__ surface in tests/llm/test_public_api.py and tests/draft/test_schema.py to include the new GeminiProvider re-export; adds DraftConfig/GradeConfig provider="gemini" acceptance tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-txe.9: #137 US-009 — operator-facing docs + CHANGELOG for Gemini docs/grade-ops.md + docs/draft-ops.md register gemini as a provider, name the [gemini] install extra + GOOGLE_API_KEY env var, and document the v0.3 no-caching cost note (DEC-013). README provider list deferred (no list exists to extend). CHANGELOG entry under [Unreleased]. .claude/rules/* deferred to Patterns & Memory (orchestrator-only). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-txe.4: #137 US-004 — FakeGeminiClient + offline provider integration tests Hand-rolled FakeGeminiClient mirrors FakeAnthropicClient's expect_* API (DEC-011). New tests/llm/test_fake_gemini.py proves the fake's contract; tests/llm/test_gemini_provider_via_fake.py drives GeminiProvider through the fake end-to-end (call_llm round-trip, safety-blocked branch, retry exhaustion). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-txe.5: #137 US-005 — provider-neutrality e2e tests (draft + grade) tests/grade/test_gemini_neutrality.py drives grade_artifacts end-to-end through FakeGeminiClient: cache_*=0 in JSONL, blake2b-8 hashes, sidecar round-trip, no dual-zero WARNING (DEC-003), safety-blocked → GradeLLMError degrade (DEC-005). tests/draft/test_gemini_neutrality.py drives draft_schema end-to-end likewise. DEC-014 two-stage scope satisfied at the offline-test layer. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-txe.8: #137 US-008 — gemini marker + live tests (raw + draft + grade) + CONTRIBUTING @pytest.mark.gemini marker registered; addopts excludes it from default CI per DEC-012. Three live smokes gated SF_RUN_GEMINI=1 + GOOGLE_API_KEY: test_gemini_live.py (raw call_llm), test_gemini_draft_live.py (draft_schema), test_gemini_grade_live.py (grade_artifacts 1-criterion × 1-artifact). CONTRIBUTING.md adds the maintainer 'uv run pytest -m gemini --no-cov' entry alongside the existing snowflake/anthropic equivalents. --estimate live test deferred to US-007 (gated on #136). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #137: harden DEC-019 cross-epic gate via bd sentinel bead DEC-019 was documentary only; bd_1-scaffolding-txe.7 (US-007) showed as bd-ready despite the plan saying "wait for #136 to merge to dev." A Ralph worker running `bd ready` would have picked it up and rebase-fought #136 on providers.py / pricing.py / cli/_estimate.py exactly as DEC-019 warns against. Created sentinel bead bd_1-scaffolding-41a ("#136 OpenAI grading PR #152 merged to dev") and wired bd_1-scaffolding-txe.7 to depend on it. `bd ready` no longer surfaces US-007 until the sentinel closes. Close the sentinel the moment #136 merges to dev → US-007 unblocks automatically. Updated DEC-019 + the beads manifest entry on US-007 to point at the sentinel. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-txe.7: #137 US-007 — GeminiProvider.estimate_input_tokens + --estimate integration Replaces the merge-resolution NotImplementedError stub with a real implementation via Gemini's native client.models.count_tokens (first-party; no tiktoken equivalent). The --estimate cost-preview path now works end-to-end for grade.provider: gemini and llm.provider: gemini. Anthropic byte-identity golden unchanged (verified against tests/fixtures/estimate/anthropic_byte_identity_golden.txt). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-txe.10: #137 Quality Gate — fix bugs from 4-reviewer pass Four parallel code-review passes surfaced 2 substantive code findings and 5 documentation findings; all fixed below. CodeRabbit run skipped (no skill available locally); maintainer can request post-merge. Code fixes (reviewer 2 — tests): - tests/test_audit_completeness.py: added the missing planted-violation regression test for the new parent_module codepath in _AttributeCallFinder. Covers Patterns 5-8 (namespace-package via from-google-import-genai, namespace alias, dotted-from-import, dotted-import-as) plus a negative check pinning that Pattern 7 requires parent_module to be set. Mandated by testing-signal.md § "AST single-construction-seam scans must catch all three bypass patterns" — without this, a refactor of the parent_module branches could silently break Scan 10 at the exact moment a real Gemini-SDK construction was added outside _gemini_client.py. - tests/grade/test_gemini_grade_live.py: added cache_*=0 assertion on every GradeEvent record (DEC-003 of #137). The docstring claimed this was tested but the assertion was missing — a grade-path bookkeeping regression would have slipped through the live smoke. Doc fixes (reviewer 3 — docs): - docs/grade-ops.md + docs/draft-ops.md: rewrote the "--estimate integration (deferred)" sections to reflect that US-007 SHIPPED in this PR. Both now describe the active behaviour (native client.models.count_tokens round-trip; failures surface as <unavailable>) instead of the pre-US-007 deferred state. - CHANGELOG.md: updated the Gemini Unreleased bullet to advertise --estimate support as part of the #137 deliverable; reserved the "follow-up" framing for explicit Gemini context caching only. - docs/cost-estimate-ops.md: added Gemini coverage in three places — the provider-aware token counting bullet list, a parallel "Gemini provider — [gemini] install extra" section (with the three registered SKUs in a table), and the maintainer live smoke set. File had ZERO Gemini coverage before this fix. - README.md: added Google Gemini to the supported-providers section alongside Anthropic + OpenAI; moved the Gemini roadmap row from "Planned v0.4" into "Shipped v0.4" since it's landing in this PR. Reviewer 1 (providers.py) and reviewer 4 (packaging + integration) reported 0 bugs. Reviewer 2's LOW-priority findings (flake-prone aggregate_complete assertion, loose substring on confinement scan, missing 5xx-retry integration test) deferred — not correctness risks. Validation after fixes: - ruff check: passed - ruff format --check: passed - pyright: 0 errors, 0 warnings, 0 informations - pytest: 2555 passed, 65 deselected, 97.51% coverage - wheel_smoke: 2/2 passed (new [gemini] extra packaging intact) - Anthropic byte-identity golden: passed (unchanged) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-txe.11: #137 Patterns & Memory — durable rules for Gemini Orchestrator-only commit (Ralph workers can't write under .claude/ in worktrees per memory ralph-worker-claude-dir-perms.md). .claude/rules/llm-drafter.md: - AST scan tally bumped 5 → 6 (Scan 10 = genai.Client confinement via _AttributeCallFinder(parent_module="google")). - New § "Gemini provider shape (#137 — the third concrete provider, no-cache via a namespace-package SDK)" with the four load-bearing patterns: .messages-over-.models.generate_content adapter, response_mime_type=application/json server-side JSON enforcement, safety-filter typed LLMResponseFormatError → grade degrade, and native models.count_tokens for --estimate (distinct from OpenAI's local tiktoken path). - New § "Namespace-package SDKs (#137 generalisation)" documenting the parent_module parameter that catches the four namespace-package import shapes the no-parent path misses. - Reference block extended with the #137 plan + the three new fakes + the two new neutrality test suites. - "new vendor lands" wording generalised (#137 is no longer "next", it's "shipped"); Anthropic→OpenAI→Gemini lineage noted explicitly. .claude/rules/grade-layer.md: - Conservative degrade taxonomy DEC-002/DEC-015 entry for GradeLLMError extended with a sentence explaining Gemini's safety-filter / no-content response routes through the same path via the typed LLMResponseFormatError that GeminiProvider raises (DEC-005 of #137). Locks in the provider-neutral contract: a future vendor with a content-filter surface MUST route through LLMResponseFormatError, not a provider-specific switch in grade_artifacts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #137: Address PR review feedback (Copilot + CodeRabbit) Five real issues fixed; three outdated threads resolved as-is. Fixed: 1. src/signalforge/llm/_gemini_client.py — Copilot flagged that GeminiClientProtocol's docstring claimed the bare SDK client satisfies it, and _make_gemini_client was annotated to return GeminiClientProtocol despite returning the raw google.genai.Client (which has no .messages namespace). The protocol is satisfied only by the wrapped _GeminiClientAdapter (in providers.py) and by the test fake. Rewrote both docstrings; relaxed _make_gemini_client's return type to `Any` and explained the wrapper-on-top contract explicitly. Replaces the misleading "satisfies structurally" claim with the honest "bare SDK does NOT satisfy; wrapper does." 2. plans/super/137-gemini-grading.md (Discovery, line 98) — CodeRabbit flagged the stale PRICE_TABLE_VERSION = "2026-05-11". Annotated the reference as "at discovery time" and noted that US-006 and #136 both bump it. 3. plans/super/137-gemini-grading.md (Worker-writability routing) — CodeRabbit flagged a duplicated "## Worker-writability routing" heading. The duplicate must have crept in during the cross-review pass. Removed the second copy; kept the first canonical version. 4. tests/llm/test_pricing.py (test_lookup_raises_estimateunknownmodelerror_for_unknown_gemini_model) — CodeRabbit flagged that the test only validates `.model`, leaving the operator-facing remediation text drifting silently from the pricing table. Added a structural pin that the rendered exception names every Gemini SKU. Future SKU additions force a lockstep remediation update. Outdated threads (3) — content already fixed by earlier commits; the threads are kept resolved for hygiene: 5. tests/test_audit_completeness.py docstring "node.module == genai" note (Copilot, two threads). The actual scan handles the dotted- form via parent_module; the comments in the implementation already reflect this since the QG pass. 6. CONTRIBUTING.md pre-release excluded-marker list (CodeRabbit). The list was already extended to include `gemini` (and `snowflake`) during the QG fix in commit f15a469. Validation after fixes: - ruff check: passed - ruff format --check: 279 files formatted - pyright: 0 errors, 0 warnings, 0 informations - pytest: 2555 passed, 65 deselected, 97.51% coverage Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #137: Address second-round PR review (Copilot, 4 new comments) Two real findings fixed; two false positives documented. Fixed: 1. pyproject.toml gemini marker description — Copilot flagged that the help text named only GOOGLE_API_KEY but the live tests also gate on SF_RUN_GEMINI=1. Updated to "requires SF_RUN_GEMINI=1 + GOOGLE_API_KEY" — mirrors the OpenAI marker's shape exactly. 2. src/signalforge/llm/providers.py::GeminiProvider.extract_usage — Copilot flagged that getattr(usage, "prompt_token_count", 0) or 0 silently swallows missing/malformed SDK response shapes and feeds misleading 0/0 figures into the audit JSONL + --estimate math. Switched to the shared _extract_usage_field helper which raises LLMResponseFormatError on missing/non-int fields. Matches the Anthropic precedent (the OpenAI provider already uses the same helper). Added test_geminiprovider_extract_usage_missing_inner_ field_raises pinning both the missing-field and non-int-type paths to keep them durable. False positives (replied in resolve-threads): 3. _gemini_client.py:116 (`_make_gemini_client` return annotation): Copilot is reviewing the pre-commit-00dc673 diff. The previous round of fixes (commit 00dc673) already changed the return type to Any and rewrote the docstring to be honest that the bare SDK client does NOT satisfy GeminiClientProtocol; only the _GeminiClientAdapter wrapper does. Current file matches. 4. plans/super/137-gemini-grading.md:777 (duplicate Worker- writability routing): Same — the duplicate was already removed in 00dc673. Only one section now exists at line 757. Validation: - ruff check: passed - ruff format --check: passed - pyright: 0 errors, 0 warnings, 0 informations - pytest: 2556 passed, 65 deselected, 97.55% coverage Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* #155: super-plan — Gemini truncation + per-provider e2e gap 12 DECs covering: - DEC-001/002/005: provider-neutral is_clean_completion ABC method - DEC-003/011: 2 new e2e siblings + BQ smoke parametrize - DEC-008/009: per-provider max_output_tokens floor table (1024/1024/2048) - DEC-010: pre-release-only cadence (~\$0.30 / suite run) - DEC-012: apply_provider_override helper 10 stories sized for Ralph contexts. Architecture review: 1 concern (seam design — resolved), 8 pass. Regression risk green (no fixture pins old reasoning string; existing test_gemini_neutrality.py:381 already pins the post-fix shape). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #155: devolved — epic + 10 tasks live in bd Epic: bd_1-scaffolding-eu0 Ready set: US-001 (.2), US-003 (.4), US-004 (.5) — three parallel-safe. 16 dep links wired. Serialization callouts captured (US-005/6/7 share _e2e_helpers + BQ smoke; US-002 + US-010 edit .claude/rules/, orchestrator-only). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-eu0.4: #155 US-003 — bump Gemini live fixture to 2048 + per-provider max_output_tokens floor docs * bd_1-scaffolding-eu0.5: #155 US-004 — apply_provider_override helper + BQ smoke uses it Adds the canonical per-test grade-provider overlay helper. Multi-provider e2e smokes (BigQuery+Anthropic / +OpenAI / +Gemini) share the committed Austin fixture and swap only grade.provider/model/max_output_tokens via this seam — no near-duplicate fixtures. The helper is non-destructive: unset knobs left alone, sibling top-level blocks (llm:/safety:/prune:) round-trip via yaml.safe_dump(sort_keys=False). Missing signalforge.yml raises FileNotFoundError rather than silently creating one (masks misconfigured tests). Unit-tested in tests/cli/test_e2e_helpers.py — runs in the default pytest set (no marker), so a regression in the YAML overlay plumbing the real e2e smokes depend on trips immediately. BigQuery smoke refactored to call apply_provider_override(project_dir, grade_provider='anthropic') as a no-op proof-of-use; OpenAI (US-005) and Gemini (US-006) sibling smokes will pass non-default values through the same seam. Traces to: DEC-012 in plans/super/155-gemini-truncation-e2e-gap.md * bd_1-scaffolding-eu0.2: #155 US-001 — LLMProvider.is_clean_completion ABC + 3 concretes + wire-in Add provider-neutral allowlist gate for response finish-reason / stop-reason to prevent silent pass-through of truncated / safety-filtered / tool-use responses (DEC-001/002/005/006/007 of plans/super/155). - LLMProvider.is_clean_completion(response) -> bool — abstract - LLMProvider.unclean_finish_reason_message(response) -> str — default + per-vendor overrides - AnthropicProvider._CLEAN_STOP_REASONS = {end_turn, stop_sequence}; tool_use is UNCLEAN (DEC-006) - OpenAIProvider._CLEAN_STOP_REASONS = {stop} - GeminiProvider._CLEAN_STOP_REASONS = {STOP} - call_llm wires the gate immediately before strategy.extract_text_blocks, raising LLMResponseFormatError (typed, non-retryable response-shape error) with the provider-specific diagnostic when the gate returns False. - _DummyProvider + FakeNoCacheProvider satisfy the new abstract by returning True (no finish-reason concept on the canned shapes). TDD: 4 happy-path tests added (one per concrete provider + 2 for Anthropic covering both clean stop reasons) and confirmed failing before implementation, then green. Canonical validation quad (ruff/format/pyright/pytest) all-green; 2560 tests pass (no regressions); AST scans 3/9/10 pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-eu0.6: #155 US-005 — new tests/cli/test_e2e_openai_smoke.py Full-pipeline live e2e gated by @pytest.mark.e2e + @pytest.mark.openai markers and a three-env-var skip gate (SF_RUN_OPENAI=1, OPENAI_API_KEY, GOOGLE_CLOUD_PROJECT — BigQuery stays the warehouse). Mirrors tests/cli/test_e2e_bigquery_smoke.py verbatim and only swaps the grader via apply_provider_override(grade_provider='openai', grade_model='gpt-4o') per DEC-011/DEC-012 of plans/super/155-gemini-truncation-e2e-gap.md; drafter stays Anthropic Sonnet per the fixture's llm.model pin and the cost-table rationale (DEC-009). Pins the seven invariants from the BQ smoke (DEC-009 of #10): exit 0, sidecar present, kept+flagged+dropped>=1, always-passes drop present (warehouse-side, provider-independent), flagged_count>=1 (tight grade thresholds), GradingReport.aggregate_complete=True (the cross-provider contract the in-isolation grade smokes can't pin), and no traceback in stderr (cli-layer.md DEC-016). Test is deselected by default addopts; maintainer runs once pre-release. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-eu0.7: #155 US-006 — new tests/cli/test_e2e_gemini_smoke.py with max_output_tokens=2048 overlay Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-eu0.3: #155 US-002 — per-provider unclean-path tests + call_llm integration (rule edit deferred to orchestrator) Per-provider unclean-path tests pinning the LLMProvider.is_clean_completion contract (#155 DEC-001/DEC-002/DEC-005/DEC-006/DEC-007): - tests/llm/test_anthropic_provider_via_fake.py: + max_tokens (with partial text), tool_use, unclean_finish_reason_message naming stop_reason. - tests/llm/test_openai_provider_via_fake.py: + length (with partial text), content_filter, tool_calls, unclean_finish_reason_message naming finish_reason. - tests/llm/test_gemini_provider_via_fake.py: + MAX_TOKENS-with-partial-text (the LOAD-BEARING #155 Finding 1 regression pin, with full context comment), SAFETY/RECITATION/OTHER parametrized, unclean_finish_reason_message naming finish_reason. - tests/llm/test_client.py: + call_llm integration test asserting LLMResponseFormatError raises at the is_clean_completion gate (post-messages.create, pre-extract_text_blocks, no retry). Existing contract pin tests/grade/test_gemini_neutrality.py:381 continues to pass unmodified — the safety-blocked Gemini path now routes through the same orchestrator gate as MAX_TOKENS, both landing at 'call failed: GradeLLMError' degrade. Note: .claude/rules/llm-drafter.md edit deferred — per memory ralph-worker-claude-dir-perms, .claude/ writes are orchestrator-only and will land in a separate commit after this merge. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-eu0.8: #155 US-007 — parametrize tests/cli/test_e2e_bigquery_smoke.py over grade.provider Adds @pytest.mark.parametrize over grade_provider ∈ [anthropic, openai, gemini] to the BigQuery e2e smoke test. Per #155 DEC-003 this covers the cross-provider diff-sidecar rendering contract the in-isolation grade smokes (tests/grade/test_*_grade_live.py) cannot pin: those smokes never exercise the diff-sidecar evidence/reasoning cascade with a non-Anthropic judge. Per #155 DEC-011 the drafter stays Anthropic Sonnet across all three variants (fixture stability — the LLM payload the drafter sees is unchanged, so the always-passes column the LLM proposes is reproducibly the same). Only the grader varies, via the canonical apply_provider_override helper (US-004 / DEC-012). Per-variant env-var gates layered on top of the existing baseline: - anthropic: baseline only (SF_RUN_BQ + ANTHROPIC_API_KEY + GOOGLE_CLOUD_PROJECT) - openai: baseline + SF_RUN_OPENAI + OPENAI_API_KEY - gemini: baseline + SF_RUN_GEMINI + GOOGLE_API_KEY Gemini variant pins max_output_tokens=2048 per #155 Finding 2 — Gemini 2.5-flash's verbose reasoning field truncates mid-string at the default 512/1024 floors and would flake assertion #6 (aggregate_complete). Sibling files (test_e2e_openai_smoke.py / test_e2e_gemini_smoke.py) remain per DEC-011 for per-provider failure ergonomics and cost transparency; this parametrize is internal to the BQ smoke. Pytest IDs: test_*[anthropic] / test_*[openai] / test_*[gemini]. All three are deselected by default addopts (gated by the e2e marker); the maintainer runs them per pre-release live suite (US-008). Validation: uv sync --dev && ruff check && ruff format --check && pyright && pytest — all green (2566 passed, 69 deselected, coverage 97.40%). * bd_1-scaffolding-eu0.3: #155 US-002 (orchestrator portion) — clarify llm-drafter.md DEC-005 for the is_clean_completion seam Updates the Gemini-section DEC-005 contract to reflect the post-#155 generalisation: - Rule now applies to ALL providers (Anthropic stop_reason, OpenAI choices[0].finish_reason, Gemini candidates[0].finish_reason.name) — not just Gemini - Enforcement moved from "no text at all" check buried in extract_text_blocks to the new LLMProvider.is_clean_completion(response) -> bool ABC method (#155 DEC-005) - Closes the Finding-1 gap: MAX_TOKENS with partial text now surfaces as LLMResponseFormatError → GradeLLMError, not GradeOutputError(json_parse) - Per-provider _CLEAN_STOP_REASONS enumerated; tool_use deliberately UNCLEAN per DEC-006 - unclean_finish_reason_message override (DEC-007) keeps vendor-native field names visible Test-side pins (worker portion of US-002, commit 9a23e3f): - tests/llm/test_{anthropic,openai,gemini}_provider_via_fake.py — unclean-path coverage - tests/llm/test_client.py — call_llm gate integration Worker (commit 9a23e3f) deferred this rule edit per memory ralph-worker-claude-dir-perms; this commit closes the bead's orchestrator-only deliverable. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-eu0.9: #155 US-008 — CONTRIBUTING.md live-suite pre-release cadence + env-var block Add a 'Live e2e suite (pre-release only)' subsection to CONTRIBUTING.md per DEC-010 of plans/super/155-gemini-truncation-e2e-gap.md. Documents the full pre-release maintainer audit: - 5 paid e2e tests (BQ smoke parametrized over 3 grader providers per US-007, OpenAI sibling, Gemini sibling with max_output_tokens=2048 floor per DEC-008, Snowflake sibling) - 6 grade-only / draft-only live-API smokes gated by anthropic / openai / gemini markers - One-shot invocation with -m 'e2e or anthropic or openai or gemini' --no-cov and the full env-var stack (SF_RUN_*, ANTHROPIC_API_KEY, OPENAI_API_KEY, GOOGLE_API_KEY, GOOGLE_CLOUD_PROJECT, SNOWFLAKE_*) Frames the cadence as pre-release only — NOT per-PR, NOT CI-gated. The addopts exclusion in pyproject.toml already keeps these out of default runs; this section just documents the maintainer-side invocation when cutting a release. Cost ceiling: ~$0.30/run × ~2–3 audits/month = ~$0.60–1.00/month. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-eu0.10: #155 US-009 — Quality Gate fixes (4-pass code review) Pass 1 (correctness): 0 real bugs. 2 doc-drift fixes: - docs/grade-ops.md floor table: pre-fix GradeOutputError narrative rewritten to post-fix LLMResponseFormatError → GradeLLMError per #155 DEC-005 - tests/cli/test_e2e_gemini_smoke.py invariant-#6 comment: same update Pass 2 (simplification): 0 refactors needed. Pass 3 (test coverage): 0 critical gaps. 3 nice-to-have safety nets added: - tests/llm/test_gemini_provider_via_fake.py — new orchestrator wire-in pin for MAX_TOKENS+partial-text via call_llm (distinct from existing SAFETY-only call_llm test) - tests/llm/test_openai_provider_via_fake.py — new orchestrator wire-in pin for length+partial-text via call_llm (no prior call_llm coverage) - tests/grade/test_gemini_neutrality.py:381 — docstring comment binding the existing safety-blocked pin to the new #155 MAX_TOKENS routing path Pass 4 (rules compliance): 1 real rule violation + 1 multi-surface drift (same root cause). Fixed: - tests/cli/test_e2e_openai_smoke.py::_skip_reason() expanded from 3 to 5 env-var checks (was missing SF_RUN_BQ + ANTHROPIC_API_KEY despite the drafter staying Anthropic per DEC-011); mirrors the Gemini sibling and the parametrized BQ smoke. Module docstring + CONTRIBUTING.md entry updated in lockstep. CodeRabbit: skipped (no MCP integration available in this environment; the project's user-triggered /code-review ultra is the cloud-billed path operators run separately). Final canonical validation: 2581 passed, 0 ruff/pyright errors, 97.47% coverage, mkdocs build clean (existing cross-repo-link warnings only, intentional per docs-publishing.md). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-eu0.11: #155 US-010 — Patterns & Memory Captures three durable lessons from #155 in the rule set + memory: 1. `.claude/rules/llm-drafter.md` § "Provider-neutral seam (#135)" — the `LLMProvider` ABC bullet now lists `is_clean_completion` + `unclean_finish_reason_message` alongside the rest of the abstract methods, cross-referencing the Gemini-shape § for the full contract + the `_CLEAN_STOP_REASONS` convention each concrete declares. This is the discoverability seam — a future provider implementer reading the abstraction-level section sees the gate immediately. 2. `.claude/rules/testing-signal.md` § "End-to-end gated tests (issue #10)" — new subsection "Per-test provider overlay via apply_provider_override" (DEC-012). Documents the canonical helper for per-test grade-provider overlays + the load-bearing 5-env-var contract (drafter Anthropic + grader-provider stays a 5-var gate even when the grader varies; the 3-var docstring drift in test_e2e_openai_smoke.py was QG Pass 4's real finding). 3. Memory: new `in-isolation-smoke-misses-pipeline-drift.md` — in-isolation grade-only smoke tests can pass while a structural contract regression slips; a new provider needs a full-pipeline `signalforge generate` e2e too. Distinct from the existing `fake-driven-byte-identity-blind-spot` (that lesson is about fake-call-shape vs real-call-shape; this is about isolation-scope vs pipeline-scope coverage). MEMORY.md index updated. ORCHESTRATOR-only commit per memory `ralph-worker-claude-dir-perms` (workers can't write to .claude/). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #155 US-009 follow-up: close codecov patch-coverage gap (7 uncovered lines → 0) Codecov flagged patch coverage at 86.54% (7 lines missing in providers.py) — the defensive-raise arms in is_clean_completion + the ABC default unclean_finish_reason_message body. Pass 3 of QG already identified these as nice-to-have safety nets and I deferred them; codecov on PR #156 called them out, so adding them now to close the gap. Files: - tests/llm/test_anthropic_provider_via_fake.py — defensive test: missing stop_reason attribute raises LLMResponseFormatError - tests/llm/test_openai_provider_via_fake.py — two defensive tests: missing/empty choices raises; missing finish_reason on first choice raises - tests/llm/test_gemini_provider_via_fake.py — two defensive tests: missing/empty candidates raises; missing finish_reason on first candidate raises - tests/llm/test_providers.py — ABC default unclean_finish_reason_message test via _DummyProvider (inherits the default unchanged) These guard against future SDK shape regressions (e.g. a vendor renames stop_reason → terminated_for; without these arms the conservative-degrade path silently swallows the structural surprise). Pinned at unit cost so the regression surfaces at unit-test time, not live-e2e time. Validation: 2587 passed (+6), 0 ruff/pyright errors, providers.py coverage 295/3 missing (only the pre-existing _count_openai_tokens defensive lines from #136 remain — out of this PR's patch scope), total coverage 97.57%. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #155: Address PR review feedback (CodeRabbit + Copilot, 0 false positives) 3 real findings across reviewers, 8 sites fixed: CodeRabbit (2 inline): test_call_llm_{gemini,openai}_..._is_clean_gate tests added in QG Pass 3 were missing @pytest.mark.unit + @pytest.mark.llm, so marker-scoped runs (`pytest -m "unit and llm"`) would silently skip the orchestrator-level regression pins. Sibling tests in the same files all use the unit+llm marker pair. Copilot (1 inline on tests/cli/test_e2e_gemini_smoke.py:31): docstring header still said "Gated by THREE env vars" with an awkward "Note: ANTHROPIC_API_KEY is also required" follow-on — but `_skip_reason()` actually checks FIVE. Mirrors the OpenAI-sibling drift QG Pass 4 already caught (`.claude/rules/testing-signal.md:169`). Rewrote the header to enumerate all five gates upfront with the same DEC-011 drafter-stays-Anthropic framing the OpenAI sibling uses post-QG. In addition: applied the same `@pytest.mark.unit + @pytest.mark.llm` fix to the 6 defensive-raise tests + the ABC default test from the codecov follow-up commit (`ce7e050`) — same root cause, all sibling tests in the same files use the marker pair, only my additions lacked it. Validation: 2587 passed (unchanged), 0 ruff/pyright errors. Marker-scoped collection (`pytest -m "unit and llm" --collect-only`) now includes all 8 newly-marked tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…e2e floor to 4096 (#160) * fix(#158): surface Gemini finish_reason in degrade reasoning + raise e2e floor to 4096 Two fixes from the post-#155 full-pipeline finding (#158): 1. **Diagnostic upgrade (`signalforge.grade.engine`).** Add `_format_degrade_reasoning(exc)` helper. When the wrapped cause is `LLMResponseFormatError`, surface its bare `message` (which names the vendor `finish_reason` field + value via `LLMProvider.unclean_finish_reason_message`) into the degraded `GradingResult.reasoning` — e.g. `"call failed: GradeLLMError: Gemini response did not finish with a clean stop reason (finish_reason='MAX_TOKENS')..."` instead of the bare `"call failed: GradeLLMError"`. For every other cause (auth / rate-limit / parser / budget) the existing class-name shape is preserved verbatim so the audit corpus stays diff-clean for the 90% case — only the response-shape branch grows the diagnostic. Two new unit tests pin the new + preserved shapes. 2. **Floor bump (e2e overlay).** Raise `grade_max_output_tokens` overlay from 2048 to 4096 in both `test_e2e_gemini_smoke.py` and the `[gemini]` parametrization of `test_e2e_bigquery_smoke.py`. The 2048 figure was the in-isolation 5-pair smoke floor verified in #155 DEC-008; the first full-pipeline run found 5–6/108 pairs still degrade at 2048 (Gemini's per-pair `reasoning` length is high-variance enough that the in-isolation floor is not the full-fixture floor). 4096 is the new #158 fixture-scale floor. Doc reframes in lockstep: - `docs/grade-ops.md` + `docs/draft-ops.md` — Gemini row of the recommended-floors table bumped 2048 → 4096, with a "fixture-scale caveat" callout explaining the floor is necessary but not sufficient and operators should watch `aggregate_complete` on their own fixture. - `plans/super/155-...md` DEC-008 — addendum reframing the 2048 "verified safe" claim as scoped to the in-isolation probe; the DEC stays as historical record of the 2048 figure's provenance. Existing `test_grade_artifacts_safety_blocked_response_degrades_pair` updated to assert the new `startswith("call failed: GradeLLMError: ")` + `"finish_reason='SAFETY'" in reasoning` shape rather than the bare class name — the contract pin moves with the broadened reasoning. The acceptance gate from #158 (full-fixture e2e against Gemini returning `aggregate_complete=True`) is only verified by re-running `tests/cli/test_e2e_bigquery_smoke.py[gemini]` + `test_e2e_gemini_smoke.py` under the live env-var stack (`SF_RUN_GEMINI=1 GOOGLE_API_KEY=… SF_RUN_BQ=1 ANTHROPIC_API_KEY=… GOOGLE_CLOUD_PROJECT=… uv run pytest -m "e2e and gemini" --no-cov`) — those are excluded from default CI. Local 2589-test suite + ruff + pyright clean. Beads: bd_1-scaffolding-we0 (diagnostic), bd_1-scaffolding-ila (floor). Closes #158. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #158: address PR review (Copilot + CodeRabbit) Four review-feedback fixes, all real: 1. `docs/grade-ops.md` (Copilot l.599): the safety-filter handling prose still pointed at `GeminiProvider.extract_text_blocks` as the raise site, but post-#155 the provider-neutral `LLMProvider.is_clean_completion` gate in `call_llm` is the primary raise site — and it fires on partial-text MAX_TOKENS too, not just zero-text-parts. Reworded to name both paths and route them through the same conservative-degrade contract. 2. `docs/draft-ops.md` (Copilot l.595): the Gemini drafter floor row leaned on grader-side evidence (#158) while the section lead-in says floors are "from live drafting runs." Qualified the 4096 figure as a *conservative mirror* of the grader floor, explicitly flagged as pending Gemini-drafter live validation when that lands. 3. `plans/super/155-...md` DEC-008 (Copilot l.56): the #158 addendum carried a broken-sentence placeholder ("See `plans/super/155-...` was the in-isolation verification" — typo from a half-edit). Replaced with a concrete pointer to issue #158 + the durable lesson in memory `in-isolation-smoke-misses-pipeline-drift`. 4. `tests/cli/test_e2e_gemini_smoke.py:69` (CodeRabbit outside-diff): module-docstring invariant #6 still said "the 2048 cap fixes the truncation bug" — stale after the floor bump. Now says "the 4096 cap (#158) fixes the full-fixture truncation bug." No code changes (doc + docstring only). Full local validation (ruff + pyright + 2589 pytest) clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* #159: drafter column-type awareness (plan) Plan for closing the test_e2e_business_rules flake where the drafter emits type-incoherent custom_sql (INT64 vs STRING comparison), prune correctly routes to kept-without-evidence, and the test asserts kept-with-evidence. Root cause: tests/fixtures/dbt_project_austin/target/manifest.json carries data_type: null for every column. The drafter / safety / prompt pipeline already supports types end-to-end — the gap is upstream. Scope (DEC-001): A + B + D. A. populate data_type in the Austin fixture B. merge target/catalog.json types into Column.data_type at load time D. add sqlglot AST type-coherence check in _validate_anchor_contract 13 decisions captured (DEC-001 … DEC-013). 4 implementation stories + Quality Gate + Patterns & Memory. No new CLI flag, no new config knob, no _PROMPT_VERSION rotation, no new error class — fix lives inside existing extensibility seams. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-crh.1: manifest loader — catalog.json sibling merge for Column.data_type (#159) Implements US-001 of the #159 plan (DEC-001, DEC-002, DEC-007, DEC-010). The manifest loader now reads a sibling 'target/catalog.json' when present and overlays its per-column 'type' onto Column.data_type via the frozen-model 'model_copy(update={...})' pattern. Case-insensitive column matching covers Snowflake (uppercased identifiers) and BigQuery (preserved-case) catalogs. Missing / malformed / unreadable catalog → silent no-op (stage-0 invariant); phantom columns are dropped; manifest columns absent from the catalog keep data_type=None. The catalog path is canonicalised through signalforge._common.path_safety.canonicalise_path, so a symlink escape raises PathContainmentError (security gate is NOT in the silent-degrade set). Tests: 19 new cases under tests/manifest/test_loader.py covering the 8 TDD acceptance cases plus parametrised shape-degrade guards. Fixtures under tests/fixtures/manifest/ (canonical/case-mismatch/phantom/partial catalogs + a minimal manifest carrying three columns). * bd_1-scaffolding-crh.2: drafter parser — sqlglot type-coherence defence for custom_sql (#159) Add a sqlglot-based type-coherence check to _validate_anchor_contract in signalforge.draft.parser. For each custom_sql test, parse the SQL with sqlglot.parse_one(dialect=...), annotate types, and append violations for direct Column <op> Column comparisons where both known types are incompatible. Skip silently on every other shape per DEC-006 (Cast / SafeCast / Coalesce / function call / subquery / literal / NULL / window / unparseable SQL / unknown types). A {{ this }} / {{ ref(...) }} placeholder is substituted with a stable identifier before parsing so sqlglot can read the surrounding SQL — the parser defence runs pre-resolution and the LLM almost always references the model via Jinja. Bidirectional COERCES_TO compatibility lets numeric-family coercions (INT64↔FLOAT64, NUMERIC↔BIGNUMERIC) pass while INT64 vs STRING / DATE is flagged. Type violations append to the existing LLMOutputAnchorContractError.violations list (DEC-011) — no new error subclass. parse_draft_response gains two keyword-only params: - model_columns_by_type: Mapping[str, str | None] | None = None - dialect_name: str = 'bigquery' draft_from_request builds the type map from model.columns_list and threads through; v0.1 hard-codes 'bigquery' with a TODO referencing v0.2 multi-warehouse work (DEC-013). sqlglot promoted to a direct runtime dep (sqlglot>=30,<31) in [project].dependencies + [project.optional-dependencies].dev + [dependency-groups].dev (mirror per python-build.md). Previously a dev-only transitive via fakesnow; type-defence correctness is load-bearing. Confinement: sqlglot imports live ONLY in src/signalforge/draft/parser.py (DEC-008). Convention only in v0.1; no AST scan. Traces to DEC-001 / DEC-003 / DEC-005 / DEC-006 / DEC-008 / DEC-011 / DEC-012 / DEC-013 in plans/super/159-drafter-column-types.md. 17 new tests in tests/draft/test_parser.py: - 3 planted positives (int64 vs string × 2 ops, int64 vs date) - 9 planted negatives (numeric coercion × 2, cast / safe_cast / coalesce / null / literal / function / subquery) - 5 robustness (unparseable, both-unknown, partial-unknown, None-map-skips-arm, collect-all-with-structural-violation) All 41 tests in tests/draft/test_parser.py pass; full canonical validation (ruff check + ruff format + pyright + pytest) green with 2604 tests passing and 97.44% coverage. * bd_1-scaffolding-crh.3: Austin fixture — populate data_type + add catalog.json (#159) Populate real BigQuery data_type values on every column of stg_bikeshare_trips in the Austin manifest fixture and ship a sibling target/catalog.json with the same types so US-001's catalog-merge read-path is exercised end-to-end by the e2e smoke. Types verified via 'bq show --schema bigquery-public-data:austin_bikeshare.bikeshare_trips': - trip_id, subscriber_type, bike_id, end_station_id STRING - start_time TIMESTAMP - start_station_id, duration_minutes INT64 end_station_id is STRING but start_station_id is INT64 — exactly the type mismatch the original #159 BQ error 'No matching signature for operator != for argument types: INT64, STRING' flagged in the production prune.jsonl. DEC-008 of #47 (demo-fixture parity) requires the shipped src/signalforge/_demo/ tree stay byte-equal to the e2e fixture except for the two documented rewrites; both manifest.json and the new catalog.json are mirrored into _demo/target/ in lockstep. Traces to DEC-001 sub-option A + DEC-004 in plans/super/159-drafter-column-types.md. * bd_1-scaffolding-crh.4: 5-surface rules + docs for column-type awareness (#159) Documents the two new behaviours added in US-001 (catalog.json sibling merge) and US-002 (sqlglot parser type-coherence defence), per cli-layer.md "Multi-surface parity for behaviour changes". Surfaces touched: - .claude/rules/manifest-readers.md — new "Catalog.json sibling merge" section: 5 load-bearing rules, no _PROMPT_VERSION rotation rationale, no drift detector required. - .claude/rules/llm-drafter.md — new "Sqlglot type-coherence check" subsection under "Whole-draft fail-loud anchor contract": dual-defence framing (prompt + parser), 6 load-bearing rules incl. skip-when-uncertain + Jinja substitution + ParseError silent skip + sqlglot confinement. - docs/manifest-loader-ops.md — operator-facing "Column types from catalog.json" section: how to run dbt docs generate, failure modes (all silent), refresh notes. - docs/draft-ops.md — operator-facing "Type-coherence defence" section inserted after "Hard JSON validation + anchor-contract": what it catches, what it skips, threading API, dialect support, dep note. - CHANGELOG.md — one Added entry (catalog.json merge), one Fixed entry (sqlglot parser defence). Validation: ruff/format/pyright/pytest all green; 2623 passed, 97.46% coverage. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-crh.5: Quality Gate — doc/code mismatch fix (#159) QG review pass 1 (low effort) surfaced one finding: docs/draft-ops.md § "Type-coherence defence" claimed the check covers "plus IN-list comparisons" but parser.py:_check_custom_sql_type_coherence only matches (EQ, NEQ, GT, LT, GTE, LTE) — no exp.In handler. Per DEC-006 skip-when-uncertain, removing the IN claim is the right fix (matches actual conservative behaviour); adding IN handling is deferred to a follow-up if real-world drift demands it. Passes 2 (medium), 3 (high), and 4 (max) review surfaced no additional real-bug findings. The implementation honors all 13 #159 DECs: - DEC-008 sqlglot confined to signalforge.draft.parser (grep verified) - DEC-006 skip-when-uncertain (Cast/SafeCast/Coalesce/IfNull/function/ subquery/literal/NULL/window/unknown-type/parse-error all skip) - DEC-009 no _PROMPT_VERSION rotation - DEC-011 violations append to existing LLMOutputAnchorContractError - DEC-012/013 model_columns_by_type + dialect_name kwargs threaded - DEC-002/007/010 catalog.json sibling lookup, case-insensitive, silent degradation CodeRabbit skill not registered in this session; manual review only. Canonical validation green: ruff/format/pyright all clean; pytest 2623 passed, 97.46% coverage. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-crh.5: cover defensive branches in parser type-coherence (#159) Codecov flagged 11 missing patch-coverage lines in src/signalforge/draft/ parser.py — all in `_check_custom_sql_type_coherence` and its model-level call site: - 121: `_types_compatible` `a == b` short-circuit (same-type case) - 126: reverse-direction `a in COERCES_TO[b]` branch - 168-171: defensive `except sqlglot.errors.SqlglotError` (non-ParseError) - 174: `if parsed is None: return ()` (parse_one can return None) - 184-187: defensive `except Exception` around `annotate_types` - 209-210: defensive `except Exception` around `DataType.build` - 353-354: model-level custom_sql type-coherence call site Per memory `qg-pass-3-defer-defensive-tests-fails-codecov` ("if a Pass-3 nice-to-have covers lines inside the patch diff, upgrade it to must-fix; codecov holds patch coverage to project standard regardless"), these defensive branches need explicit tests rather than skipping them. 7 new tests pin each branch: - test_custom_sql_same_type_comparison_skipped (121) - test_custom_sql_reverse_coerce_direction_accepted (126) - test_check_custom_sql_type_coherence_sqlglot_non_parse_error_skipped (168-171, monkeypatch) - test_check_custom_sql_type_coherence_parser_returns_none_handled (174, monkeypatch — parse_one returning None varies by sqlglot release) - test_check_custom_sql_type_coherence_annotate_types_failure_skipped (184-187, monkeypatch) - test_custom_sql_invalid_type_string_skipped (209-210, opaque type string) - test_model_level_custom_sql_type_mismatch_is_rejected (353-354) Mix of organic-input tests where possible (lines 121/126/209-210/353-354) and monkeypatch tests for the sqlglot-internals defensive catches that can't be triggered organically without mocking (168-171/174/184-187). Imports the private `_check_custom_sql_type_coherence` helper alongside `_LLMResultMeta` per the existing pattern (private-under-test). Result: parser.py at 100% coverage (was 92% / 11 missing). Full project coverage up to 97.61% (from 97.46%). 2630 tests pass (+7). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-crh.5: cover pre-existing defensive gaps in loader+schema (#159) Codecov flags any uncovered line in a file modified by the PR — even pre-existing defensive branches that #159 never touched. The catalog overlay landed in loader.py and the type-coherence kwargs threaded into schema.py, so both files surfaced their pre-existing gaps on the PR view. Closing those completes the project-coverage picture for #159. schema.py (2 lines): - test_draft_from_request_record_too_large_propagates_typed (line 250): pin that LLMResponseAuditRecordTooLargeError raised by write_response_ event is re-raised as-is, NOT wrapped under LLMResponseAuditWriteError - test_draft_from_request_keyboard_interrupt_propagates_untouched (line 254): pin that a Ctrl-C signal mid-audit-write propagates with its identity intact — never silently demoted to an audit error loader.py (6 lines): - test_load_project_dir_is_a_file_raises_manifest_not_found (line 231): non-directory project_dir → ManifestNotFoundError (distinct from FileNotFoundError caught one branch up) - test_load_non_dict_metadata_raises_manifest_error (line 283): manifest with metadata as a non-dict → ManifestError before version-sniff - test_load_disabled_with_non_list_value_continues (line 308): defensive shape-drift skip in the disabled-models loop - test_schema_version_non_string_returns_empty_string (line 485): the free-function schema_version() returns "" for a non-string version - test_get_model_resolver_index_cache_hit_returns_cached (line 598): the in-memory resolver-index cache is reused on second get_model call - test_get_model_by_file_path_for_disabled_model_raises_disabled (line 679): file-path lookup for disabled model raises ModelDisabledError (the path-lookup branch needs the same disabled detection the unique_id branch already has) Result: - src/signalforge/draft/parser.py: 100% (was 100%) - src/signalforge/draft/schema.py: 100% (was 96%, 2 missing) - src/signalforge/manifest/loader.py: 100% (was 97%, 6 missing) - Project coverage: 97.72% (was 97.61%) - 2638 tests pass (+8) Per memory `qg-pass-3-defer-defensive-tests-fails-codecov`. Most tests use monkeypatch or carefully-crafted JSON fixtures to hit the precise defensive branch; lines that can be exercised organically (231, 283, 598, 679) use real Manifest construction. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #159: address PR #161 review feedback (5 threads) Addresses all 5 unresolved review comments on PR #161: Thread 1 (CodeRabbit, plan US-003 "Done when"): - Rewrote the criterion to be story-local: scopes only THIS story's own commit, not the cumulative diff against the base branch. Adds the demo mirror files explicitly (the DEC-008 of #47 parity gate the original scope missed). Clarifies why US-001 + US-002 are real dependencies. Thread 2 (CodeRabbit, duplicate "Beads manifest" heading): - Already fixed during devolve (Phase 7 cleanup committed earlier); only one heading remains at line 302. No code change. Thread 3 (Copilot, test_loader.py Windows skipif chain): - `@pytest.mark.skipif(os.geteuid() == 0, ...)` ran at collection time on Windows even when the prior `sys.platform == "win32"` skipif fired, because pytest evaluates ALL skipif predicates. On Windows os.geteuid doesn't exist → AttributeError breaking collection. Combined into a single skipif with `hasattr(os, "geteuid")` short-circuit. Thread 4 (Copilot, docs/manifest-loader-ops.md "all silent" framing): - Added the path-containment caveat: catalog.json that escapes the project via symlink raises PathContainmentError loud — the one non-silent failure mode. Reworded section header from "all silent" to "all silent except the path-safety gate". Thread 5 (Copilot, CHANGELOG mechanism wording): - The previous wording implied sqlglot's annotator received a `schema=` kwarg. The actual mechanism walks binary comparison nodes and looks each operand's column name up in the model's data_type map directly, then tests via TypeAnnotator.COERCES_TO. Reworded to match. Validation: ruff + format + pyright + pytest all green; 2638 tests pass, 97.72% coverage. No production-code changes (docs / plan / test decorator only). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* #157: super plan — e2e cost docs + parallelization (plan) Scope: both ticket asks together. Adds a re-runnable cost-rollup helper (`signalforge.llm.cost.rollup_audit_dir`) + a `scripts/measure_e2e_cost.py` wrapper, adds `pytest-xdist` as an opt-in dev dep (no addopts change), then lifts the measured baseline into the three drifted doc surfaces (plans/super/155-…md DEC-010, CONTRIBUTING.md § Live e2e suite, docs/grade-ops.md § Cost guidance). 6 implementation stories + Quality Gate + Patterns & Memory. US-005 is maintainer-only (live re-run + measurement capture); the rest are Ralph-eligible. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #157: devolve plan to beads Epic bd_1-scaffolding-e1a + 8 children with deps wired. US-005 (maintainer live re-run) assigned to @wjduenow. Phase: devolved. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-e1a.1: scaffold signalforge.llm.cost subpackage Adds CostError(LLMError) base + 3 concrete errors (audit-missing, malformed-record, unknown-model); registers each at CLI tier 2 and CostError as a dual-registration safety-net + excluded base. Ships frozen-dataclass shapes (CostReport / ProviderRollup / ModelRollup); rollup_audit_dir() is a NotImplementedError stub that US-002 fills in. Extends scan-7 to depth-2 glob; bumps expected per-stage errors.py count 11 → 12. Refs #157. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-e1a.2: implement signalforge.llm.cost.rollup_audit_dir Walks .signalforge/llm_responses.jsonl + grade.jsonl under a canonicalised project_dir, deserialises each record via the existing LLMResponseEvent / GradeEvent models, and aggregates per-provider per-model USD against the frozen PRICES table. Symlink-hardened path entry; typed errors for missing audit, malformed record, unknown model. Degraded report when only one JSONL present; CostRollupAuditMissingError when neither. 16 TDD cases (test_rollup_*) cover provider mixes, error paths, path safety, and the frozen-dataclass invariants. Refs #157. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-e1a.3: add scripts/measure_e2e_cost.py wrapper Thin argparse entrypoint around signalforge.llm.cost.rollup_audit_dir. --format {text,json} controls output shape; CostError -> exit 2 via the boundary catch; no Traceback ever leaks to stderr (cli-layer.md floor). scripts/ stays repo-only -- the wheel_smoke gate asserts it is excluded from the built wheel. Refs #157. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-e1a.4: pytest-xdist dev dep + parallel e2e docs Adds pytest-xdist as an opt-in dev dep (both [dependency-groups].dev and [project.optional-dependencies].dev, mirrored). Rewrites CONTRIBUTING § "Live e2e suite (pre-release only)" to document `pytest -m e2e -n 3 --no-cov` as the recommended invocation with the Anthropic 50-RPM rate-limit caveat + downgrade path. Lists all 5 paid e2e files (adds test_e2e_business_rules.py — latent doc gap). Notes cli_subprocess / wheel_smoke stay serial. Adds parity gate test that asserts every e2e file basename + the parallel invocation + the rate-limit caveat phrase + the measure_e2e_cost.py pointer appear in CONTRIBUTING. No addopts change. Cost figures left stale; US-006 lifts the measured baseline after US-005's live re-run. Refs #157. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-e1a.5: US-005 measured baseline — live e2e run 2026-05-29 `-n 3` wall-clock 14:53 (vs 39 min serial = 2.6× speedup). Grand total $1.38 / full-suite at PRICE_TABLE_VERSION=2026-05-28 (vs the stale $0.30 figure). Zero Anthropic rate-limit retries observed; no -n downgrade needed. Per-test + per-provider rollup tables added to the plan's refinement log; US-006 lifts these into the 3 user-facing doc surfaces. Pre-existing flake observed in test_e2e_business_rules.py (custom_sql LLM-determinism); rollup unaffected. Refs #157. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-e1a.6: lift measured baseline into 3 doc surfaces Replaces the stale $0.30/full-suite figure across plans/super/155-...md DEC-010 + the architecture-review row, CONTRIBUTING.md § "Live e2e suite", and docs/grade-ops.md § Cost guidance. New 3-row per-provider table in grade-ops: Anthropic ~$0.38/model, OpenAI ~$0.21, Gemini ~$0.045. Each surface carries the date-stamp + PRICE_TABLE_VERSION + "calibration signal, not a billing guarantee" framing. Parity gate extended to pin 2026-05-28 across all three surfaces; planted-violation self-check passes. Refs #157. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-e1a.7: Quality gate — fix bugs from code review Four code-review passes ran against the cbcaeec..HEAD diff (2389 LOC, 18 files). Pass 1 (correctness) found zero bugs. Passes 2/3/4 surfaced 2 real bugs + multiple test-signal / coverage / parity-gate gaps inside the patch diff. CodeRabbit not invoked (no skill available in this environment); auto-review will pick up the PR. Fixes applied: Pass-2 F1 (BUG): test_rollup_grand_total_equals_sum_of_provider_subtotals re-derived the sum the same way the engine does → tautology. Now asserts hand-computed $10.50 / $15.00 / $0.90 / $26.40 against the locked PRICES table. Pass-2 F2 (BUG): scripts/measure_e2e_cost.py panic-path (exit 1) was untested. New test_measure_e2e_cost_main_exits_1_on_unexpected_error monkeypatches rollup_audit_dir to raise ValueError; asserts exit 1 + "ERROR: ValueError" in stderr + no traceback. Pass-2 F3: audit_files_consumed asserted as set → now tuple, pins the drafter-then-grader ordering invariant. Pass-2 F4: rate-limit caveat anchored on common words ("Anthropic" + "rate limit") → now pinned to verbatim "Anthropic 50 RPM rate-limit caveat" (the unique compound phrase). Pass-2 F5: degraded-report tests skipped USD assertion → both now hand-compute $0.0105 for the 1000/500-token claude-sonnet-4-6 record. Pass-3 F1 (BUG/coverage): no-prefix-match unknown-model branch in _ingest_jsonl was unreachable. New test_rollup_unknown_provider_prefix_raises_typed_error covers databricks-llama-3-70b-shape model ids. Pass-3 F3: line_num literal-file-position invariant pinned via test_rollup_line_num_reflects_literal_file_position_with_blank_lines (blank lines 2-3, malformed on line 4, asserts line_num == 4). Pass-3 F4: no opus-tier coverage → new test_rollup_anthropic_opus_uses_opus_pricing_not_sonnet hand-computes $52.50 (5× sonnet's $10.50 for the same token shape) so a wrong-tier wire-formula bug fails loud. Pass-3 F5: empty-providers _print_text branch untested → new test_measure_e2e_cost_format_text_no_priced_records_still_prints_total asserts both "(no priced records" and "TOTAL: $0.0000" appear. Pass-3 F7: --format=json audit_files_consumed ordering deterministic- by-accident → now pinned to ["llm_responses.jsonl", "grade.jsonl"] in the JSON test. Pass-4 F1 (BUG): _PARALLEL_INVOCATION substring-matched in CONTRIBUTING, but the same string appears in two distinct spots (canonical recommendation + monitoring example). A partial rotation would silently pass. Now asserts the count is exactly _PARALLEL_INVOCATION_EXPECTED_COUNT (= 2). Pass-4 F4 prose: per-test vs per-provider rollup sums could read confusingly → added one-line clarifier "Sums agree to rounding: per-test sum = $1.3790; per-provider sum = $1.3788; headline = $1.38." Pass-4 F5 prose: leader-test attribution "gemini_smoke on gw2" was undersourced → softened to "gw2 worker's first test (Gemini-grader sibling, inferred from progress output)". Pass-4 F6 prose: plan-157 flake call-out didn't backref #163 → added explicit link to issue #163 with the observed-SQL summary. Canonical validation: 2634 passed (+4 new test cases), pyright 0/0/0, ruff clean, format clean, coverage 97.59%. Refs #157. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-e1a.8: Patterns & Memory — capture #157 conventions Extends testing-signal.md § "End-to-end gated tests" with the parallel-safe e2e contract (copy_fixture_to_tmp / apply_provider_override / inject_model_business_rules per-test isolation; pytest-xdist -n 3 recommended invocation; Anthropic 50-RPM caveat; cli_subprocess / wheel_smoke stay serial; cross-reference signalforge.llm.cost.rollup_audit_dir and the pricing-table-version parity gate). Extends cli-layer.md § "7th AST scan" with the depth-2 glob generalisation (signalforge.llm.cost.errors is the first sub-stage errors.py; scan-7 now walks */errors.py union */*/errors.py; expected path count 11 -> 12; CostError joins _EXCEPTION_MAPPING_EXCLUDED_BASES with tier-2 dual registration). Memory: #163 drafter business_rules hallucination pointer; bd worktree remove no-op gotcha. Refs #157. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #157: Address PR review feedback 5 issues from CodeRabbit + Copilot inline reviews on PR #162: 1. CONTRIBUTING.md L152 (Copilot): Gemini e2e said `grade.max_output_tokens=2048` but the actual test uses 4096 (the #158 floor bump after the full-Austin-fixture e2e found 5-6/108 pairs still degrading at 2048). Fixed to 4096 with the historical 2048→4096 context. 2. CONTRIBUTING.md L252+266 (Copilot + CodeRabbit): contradictory marker guidance for the Snowflake live tier. Line 157 (my US-004 doc) said `@pytest.mark.snowflake` (correct — the test file confirms); line 266 (pre-existing, stale) said "reached via the `e2e` marker" (wrong). The "Full pre-release invocation" example silently dropped Snowflake by listing only `e2e or anthropic or openai or gemini`. Aligned both: added ` or snowflake` to the example expression and rewrote the trailing prose to describe the `snowflake` marker's two tiers (offline fakesnow + sqlglot AND live warehouse) with the `SF_RUN_SNOWFLAKE=1` opt-in story. 3. src/signalforge/llm/cost/errors.py L75 (Copilot): `CostRollupAuditMissingError` built the audit-root display as `_format_value(project_dir) + '/' + _format_value(audit_dir)`, which renders as two separately-quoted reprs (e.g. `'/tmp/proj'/'.signalforge'`). Now formats the combined path first, then passes through `_format_value` once — single quoted, copy-pasteable. 4. src/signalforge/llm/cost/_rollup.py L122 (Copilot): used `assert` for the import-time pricing/provider sanity check, which `python -O` strips. The check is load-bearing for provider-dispatch correctness — a missing prefix would silently route the SKU to CostRollupUnknownModelError at rollup time instead of failing loud at import. Replaced with an explicit `if/raise RuntimeError` that includes the missing-models list. Canonical validation: 2685 passed (unchanged from the merge-resolve baseline), pyright 0/0/0, ruff clean, coverage 97.71%. Refs #157. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #157: Address codecov patch-coverage gaps (4 lines in _rollup.py) Codecov flagged 97.57% patch coverage on PR #162 with 4 missed lines: Lines 124-125 (post-PR-review-fix import-time guard): the new `if/raise RuntimeError(...)` block that replaced the stripped-under-O assert can't be triggered at module-load without an importlib.reload dance. Refactored: extracted the predicate to a module-level `_verify_provider_prefix_coverage(prices, model_to_provider)` helper that the module body calls inline. Helper is unit-testable directly with two cases (matching + mismatched). Lines 298-299 (ValidationError branch in _ingest_jsonl): the existing `test_rollup_malformed_jsonl_line_raises_typed_error` exercises only the JSONDecodeError branch. Added `test_rollup_valid_json_but_invalid_event_shape_raises_typed_error` that writes a JSONL line that parses as JSON but fails Pydantic's LLMResponseEvent.model_validate, exercising the `except ValidationError` arm. Also strengthened the existing test to assert `JSONDecodeError` appears in `reason`, so the two branches are pinned by name. _rollup.py now at 100% coverage (135/135 lines). Total project coverage 97.71% → 97.77%. 3 new test cases (2688 total). Refs #157. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* #163: drafter business-rules fidelity (plan) Add super-plan for #163 covering: - DEC-001: Lever B + C1 (dynamic-block envelopes + parser cardinality gate) - DEC-002: at-least-one-per-rule cardinality - DEC-004: thread business_rules through parse_draft_response (mirror #159) - DEC-005: reuse PromptEnvelopeBreachError with parameterised envelope - DEC-008: exclude_tests=('custom_sql',) short-circuits both surfaces - DEC-009: <BUSINESS_RULE id="N">...</BUSINESS_RULE> envelope format Four stories: dynamic-block hardening, parser gate, Quality Gate, Patterns & Memory. ~100 LOC code + ~30 LOC prose. No _PROMPT_VERSION rotation; pinned by US-001 AC #7 as the load-bearing cache-stability gate. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #163: devolve plan to beads (epic bd_1-scaffolding-74b) Phase 7 — populate Beads manifest with epic + 4 task IDs and update phase marker to 'devolved'. Epic: bd_1-scaffolding-74b .1 US-001 — Dynamic-block envelope hardening + parameterised breach guard .2 US-002 — Parser cardinality gate + business_rules threading (blocked on .1) .3 US-003 — Quality Gate (blocked on .1, .2) .4 US-004 — Patterns & Memory (blocked on .3) Ready next: bd_1-scaffolding-74b.1. Run /ralph-run from this worktree to execute the chain. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-74b.1: dynamic-block envelope hardening + parameterised breach guard (US-001 of #163) Replace the bulleted `## BUSINESS RULES` list in `_render_business_rules_section` with numbered `<BUSINESS_RULE id="N">…</BUSINESS_RULE>` envelopes (DEC-009). Each rule body indented exactly 2 spaces, preserving the existing `(model) `/`(column X) ` scope prefix verbatim. Extend `PromptEnvelopeBreachError` to be envelope-parameterised (DEC-005): new keyword-only `envelope: str = "MODEL_SQL"` and `rule_index: int | None` kwargs. The default rendering is byte-equal to the pre-#163 message so the existing call site in `prompts.py` keeps working unchanged. When `envelope="BUSINESS_RULE"` with `rule_index` set, the message names the 1-indexed offending rule. Add a pre-render breach guard: `_render_business_rules_section` scans each rule for the literal `</BUSINESS_RULE>` substring (boring substring match per the `</MODEL_SQL>` precedent — no whitespace/case normalisation) and raises `PromptEnvelopeBreachError` with the 1-indexed rule index. Short-circuit the section to `""` when `"custom_sql"` is in `DraftConfig.exclude_tests` (DEC-008) — don't tell the LLM to draft rules it can't emit. Thread `exclude_tests` through `_render_dynamic_block` and `render_prompt` so the caller's already-known exclusion set reaches the business-rules renderer without re-reading config. `_PROMPT_VERSION` is unchanged (`c9e7ee1f6f465933`) — all changes scope to the dynamic block and the errors module; the cached system prompt templates are untouched. `tests/llm/test_prompt_cache_stability.py` stays green. Traces to: DEC-001, DEC-005, DEC-008, DEC-009 of plans/super/163-drafter-business-rules-fidelity.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-74b.2: parser cardinality gate + business_rules threading (US-002 of #163) Adds keyword-only `business_rules: tuple[str, ...] = ()` to `parse_draft_response` and `_validate_anchor_contract`. Threads it from `draft_from_request` (in `signalforge.draft.schema`) using the existing `_read_business_rules(model)` helper from `prompts.py` as the single source of truth — mirrors the `model_columns_by_type` threading style from issue #159. Gate (DEC-002, DEC-008): when `business_rules` is non-empty AND `"custom_sql"` is NOT in `exclude_tests`, count `custom_sql` tests across both `candidate.tests` (model-level) and every `column.tests` (column-level), and append one collect-all violation when the count falls below `len(business_rules)`. At-least-one-per-rule semantics: excess is allowed (legitimate multi-test decomposition of a complex rule). Violation message (DEC-006) names every declared rule verbatim via `repr()` so the operator sees exactly which rules they declared: "Expected ≥N custom_sql test(s) (one per declared business rule), got M. Declared rules: '(model) rule one', '(column X) rule two'." 11 new TDD tests in `tests/draft/test_parser.py` cover under-coverage rejection, coverage match, over-coverage allowed, both no-op paths (empty rules with/without custom_sql), exclude_tests short-circuit, model-level / column-level / mixed counting, the pinned violation message shape, and collect-all parity with other violations. The gate is a no-op when `business_rules=()` so all 36 existing parser-test call sites continue to work unchanged. No new error class, no new exit-code-table entry, no prompt rendering touched, no `_PROMPT_VERSION` rotation — cache-stability test still passes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-74b.3: Quality gate — fix bugs from code review + CodeRabbit US-003 (#163) inline Quality Gate. 4 code-review passes ran in parallel on the US-001+US-002 diff; this commit lands the real signal from those passes as 5 invariant tests (no production-code changes — the diff itself was found contract-compliant and correctness-clean). Findings landed: 1. Pass 3 (coverage) finding 1 — pin the constructor invariant for the state-mismatched 'envelope="BUSINESS_RULE", rule_index=None' call so a future refactor (e.g. silent default to rule_index=1) can't regress the BUSINESS_RULE-side rendering. Production call sites always pair the two; this is the latent-invariant gate. 2. Pass 3 (coverage) side observation — pin the rotated default_remediation text. AC #5's byte-equality applies to .message (already pinned); the accompanying remediation hint was deliberately rotated to cover BOTH envelopes ('</MODEL_SQL>' and '</BUSINESS_RULE>') so operators of the new envelope see an actionable steer. Pinning the new text prevents a silent re-rotation back to MODEL_SQL-only wording. 3. Pass 3 (coverage) finding 2a — boring substring match invariant: a rule containing the bare opening tag '<BUSINESS_RULE>' (no slash) does NOT terminate the envelope and must render. Mirrors the safety / grade envelope contract. 4. Pass 3 (coverage) finding 2b — truncated closing fragment '</BUSINESS_RUL' is NOT a breach. Pins the exact-substring contract so a future 'be helpful' regex / case-normalisation refactor can't silently widen the match. 5. Pass 3 (coverage) finding 3 — branch-order invariant: the exclude_tests short-circuit beats the breach scan. An adversarial rule containing '</BUSINESS_RULE>' does NOT raise when 'custom_sql' is excluded because the section never renders. Pins the order. Pass 4 (cross-cutting) finding — 'import pytest as _pytest' alias dropped in favour of plain 'import pytest' to match the in-file precedent at test_render_dynamic_block_rejects_closing_tag_in_raw_code (line 515). Pass 1 (correctness): no P0/P1 bugs. Pass 2 (contract): no DEC / rule-file deviations. Validation: 2662 passed (+5 new), 97.72% coverage, _PROMPT_VERSION unchanged at c9e7ee1f6f465933 (cache-stability gate green). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-74b.4: Patterns & Memory — rule files + ops docs + plan-doc closeout US-004 (#163) inline Patterns & Memory. Updates the durable conventions established by #163: - docs/draft-ops.md — three new subsections: 1. 'Numbered envelope shape (<BUSINESS_RULE id="N">...</BUSINESS_RULE>)' documenting the envelope format + breach guard contract (boring substring match, opening tag / truncated fragments allowed) 2. 'Cardinality contract (at-least-one-per-rule)' documenting the parser-side gate, the verbatim violation message shape, and the collect-all preservation 3. ''exclude_tests' short-circuit' documenting the no-op on both surfaces when 'custom_sql' is in DraftConfig.exclude_tests - .claude/rules/business-rule-tests.md — 'Two input paths' section extended with the #163 'gate-over-prompt' subsection covering: * Dynamic-block envelope (DEC-009 of #163) * Parser cardinality gate (DEC-002 / DEC-006 of #163) * exclude_tests short-circuit (DEC-008 of #163) * Load-bearing 'no _PROMPT_VERSION rotation' note pointing at tests/llm/test_prompt_cache_stability.py as the regression gate - .claude/rules/llm-drafter.md — '<MODEL_SQL> prompt-injection envelope' section heading updated to call out the #163 parameterisation; a new paragraph documents the envelope-parameterised PromptEnvelopeBreachError pattern (one class, N raise sites; never a new error subclass). Future envelopes follow this shape. - plans/super/163-drafter-business-rules-fidelity.md — Beads manifest updated with final statuses (all 4 stories closed) + commit hashes; phase marker advanced to 'complete'. Validation: 2662 passed, 97.72% coverage. _PROMPT_VERSION unchanged at c9e7ee1f6f465933. Note: workers cannot write to .claude/ in worktrees (orchestrator-only per the ralph-worker-claude-dir-perms convention), so US-004's bead description flagged that the .claude/rules/*.md edits would be applied inline by the orchestrator. That's what happened here. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #163: Address PR review feedback — markdownlint MD038 + MD040 CodeRabbit found 3 markdownlint warnings on the docs/rules added by US-004: 1. .claude/rules/business-rule-tests.md:19 — MD038 (spaces inside inline code spans). Dropped trailing spaces inside backticks: '(model) ' -> '(model)', '(column X) ' -> '(column X)'. 2. docs/draft-ops.md:304 — same MD038 fix. 3. plans/super/163-drafter-business-rules-fidelity.md:30 — MD040 (fenced code block missing language tag). Added 'text' language to the 'BUSINESS RULES' rendered-output illustration. Also fixed the line-122 MD038 same as (1)/(2). All three are valid markdownlint findings; fixed inline per the closeout rule 'NEVER SILENTLY DEFER ANYTHING.' Validation: 2662 passed, 97.72% coverage. _PROMPT_VERSION unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(#134): LLM-provider deep-dive + README cross-links #134 (the pluggable-provider epic) shipped via #135 / #136 / #137 but the user-facing docs covered provider selection only inside the per-stage drafter and grader ops pages. Add a top-level `docs/llm-providers-ops.md` that pulls the story together — where LLMs are (and aren't) used, the provider-neutral seam shape, a capability matrix across Anthropic / OpenAI / Gemini, per-provider sections, choosing-a-provider guidance, cost + caching tradeoffs, the reliability + audit story, and an "adding a fourth provider" recipe. README changes: - New "Supported LLM providers" section mirroring the existing "Supported warehouses" section structure (table + one-paragraph intro) so first-time readers see the three providers and the install / env-var shape immediately. - Trim the long OpenAI / Gemini bullet list in Quick start §2 to a single paragraph pointing at the deep dive (the table above already carries the install + env-var info). - Fix a stale `call_anthropic` reference in the "How drafting works" ASCII diagram to `call_llm` (renamed in #135) and add a one-line pointer to the new deep dive. mkdocs nav gains an "LLM Providers" entry between Pipeline Stages and Cost Estimate. `mkdocs build` succeeds with no new warnings beyond the existing false-positive heading-anchor pattern that `cost-estimate-ops.md` already triggers (mkdocs and pymdownx disagree on a few anchor slugs; the headings exist and resolve correctly on GitHub Pages). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(#134): add prompt-caching explainer + per-provider rationale Reviewer feedback on the deep dive: the capability matrix marks Anthropic ✅ and OpenAI / Gemini ❌ for prompt caching, but doesn't say what caching IS in business terms or WHY the asymmetry exists. Closing that gap is the single most consequential row for anyone budgeting a real workload. Adds a "Prompt caching — what it is and why providers differ" section between Supported providers and Choosing a provider: - Business framing: caching is the provider's "pay a write premium once, get steep cache-read discount on subsequent calls" offer. Anthropic Sonnet 4.6 rate table (write +25% premium / read 90% discount) makes the break-even concrete (one follow-up call). - Where it matters in SignalForge: modest on the 1-call drafter, load-bearing on the ~48-call grader fan-out where the cached rubric block hits cache 47 of 48 times. - Why OpenAI isn't supported: their Chat Completions prompt-caching is automatic + opaque (no `cache_control` marker, no public per-MTok rates, no `cache_creation_input_tokens` field in usage). We can't steer, price, or audit it — so the honest posture is `supports_prompt_caching=False` + silently-ignored `cache_ttl`. - Why Gemini isn't supported yet: their context caching is real but ships as a separate API surface (`CachedContent.create` + handle reference) with a distinct pricing dimension (storage per hour + discounted reads). Different code path than Anthropic's inline marker; tracked as a follow-up to #137. - Practical bottom line: per-token cheaper providers (Gemini Flash, gpt-4o-mini) typically still beat Anthropic-with-caching on absolute dollars — they're just leaving optimization on the table. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: begin 0.4.0.dev0
* #135: provider-neutral LLM seam (plan) (#148)
* Add super plan for #135: provider-neutral LLM seam
* Mark #135 plan published (PR #148)
* Mark #135 plan devolved (beads epic bd_1-scaffolding-j2c)
* bd_1-scaffolding-j2c.1: US-001 provider foundation (ExceptionCategory, UsageMetrics, LLMProvider ABC, registry)
Adds the provider-neutral LLM seam foundation (DEC-001/002/003 of #135):
- src/signalforge/llm/providers.py: ExceptionCategory (5-member enum),
UsageMetrics (frozen pydantic value object, cache fields default 0),
LLMProvider ABC, and the process-level registry (register_provider /
provider_for). No vendor behaviour wired; registry starts empty.
- UnknownProviderError(LLMError) in errors.py — lists available registered
provider names, repr-safe message, default_remediation; registered at
CLI exit-code tier 2 (looked-up-identifier-not-in-table input failure).
- Exported the new public names from signalforge.llm.__all__; updated the
documented-surface lists in test_public_api.py / test_schema.py and the
errors count in test_errors.py.
* bd_1-scaffolding-j2c.2: US-002 AnthropicProvider strategy + _anthropic_client rename + AST scan
* bd_1-scaffolding-j2c.3: US-003 generic call_llm orchestrator (rename, capability gating, strategy dispatch)
Collapse call_anthropic into a provider-neutral call_llm orchestrator:
- Resolve strategy via provider_for(provider); build client via
strategy.make_client() when client is None (DEC-006).
- Pre-send count gate now gated on strategy.supports_token_count; cache
marker / beta header + dual-zero anomaly WARNING gated on
supports_prompt_caching (DEC-008). Anthropic = both True ⇒ byte-identical.
- Retry loop dispatches on strategy.classify_exception -> ExceptionCategory
(DEC-001) instead of catching Anthropic SDK classes directly; per-class
budgets, backoff math, WARNING shape, exhaustion raises preserved.
- Response assembled via strategy.extract_text_blocks/extract_usage.
- Neutral _LLMClientProtocol narrows the resolved client so no vendor-SDK
type / type-checker suppression leaks into llm.client (DEC-012 confinement).
Drop call_anthropic (name + __all__); add call_llm. Migrate draft_from_request
and grade._grade_one call sites + all client/retry/public-api/schema tests.
* bd_1-scaffolding-j2c.4: US-004 provider config field + stage threading + CLI client-construction migration
DEC-006/007 of #135. Adds a registry-validated `provider: str = "anthropic"`
field to DraftConfig and GradeConfig, each with a field_validator that
delegates to providers.provider_for (raises UnknownProviderError listing
available names on an unknown value — fails loud at config load). Threads
provider=config.provider into call_llm from draft_from_request and
grade._grade_one.
Migrates the CLI generate path off the in-CLI _make_anthropic_client helper
(removed): the real-run draft/grade stages now pass client=None so call_llm
lazy-builds the client via the configured provider. The --estimate
short-circuit (Anthropic-specific count_tokens) builds its concrete client via
provider_for(draft_config.provider).make_client(). Migrates the 7 CLI
monkeypatch tests: 5 batch/select tests drop the orphaned make_anthropic_client
mock; the estimate test patches AnthropicProvider.make_client; test_lint
unchanged (it patches the LLM shim, which still exists).
* bd_1-scaffolding-j2c.5: US-005 no-cache fake provider neutrality proof (AC #2/#3)
* bd_1-scaffolding-j2c.6: US-006 docs + surface parity (docs/ portion)
Update operator-facing docs for the provider-neutral LLM seam (#135):
- call_anthropic -> call_llm; llm._client -> llm._anthropic_client
- document the provider config knob (llm.provider / grade.provider),
default "anthropic", registry-validated, forward-looking plugin seam
- note prompt caching + count_tokens are now provider capabilities
(supports_prompt_caching / supports_token_count); Anthropic unchanged
* bd_1-scaffolding-j2c.6: US-006 .claude/rules seam-rename + provider-seam accuracy
Orchestrator-only rule-file edits (workers can't write .claude/ in worktrees).
* bd_1-scaffolding-j2c.7: Quality gate — fix bugs from code review
- providers.build_count_tokens_kwargs: correct docstring (probe omits the
cache_control marker because it doesn't affect the token count; the old
claim of "matching the inline call_anthropic probe" was inaccurate).
- call_llm: gate cache_marker_active on BOTH supports_prompt_caching AND
supports_token_count, so a future caching=True/token_count=False provider
degrades to no-caching rather than sending an unvalidated cache_control
marker (no sub-minimum drop / no oversize cap). Anthropic is True/True so
the default path is a no-op. (4-pass review latent-trap finding.)
CodeRabbit: skill not available in this environment — skipped.
* bd_1-scaffolding-j2c.8: Patterns & Memory — provider-seam convention + QG lesson
- llm-drafter.md: capability-gating lesson (gate the cache marker on BOTH
supports_prompt_caching AND supports_token_count — the #135 QG latent-trap
finding) for future #136/#137 provider authors.
- Added `provider` to the documented `llm:` config block.
- Reference section now cites plans/super/135 + the no-cache neutrality proof.
* #135: close Codecov patch gaps — count_tokens probe error mapping + classify fallthrough
The provider-neutral refactor rewrote the count_tokens pre-send probe's
error-mapping as new lines (auth/rate-limit/conn/5xx/other -> typed LLMError,
no retry on the probe) that the existing retry tests (messages.create path)
never exercised. Add 6 probe-failure tests + a missing-input_tokens case
(client.py 320-348 now covered, 87% -> 96%).
Also cover AnthropicProvider.classify_exception's defensive fallthrough for a
non-5xx/non-4xx-non-auth APIStatusError (3xx) -> NO_RETRY (providers.py -> 100%).
Remaining uncovered client.py helper branches (_extract_text_blocks /
_extract_usage_field malformed-response raises) are pre-existing and not part
of the #135 diff.
* #135: address PR review (CodeRabbit/Copilot)
- Stale-symbol docstrings: rename every `call_anthropic` :func: reference to
`call_llm` across llm/{__init__,models,_anthropic_client,providers}.py,
draft/schema.py, grade/{engine,config,errors}.py, cli/_estimate.py; reword
the AnthropicProvider "preparatory refactor" note (US-003 already collapsed
the duplication); rewrite the llm package docstring to the provider-neutral
seam.
- client.py: count_tokens NO_RETRY fallback message no longer claims a "status"
(it also covers non-status exceptions) — now "failed with a non-retryable error".
- generate.py --estimate: fail fast with CliInputError when draft.provider !=
grade.provider or the provider isn't 'anthropic' (the estimate engine drives a
single Anthropic-cast count_tokens client across both configs). + 2 regression
tests.
- plan SD-1: drop the stale `Literal` wording (DEC-007 settled it as a
registry-validated str).
---------
* docs(readme): refresh status, generalize config, redraft roadmap (#153)
* docs(readme): refresh status, generalize config, redraft roadmap
The v0.1 "Status:" line was four releases stale and BigQuery-only.
Replace with a Supported-warehouses block covering BigQuery (v0.1) and
Snowflake (v0.3 — incl. the deferred aggregate-only / column_stats
limitation), generalize Configuration to dispatch on dbt profile type,
and redraft the Roadmap into Shipped (v0.1/v0.2/v0.3 with release dates)
+ Planned (Gemini/OpenAI → installable skill → Airflow → GitHub Action
→ rubric customization → dbt Fusion). Drop the now-stale "prune-existing
is dev-branch only" and "first four ship; prune-existing is v0.2 dev"
callouts.
* docs(readme): add custom-business-logic callout, strip non-roadmap version markers
- Add a **Custom business logic** paragraph alongside Supported warehouses,
explaining `meta.signalforge.business_rules` → `custom_sql` drafting + the
ownership-marker write semantics + always-pass drop behaviour.
- Remove every version marker outside the Roadmap section: "since v0.1/v0.3"
in the warehouses block, "*(v0.2)*" on the prune-existing bullet, the "v0.2
default" config comment, the "(v0.1)"/"(v0.3)" subsection headers, and the
"pre-alpha … v0.1 milestone" line in Contributing.
* docs(readme): reorganize scope callouts; promote Supported warehouses to its own H2
The two bolded standalone paragraphs at the top of the README (Supported
warehouses, Custom business logic) sat before the "Why this exists" pitch
and front-loaded dense capability detail. The custom-business-logic block
was also triple-covered (standalone block + "What it does" bullet +
Quick-start worked example).
- Drop both standalone bolded paragraphs from the top.
- Promote Supported warehouses to a proper H2 between "How it works"
and the "Live on PyPI" callout — brief, points to Configuration for
per-warehouse setup; Snowflake's known limitation stays in its single
home under Configuration > Snowflake.
- Strengthen the custom-business-rule bullet in "What it does" with the
"fifth test type" framing and a direct link to the worked example, so
the feature stays prominent without a redundant top-of-readme banner.
New flow: pitch → features → architecture → scope → quickstart → reference.
---------
* #136: OpenAI grading provider (plan) (#152)
* #136: super plan for OpenAI grading provider
Phase: detailing (awaiting approval).
13 DECs captured: Chat Completions API, no provider/model cross-validation,
fully wire --estimate for OpenAI (new estimate_input_tokens ABC), default
judge gpt-4o, scope both grade + draft, server-side response_format=json_object,
four pricing SKUs (gpt-4o, -mini, 4.1, 4-turbo), live smoke covers grade +
estimate, .messages adapter wraps chat.completions, AST Scan 3 extension,
[openai] extra + lazy tiktoken, Anthropic estimate byte-identity floor.
9 stories: shim + AST confinement, OpenAIProvider + registration, FakeOpenAIClient
+ neutrality test, pricing entries, --estimate provider-aware counting,
live gated smokes, docs, Quality Gate, Patterns & Memory.
* #136: link plan to PR #152 (phase=published)
* #136: apply 5 plan revisions from #137 cross-review
Cross-review against #137 (Gemini grading) plan surfaced 5 substantive gaps
in the OpenAI plan:
1. DEC-010 — fix wording: a NEW 9th AST scan (Scan 3 is Anthropic-specific),
not an extension. Reuse _QualifiedNameCallFinder + three-pattern bypass
regression test per testing-signal.md.
2. DEC-012 — fix pyproject.toml listing: three slots in lockstep
([project.optional-dependencies].openai + .dev + [dependency-groups].dev),
mirroring Snowflake precedent per python-build.md. Plan previously named
only two.
3. DEC-014 (new) — codify the _load_openai_exception_classes empty-tuple
fallback on ImportError (mirrors AnthropicProvider). Includes refusal /
content-filter symmetry note explaining why OpenAI needs no Gemini-style
safety_filter typed-degrade DEC.
4. US-006 — add a third live test (draft_schema) honouring DEC-005's
"scope both stages" commitment at live level too. DEC-008 updated to
match (three tests, not two).
5. New "Open notes for implementation" section — pragmatic SDK-class-name
verification, .messages adapter shape confirmation, tiktoken fallback
table, response_format prompt-requirement check, Anthropic byte-identity
snapshot capture protocol.
Plus a new "Worker-writability routing" section codifying that US-009
(Patterns & Memory) is orchestrator-only because it edits .claude/rules/,
mirroring #137's same routing.
Plan grows from 319 → 343 lines (+40/-17).
* #136: devolve plan into bd beads (epic bd_1-scaffolding-4tw + 9 tasks)
Fills in the Beads manifest section. Epic bd_1-scaffolding-4tw with 9
child tasks (US-001 … US-009) created with dep edges per the plan's
"Depends on:" lines:
US-001 → none (READY)
US-002 → US-001
US-003 → US-002
US-004 → none (parallel-safe) (READY)
US-005 → US-002, US-004
US-006 → US-002, US-005
US-007 → US-001, US-002, US-005
US-008 → US-001..US-007 (QG)
US-009 → US-008 (P&M; orchestrator-only)
Cross-epic gate wired downstream: #137's sentinel bd_1-scaffolding-41a
now DEPENDS ON US-009, so #137 US-007 stays mechanically blocked until
this epic completes + sentinel closes after PR #152 merges to dev.
Parallel-safe at the entry points: US-001 and US-004 edit disjoint
files (shim/pyproject/AST vs pricing) — ralph-serialize-shared-registry
does not apply. Run concurrently.
bd ready (post-devolve) surfaces .1 + .4 of this epic + the two epics
themselves + the unrelated Airflow integration.
* bd_1-scaffolding-4tw.4: #136 US-004 — OpenAI pricing SKUs (gpt-4o + 3 siblings)
Adds four OpenAI SKUs to _PRICES_MUTABLE per DEC-007: gpt-4o (default
judge per DEC-004), gpt-4o-mini (budget), gpt-4.1 (newer flagship),
gpt-4-turbo (back-compat). Cache fields = 0.0 (OpenAI has no equivalent
cache discount). PRICE_TABLE_VERSION bumped to 2026-05-28.
Anthropic SKUs unchanged (byte-identical) — Anthropic estimate
byte-identity floor (DEC-013) preserved at the pricing layer.
Tests assert: non-zero input/output rates + zero cache fields for all
four; unknown model still raises EstimateUnknownModelError; Anthropic
SKUs unchanged; version bump pinned.
OpenAI per-MTok rates are calibration figures captured at PR-prep time
pending operator verification against https://openai.com/api/pricing/;
the figures are sanity-check baselines, not billing guarantees.
Traces: DEC-003, DEC-004, DEC-007.
* bd_1-scaffolding-4tw.1: #136 US-001 — _openai_client.py shim + [openai] extra + 9th AST scan
Implements the per-vendor shim confining every openai SDK type ignore.
Adds [openai] optional-dep extra (openai + tiktoken) in three-slot
lockstep across [project.optional-dependencies].openai, .dev, and
[dependency-groups].dev. Adds the new 9th AST scan in
tests/test_audit_completeness.py reusing _QualifiedNameCallFinder
(NOT an extension of Scan 3 — Scan 3 is Anthropic-specific) and a
companion per-file confinement test mirroring Snowflake's. Bumps the
docstring tally 8→9.
DEC-014 empty-tuple ImportError fallback on _load_openai_exception_classes
mirrors AnthropicProvider; tiktoken cl100k_base fallback for unknown
model ids.
Traces: DEC-001, DEC-009, DEC-010, DEC-012, DEC-014.
* bd_1-scaffolding-4tw.2: #136 US-002 — OpenAIProvider + registration + config-validator coverage
Implements OpenAIProvider(LLMProvider) with both capability flags False
(no prompt caching, no pre-send token count) — mirrors FakeNoCacheProvider
end-to-end shape. build_create_kwargs attaches response_format=
{type: json_object} per DEC-006 (server-side JSON enforcement); no
cache_control / extra_headers. extract_text_blocks reads
response.choices[0].message.content; extract_usage maps usage.prompt_tokens
+ completion_tokens → UsageMetrics with cache fields 0. classify_exception
covers all five ExceptionCategory branches via _load_openai_exception_classes
from the US-001 shim.
register_provider(OpenAIProvider()) at module scope so GradeConfig/
DraftConfig validators accept provider='openai'. Exported from
signalforge.llm.__init__.
Tests pin: each ABC method's contract; config validator acceptance for
both stages; UnknownProviderError lists both anthropic and openai.
Traces: DEC-001, DEC-005, DEC-006, DEC-009, DEC-011.
* bd_1-scaffolding-4tw.3: #136 US-003 — FakeOpenAIClient + grade neutrality e2e
Adds tests/llm/_fake_openai.py with FakeOpenAIClient + expect_messages_create
API mirroring FakeAnthropicClient verbatim. Support dataclasses match the
real chat-completions response shape (choices[0].message.content; usage
with prompt_tokens + completion_tokens).
Adds tests/grade/test_provider_neutrality_openai.py — the OpenAI analogue
of the FakeNoCacheProvider neutrality proof. Asserts cache_*=0 in JSONL
audit, blake2b-8 reproducibility hashes present, sidecar drift-detector
round-trip, no dual-zero cache-anomaly WARNING in caplog, all expectations
consumed.
Traces: DEC-001, DEC-005, DEC-006, DEC-009, DEC-011.
* bd_1-scaffolding-4tw.5: #136 US-005 — --estimate provider-aware token counting
Adds LLMProvider.estimate_input_tokens(model, text, *, client=None) -> int
as an abstract method on the ABC (DEC-003). AnthropicProvider impl
delegates to the SDK's messages.count_tokens with a single user-message
text envelope; OpenAIProvider impl delegates to _count_openai_tokens
(tiktoken with cl100k_base fallback per DEC-012); FakeNoCacheProvider
and _DummyProvider impls return trivial deterministic answers so the
existing neutrality + registry tests still pass.
Refactors cli/_estimate.py: _count_draft_tokens and the grader-side
equivalent dispatch through provider_for(config.provider).estimate_input_tokens
— no more hard-coded anthropic_client.messages.count_tokens calls. The
engine signature relaxes anthropic_client to 'object | None' so the
OpenAI path can pass None (tiktoken is local). Lifts the
'--estimate currently supports only provider=anthropic' gate in
cli/generate.py; the divergent-providers check stays (the engine still
takes one optional client).
Pins DEC-013 Anthropic byte-identity floor via
tests/cli/test_estimate.py::test_estimate_anthropic_byte_identity_golden
and tests/fixtures/estimate/anthropic_byte_identity_golden.txt (captured
2026-05-28 before the refactor; reproduced verbatim after). New
companion tests prove the OpenAI path produces non-zero token counts
+ non-zero USD, ignores the threaded anthropic_client, and goes through
_count_openai_tokens (patched-helper assertion).
Traces: DEC-003, DEC-007, DEC-012, DEC-013.
* bd_1-scaffolding-4tw.7: #136 US-007 — operator docs + CHANGELOG (worker-writable portion)
Updates operator-facing documentation surfaces for OpenAI grading +
drafting:
- docs/grade-ops.md — OpenAI provider section (config snippet,
OPENAI_API_KEY, no-cache caveat, live smoke gating link)
- docs/draft-ops.md — equivalent for llm.provider: openai
- docs/cost-estimate-ops.md — tiktoken note, [openai] extra install,
four pricing SKUs
- CHANGELOG.md — [Unreleased] Added entry for #136
- README.md — provider enumeration extended (if applicable)
The .claude/rules/llm-drafter.md update is deferred to US-009
(Patterns & Memory) per the project's worker-writability rule
(orchestrator-only edits under .claude/).
Traces: DEC-001 through DEC-014.
* bd_1-scaffolding-4tw.6: #136 US-006 — live gated smoke tests (openai marker + 3 tests)
Adds @pytest.mark.openai marker + three gated live-API tests:
- tests/grade/test_smoke_real_api_openai.py — grade_artifacts(provider=openai)
- tests/draft/test_smoke_real_api_openai.py — draft_schema(provider=openai)
(honors DEC-005 both-stages at live level)
- tests/cli/test_e2e_estimate_openai.py — generate --estimate openai
All three env-gated on SF_RUN_OPENAI=1 + OPENAI_API_KEY via belt-and-
suspenders _skip_reason() helper per testing-signal.md. Marker registered
in pyproject.toml + added to addopts exclusion so default pytest doesn't
collect them. CONTRIBUTING.md documents the maintainer-only run pattern.
Traces: DEC-001, DEC-004, DEC-005, DEC-008.
* bd_1-scaffolding-4tw.8: #136 Quality Gate — fix 2 majors + 1 minor from code review x4
Code-review pass 3 surfaced two correctness MAJORS + pass 4 surfaced
a doc MINOR; all fixed inline per /ralph-run Step 3b. Pass 1, 2, 4
(other surfaces) reported no correctness bugs.
MAJOR 1 — EstimateUnknownModelError.default_remediation enumerated
only the three Anthropic SKUs. An operator typo on an OpenAI SKU
got pointed at Anthropic options (violates "errors carry remediation"
in manifest-readers.md). Updated to enumerate all 7 SKUs
(claude-sonnet-4-6, claude-opus-4-7, claude-haiku-4-5, gpt-4o,
gpt-4o-mini, gpt-4.1, gpt-4-turbo) + refreshed the locked-text test
that pinned the stale string.
MAJOR 2 — AnthropicProvider.estimate_input_tokens dropped the
pre-refactor `system=` kwarg, concatenating system+cached+dynamic
into a single user-content string. Anthropic's server-side tokenizer
counts the system block with its own envelope tokens; without the
kwarg, real-API counts under-report by the system-envelope size.
The fake-driven byte-identity test passed only because the fake
returned canned input_tokens regardless of kwargs (so rendered-output
identity held, but DEC-013's "byte-identity vs the pre-refactor
Anthropic count_tokens call" spirit broke at the real-API level).
Fix extends the ABC signature with a keyword-only `system: str = ""`
parameter. AnthropicProvider passes it via `system=` kwarg when
non-empty (matches pre-refactor real-API call shape). OpenAIProvider
concatenates `system + text` before tiktoken (tiktoken has no
system-envelope distinction; the total still counts every token).
FakeNoCacheProvider concatenates for the word-count proxy.
_DummyProvider gains the new kwarg for ABC parity. cli/_estimate.py
callers (_count_draft_tokens, _count_grade_criterion_tokens) thread
`system=` separately; grade-side preserves the pre-existing
double-count of the rubric (passed as both system= AND in user content)
to keep byte-identity with the pre-refactor shape.
MINOR — docs/grade-ops.md:118 config-snippet comment said "only
anthropic registered today", contradicting the same file's later
"## OpenAI provider" section and docs/draft-ops.md's correct
wording. Updated to mirror draft-ops's form.
Validation: ruff/pyright/pytest all green (2498 passed, 62 deselected,
97.34% coverage); wheel_smoke green (2 passed).
Note: CodeRabbit skill not available in this environment, skipped
per Step 3b "(if available)".
* bd_1-scaffolding-4tw.9: #136 Patterns & Memory — codify OpenAI seam conventions
Orchestrator-only commit (workers can't write under .claude/ in
worktrees per ralph-worker-claude-dir-perms.md).
Updates .claude/rules/llm-drafter.md to capture three durable patterns
from #136:
1. Provider-neutral seam — adds OpenAI as the second registered
provider (name="openai", both capability flags False) and extends
the LLMProvider ABC surface enumeration with the new
estimate_input_tokens(model, text, *, system="", client=None)
method (#136 US-005).
2. "OpenAI provider shape" subsection — codifies the .messages.create
façade adapter pattern (SimpleNamespace delegating to
chat.completions.create), the response_format={type:json_object}
server-side JSON enforcement (with the cross-ref to issue #144's
tolerant parser as fallback), and the tiktoken cl100k_base
fallback for unknown model ids. This becomes the canonical
precedent for #137 Gemini's response_mime_type=application/json
equivalent.
3. estimate_input_tokens(*, system=...) — documents the load-bearing
reason the system envelope is threaded separately (Anthropic's
server-side tokenizer applies system-block envelope tokens; dropping
the kwarg under-reports real-API counts silently because
fake-driven byte-identity tests can't catch call-shape drift). Also
pins the deliberate double-count of system_and_rubric in
_count_grade_criterion_tokens as pre-existing behaviour that must
be preserved.
Bumps AST-scan tally section from "four" → "five" (adds the openai.OpenAI
9th-project-scan); flags #137 Gemini as the 10th. Updates the
References block with #136 plan + new files.
Memory entry: ~/.claude/projects/.../memory/fake-driven-byte-identity-
blind-spot.md captures the QG lesson (fakes return canned values
regardless of kwargs → call-shape drift slips past rendered-output
snapshots → needs explicit kwargs-shape assertion OR live test).
Indexed in MEMORY.md.
Validation: all four canonical gates green (2498 passed, 97.34%
coverage).
* #136: address PR #152 review (6 actionable + meta)
Copilot:
- MAJOR _estimate.py:359 — OpenAI grade-side over-counted rubric. The
first QG fix preserved a pre-existing Anthropic double-count of
system_and_rubric (passed as both system= AND in user content),
which triple-counted on OpenAI (system→system+text concat → rubric
prefix in text). Corrected to match the runtime grader call: rubric
in system= once, artifact envelope in user content. Anthropic real-
API counts drop one rubric copy from the buggy pre-refactor bytes;
OpenAI counts each input once. Fake-driven byte-identity golden
still passes (canned token counts are call-shape-agnostic).
CHANGELOG documents the calibration shift.
- NIT _estimate.py:421 — renamed anthropic_client → client. Type was
already object | None and forwarded to whichever provider strategy
is active; old name implied Anthropic-only and would mislead a
future #137 Gemini wiring. CLI in generate.py already passed None
for non-Anthropic; rename surfaces the contract without behaviour
change.
- META plan/PR description — addressed by updating the PR body
separately (not part of this commit).
CodeRabbit:
- CONTRIBUTING.md:64 — added snowflake to the gated-marker audit
command + the corresponding SNOWFLAKE_* env vars.
- tests/llm/_fake_provider.py:234 — boundary-word undercount fix in
the FakeNoCacheProvider word-count proxy (f"{system} {text}" with
a delimiter; was system+text which merged the last word of system
with the first word of text under .split()).
- tests/llm/test_openai_client_confinement.py:47 — glob("*.py") →
rglob("*.py") so the confinement scan catches openai-mentioning
ignore directives in any future nested signalforge/llm/ subpackage,
not just top-level files. Path display becomes relative-to-_LLM_DIR.
- tests/test_audit_completeness.py:1080 — closed the "call-before-
import alias" bypass on _AttributeCallFinder by adding a two-pass
visit_Module that pre-collects every alias module-wide before
visiting any Call node. Mirrors the same fix already shipped on
_QualifiedNameCallFinder (PR #69 / DEC-013). Pattern-4 regression
test added.
Validation: ruff/format/pyright/pytest all green (2498 passed,
97.34% coverage); wheel_smoke green.
* #136: close PR #152 codecov gaps (9 lines → 0)
Codecov flagged 91.82% patch coverage with 9 lines missing across
3 files. Closed all 9 with 6 focused unit tests:
src/signalforge/llm/_openai_client.py (was 85%, now 100%):
- _OpenAIClientAdapter.__init__ + ._messages_create — no production
caller exercises the adapter (orchestrator drives via
FakeOpenAIClient which has its own .messages.create); added
test_openai_client_adapter_messages_create_delegates_to_chat_completions
that builds a SimpleNamespace raw client and pins the delegation
+ kwargs forwarding.
- _count_openai_tokens cl100k_base fallback (DEC-012 unknown-model
branch) — added test_count_openai_tokens_falls_back_to_cl100k_base_
for_unknown_model.
src/signalforge/llm/providers.py (was 98%, now 100%):
- AnthropicProvider.estimate_input_tokens no-system-kwarg arm (the
else branch added in QG) — added test_anthropic_provider_estimate_
input_tokens_skips_system_kwarg_when_empty, which ALSO pins the
load-bearing invariant that an empty system MUST be omitted from
the SDK kwargs (otherwise the count carries a spurious system=""
block).
- LLMResponseFormatError raise on missing input_tokens — added
test_anthropic_provider_estimate_input_tokens_raises_on_missing_
input_tokens.
- OpenAIProvider.extract_text_blocks missing-message arm — added
test_openai_provider_extract_text_blocks_missing_message_attr_
raises (distinct from the existing content=None and empty-choices
cases — those exercise different arms).
src/signalforge/cli/generate.py:846 (was 66%, now covered):
- The non-Anthropic `client = None` branch in cmd_generate's
--estimate short-circuit — exercised only by the
@pytest.mark.openai live smoke before. Added test_generate_
estimate_openai_provider_passes_client_none_to_engine that drives
cmd_generate with llm.provider: openai + grade.provider: openai
in a tmp signalforge.yml, spies on AnthropicProvider.make_client
to confirm it's NEVER called on the openai path, and captures the
client kwarg into estimate(...) to pin client=None. The remaining
7 uncovered lines in generate.py are pre-PR and out of scope.
Total: 2504 passed (was 2498), 97.47% coverage (was 97.34%), no
new gated tests added (all unit tests, default-CI-included).
---------
* #137: Gemini grading provider (plan) (#151)
* #137: Gemini grading provider (plan)
Super plan for #137 (Gemini grading provider) — depends on #135 (merged).
9 stories: shim + provider + extra + fake/unit + neutrality + live +
docs + QG + P&M. Caching deferred (both capability flags False);
safety-filter no-content surfaces as LLMResponseFormatError →
grade GradeLLMError degrade. Both drafter and grader covered.
* #137: extend plan after comparison with #136 OpenAI plan
Closes the three gaps surfaced by the #136 comparison:
- Add --estimate integration (DEC-016): GeminiProvider.estimate_input_tokens
via native client.models.count_tokens (cleaner than #136's tiktoken path —
Gemini has a first-party count endpoint).
- Add 3 Gemini pricing SKUs to pricing.py (DEC-017): gemini-2.5-pro,
gemini-2.5-flash, gemini-2.0-flash; bump PRICE_TABLE_VERSION.
- Add server-side JSON enforcement (DEC-018): response_mime_type=
application/json — mirrors #136 DEC-006.
- Add CHANGELOG entry to docs story (US-009).
- Add wheel_smoke to QG (new [gemini] extra changes packaging).
- Sequence after #136 (DEC-019): inherit ABC extension + estimate refactor.
New stories: US-006 (pricing, parallel-safe), US-007 (estimate impl,
depends on #136). Old US-006/007 renumbered to US-008/009.
* #137: apply 3 plan refinements from #136 cross-review
Closes the smaller gaps surfaced by re-reviewing #137 against #136 after
the #136 plan was tightened (520b36c):
1. DEC-009 — merge-order-conditional AST scan tally. Currently said "8 → 9"
unconditionally, but DEC-019 specifies #137 ships AFTER #136. Realistic
bump is 9 → 10 (since #136's openai.OpenAI scan bumps 8 → 9 first);
added the fallback wording for the slipped-sequencing case.
2. Dedicated "Worker-writability routing" top-level section codifying that
Patterns & Memory is orchestrator-only (.claude/rules/ writes). Was a
one-line architecture-review row; now a durable section that mirrors
#136's codification — future #138+ provider plans copy the pattern.
3. Open notes for implementation — two additions:
- response_mime_type requires NO prompt keyword (contrast with OpenAI's
response_format which fails without "json" in prompt). Prevents a
future maintainer adding defensive prompt text.
- pricing.lookup zero cache fields math validation (symmetric to #136
verification item).
Major gaps from the prior cross-review (--estimate, pricing, dual-listing,
JSON enforcement, CHANGELOG, three live tests, sequencing) were already
addressed in b28c0b5.
* #137: devolve plan to beads (phase=devolved)
Epic: bd_1-scaffolding-txe
Tasks: 11 (US-001 through US-009 + Quality Gate + Patterns & Memory)
Dependencies: 19 edges per the plan's dependency graph
Ready queue: US-001 (shim) + US-006 (pricing, parallel-safe)
US-007 (--estimate integration) is blocked on US-002 + US-006 AND
carries a cross-epic gate on #136 landing first per DEC-019 — that
sequencing is documented in the task's description rather than wired
as a bead dep since #136's beads live in a different epic.
US-011 (Patterns & Memory) is flagged orchestrator-only because it
edits .claude/rules/ which Ralph workers can't write under in
worktrees (per the Worker-writability routing section of the plan).
* #137: adjust plan now that #136 is being implemented first
Folds devolve-time annotations (phase=devolved, beads epic
bd_1-scaffolding-txe, per-task status + cross-epic blocker on US-007)
together with five #136-driven adjustments:
1. AST-scan tally locked to 10th (not 9th) — #136 owns the 8→9 bump
for openai.OpenAI; #137 takes 9→10 for genai.Client. Updated in
DEC-009, US-001, P&M, and Open notes.
2. DEC-019 sharpened: #136 plan → #136 implementation; explicit rebase
guidance (rebase on dev after #136 merges) rather than contingency
wording.
3. DEC-005 gains a refusal/content-filter symmetry note cross-ref'ing
#136 DEC-014 — explains why OpenAI deliberately ships no
safety-filter typed-degrade and why Gemini deliberately does.
4. New "Worker-writability routing" section mirroring #136's, codifying
that P&M is orchestrator-only because it edits .claude/rules/.
5. Open notes extended with two pragmatic items parallel to #136's:
- google-genai count_tokens response field-name verification
(.total_tokens vs .total_token_count) at US-007 implementation.
- Anthropic byte-identity snapshot ownership: #136 owns it; #137
inherits it; the wiring is wrong if it moves the snapshot.
Plan grows 762 → 825 lines (+70/-30).
* bd_1-scaffolding-txe.6: #137 US-006 — add Gemini pricing SKUs (gemini-2.5-pro, gemini-2.5-flash, gemini-2.0-flash)
Adds three Gemini SKUs to _PRICES_MUTABLE per DEC-017, each with positive
input_per_mtok / output_per_mtok per Google's public Gemini API price page
and cache fields = 0.0 (v0.3 ships Gemini without an Anthropic-equivalent
prompt-cache discount). Bumps PRICE_TABLE_VERSION to "2026-05-27". Tests
parametrise the three SKUs, pin Anthropic SKU byte-identity (additive-only),
and assert lookup("gemini-unknown") still raises EstimateUnknownModelError.
cli/_estimate.py reads only input_per_mtok and output_per_mtok in the USD
math (lines 468, 511), so zero cache fields are safe — no divide-by-zero
or NaN risk. The EstimateUnknownModelError.default_remediation in
llm/errors.py still lists only the three Claude SKUs by name; per the
story scope ("pricing-only; DO NOT touch any other file") this is left
for a follow-up to broaden once the wider Gemini surface (provider class
+ --estimate integration in US-007) lands.
* bd_1-scaffolding-txe.1: #137 US-001 — _gemini_client.py shim + new AST confinement scan
Land the per-vendor Gemini SDK shim and the 9th AST audit-completeness
scan that confines genai.Client(...) constructions to it. Mirrors the
Anthropic shim shape verbatim — GeminiClientProtocol (with .messages
façade for US-002 and .models for US-007), lazy-import factory, frozen
_GeminiExceptionClasses dataclass with empty-tuple fallback per DEC-015
so the module imports cleanly without the [gemini] extra installed.
The new scan extends _AttributeCallFinder with an optional parent_module
parameter so the namespace-package shape `from google import genai;
genai.Client(...)` is caught alongside `from google.genai import Client`
and its alias variant — five planted-violation regression tests pin the
three bypass patterns plus a negative case and an Anthropic-path-unchanged
guard. A line-based confinement test
(tests/llm/test_gemini_client_confinement.py) rejects any
google.genai-mentioning `# type: ignore` outside the shim, mirroring
tests/warehouse/test_snowflake_client_confinement.py. The existing
Anthropic shim's regex-level confinement test narrowed from "any
ignore" to "anthropic-mentioning ignore" so the new Gemini shim's own
SDK ignores don't trip it. All four canonical validation steps pass;
2474 tests; coverage 97.33%.
* bd_1-scaffolding-txe.3: #137 US-003 — pyproject.toml [gemini] extra + dev-group sync
google-genai>=0.5,<1 listed in three pyproject slots in lockstep per DEC-010
(operator install, pip dev back-compat, uv dev group). uv.lock regenerated.
* bd_1-scaffolding-txe.2: #137 US-002 — GeminiProvider(LLMProvider) + registration
GeminiProvider concrete strategy registered at module import; both capability
flags False per DEC-003. build_create_kwargs sets response_mime_type=
application/json (DEC-018) and maps system → system_instruction + single user
turn (DEC-004). Safety-blocked / no-content responses → LLMResponseFormatError
per DEC-005. Exception taxonomy per DEC-006 — verified against
google-genai==0.8.0.
Notes from verification against the installed SDK:
- google.genai.errors.APIError (the shared parent of ClientError +
ServerError) stores the HTTP code on .code (an int), not .status_code —
the classifier reads .code. ClientError covers 4xx (incl. 401/403/429);
ServerError covers 5xx. ServerError is checked BEFORE ClientError so the
narrower 5xx bucket cannot be shadowed by a future shared parent class.
- The SDK's models.generate_content(config=) accepts both
GenerateContentConfig and the plain-dict GenerateContentConfigDict form.
build_create_kwargs returns the plain-dict form so providers.py never
imports google.genai.types at any scope — keeping
test_gemini_client_confinement.py green and base-install module-import
clean.
- The .messages.create façade required by the orchestrator is wired in
GeminiProvider.make_client via a small _GeminiClientAdapter /
_GeminiMessagesAdapter pair (the shim returns the bare client; US-002
owns the adapter per the shim's docstring). The adapter forwards
**kwargs straight to client.models.generate_content / count_tokens.
- Connection-flavoured exceptions (httpx.ConnectError /
httpx.TimeoutException) leak through the SDK on hard network failures;
the classifier handles them with a lazy httpx import and an
ImportError-safe fallback to NO_RETRY.
Updates the pinned signalforge.llm __all__ surface in
tests/llm/test_public_api.py and tests/draft/test_schema.py to include the
new GeminiProvider re-export; adds DraftConfig/GradeConfig provider="gemini"
acceptance tests.
* bd_1-scaffolding-txe.9: #137 US-009 — operator-facing docs + CHANGELOG for Gemini
docs/grade-ops.md + docs/draft-ops.md register gemini as a provider, name the
[gemini] install extra + GOOGLE_API_KEY env var, and document the v0.3
no-caching cost note (DEC-013). README provider list deferred (no list exists
to extend). CHANGELOG entry under [Unreleased]. .claude/rules/* deferred to
Patterns & Memory (orchestrator-only).
* bd_1-scaffolding-txe.4: #137 US-004 — FakeGeminiClient + offline provider integration tests
Hand-rolled FakeGeminiClient mirrors FakeAnthropicClient's expect_* API
(DEC-011). New tests/llm/test_fake_gemini.py proves the fake's contract;
tests/llm/test_gemini_provider_via_fake.py drives GeminiProvider through
the fake end-to-end (call_llm round-trip, safety-blocked branch, retry
exhaustion).
* bd_1-scaffolding-txe.5: #137 US-005 — provider-neutrality e2e tests (draft + grade)
tests/grade/test_gemini_neutrality.py drives grade_artifacts end-to-end through
FakeGeminiClient: cache_*=0 in JSONL, blake2b-8 hashes, sidecar round-trip, no
dual-zero WARNING (DEC-003), safety-blocked → GradeLLMError degrade (DEC-005).
tests/draft/test_gemini_neutrality.py drives draft_schema end-to-end likewise.
DEC-014 two-stage scope satisfied at the offline-test layer.
* bd_1-scaffolding-txe.8: #137 US-008 — gemini marker + live tests (raw + draft + grade) + CONTRIBUTING
@pytest.mark.gemini marker registered; addopts excludes it from default CI per
DEC-012. Three live smokes gated SF_RUN_GEMINI=1 + GOOGLE_API_KEY:
test_gemini_live.py (raw call_llm), test_gemini_draft_live.py (draft_schema),
test_gemini_grade_live.py (grade_artifacts 1-criterion × 1-artifact).
CONTRIBUTING.md adds the maintainer 'uv run pytest -m gemini --no-cov' entry
alongside the existing snowflake/anthropic equivalents. --estimate live test
deferred to US-007 (gated on #136).
* #137: harden DEC-019 cross-epic gate via bd sentinel bead
DEC-019 was documentary only; bd_1-scaffolding-txe.7 (US-007) showed
as bd-ready despite the plan saying "wait for #136 to merge to dev."
A Ralph worker running `bd ready` would have picked it up and
rebase-fought #136 on providers.py / pricing.py / cli/_estimate.py
exactly as DEC-019 warns against.
Created sentinel bead bd_1-scaffolding-41a ("#136 OpenAI grading
PR #152 merged to dev") and wired bd_1-scaffolding-txe.7 to depend on
it. `bd ready` no longer surfaces US-007 until the sentinel closes.
Close the sentinel the moment #136 merges to dev → US-007 unblocks
automatically.
Updated DEC-019 + the beads manifest entry on US-007 to point at the
sentinel.
* bd_1-scaffolding-txe.7: #137 US-007 — GeminiProvider.estimate_input_tokens + --estimate integration
Replaces the merge-resolution NotImplementedError stub with a real
implementation via Gemini's native client.models.count_tokens
(first-party; no tiktoken equivalent). The --estimate cost-preview
path now works end-to-end for grade.provider: gemini and
llm.provider: gemini. Anthropic byte-identity golden unchanged
(verified against tests/fixtures/estimate/anthropic_byte_identity_golden.txt).
* bd_1-scaffolding-txe.10: #137 Quality Gate — fix bugs from 4-reviewer pass
Four parallel code-review passes surfaced 2 substantive code findings
and 5 documentation findings; all fixed below. CodeRabbit run skipped
(no skill available locally); maintainer can request post-merge.
Code fixes (reviewer 2 — tests):
- tests/test_audit_completeness.py: added the missing planted-violation
regression test for the new parent_module codepath in
_AttributeCallFinder. Covers Patterns 5-8 (namespace-package via
from-google-import-genai, namespace alias, dotted-from-import,
dotted-import-as) plus a negative check pinning that Pattern 7
requires parent_module to be set. Mandated by testing-signal.md
§ "AST single-construction-seam scans must catch all three bypass
patterns" — without this, a refactor of the parent_module branches
could silently break Scan 10 at the exact moment a real Gemini-SDK
construction was added outside _gemini_client.py.
- tests/grade/test_gemini_grade_live.py: added cache_*=0 assertion on
every GradeEvent record (DEC-003 of #137). The docstring claimed
this was tested but the assertion was missing — a grade-path
bookkeeping regression would have slipped through the live smoke.
Doc fixes (reviewer 3 — docs):
- docs/grade-ops.md + docs/draft-ops.md: rewrote the "--estimate
integration (deferred)" sections to reflect that US-007 SHIPPED in
this PR. Both now describe the active behaviour (native
client.models.count_tokens round-trip; failures surface as
<unavailable>) instead of the pre-US-007 deferred state.
- CHANGELOG.md: updated the Gemini Unreleased bullet to advertise
--estimate support as part of the #137 deliverable; reserved the
"follow-up" framing for explicit Gemini context caching only.
- docs/cost-estimate-ops.md: added Gemini coverage in three places —
the provider-aware token counting bullet list, a parallel "Gemini
provider — [gemini] install extra" section (with the three
registered SKUs in a table), and the maintainer live smoke set.
File had ZERO Gemini coverage before this fix.
- README.md: added Google Gemini to the supported-providers section
alongside Anthropic + OpenAI; moved the Gemini roadmap row from
"Planned v0.4" into "Shipped v0.4" since it's landing in this PR.
Reviewer 1 (providers.py) and reviewer 4 (packaging + integration)
reported 0 bugs. Reviewer 2's LOW-priority findings (flake-prone
aggregate_complete assertion, loose substring on confinement scan,
missing 5xx-retry integration test) deferred — not correctness risks.
Validation after fixes:
- ruff check: passed
- ruff format --check: passed
- pyright: 0 errors, 0 warnings, 0 informations
- pytest: 2555 passed, 65 deselected, 97.51% coverage
- wheel_smoke: 2/2 passed (new [gemini] extra packaging intact)
- Anthropic byte-identity golden: passed (unchanged)
* bd_1-scaffolding-txe.11: #137 Patterns & Memory — durable rules for Gemini
Orchestrator-only commit (Ralph workers can't write under .claude/ in
worktrees per memory ralph-worker-claude-dir-perms.md).
.claude/rules/llm-drafter.md:
- AST scan tally bumped 5 → 6 (Scan 10 = genai.Client confinement
via _AttributeCallFinder(parent_module="google")).
- New § "Gemini provider shape (#137 — the third concrete provider,
no-cache via a namespace-package SDK)" with the four load-bearing
patterns: .messages-over-.models.generate_content adapter,
response_mime_type=application/json server-side JSON enforcement,
safety-filter typed LLMResponseFormatError → grade degrade, and
native models.count_tokens for --estimate (distinct from OpenAI's
local tiktoken path).
- New § "Namespace-package SDKs (#137 generalisation)" documenting
the parent_module parameter that catches the four namespace-package
import shapes the no-parent path misses.
- Reference block extended with the #137 plan + the three new fakes
+ the two new neutrality test suites.
- "new vendor lands" wording generalised (#137 is no longer "next",
it's "shipped"); Anthropic→OpenAI→Gemini lineage noted explicitly.
.claude/rules/grade-layer.md:
- Conservative degrade taxonomy DEC-002/DEC-015 entry for
GradeLLMError extended with a sentence explaining Gemini's
safety-filter / no-content response routes through the same path
via the typed LLMResponseFormatError that GeminiProvider raises
(DEC-005 of #137). Locks in the provider-neutral contract: a
future vendor with a content-filter surface MUST route through
LLMResponseFormatError, not a provider-specific switch in
grade_artifacts.
* #137: Address PR review feedback (Copilot + CodeRabbit)
Five real issues fixed; three outdated threads resolved as-is.
Fixed:
1. src/signalforge/llm/_gemini_client.py — Copilot flagged that
GeminiClientProtocol's docstring claimed the bare SDK client
satisfies it, and _make_gemini_client was annotated to return
GeminiClientProtocol despite returning the raw google.genai.Client
(which has no .messages namespace). The protocol is satisfied only
by the wrapped _GeminiClientAdapter (in providers.py) and by the
test fake. Rewrote both docstrings; relaxed _make_gemini_client's
return type to `Any` and explained the wrapper-on-top contract
explicitly. Replaces the misleading "satisfies structurally" claim
with the honest "bare SDK does NOT satisfy; wrapper does."
2. plans/super/137-gemini-grading.md (Discovery, line 98) — CodeRabbit
flagged the stale PRICE_TABLE_VERSION = "2026-05-11". Annotated
the reference as "at discovery time" and noted that US-006 and
#136 both bump it.
3. plans/super/137-gemini-grading.md (Worker-writability routing) —
CodeRabbit flagged a duplicated "## Worker-writability routing"
heading. The duplicate must have crept in during the cross-review
pass. Removed the second copy; kept the first canonical version.
4. tests/llm/test_pricing.py
(test_lookup_raises_estimateunknownmodelerror_for_unknown_gemini_model)
— CodeRabbit flagged that the test only validates `.model`,
leaving the operator-facing remediation text drifting silently
from the pricing table. Added a structural pin that the rendered
exception names every Gemini SKU. Future SKU additions force a
lockstep remediation update.
Outdated threads (3) — content already fixed by earlier commits; the
threads are kept resolved for hygiene:
5. tests/test_audit_completeness.py docstring "node.module == genai"
note (Copilot, two threads). The actual scan handles the dotted-
form via parent_module; the comments in the implementation
already reflect this since the QG pass.
6. CONTRIBUTING.md pre-release excluded-marker list (CodeRabbit). The
list was already extended to include `gemini` (and `snowflake`)
during the QG fix in commit f15a469.
Validation after fixes:
- ruff check: passed
- ruff format --check: 279 files formatted
- pyright: 0 errors, 0 warnings, 0 informations
- pytest: 2555 passed, 65 deselected, 97.51% coverage
* #137: Address second-round PR review (Copilot, 4 new comments)
Two real findings fixed; two false positives documented.
Fixed:
1. pyproject.toml gemini marker description — Copilot flagged that
the help text named only GOOGLE_API_KEY but the live tests also
gate on SF_RUN_GEMINI=1. Updated to "requires SF_RUN_GEMINI=1 +
GOOGLE_API_KEY" — mirrors the OpenAI marker's shape exactly.
2. src/signalforge/llm/providers.py::GeminiProvider.extract_usage —
Copilot flagged that getattr(usage, "prompt_token_count", 0) or 0
silently swallows missing/malformed SDK response shapes and feeds
misleading 0/0 figures into the audit JSONL + --estimate math.
Switched to the shared _extract_usage_field helper which raises
LLMResponseFormatError on missing/non-int fields. Matches the
Anthropic precedent (the OpenAI provider already uses the same
helper). Added test_geminiprovider_extract_usage_missing_inner_
field_raises pinning both the missing-field and non-int-type
paths to keep them durable.
False positives (replied in resolve-threads):
3. _gemini_client.py:116 (`_make_gemini_client` return annotation):
Copilot is reviewing the pre-commit-00dc673 diff. The previous
round of fixes (commit 00dc673) already changed the return type
to Any and rewrote the docstring to be honest that the bare SDK
client does NOT satisfy GeminiClientProtocol; only the
_GeminiClientAdapter wrapper does. Current file matches.
4. plans/super/137-gemini-grading.md:777 (duplicate Worker-
writability routing): Same — the duplicate was already removed
in 00dc673. Only one section now exists at line 757.
Validation:
- ruff check: passed
- ruff format --check: passed
- pyright: 0 errors, 0 warnings, 0 informations
- pytest: 2556 passed, 65 deselected, 97.55% coverage
---------
* #155: Gemini truncation + per-provider e2e gap (#156)
* #155: super-plan — Gemini truncation + per-provider e2e gap
12 DECs covering:
- DEC-001/002/005: provider-neutral is_clean_completion ABC method
- DEC-003/011: 2 new e2e siblings + BQ smoke parametrize
- DEC-008/009: per-provider max_output_tokens floor table (1024/1024/2048)
- DEC-010: pre-release-only cadence (~\$0.30 / suite run)
- DEC-012: apply_provider_override helper
10 stories sized for Ralph contexts. Architecture review: 1 concern (seam
design — resolved), 8 pass. Regression risk green (no fixture pins old
reasoning string; existing test_gemini_neutrality.py:381 already pins the
post-fix shape).
* #155: devolved — epic + 10 tasks live in bd
Epic: bd_1-scaffolding-eu0
Ready set: US-001 (.2), US-003 (.4), US-004 (.5) — three parallel-safe.
16 dep links wired.
Serialization callouts captured (US-005/6/7 share _e2e_helpers + BQ smoke;
US-002 + US-010 edit .claude/rules/, orchestrator-only).
* bd_1-scaffolding-eu0.4: #155 US-003 — bump Gemini live fixture to 2048 + per-provider max_output_tokens floor docs
* bd_1-scaffolding-eu0.5: #155 US-004 — apply_provider_override helper + BQ smoke uses it
Adds the canonical per-test grade-provider overlay helper. Multi-provider
e2e smokes (BigQuery+Anthropic / +OpenAI / +Gemini) share the committed
Austin fixture and swap only grade.provider/model/max_output_tokens via
this seam — no near-duplicate fixtures.
The helper is non-destructive: unset knobs left alone, sibling top-level
blocks (llm:/safety:/prune:) round-trip via yaml.safe_dump(sort_keys=False).
Missing signalforge.yml raises FileNotFoundError rather than silently
creating one (masks misconfigured tests).
Unit-tested in tests/cli/test_e2e_helpers.py — runs in the default
pytest set (no marker), so a regression in the YAML overlay plumbing the
real e2e smokes depend on trips immediately.
BigQuery smoke refactored to call apply_provider_override(project_dir,
grade_provider='anthropic') as a no-op proof-of-use; OpenAI (US-005) and
Gemini (US-006) sibling smokes will pass non-default values through the
same seam.
Traces to: DEC-012 in plans/super/155-gemini-truncation-e2e-gap.md
* bd_1-scaffolding-eu0.2: #155 US-001 — LLMProvider.is_clean_completion ABC + 3 concretes + wire-in
Add provider-neutral allowlist gate for response finish-reason / stop-reason
to prevent silent pass-through of truncated / safety-filtered / tool-use
responses (DEC-001/002/005/006/007 of plans/super/155).
- LLMProvider.is_clean_completion(response) -> bool — abstract
- LLMProvider.unclean_finish_reason_message(response) -> str — default + per-vendor overrides
- AnthropicProvider._CLEAN_STOP_REASONS = {end_turn, stop_sequence}; tool_use is UNCLEAN (DEC-006)
- OpenAIProvider._CLEAN_STOP_REASONS = {stop}
- GeminiProvider._CLEAN_STOP_REASONS = {STOP}
- call_llm wires the gate immediately before strategy.extract_text_blocks,
raising LLMResponseFormatError (typed, non-retryable response-shape error)
with the provider-specific diagnostic when the gate returns False.
- _DummyProvider + FakeNoCacheProvider satisfy the new abstract by
returning True (no finish-reason concept on the canned shapes).
TDD: 4 happy-path tests added (one per concrete provider + 2 for Anthropic
covering both clean stop reasons) and confirmed failing before implementation,
then green. Canonical validation quad (ruff/format/pyright/pytest) all-green;
2560 tests pass (no regressions); AST scans 3/9/10 pass.
* bd_1-scaffolding-eu0.6: #155 US-005 — new tests/cli/test_e2e_openai_smoke.py
Full-pipeline live e2e gated by @pytest.mark.e2e + @pytest.mark.openai
markers and a three-env-var skip gate (SF_RUN_OPENAI=1, OPENAI_API_KEY,
GOOGLE_CLOUD_PROJECT — BigQuery stays the warehouse). Mirrors
tests/cli/test_e2e_bigquery_smoke.py verbatim and only swaps the grader
via apply_provider_override(grade_provider='openai', grade_model='gpt-4o')
per DEC-011/DEC-012 of plans/super/155-gemini-truncation-e2e-gap.md;
drafter stays Anthropic Sonnet per the fixture's llm.model pin and the
cost-table rationale (DEC-009).
Pins the seven invariants from the BQ smoke (DEC-009 of #10): exit 0,
sidecar present, kept+flagged+dropped>=1, always-passes drop present
(warehouse-side, provider-independent), flagged_count>=1 (tight grade
thresholds), GradingReport.aggregate_complete=True (the cross-provider
contract the in-isolation grade smokes can't pin), and no traceback in
stderr (cli-layer.md DEC-016).
Test is deselected by default addopts; maintainer runs once pre-release.
* bd_1-scaffolding-eu0.7: #155 US-006 — new tests/cli/test_e2e_gemini_smoke.py with max_output_tokens=2048 overlay
* bd_1-scaffolding-eu0.3: #155 US-002 — per-provider unclean-path tests + call_llm integration (rule edit deferred to orchestrator)
Per-provider unclean-path tests pinning the LLMProvider.is_clean_completion
contract (#155 DEC-001/DEC-002/DEC-005/DEC-006/DEC-007):
- tests/llm/test_anthropic_provider_via_fake.py: + max_tokens (with partial
text), tool_use, unclean_finish_reason_message naming stop_reason.
- tests/llm/test_openai_provider_via_fake.py: + length (with partial text),
content_filter, tool_calls, unclean_finish_reason_message naming
finish_reason.
- tests/llm/test_gemini_provider_via_fake.py: + MAX_TOKENS-with-partial-text
(the LOAD-BEARING #155 Finding 1 regression pin, with full context comment),
SAFETY/RECITATION/OTHER parametrized, unclean_finish_reason_message naming
finish_reason.
- tests/llm/test_client.py: + call_llm integration test asserting
LLMResponseFormatError raises at the is_clean_completion gate
(post-messages.create, pre-extract_text_blocks, no retry).
Existing contract pin tests/grade/test_gemini_neutrality.py:381 continues
to pass unmodified — the safety-blocked Gemini path now routes through the
same orchestrator gate as MAX_TOKENS, both landing at
'call failed: GradeLLMError' degrade.
Note: .claude/rules/llm-drafter.md edit deferred — per memory
ralph-worker-claude-dir-perms, .claude/ writes are orchestrator-only and
will land in a separate commit after this merge.
* bd_1-scaffolding-eu0.8: #155 US-007 — parametrize tests/cli/test_e2e_bigquery_smoke.py over grade.provider
Adds @pytest.mark.parametrize over grade_provider ∈ [anthropic, openai,
gemini] to the BigQuery e2e smoke test. Per #155 DEC-003 this covers
the cross-provider diff-sidecar rendering contract the in-isolation
grade smokes (tests/grade/test_*_grade_live.py) cannot pin: those
smokes never exercise the diff-sidecar evidence/reasoning cascade with
a non-Anthropic judge.
Per #155 DEC-011 the drafter stays Anthropic Sonnet across all three
variants (fixture stability — the LLM payload the drafter sees is
unchanged, so the always-passes column the LLM proposes is
reproducibly the same). Only the grader varies, via the canonical
apply_provider_override helper (US-004 / DEC-012).
Per-variant env-var gates layered on top of the existing baseline:
- anthropic: baseline only (SF_RUN_BQ + ANTHROPIC_API_KEY + GOOGLE_CLOUD_PROJECT)
- openai: baseline + SF_RUN_OPENAI + OPENAI_API_KEY
- gemini: baseline + SF_RUN_GEMINI + GOOGLE_API_KEY
Gemini variant pins max_output_tokens=2048 per #155 Finding 2 — Gemini
2.5-flash's verbose reasoning field truncates mid-string at the
default 512/1024 floors and would flake assertion #6 (aggregate_complete).
Sibling files (test_e2e_openai_smoke.py / test_e2e_gemini_smoke.py)
remain per DEC-011 for per-provider failure ergonomics and cost
transparency; this parametrize is internal to the BQ smoke.
Pytest IDs: test_*[anthropic] / test_*[openai] / test_*[gemini]. All
three are deselected by default addopts (gated by the e2e marker);
the maintainer runs them per pre-release live suite (US-008).
Validation: uv sync --dev && ruff check && ruff format --check &&
pyright && pytest — all green (2566 passed, 69 deselected, coverage
97.40%).
* bd_1-scaffolding-eu0.3: #155 US-002 (orchestrator portion) — clarify llm-drafter.md DEC-005 for the is_clean_completion seam
Updates the Gemini-section DEC-005 contract to reflect the post-#155 generalisation:
- Rule now applies to ALL providers (Anthropic stop_reason, OpenAI choices[0].finish_reason,
Gemini candidates[0].finish_reason.name) — not just Gemini
- Enforcement moved from "no text at all" check buried in extract_text_blocks to the new
LLMProvider.is_clean_completion(response) -> bool ABC method (#155 DEC-005)
- Closes the Finding-1 gap: MAX_TOKENS with partial text now surfaces as
LLMResponseFormatError → GradeLLMError, not GradeOutputError(json_parse)
- Per-provider _CLEAN_STOP_REASONS enumerated; tool_use deliberately UNCLEAN per DEC-006
- unclean_finish_reason_message override (DEC-007) keeps vendor-native field names visible
Test-side pins (worker portion of US-002, commit 9a23e3f):
- tests/llm/test_{anthropic,openai,gemini}_provider_via_fake.py — unclean-path coverage
- tests/llm/test_client.py — call_llm gate integration
Worker (commit 9a23e3f) deferred this rule edit per memory ralph-worker-claude-dir-perms;
this commit closes the bead's orchestrator-only deliverable.
* bd_1-scaffolding-eu0.9: #155 US-008 — CONTRIBUTING.md live-suite pre-release cadence + env-var block
Add a 'Live e2e suite (pre-release only)' subsection to CONTRIBUTING.md
per DEC-010 of plans/super/155-gemini-truncation-e2e-gap.md.
Documents the full pre-release maintainer audit:
- 5 paid e2e tests (BQ smoke parametrized over 3 grader providers per
US-007, OpenAI sibling, Gemini sibling with max_output_tokens=2048
floor per DEC-008, Snowflake sibling)
- 6 grade-only / draft-only live-API smokes gated by anthropic /
openai / gemini markers
- One-shot invocation with -m 'e2e or anthropic or openai or gemini'
--no-cov and the full env-var stack (SF_RUN_*, ANTHROPIC_API_KEY,
OPENAI_API_KEY, GOOGLE_API_KEY, GOOGLE_CLOUD_PROJECT, SNOWFLAKE_*)
Frames the cadence as pre-release only — NOT per-PR, NOT CI-gated. The
addopts exclusion in pyproject.toml already keeps these out of default
runs; this section just documents the maintainer-side invocation when
cutting a release. Cost ceiling: ~$0.30/run × ~2–3 audits/month =
~$0.60–1.00/month.
* bd_1-scaffolding-eu0.10: #155 US-009 — Quality Gate fixes (4-pass code review)
Pass 1 (correctness): 0 real bugs. 2 doc-drift fixes:
- docs/grade-ops.md floor table: pre-fix GradeOutputError narrative
rewritten to post-fix LLMResponseFormatError → GradeLLMError per #155 DEC-005
- tests/cli/test_e2e_gemini_smoke.py invariant-#6 comment: same update
Pass 2 (simplification): 0 refactors needed.
Pass 3 (test coverage): 0 critical gaps. 3 nice-to-have safety nets added:
- tests/llm/test_gemini_provider_via_fake.py — new orchestrator wire-in pin
for MAX_TOKENS+partial-text via call_llm (distinct from existing
SAFETY-only call_llm test)
- tests/llm/test_openai_provider_via_fake.py — new orchestrator wire-in pin
for length+partial-text via call_llm (no prior call_llm coverage)
- tests/grade/test_gemini_neutrality.py:381 — docstring comment binding the
existing safety-blocked pin to the new #155 MAX_TOKENS routing path
Pass 4 (rules compliance): 1 real rule violation + 1 multi-surface drift
(same root cause). Fixed:
- tests/cli/test_e2e_openai_smoke.py::_skip_reason() expanded from 3 to 5
env-va…
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* #141: SignalForge skill + install-skill (plan) Phase 4 detailing complete. 24 decisions locked across: - skill source path (`src/signalforge/skills/signalforge/` package-data tree; `src/signalforge/skill/` Python lib) - destination policy (always overwrite SKILL.md, preserve siblings, no --force) - symlink/cycle defence (mirrors copy_demo verbatim) - error hierarchy (SkillError base + 3 concretes; spans tiers 1+2 → excluded base) - wheel packaging + wheel_smoke gate - AST scan #7 bump (12→13) - SKILL ↔ CLI parity gate (new test scans live argparse + key demo commands) - 5-surface parity for install-skill - self-grade ops (pre-release manual; pinned in eval.json + README badge) - e2e demo paths (zero-cred default + opt-in live) - skill-parity.md rule + cli-layer.md update (orchestrator-only) 11 stories laid out: US-001…US-009 implementation + US-010 Quality Gate + US-011 Patterns & Memory. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #141: mark plan phase=published, link PR #166 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(#141): add skill-parity rule documenting CLI/skill parity gate Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #141: devolved to beads (epic bd_1-scaffolding-ezn, 11 tasks) Approved + devolved. US-001 is at the front of the ready queue. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-ezn.1: US-001 bootstrap skills tree + wheel packaging Lays down src/signalforge/skills/signalforge/{SKILL.md,assets/SKILL.eval.json} as structural placeholders. Wires [tool.hatch.build.targets.wheel].include for the skills tree. Extends wheel_smoke with _EXPECTED_SKILL_FILES (positive) and a negative assertion that no .claude/skills/* paths appear in the wheel. Plan: plans/super/141-claude-skill-install.md US-001 / DEC-001, 010, 011, 022. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-ezn.2: US-002 signalforge.skill lib + typed errors + AST scan #7 bump Adds signalforge.skill subpackage with install_skill(dest) -> Path and the three-class typed-error hierarchy (SkillDestPathError tier 1, SkillDestUnsafeError tier 2, SkillPackageDataMissingError tier 1). Mirrors signalforge.demo.copy_demo verbatim for symlink-cycle defence and importlib.resources lookup; never rmtree. Registers the three lib concretes in _EXCEPTION_TO_EXIT_CODE; adds SkillError to _EXCEPTION_MAPPING_EXCLUDED_BASES; bumps AST scan #7 count 12 -> 13. Plan: plans/super/141-claude-skill-install.md US-002 / DEC-002, 003, 005, 006, 007, 008, 009. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-ezn.3: US-003 CLI install-skill subcommand + handler + exit-code mapping + subprocess smoke Adds 'signalforge install-skill [<dest>]' wired via add_parser/cmd_install_skill mirroring init_demo shape. Three CliInstallSkill*Error wrappers (Path/DestUnsafe/ PackageDataMissing) registered in _EXCEPTION_TO_EXIT_CODE (tier 1/2/1). Stdout success line + (replaced existing SKILL.md) on overwrite per DEC-017. Subprocess --help smoke under cli_subprocess marker. Per-class construction branches added to test_exit_codes.py for the three new wrappers. Plan: plans/super/141-claude-skill-install.md US-003 / DEC-002, 003, 004, 008, 009, 017, 024. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-ezn.7: US-007 author SKILL.md prose (frontmatter + 7 body sections) Replaces the US-001 placeholder with the real user-facing workflow per DEC-020 (frontmatter) and DEC-021 (seven sections). Body teaches the full pipeline: point-at-project, zero-cred demo, real draft+prune, prune-existing, diff-reading, gated live e2e, troubleshooting. Contains every canonical token the US-004 parity gate scans for (subcommand names + demo command lines + install-skill bootstrap). Plan: plans/super/141-claude-skill-install.md US-007 / DEC-012, 013, 020, 021. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-ezn.4: US-004 SKILL ↔ CLI parity gate NEW tests/cli/test_skill_cli_parity.py scans src/signalforge/skills/signalforge/ SKILL.md for three token categories per DEC-015: 1. Every subcommand name from the live argparse parser (auto-grows) 2. Four canonical demo command lines (signalforge init-demo / generate <model> --write / prune-existing <model> --schema <path> / install-skill) 3. The install-skill bootstrap line (covered by category 2's fourth entry) Plain substring match; no normalisation. Planted-violation self-check proves the gate can fail loud — per testing-signal.md AST-source-scan-gate philosophy. The gate runs inside the canonical VALIDATE_CMD (uv run pytest) so /ralph-run keeps the skill current automatically without relying on the model remembering. Plan: plans/super/141-claude-skill-install.md US-004 / DEC-015, 016, 019. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-ezn.9: US-009 skill-parity.md + cli-layer.md update Replaces the anticipatory skill-parity.md with the post-implementation contract: names the actual shipped artefacts (src/signalforge/skill/, src/signalforge/skills/signalforge/SKILL.md, signalforge install-skill, tests/cli/test_skill_cli_parity.py), documents the two-name convention (skills/ plural for package-data vs skill/ singular for the Python lib), the planted-violation self-check, and the wheel exclusion defence for maintainer-only .claude/skills/. cli-layer.md § Multi-surface parity gains a paragraph naming the bundled skill as the 6th parity surface, cross-linking to skill-parity.md and the test_skill_cli_parity.py gate. ORCHESTRATOR-ONLY commit per ralph-worker-claude-dir-perms memory — workers cannot Write under .claude/ in worktrees. Plan: plans/super/141-claude-skill-install.md US-009 / DEC-018, 019. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-ezn.6: US-006 docs (skills.md + mkdocs nav + cli-ops + README pointer) NEW docs/skills.md catalog page describing the bundled SignalForge skill, the install-skill subcommand, the two demo paths (zero-cred default + opt-in live e2e), the parity gate, and the maintainer-only-skill exclusion. mkdocs.yml nav gains 'Claude Code Skill' entry. docs/cli-ops.md gains the install-skill subcommand entry with stderr shapes + exit codes. README Quick start gains a one-sentence pointer after pip install. clauditor badge intentionally NOT added here — US-008 owns that surface. Plan: plans/super/141-claude-skill-install.md US-006 / DEC-021, 023. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-ezn.5: US-005 5-surface parity test for install-skill Mirrors tests/cli/test_5_surface_parity_init_demo.py for the install-skill subcommand. v0.1 canonical token: 'install-skill' (no flags). Pins the token across argparse help, handler docstring, docs/cli-ops.md, plan document, and the test docstring itself. Orthogonal to test_skill_cli_parity.py (US-004): that scans the FULL CLI surface against ONE skill body; this pins ONE subcommand across FIVE surfaces. Plan: plans/super/141-claude-skill-install.md US-005 / DEC-024. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-ezn.8: US-008 clauditor self-grade + README badge Adds clauditor-eval to [dependency-groups].dev (PyPI dist name; provides the 'clauditor' CLI entry point — the upstream LLM-as-judge framework SignalForge's own grading layer shares its methodology with). Grade pending — clauditor-eval installs cleanly and 'uv run clauditor grade' runs, but a meaningful score requires a maintainer-crafted EvalSpec (SignalForge-specific assertions + grading criteria); the auto-scaffolded 'clauditor init' template is generic boilerplate that would grade noise-against-noise. Refined assets/SKILL.eval.json to a 'pending-first-grade' shape that pins the current signalforge.__version__, names the grader and regen command, and explains why the maintainer must hand-tune an EvalSpec before the first real grade. README shields.io badge surfaces the pending state ('clauditor: pending', lightgrey). New 'Self-grade' section in docs/skills.md documents the regen flow for the maintainer's pre-release workflow per DEC-014. VALIDATE_CMD green: 2745 passed, all four checks. wheel_smoke + cli_subprocess gated markers also green. Plan: plans/super/141-claude-skill-install.md US-008 / DEC-014. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-ezn.10: Quality Gate — fix bugs from code review Four code-review passes surfaced 8 real findings; this commit addresses them. CORRECTNESS - src/signalforge/__init__.py: revert __version__ from 0.4.0.dev0 back to 0.5.0.dev0 — the dev branch is the 0.5.0 development line; the previous value was a stale-rebase artifact (Review 1). - src/signalforge/skill/__init__.py: extend symlink defence (DEC-005) to cover every install-tree ancestor (.claude/, .claude/skills/, .claude/skills/signalforge/) — not just SKILL.md itself. A symlinked ancestor dir would otherwise smuggle writes through copytree (Review 1). - tests/skill/test_install.py: new test_install_skill_refuses_when_install_dir_ancestor_is_symlink pins the ancestor-symlink defence with a concrete attacker-elsewhere repro. SKILL.md PROSE (drift between skill and live CLI surface) - Remove the "--force" example (DEC-003 explicitly forbids the flag; re-running install-skill overwrites SKILL.md by default and preserves siblings). - Frontmatter signalforge-version: "0.X.Y" → "0.5.0.dev0" (matches __version__; was a literal placeholder shipped to operators). - Section 2: "no warehouse, no API keys, no dbt profile" was misleading — the drafter always calls Anthropic. Reword to "no dbt profile / no warehouse credentials of your own" + name the ANTHROPIC_API_KEY requirement. - LLMCacheTooLargeError mis-labelled tier-3 → corrected to tier-2 (the cache-too-large check is a pre-LLM-call input-validation gate per cli-layer.md's four-tier taxonomy). docs/skills.md DRIFT - Canonical-commands list said "signalforge --version" — DEC-015's hardcoded list is init-demo / generate <model> --write / prune-existing <model> --schema <path> / install-skill (no --version variant; the parity gate scans for `version` subcommand separately via category 1). - "signalforge --version is the first thing it runs" → "signalforge version (the subcommand)" — flag vs subcommand mismatch. PARITY GATE EXTENSION (the gate would have caught the --force bug) - tests/cli/test_skill_cli_parity.py: add fourth category that scans SKILL.md for `signalforge <subcommand> --<flag>` patterns and asserts each flag exists on the live subparser. Pinned with the same planted-violation philosophy (verified manually: reinstating --force fails the gate loud). Closes the "skill prose teaches a flag that doesn't exist" failure mode the original gate could not catch. Validation: - uv run ruff check . — clean - uv run ruff format --check . — clean - uv run pyright — 0 errors - uv run pytest — 2747 passed (was 2745 baseline; +2 new tests) - uv run pytest -m wheel_smoke --no-cov — 5/5 - uv run pytest -m cli_subprocess --no-cov — 8/8 Plan: plans/super/141-claude-skill-install.md US-010. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #141: plan phase=implemented; record Ralph run completion + QG findings Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #141: close codecov patch-coverage gaps (5 lines, 3 files → 100%) codecov flagged 5 missed lines in the PR diff: - src/signalforge/cli/install_skill.py:177-178 — except OSError around the existed_before probe (silently downgrades on probe failure; the install should still proceed) - src/signalforge/skill/__init__.py:150 — resolve(strict=False) fallback for a dest that doesn't exist yet (the common fresh-project case) - src/signalforge/skill/__init__.py:157 — non-ELOOP OSError raw re-raise (narrow ELOOP-only routing is load-bearing — a PermissionError must NOT be mis-attributed as a symlink cycle) - src/signalforge/skill/errors.py:66 — SkillError.__str__'s no-footer branch (when neither remediation kwarg nor default_remediation is set) Per the qg-pass-3-defer-defensive-tests-fails-codecov memory, codecov holds patch coverage to project standard regardless of "is this a real bug today" — defensive branches need test coverage even when they're fallbacks. Adds 4 tests: - test_skill_error_str_omits_footer_when_remediation_is_none (errors.py:66) - test_install_skill_propagates_non_eloop_oserror_unchanged (skill:157) - test_install_skill_resolves_nonexistent_dest_via_strict_false_fallback (skill:150) - test_install_skill_handles_oserror_in_existed_before_probe (cli:177-178) Coverage: install_skill.py 95% → 100%, skill/__init__.py 95% → 100%, skill/errors.py 95% → 100%. Full pytest: 2751 passed (was 2747). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #141: address CodeRabbit + Copilot PR review (13 threads) Trivial: - assets/SKILL.eval.json: version 0.4.0.dev0 → 0.5.0.dev0 (matches the __version__ revert in the previous QG commit; flagged by both reviewers). - SKILL.md fenced ASCII pipeline diagram: add `text` language tag (MD040). - plans/super/141: drop the leftover duplicate "_Pending Phase 3/4._" placeholders at the end of the doc. Substantive — symlink defence broadened to EVERY bundled path: - src/signalforge/skill/__init__.py: the symlink defence used to cover only `<dest>/.claude/skills/signalforge/SKILL.md`. A symlinked `assets/SKILL.eval.json` (or symlinked `assets/` directory) would smuggle writes through copytree. Now enumerates every relative path under the bundled source tree via `rglob` and refuses to overwrite any of them through a symlink. Pinned by `test_install_skill_refuses_when_assets_dir_is_symlink`. - Also wrap `mkdir(parents=True)` in try/except NotADirectoryError → raise SkillDestUnsafeError so a non-dir component along the install chain (e.g. `<dest>/.claude` is a regular file) yields a typed remediation-bearing message instead of a raw OSError. Pinned by `test_install_skill_wraps_notadirectoryerror_from_mkdir_chain`. Substantive — CLI existed_before probe: - src/signalforge/cli/install_skill.py: probe was `.exists()` which follows symlinks AND returns False for broken symlinks. The DEC-017 contract says "True for files and symlinks (both shapes are replaced from the operator's POV)" — a broken symlink is a third shape that the probe silently downgrades. Probe now OR's `.is_symlink()` to catch the broken-symlink case (semantics stay honest even though the lib seam then raises SkillDestUnsafeError on the same path). Substantive — SKILL.md ADC contradiction: - The frontmatter `compatibility:` field and Section 2 body claimed "zero-credential demo (no warehouse needed)" while simultaneously describing the demo as sampling a public BigQuery dataset under ADC (which requires `gcloud auth application-default login` + GOOGLE_CLOUD_PROJECT). Both surfaces rewritten to honestly describe the demo's real posture: removes the dbt-project setup cost, but needs ANTHROPIC_API_KEY + ADC + GOOGLE_CLOUD_PROJECT. Section 2 also surfaces `signalforge lint --model <name>` as the truly-offline fallback (manifest-only, no LLM, no warehouse). Substantive — dbt parse invocation: - SKILL.md Section 1 suggested running `dbt parse` but the `allowed-tools` frontmatter does NOT include `Bash(dbt *)`. Reworded to "ask the user to run dbt parse themselves" rather than implying the skill runs it — preserves the narrow tool grant. Substantive — parity gate (`test_skill_cli_parity.py`): - Broaden `_SKILL_FLAG_USAGE_RE` to match `signalforge <subcommand> [<positional> ...] --<flag>` so canonical shapes like `generate <model> --write` and `prune-existing <model> --schema <path>` are no longer skipped (CodeRabbit + Copilot). Constrain to same-line `[ \t]` (not `\s`) so the match cannot span newlines — without this, prose like "signalforge installed (pip install ...)" plus "signalforge lint --model" two paragraphs later yields a spurious `installed --model` capture (caught during validation). - Unknown-subcommand branch now FAILS instead of skipping (Copilot). A typo like `signalforge instal-skill --force` would previously skip silently; now it surfaces in the assertion message. - Verified via planted-violation: `signalforge generate <model> --xyzbogus` injection trips the gate; restoring SKILL.md returns the gate to green. Validation: - uv run ruff check . — clean - uv run ruff format --check . — clean - uv run pyright — 0 errors - uv run pytest — 2753 passed (was 2751; +2 new tests) - uv run pytest -m wheel_smoke --no-cov — 5/5 - Patch coverage: install_skill.py 100%, skill/__init__.py 100%, skill/errors.py 100% — codecov-clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Combined post-release housekeeping per the "dev ahead of main" pattern: 1. Bump `__version__` 0.5.0.dev0 → 0.6.0.dev0 2. Bump skill metadata version strings (SKILL.md frontmatter + assets/ SKILL.eval.json) to match 3. Backmerge origin/main (v0.5.0) into dev so dev = main + future work 4. Clear dev's stale `[Unreleased]` CHANGELOG carryover from v0.4.0 by taking main's post-release CHANGELOG shape verbatim (empty placeholder + correct version-descending order) The CHANGELOG cleanup is the load-bearing fix that prevents the v0.4.0 → v0.5.0 release-PR conflict from repeating: dev's `[Unreleased]` had been carrying the v0.4.0 entries since the v0.4.0 release didn't clear them, which then duplicated into the proposed v0.5.0 release notes via the release skill's CHANGELOG promotion. Taking main's CHANGELOG resets dev to the correct empty-placeholder shape. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…erge chore: begin 0.6.0.dev0 + backmerge main into dev (post-0.5.0)
Five edits, each capturing one mistake from cutting v0.5.0: 1. **Pre-flight branch check** — recognise the dev-ahead-of-main case (common when main is at v(N-1) and dev carries v(N) work). Skill previously required `main`; now detects the deviation and points to Step 3's two-shape guidance. 2. **Pre-flight item 5 — stale-`[Unreleased]` carryover guard** — diff `[Unreleased]` against the previous `[X.(Y-1).0]` section; identical → STOP. This is the load-bearing check that would have caught v0.5.0's "release notes byte-identical to v0.4.0" bug at pre-flight time instead of at PR-merge-conflict time. Linked to Step 9's cleanup as the long-term fix. 3. **Step 1 — skill metadata lockstep bump** — `signalforge-version` stamps in `src/signalforge/skills/signalforge/SKILL.md` frontmatter and `assets/SKILL.eval.json` must match `__version__` (and bump in lockstep at every version edit). v0.5.0 shipped without these bumps on the first attempt; the patched skill grep+edits both. 4. **Step 3 — dev-ahead-of-main branching shape** — two variants documented inline. Normal case = branch from main. Dev-ahead case = branch from main + cherry-pick the post-v(N-1) commits (often a single squash-merge). Verification recipe: `git diff release/X.Y.Z origin/dev --stat` should show only the release-prep edits. 5. **Steps 8 + 9 combined** — one PR per release-cycle housekeeping pass (next-dev bump + skill metadata + backmerge + CHANGELOG cleanup). Crucially: `git checkout --theirs CHANGELOG.md` during the merge resets dev to main's clean post-release shape, which is what prevents pre-flight item 5 from firing next cycle. The old separate-PR shape (Step 8 = bump, Step 9 = backmerge) loses signal when dev is ahead. Also bumps the skill's own `signalforge-version` metadata 0.1.0 → 0.5.0 (was stale from when the skill was authored). Memory cross-link: [[release-clear-unreleased-on-backmerge]] (the load-bearing CHANGELOG cleanup, now codified in Step 9 itself) and [[release-full-from-dev-ahead-of-main]] (the parent shape). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ll-from-v0.5.0-lessons chore: patch release-manager skill with v0.5.0 lessons learned
Surfaces the v0.5.0 install-skill feature where prospective users decide
whether SignalForge is for them. Three edits:
1. New "Drives end-to-end from Claude Code" bullet in § What it does
(8th entry in the feature list), linking to docs/skills.md as the
deep dive.
2. New top-level § Claude Code section between § How it works and §
Supported warehouses — gives the install command, three example
prompts ("draft tests for dim_customers", "prune my schema.yml",
"run the demo"), an explanation of what Claude actually does with
the skill (picks subcommand + flags, runs the pipeline, explains
the diff), and a pointer to docs/skills.md for the full reference
(install path, 7-step workflow, demo flows, parity gate, clauditor
self-grade).
3. Tightens the pre-existing Quick-start install hint at Step 1 to
cross-link to the new § Claude Code section rather than be the
first introduction.
Also adds an [Unreleased] § Docs CHANGELOG bullet recording the
overview promotion (the v0.5.0 entry already documents the feature
itself).
docs/skills.md is the deep dive — no change required there, it
already covers everything the new README section points to. mkdocs.yml
nav already has 'Claude Code Skill: skills.md' from v0.5.0.
`uv run --only-group docs mkdocs build` passes; all warnings are
pre-existing (per docs-publishing.md, repo-internal plans/super/ links
are deliberately not in the built site).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…-overview docs: promote Claude Code skill into the README overview
* #169: super-plan — row_count_between 6th test primitive Phase 1 discovery + Phase 2 architecture review + Phase 3 refinement (15 DECs) + Phase 4 detailing (12 implementation stories + Quality Gate + Patterns & Memory). Refs #169. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #169: devolved to beads (epic bd_1-scaffolding-tt8 + 14 tasks) Phase 6 → Phase 7. Plan-doc Beads Manifest filled in. Refs #169. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-tt8.1: #169 US-001 — CandidateTestRowCountBetween model + drift detector Land the 6th first-class CandidateTest variant per #169 DEC-001 / DEC-008 / DEC-013: * New `CandidateTestRowCountBetween` model in `signalforge.draft.models`: - `type: Literal["row_count_between"]` - `column: None = None` (model-level only — matches CandidateTestCustomSQL's posture as the other model-level-capable variant, but hard-coded to None since a row-count is always a model-scoped aggregate) - `minimum: int | None` / `maximum: int | None` with non-negative field validators (DEC-008 — prefix-free naming matching `values` / `to` / `field` precedent; ingest + diff emitter handle the dbt-expectations `min_value` / `max_value` mapping) - `where: str | None` with non-empty-after-strip field validator - `@model_validator(mode="after")` enforces at-least-one-of-(min, max) and `minimum <= maximum` - `frozen=True, extra="ignore"` per the _BASE_CONFIG read-back posture * Extend the `CandidateTest` discriminated union with the new variant (`Field(discriminator="type")` preserved); extend `__all__`. * Drift detector mirror `StrictCandidateTestRowCountBetween(extra="forbid")` in `tests/draft/test_drift_detector.py` + extend the strict union; existing field-set parity test covers the new variant automatically via the discriminated-union walk. * Add one model-level fixture row to `tests/fixtures/draft/candidate_schema_v1.json` exercising all four fields (`minimum`, `maximum`, `where`, `rationale`). No v1→v2 rename — schema-version is forward-compat per DEC-013. * 17 new validator tests in `tests/draft/test_models.py` covering every Pydantic invariant: both bounds none, negative bounds, min>max, empty/ whitespace-only where, frozen-mutation rejection, round-trip byte-stability, discriminated-union resolution, `column != None` rejection, `extra="ignore"` forward-compat. Validation: `uv run pytest tests/draft/test_models.py tests/draft/test_drift_detector.py` green (42 tests). Full suite green (2770 tests, 97.82% coverage). ruff + ruff-format + pyright all green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-tt8.2: VALID_TEST_TYPES token + exclude_tests round-trip US-002 of #169. Add "row_count_between" to VALID_TEST_TYPES (six variants now) and refresh the constant + exclude_tests field docstrings to reflect the new sixth member. Tests: - tests/draft/test_config.py — two new round-trip checks: accept exclude_tests=("row_count_between",) and reject the typo ("row_count_betwen",) with the "not a valid test type" error that lists every VALID_TEST_TYPES member (including the new token). - tests/draft/test_exclude_tests.py — bump the pinned VALID_TEST_TYPES set to include "row_count_between" so the canonical-set sentinel test stays green. Traces to: DEC-001 of #169. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-tt8.6: #169 US-007 — prune compiler arm + BigQuery & Snowflake snapshots Implements ``_compile_row_count_between`` in ``signalforge.prune.compiler`` plus the ``_compile_test`` dispatcher arm for the 6th candidate-test variant. Always emits ``SELECT COUNT(*) FROM <quoted-table> [WHERE <where>]`` regardless of ``prune.scope`` (DEC-003) — a sampled COUNT(*) cannot be compared against full-table bounds, so the variant deliberately bypasses ``scope`` / ``sample_size`` / ``sample_bucket`` / ``partition_filter``. Under materialised-sample the orchestrator passes ``table_ref=<temp table>`` so the count lands cheap. Composes the full statement then validates via existing ``validate_test_sql`` (DEC-005); a hostile ``where`` (containing ``;`` / ``--`` / unbalanced parens) routes via ``_InvalidIdentifier`` to ``kept-without-evidence`` (DEC-011). No new ``validate_where_fragment`` helper — reuses the ``custom_sql`` validation surface verbatim. Identifier quoting + case folding read entirely from existing ``Dialect`` fields (``quote_char``, ``identifier_case``, ``quote_qualified_per_component``); no new ``Dialect`` fields, no ``if dialect.name ==`` branches, no new vendor SDK imports under ``signalforge/prune/`` — the import-guard test remains green. Snapshot fixtures cover BigQuery (no-where / with-where / only-min / only-max) and Snowflake (no-where / with-where, per-component double- quoted, UPPER-folded). 14 new tests pin the byte-exact output across dialects, the conservative-bias routing on hostile ``where``, the scope-sample bypass invariant, the partition_filter bypass invariant, and the #116-shaped materialised-sample correctness (compiled SQL references the ``_SESSION._sf_sample_*`` temp table, never the source). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-tt8.3: US-004 parser anchor-contract arm + where-clause type-coherence Extend `_validate_anchor_contract` in `signalforge.draft.parser` with a model-level arm for `CandidateTestRowCountBetween` (the #169 6th variant): - Special-case `row_count_between` AHEAD of the generic `test.column not in model_columns` branch — `column` is always `None` by Pydantic invariant, so the generic check would mishandle it (same precedent as the `custom_sql` arm). - Optional `where` clause validated via a new sibling helper `_check_row_count_between_where`, modelled on `_check_custom_sql_type_coherence` (#159, DEC-003) and reusing the same sqlglot machinery. The helper composes `SELECT 1 FROM __sf_where_placeholder__ WHERE <where>` so sqlglot can parse the freestanding clause, then walks comparison nodes for two violation classes: - Unknown column reference (bare `exp.Column` operand whose name is absent from `model_columns`) — the row_count_between equivalent of the structural `test references nonexistent column` check. - Type incompatibility (bidirectional `COERCES_TO` check on `Column <op> Column` pairs with known types). - Skip-when-uncertain posture preserved per DEC-006: comparisons inside an `exp.Subquery` skip entirely (a `(SELECT ...) > 0` shape does not flag inner column refs); ParseError / annotator failure / unparseable type strings / Cast / Coalesce / function calls / NULL all skip silently. The warehouse adapter remains the safety net for real-SQL breakage via `kept-without-evidence` routing. - `exclude_tests=("row_count_between",)` rejects a drafted row_count_between via the existing model-level dual-defence backstop (prompt-builder filter is the primary defence; this is the parser-side rejection for an LLM that ignores the prompt). - Collect-all preserved: a candidate with both an unknown-column `where` AND a hallucinated CandidateColumn produces BOTH violations in one error. sqlglot imports stay confined to `signalforge.draft.parser` (DEC-008 of #159); no new vendor SDK imports. Traces to: DEC-004, DEC-005, DEC-006, DEC-013 of #169. Tests added (`tests/draft/test_parser.py`): - Valid row_count_between with both bounds + no where → no violations. - `where: "user_id > 100"` referencing a real column → no violations. - `where: "phantom_col > 1"` (unknown column) → violation. - `where: "(SELECT 1 FROM foo) > 0"` (subquery, skip-when-uncertain) → no violations. - `exclude_tests=("row_count_between",)` + drafted row_count_between → violation. - Collect-all preserved with concurrent CandidateColumn violation. Validation: `uv run pytest` 2776 passed, `uv run ruff check` clean, `uv run ruff format --check` clean, `uv run pyright` 0 errors. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-tt8.9: drafter prompt catalogue + _PROMPT_VERSION rotation for row_count_between (US-003) Adds the row_count_between catalogue line to _TEST_CATALOGUE_LINES, illustrating both the no-where (whole-table bound) and with-where (filtered bound) JSON shapes per DEC-012 of #169. Rotates _PROMPT_VERSION from c9e7ee1f6f465933 -> 77e9ee8a6ae7d875; _CACHED_BLOCK_GOLDEN (manifest summary) is unchanged. Extends tests/draft/test_prompts.py with: - system prompt advertises the new type - catalogue illustrates both where forms (two occurrences of the type literal) - default render includes row_count_between - exclude_tests=("row_count_between",) drops the catalogue line + SCOPE entry - two consecutive renders are byte-stable Updates tests/draft/test_exclude_tests.py existing scope-line tests to account for the 5th standard type when excluding everything below custom_sql. Updates tests/llm/test_prompt_cache_stability.py: - _EXPECTED_PROMPT_VERSION to the new hash - rotation-history bullet documenting the #169 rotation Traces to: DEC-012 of #169. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-tt8.5: #169 US-006 _common.artifact_id row_count_between arm Extend signalforge._common.artifact_id.model_test_args_hash with a CandidateTestRowCountBetween arm. Hash domain: type + column (always None for this variant, included for shape-parity) + minimum + maximum + where, serialised as canonical JSON. Identity parity across the three re-exporters (signalforge._common, signalforge.diff, signalforge.grade.engine) holds automatically — the shared seam means the arm propagates everywhere with no per-layer changes; test_cross_stage_parity_is_function_identity continues to pass on `is` equality. Three collision/distinctness tests: - Identical (minimum, maximum, where) -> identical hash - Differing minimum -> different hash - Differing where (None vs "x > 1") -> different hash Traces to: DEC-013 of #169. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-tt8.10: #169 US-008 prune engine routing matrix coverage Pin every routing path for `CandidateTestRowCountBetween` through the existing test-type-agnostic `_decide_from_test_result` matrix (DEC-011 of #169). NO engine code change — the matrix dispatches on compiler return shape, warehouse `failure_count`, and raised `WarehouseError`, agnostic to candidate variant. A regression that added a bespoke row_count_between arm would fail loud here. Tests added (8 total, all in tests/prune/test_engine.py): * always-pass on warehouse → `dropped` / `always-passes` * failing rows untrusted → `kept` / `kept` * failing rows trusted model → `dropped` / `failed-on-known-clean-data` * hostile `where` (`;` rejected by compose-then-validate) → compiler `_InvalidIdentifier` → `kept` / `kept-without-evidence` with locked "row_count_between rejected by SQL safety check" `why`. No warehouse call dispatched (fake has zero queued expectations). * `TableNotFoundError` from warehouse → `kept` / `kept-without-evidence` * total-budget exhausted mid-run → remaining tests drain to `kept` / `kept-without-evidence` with locked "Total prune budget" `why` text. Stubs `_now_monotonic_ms` per the established budget-test pattern. * Empty-table carve-out (DEC-010 of #169): a violating bound routes to `kept` / `kept` because that IS what the test exists to catch (real signal). Documented engine posture, NO special-case. * `DropReason` literal still exactly 5 values — closed-set lockdown pin via `typing.get_args`. Cross-checked against the existing drift detector fixture (prune_event_v1.jsonl covers all 5) so the two pins catch regressions independently. Validation: `uv run pytest tests/prune/test_engine.py tests/prune/test_drift_detector.py` → 76 passed. Full suite: 2800 passed. `ruff check` / `ruff format --check` / `pyright` all green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-tt8.4: ingest parser recognises expect_table_row_count_to_be_between (#169 US-005) Promote dbt_expectations.expect_table_row_count_to_be_between from the generic custom-or-generic skip to a first-class supported variant via a new dispatch arm in signalforge.ingest.parser._parse_named_test + helper _parse_row_count_between. Inbound name translation per DEC-008: min_value → minimum, max_value → maximum, where → where. Skip routes (DEC-007), all reusing the existing 'malformed-supported-test' literal (DEC-011 — SkipReason stays the closed 3-value taxonomy): column-scoped usage (variant is model-level only); both bounds missing; non-int or negative bound; bool bound (silent isinstance(True, int) trap); min > max; non-string or whitespace-only where; non-dict body. Sibling dbt-expectations macros (e.g. expect_table_row_count_to_equal) keep falling through to the existing namespaced custom-or-generic-test skip — no behaviour change. The where arg is NOT routed through _extract_args: that helper strips 'where' as a generic dbt test-config key, which is correct for the four standard variants but wrong here (where is a first-class arg of this macro). The new helper reads body directly, honouring the dbt 1.8+ arguments:-nested shape. Files: - src/signalforge/ingest/parser.py: new _ROW_COUNT_BETWEEN_NAME constant + _parse_row_count_between helper + dispatch arm. - tests/ingest/test_parser.py: 17 new tests covering happy paths (inline bounds, arguments-nested, only-min, only-max, with where), every skip route, the sibling-macro fall-through, and the closed 3-value SkipReason invariant. Plus a fixture round-trip test driving every entry in the new schema.yml. - tests/fixtures/ingest/row_count_between_schema.yml: kept + skipped fixture cases exercising every documented path. Traces to DEC-007, DEC-008, DEC-011 of plans/super/169-row-count-between.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-tt8.7: grade rubric calibration + grade _PROMPT_VERSION rotation for vacuous row_count_between bounds (US-009) Extends the existing no-redundant rubric criterion (NOT a 5th criterion — DEC-009 of #169 explicitly forbids that to avoid the +25% LLM cost of an extra per-artifact call) with calibration prose teaching the judge to score vacuous row_count_between bounds low: "...or trivially satisfiable? For tests carrying numeric bounds (e.g. `row_count_between`), is each bound a meaningful guardrail calibrated to the model's expected size, rather than a vacuous floor or ceiling (`minimum=0` with no `maximum`, or a `maximum` so high it cannot fire)?" Rotates the grade-side prompt_version_template a35012b627b8ba6a → 5a70561088930c97 and the _canonical_rubric_hash 280aa6db7fde2b24 → 22a0231690aca6ef. The no-redundant criterion_prompt_hash rotates f89695e3daf7d559 → 60690cb4ef9246ee; the other three criterion hashes are unchanged. The grader's 3-trigger degrade taxonomy (DEC-011) stays locked: LLMError retries / GradeOutputError / total_budget_seconds. Vacuous bounds route through the rubric criterion's low score → existing passed: bool threshold → ship as `flagged` (NOT `kept-uncertain`, which is reserved for prune couldn't-evaluate). NOT a 4th degrade trigger. Adds tests: - no-redundant calibration prose names row_count_between, minimum=0, vacuous, trivially satisfiable - the extension is additive (preserves "semantically identical" + "always-passing" wording) - DEFAULT_RUBRIC stays at exactly four criteria after DEC-009 - vacuous-bound (kept + score=0.2 + passed=False) routes to tier "flagged" via diff.engine._tier_for_kept — NOT "kept-uncertain" - healthy-bound (kept + score=0.9 + passed=True) routes to tier "kept" - 3-trigger degrade taxonomy still has "call failed: " and "grade budget exceeded" reasoning strings in grade.engine Updates rotation-history bullets in test_prompts.py, test_rubric.py, and the DEFAULT_RUBRIC docstring. Traces to: DEC-009, DEC-011, DEC-012 of #169. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-tt8.15: row_count_between failing-rows shape (US-007a) US-007 (#169) landed _compile_row_count_between emitting `SELECT COUNT(*) FROM <table> [WHERE <where>]`. The warehouse adapters wrap any compiler output as `SELECT COUNT(*) AS failures FROM (<sql>) AS t` (BigQueryAdapter line 942, SnowflakeAdapter line 852). Under the wrap, the inner returned 1 row (the count value); the outer COUNT(*) over a 1-row result was always 1, so `failures=1` regardless of bounds. The engine's `failures>0 → kept` matrix routed every real row_count_between to `kept` (or `failed-on-known-clean-data` if trusted) without ever checking the bounds. Rewrite _compile_row_count_between to emit the failing-rows CTE shape: SELECT n FROM (SELECT COUNT(*) AS n FROM <table> [WHERE <where>]) AS rc WHERE <bound-violation-predicate> The bound-violation predicate is one of: - only-minimum (maximum is None) → n < <minimum> - only-maximum (minimum is None) → n > <maximum> - both → n < <minimum> OR n > <maximum> When the bound holds, the inner WHERE filters out the count row → outer failures=0 → engine routes `always-passes`. When the bound is violated, the row passes the WHERE → outer failures=1 → engine routes `kept`. The shape now matches the failing-rows-SELECT contract the other 4 built-in tests follow. Changes: - src/signalforge/prune/compiler.py — rewrite _compile_row_count_between; compose-then-validate (DEC-005) preserved. - 6 fixture files regenerated (4 BigQuery + 2 Snowflake). - tests/prune/test_compiler.py — 12 of 14 existing snapshot tests pass automatically against new bytes; 2 tests with shape-invariant assertions (`WHERE not in actual`, `startswith("SELECT COUNT(*) FROM ")`) updated to the new shape. ONE new test (test_compile_row_count_between_adapter_wrapped_failing_rows_contract) pins the failing-rows contract via literal-string assertions on the inner CTE + outer WHERE bound-violation predicate for all three bound variants. - plans/super/169-row-count-between.md — DEC-003 prose updated to describe the CTE+WHERE shape; correction note explains the adapter-wrap interaction bug. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-tt8.8: diff emitter arm for row_count_between (#169 US-010) Extend ``signalforge.diff._emitter._render_test`` with the ``CandidateTestRowCountBetween`` arm. Emits the dbt-expectations YAML form per DEC-002 of #169: {dbt_expectations.expect_table_row_count_to_be_between: {min_value: <N>, max_value: <M>, where: "..."}} THIS function is the outbound mapping seam — the model carries ``minimum`` / ``maximum`` (DEC-008, prefix-free naming on the model); the dbt-expectations macro uses ``min_value`` / ``max_value``. Maps outbound here. ``None``-valued fields are omitted so the emitted YAML stays minimal. ``proposed_test_files`` stays ``custom_sql``-only (DEC-002). Tier classification is variant-agnostic — a kept ``row_count_between`` flows through the generic ``_tier_for_kept`` classifier as ``tier=kept``, so no engine-level changes are needed. Test pin guards the contract. TDD coverage: * No-where YAML shape (only ``min_value`` / ``max_value``) * With-where YAML shape (verifies the dict carries the ``where`` field) * Only-minimum omits ``max_value``; only-maximum omits ``min_value`` * Hostile ``where`` content (multi-line, embedded quotes, YAML metacharacters) round-trips via ``yaml.safe_dump`` / ``yaml.safe_load`` * Dropped decision filtered out (no model ``tests:`` key) * Variant does NOT appear in ``proposed_test_files`` (custom_sql-only) * Tier-classification pin: kept ``row_count_between`` lands in the kept-table via the generic classifier; proposed_yaml carries the dbt_expectations namespace; proposed_test_files stays empty Traces to DEC-002, DEC-008, DEC-013 of #169. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-tt8.11: #169 US-011 docs + CHANGELOG + SKILL.md (5-surface parity) Land the 5-surface-parity doc pass for `row_count_between` (the 6th first-class `CandidateTest` variant) plus the 6th-surface SKILL.md update per DEC-015 of #169: - docs/draft-ops.md — new § "Row-count tests (`row_count_between`)" covering the catalogue entry (no-where + with-where), when the drafter proposes it, the `exclude_tests` short-circuit, and the worked example (intuit_airflow `weekly_query_cost` analog from DEC-012). - docs/prune-ops.md — new § "Row-count cost model" covering the failing-rows CTE wrap, sample-mode bypass + materialised-sample substitution, partition-aligned `where`-scan cost guidance, `maximum_bytes_billed` + `total_budget_seconds` safety nets, and the empty-table → `kept` carve-out (DEC-010 — load-bearing: real signal, not a degenerate case). - docs/grade-ops.md — new § "Row-count calibration" naming the `no-redundant` criterion's extended language (DEC-009) and confirming the 3-trigger degrade taxonomy stays locked; vacuous bounds route through `flagged`, not `kept-uncertain` and not a 4th degrade slot. - docs/diff-ops.md — new § "Row-count YAML emission" with the `dbt_expectations.expect_table_row_count_to_be_between` block, `minimum`/`maximum` → `min_value`/`max_value` outbound mapping, null-field omission, no `.sql` fallback (DEC-002), and the operator's responsibility for `dbt-expectations` in `packages.yml`. - docs/ingest-ops.md — new § "Recognition of `expect_table_row_count_to_be_between`" closing AC-5: promotion to the typed variant, inbound mapping table, the closed SkipReason literal preserved at 3 values, and the narrow recognition scope (other `dbt_expectations.*` macros still skip-record). Updated the SkipReason table entry for `custom-or-generic-test` to point at the new section. - CHANGELOG.md — Unreleased Added bullet covering all 5 layers + the ingest promotion + the empty-table semantic + the suppression knob. - src/signalforge/skills/signalforge/SKILL.md — one-paragraph operator-facing description of the 6-variant catalogue and the `exclude_tests` knob. Validation: `uv run pytest tests/cli/test_skill_cli_parity.py` green (no new CLI subcommands/flags/demo-commands introduced); full suite 2842 passed; ruff check + ruff format --check + pyright clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-tt8.12: engineered-determinism live e2e for row_count_between (#169 US-012) Adds the gated full-pipeline smoke that pins the row_count_between variant's kept/dropped contract under a real BigQuery warehouse. * tests/cli/test_e2e_row_count_between.py — the @pytest.mark.e2e smoke. Drives signalforge prune-existing against the committed Austin-bikeshare fixture with a hand-crafted external schema.yml carrying two engineered row_count_between tests: - engineered-failure-kept (load-bearing AC-1): min_value: 1, where: "1 = 0". The compiler emits SELECT n FROM (SELECT COUNT(*) AS n FROM <table> WHERE 1 = 0) AS rc WHERE n < 1. Inner COUNT is mathematically 0 on any table; outer WHERE returns one row; adapter wrap yields failures=1; engine routes to decision='kept', reason='kept'. Independent of warehouse state or sample bytes — proves the variant produces signal end-to-end. - engineered-always-pass-and-drop (AC-2): min_value: 0 (no upper bound). Predicate n < 0 vacuously false (COUNT(*) >= 0 always); failures=0; routes to always-passes -> dropped. Mathematically deterministic without LLM cooperation. Gated by SF_RUN_BQ=1 + GOOGLE_CLOUD_PROJECT only (deliberately narrower than the three-env-var bigquery smoke — prune-existing makes no LLM call). The default pytest suite deselects via the e2e marker; missing env vars route through pytest.skip cleanly. * src/signalforge/ingest/anchor.py — adds the row_count_between arm to the model-level test loop. Mirrors the drafter-side exemption in signalforge.draft.parser._validate_anchor_contract: the Pydantic model fixes column=None, so None not in model_columns would otherwise fire a spurious 'references nonexistent column None' violation that blocks the variant through ingest. Without this, the e2e test (and any prune-existing run with a hand-authored expect_table_row_count_to_be_between) is dead on arrival. * src/signalforge/ingest/reader.py — extends _test_dedupe_key with a row_count_between arm. The previous (type, column) fallback collapsed two row_count_between entries on the same model (column=None always) to one, dropping the second silently. The new key includes (minimum, maximum, where) so distinct bound configs survive as separate candidates. Mirrors the existing accepted_values / relationships arm-extension pattern (DEC-008). * tests/ingest/test_anchor.py — pins the model-level row_count_between with column=None passes the anchor validator cleanly. * tests/ingest/test_reader.py — pins (a) distinct row_count_between configs survive dedupe; (b) byte-identical configs across tests:/ data_tests: still collapse. Traces to: AC-1, AC-2, AC-7 of #169. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-tt8.13: Quality gate — fix HIGH findings from QG review QG Pass 1 (Correctness) HIGH finding: row_count_between returned wrong verdict at default config (scope=sample + sample_strategy=materialised). The engine substituted compile_table_ref = materialised_ref (a temp table with sample_size rows, e.g. 100K), so the compiled COUNT(*) returned the sample size — not the model's true row count. Bounds checked against sample size were semantically meaningless. Fix: in signalforge.prune.engine, add a per-test arm that overrides compile_table_ref to source_table_ref when the test is a CandidateTestRowCountBetween. The COUNT(*) against the source is a single aggregate scan — cheap even on petabyte tables — so there is no cost argument for routing through the temp table. Other variants (not_null / unique / accepted_values / relationships / custom_sql) continue to read row-level data from the materialised sample correctly. The bypass is row_count_between-specific. Plan-doc DEC-003 updated to reflect the corrected behaviour. Pinned by test_prune_tests_row_count_between_under_materialised_references_source_not_temp_table. QG Pass 4 (Docs+UX) HIGH findings F5-F9: 5 doc-vs-code drifts in the #169 surface: - docs/draft-ops.md: VALID_TEST_TYPES "five" -> "six" + row_count_between - docs/prune-ops.md:50: CandidateTest variants "five" -> "six" - docs/prune-ops.md:674: dbt-expectations claim — promoted one macro in #169 - docs/ingest-ops.md: overview omitted expect_table_row_count_to_be_between - src/signalforge/draft/models.py: docstring described pre-US-007a shape Triangulated stale-base findings (Conventions + Docs both flagged it) — will be cleared by a follow-up git merge dev in the same QG turn. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-tt8.14: Patterns & Memory — rules + memory for #169 row_count_between Update .claude/rules/business-rule-tests.md to reflect the 2-instance precedent for variant extension (custom_sql #116 + row_count_between #169). Key changes: - Title + intro: "Custom business-rule tests" → "Custom business-rule tests + variant-extension pattern"; describes both variants. - "The variants" § lists both Pydantic models + their validators. - NEW § "The 6 production dispatch sites (was 5; #169 added the 6th ingest anchor exemption)" enumerates the audit list with the new signalforge.ingest.anchor.validate_anchor_contract entry from US-012's inline ingest bugfix; also calls out which sites stay automatic (string-discriminator, not isinstance). - Materialised-sample substitution § now covers TWO directions: - Direction 1 (#116): row-level test bypass-to-temp via FROM-clause rewrite. - Direction 2 (#169): metadata/aggregate test bypass-the-substitution via per_test_table_ref source override at the engine. Pinned by test_prune_tests_row_count_between_under_materialised_references _source_not_temp_table. Decision rule: ask first what the test queries (row-level vs metadata/aggregate); the compiler snapshot can't catch this. - NEW § "Lockstep _PROMPT_VERSION rotation when extending the catalogue (#169 DEC-012)" — two independent _PROMPT_VERSION constants (drafter + grade); both rotate when a variant adds both a catalogue entry AND a rubric criterion refinement. Cross-link to ralph-serialize-shared-registry-beads. - NEW § "where-clause sqlglot type-coherence — reuse, don't fork (#169 DEC-004, DEC-006)" — #169 reuses #159's _check_custom_sql_type_coherence verbatim on the composed SELECT. - Ingest § now describes both recognition paths (custom_sql singular files vs dbt_expectations.expect_table_row_count_to_be_between schema.yml) and the inbound/outbound naming mapping seam (minimum internally, min_value externally). - On-disk artifact § distinguishes per-variant emission form: custom_sql → proposed_test_files; row_count_between → YAML block. Decision rule: default to YAML, promote to proposed_test_files only on a real standalone-artefact need (the 6th fail-closed writer is expensive ceremony). - Testing § adds the engineered-determinism trick for row_count_between (where: false + minimum: 1) and the engine-routing pin pattern (assert source vs _SESSION._sf_sample_*). - Reference § cross-links plans/super/169-row-count-between.md. Also (outside the repo, under ~/.claude/projects/...): new memory signalforge-row-count-between-pattern.md + index entry in MEMORY.md capturing the 6 dispatch sites, the metadata-vs-row-level decision rule for materialised-sample bypass, and the lockstep _PROMPT_VERSION rotation across drafter + grade. No code changes. uv run pytest green (2846 passed, 75 deselected). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #169: Address PR #176 review feedback (14 threads) 14 of 14 review threads fixed (0 false positives). Categories: (1) Real new bug — CodeRabbit Thread w2h: when every candidate is row_count_between, the engine still ran materialise_sample / _resolve_sample_bucket pre-work before the per-test override fired. If those adapter calls raised, every test routed to kept-without-evidence even though they could have run directly against source. Added all_bypass_to_source short-circuit in prune.engine.prune_tests that skips the materialised/oneshot setup entirely when every candidate is row_count_between (the per-test override would route them all to source anyway). Updated the existing test_prune_tests_row_count_between_under_materialised_references_source_not_temp_table to pin the no-pre-work invariant directly. (2) Real cost-model correction — Copilot Thread u_g: docs claimed BigQuery COUNT(*) is metadata-cheap; it actually bills as a scan. Reworded prune-ops.md cost guidance. (3) Post-QG drift cleanup — Copilot Threads u_S / u_a / u_i / u_k / u_n / u_q / u_x / u_- / u_0 / u_4 and CodeRabbit Thread w2f. The QG correction (engine bypasses materialised substitution for row_count_between) wasn't followed through into: - compiler.py docstring (still said materialised → temp table) - prune-ops.md sample-mode section (same) - test_compiler.py docstring on the materialised-temp-table test (test still useful as compiler-contract pin, but engine never actually passes a temp ref for row_count_between) - plan-doc US-007 TDD bullet (same) - draft-ops.md / grade-ops.md vacuous-bound routing prose (claimed minimum=0 maximum=None ships as flagged after grading; actually dropped by prune as always-passes because failing-rows CTE's `WHERE n < 0` never matches) - test_rubric.py docstrings + two test fixtures (constructed PruneDecision(reason="kept", failures=0) which is unreachable — the real prune matrix routes failures=0 to always-passes/dropped; flipped fixtures to failures=1 to match the only reachable kept state) (4) Lint nit — CodeRabbit Thread w2g: MD040 markdownlint on plan-doc fenced code blocks. Added `text` language tag to the two untyped opening fences (story dep graph and devolve output). Validation: ruff / ruff format / pyright (0/0/0) clean; full suite 2846 passed, 75 deselected. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #169: Address PR #176 review feedback (wording fix) CodeRabbit Thread ANeH on docs/grade-ops.md:297 — minor wording: 'at-least-one-bound' reads as an accidental typo in user-facing prose. Replaced with 'at least one bound'. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add super-plan for #170 — unique_combination 7th test primitive Phase 1 (discovery) — 3 parallel subagents mapped the 6 dispatch sites from the #169 precedent, surfaced the genuinely-new deltas (sample-mode semantics, columns canonical form, grade-side cache-stability gap). Phase 2 (architecture) — 5 parallel reviews (security, performance, data model + API, testing strategy, observability + ops). Cross-review consensus auto-decided sample-mode routing (engine source-override per #169 US-007a), rubric refinement (extend no-redundant, no 5th criterion), ingest strictness, mechanic exhaustiveness gate, and e2e-fixture seeding strategy. Phase 3 (refinement) — 17 DECs across all three earlier phases including three contested decisions resolved by the user: SORT the columns tuple in args_hash, establish the grade-side _PROMPT_VERSION surface, establish __repr__ redaction retroactively across CandidateTestRowCountBetween and CandidateTestCustomSQL. Phase 4 (detailing) — 16 implementation stories + Quality Gate + Patterns & Memory = 18 beads. US-005 split into compiler+snapshots (US-005a) and engine override+behavioural pin (US-005b) per user feedback. Plan doc is the tracking surface; beads land in Phase 7 after plan approval. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Update #170 plan with Phase 7 beads manifest Devolved to 17 beads (1 epic + 14 implementation + Quality Gate + Patterns & Memory). User-approved on 2026-06-01. Ready-to-start beads: US-001 (variant model + plumbing) and US-011 (engineered fixture + manifest seed) — the two independent leaves of the dependency graph. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-0tq.1: US-001 — CandidateTestUniqueCombination variant model + plumbing Add the 7th CandidateTest Pydantic variant + every supporting registration. Foundation for every downstream story in #170. Traces to #170 DEC-001 (variant name), DEC-002 (where shipped v1), DEC-014 (per-column identifier check at anchor-contract arm — NOT Pydantic), DEC-016 (len>=2 cardinality + no-duplicates invariants). Variant shape: type: Literal['unique_combination'] = 'unique_combination' column: None = None (model-level only — third variant after custom_sql and row_count_between to share this constraint) columns: tuple[str, ...] (len>=2, no duplicates — both invariants enforced at Pydantic; raw identifier strings carried through, shape check is US-004) where: str | None = None (non-empty after strip when set, matches the CandidateTestRowCountBetween precedent) rationale: str | None = None Registrations: - Added to CandidateTest discriminated union + __all__ export (src/signalforge/draft/models.py) - VALID_TEST_TYPES frozenset gains 'unique_combination' (src/signalforge/draft/config.py) - StrictCandidateTestUniqueCombination drift mirror + added to _StrictCandidateTest union (tests/draft/test_drift_detector.py) - New row in tests/fixtures/draft/candidate_schema_v1.json (after the row_count_between row, before the closing bracket) - test_valid_test_types_constant_matches_known_set updated to include the new variant (the load-bearing fail-loud gate) Deferred to downstream beads per the plan: - US-002: __repr__ redaction (security review gap) - US-003: drafter prompt catalogue + _PROMPT_VERSION rotation - US-004: anchor-contract arm (identifier shape + per-column check) - US-005a/b: prune compiler arm + sample-mode routing - US-006: artifact_id (SORTED) + diff emitter arm - US-007: ingest parser arm + anchor exemption - US-008: grade rubric no-redundant extension Validation: ruff check + ruff format + pyright + pytest all green (2860 passed, 75 deselected, coverage 97.53%). * bd_1-scaffolding-0tq.12: US-011 — Engineered fixture stg_bikeshare_station_pairs.sql + manifest seed Add a new fixture model with a natural multi-column GROUP BY pattern on (start_station_id, end_station_id, subscriber_type) so the drafter's prompt example reliably steers claude-sonnet-4-6 toward proposing the structured `unique_combination` test (#170 AC-1) instead of freeform `custom_sql GROUP BY HAVING COUNT(*) > 1`. Source-as-model alias trick per .claude/rules/testing-signal.md § 'WHERE the always-pass column must live depends on whether the model is materialised' — the model's `alias` is overridden to `bikeshare_trips` so its relation_name resolves directly to the public source table, no `dbt run` materialisation needed. Real source columns only; no engineered literal/COALESCE columns. Hand-crafted manifest seed per testing-signal.md § 'Hand-crafted manifest seed when workers can't run live tooling': Ralph workers in worktrees can't reach live BigQuery. The committed manifest.json + catalog.json entries mirror the dbt-bigquery 1.8 shape used for `stg_bikeshare_trips`. A future maintainer with live credentials can re-run regenerate.sh to refresh both models in one shot. Loads-only tests in tests/manifest/test_austin_fixture_loads.py verify the seed survives Pydantic parsing through signalforge.manifest.load without env vars / live calls. The existing iter_models test graduated from 'exactly one model' to 'staging models include both'. Demo parity: src/signalforge/_demo/ is mirrored with the same SQL + manifest + catalog updates so tests/test_demo_fixture_parity.py stays green (DEC-015 of #47 — the regenerate.sh's rsync step naturally covers the new file on a real regen). Traces to plans/super/170-unique-combination.md DEC-005, DEC-010, US-011. * bd_1-scaffolding-0tq.9: US-008 — Grade rubric no-redundant extension for grain-meaningfulness Extend the no-redundant criterion (rubric.py:189-198) with sibling calibration prose for unique_combination, naming the vacuously-unique tuple shape (primary_key, anything) — analogous to the row_count_between vacuous-bound extension from #169 DEC-009. Stays at 4 criteria per DEC-007 (no 5th criterion); same routing as the prior extension (low score → existing passed: bool threshold → flagged tier). Rotated three pinned hashes in lockstep (only the no-redundant criterion text changed; clarity/consistency/rationale hashes unchanged): - _canonical_rubric_hash: 22a0231690aca6ef → 30a9fda975b6d45c - prompt_version_template: 5a70561088930c97 → 4dae4421972e9c2d - criterion_prompt_hash[no-redundant]: 60690cb4ef9246ee → 7b96cfdfe63bc8bc Rotation-history comments updated in rubric.py + both pin sites with the #170 DEC-007 rationale. Grader's 3-trigger degrade taxonomy stays locked — a vacuous composite key is a low score, not a 4th degrade trigger. * bd_1-scaffolding-0tq.4: US-004 — Draft parser anchor-contract arm + collect-all matrix Extend _validate_anchor_contract with a unique_combination arm (model-level only): per-column membership check on each entry of test.columns + sqlglot-driven column-existence + type-coherence validation on the optional where clause. Collect-all preserved: every violation surfaces in one LLMOutputAnchorContractError, never short-circuits (DEC-022 of #5; DEC-014/015/016 of #170). Generalised _check_row_count_between_where -> _check_where_clause with a test_type prefix parameter so the same sqlglot machinery serves both where-bearing variants (DEC-005 of #169 'reuse, don't fork'). Existing row_count_between violation messages preserved byte-equal via the test_type='row_count_between' call site. Tests (7 new, all under -k unique_combination): - valid pair (no where) - valid 3-column tuple + where on a coercible-type column - hallucinated column in the columns tuple - hallucinated column in the where clause - type-incoherent where (INT64 vs STRING comparison) - exclude_tests=('unique_combination',) backstop - collect-all multi-violation (CandidateColumn + tuple + where) Validation: ruff/ruff-format/pyright/pytest all green; 2867 passed, coverage 97.68%; the 6 pre-existing row_count_between parser tests still pass byte-equal against the renamed helper. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-0tq.7: US-006 — _common.artifact_id arm (SORTED columns) + diff emitter arm Add CandidateTestUniqueCombination arms in two related sites per #170 DEC-011 and DEC-002: - src/signalforge/_common/artifact_id.py: new isinstance arm in model_test_args_hash with sorted(test.columns) (load-bearing — DEC-011). Mirrors accepted_values.values precedent. Cross-stage parity holds by re-export identity through signalforge.diff._artifact_id and signalforge.grade.engine (no per-module change). - src/signalforge/diff/_emitter.py: new isinstance arm in _render_test emitting {dbt_utils.unique_combination_of_columns: {combination_of_columns: [...]}}. Pydantic field 'columns' maps to dbt-utils macro key 'combination_of_columns' on emission only (field-name mapping seam). Emission preserves the LLM's declared order — sorting is for the canonical-hash domain only. Optional 'where' rendered verbatim under 'where' key when set; omitted when None. Tests: - tests/diff/test_artifact_id.py: 7 new unique_combination tests covering sort invariance (a,b) == (b,a), 3-column permutation, distinct columns → distinct hash, distinct where → distinct hash, collision suffix via compute_args_hashes, exact-duplicate ordinal suffix, cross-stage parity. - tests/diff/test_emitter.py: 5 new unique_combination tests covering no-where YAML shape, with-where YAML shape, declared-order preservation (the sort/no-sort load-bearing distinction), dropped-decision filtering, and the contract that unique_combination does NOT flow to emit_proposed_test_files. Validation: uv run ruff check / format / pyright / pytest all green. All 2873 tests pass; coverage 97.54%. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-0tq.10: US-009 — Establish grade-side _PROMPT_VERSION snapshot surface Establishes the grade-side cache-stability surface that .claude/rules/business-rule-tests.md § "Lockstep _PROMPT_VERSION rotation when extending the catalogue (#169 DEC-012)" claims exists but actually doesn't pre-#170. Closes the overstated rule-file claim with reality (#170 DEC-012 / US-009). Changes: - src/signalforge/grade/prompts.py: add module-level _PROMPT_VERSION constant computed at import as prompt_version_template(DEFAULT_RUBRIC). Mirrors the drafter shape (signalforge.draft.prompts._PROMPT_VERSION). Recipe documented inline: blake2b-8 over _SYSTEM_PROMPT + render_rubric_block(DEFAULT_RUBRIC) + envelope tags. Rotates when system prompt, any of the 4 default criterion texts, or envelope tags change. Exported in __all__. - tests/grade/test_prompt_cache_stability.py (new): pin the constant + the rendered rubric-block bytes. Two tests: hash-pin assertion and difflib-diffing byte-equality against an inline golden. Mirrors tests/llm/test_prompt_cache_stability.py shape verbatim. Rotation history documents the current value 4dae4421972e9c2d as the #170 US-009 establishment. - pyproject.toml: per-file-ignore E501 on the new test (inline rubric-block golden carries long single-line criterion texts that render together on the wire; refactoring would change the bytes the test pins). Pinned values (current, also matching the live helpers after US-008): - _PROMPT_VERSION = "4dae4421972e9c2d" - _canonical_rubric_hash(DEFAULT_RUBRIC) = "30a9fda975b6d45c" (pinned elsewhere by US-008; this commit does not touch it). Validation: ruff check, ruff format --check, pyright, pytest all green (2863 passed, 97.54% coverage). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-0tq.8: US-007 — Ingest parser arm + ingest anchor exemption Adds the recognition arm + helper for the ``dbt_utils.unique_combination_of_columns`` macro in ``signalforge.ingest.parser`` (dispatch site #4 per ``.claude/rules/business-rule-tests.md`` § "The 6 production dispatch sites") and the model-level loop exemption in ``signalforge.ingest.anchor.validate_anchor_contract`` (dispatch site #6 — the third model-level-only variant after ``custom_sql`` and ``row_count_between``). Inbound mapping (DEC-002 / DEC-008): YAML ``combination_of_columns: list[str]`` ↔ Pydantic ``columns: tuple[str, ...]``; optional ``where: str`` carried verbatim. The mapping seams are exactly these two functions (inbound here, outbound in ``diff/_emitter.py``) — the dbt_utils macro-name field does NOT bleed into the internal model. ``_UNIQUE_COMBINATION_NAME`` mirrors the ``_ROW_COUNT_BETWEEN_NAME`` precedent. Skip routes (all ``reason="malformed-supported-test"`` — closed 3-value ``SkipReason`` enum stays locked per ``.claude/rules/ingest-layer.md``): column-scoped usage (variant is model-level only); non-dict body; missing ``combination_of_columns`` key; non-list value under the key; empty list; ``len < 2`` (single-column is just ``unique``); non-string items (incl. bool guard mirroring ``_parse_row_count_between``); duplicate items; non-empty non-string or whitespace-only ``where``. Different sibling ``dbt_utils.*`` macros stay ``custom-or-generic-test``. Tests (15 new parser + 1 anchor): * tests/ingest/test_parser.py — happy paths (inline 2-col, with where, 3-col, arguments:-nested), 9 malformed routes, sibling-macro custom-skip, config-keys-ignored. * tests/ingest/test_anchor.py — model-level + ``column=None`` does not raise (mirrors ``test_model_level_row_count_between_with_none_column_does_not_raise``). Fixture updates: ``schema_codegen_shaped.yml`` and ``schema_austin_bikeshare.yml`` (and their consumers ``test_reader.py``, ``test_prune_existing.py``, and the column-scoped custom-skip pin in ``test_parser.py``) switched the example namespaced/custom test from ``dbt_utils.unique_combination_of_columns`` to ``dbt_utils.not_null_proportion`` — the original macro is now a first-class variant and column-scoped usage now correctly routes to malformed-supported-test. Also: pre-existing format drift in ``tests/draft/test_parser.py`` fixed in-passing so the pipeline gate stays green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-0tq.5: US-005a — Prune compiler arm _compile_unique_combination + BQ/Snowflake snapshots Implements the 7th first-class CandidateTest variant's compiler arm (#170 US-005a, traces to DEC-014 / DEC-015). * _compile_unique_combination in src/signalforge/prune/compiler.py emits the composite-grain failing-rows SELECT shape 'SELECT <cols> FROM <table_ref> [WHERE <where>] GROUP BY <cols> HAVING COUNT(*) > 1'. Dialect-driven via the existing _fold_identifier + _quote + _qualified_table_name helpers (no dialect.name branching — load-bearing per .claude/rules/prune-engine.md § 'Compiler is dialect-driven'). * Per-column DEC-014 identifier shape gate (defence-in-depth on top of the anchor-contract arm). Identifier rejection routes via _InvalidIdentifier → kept-without-evidence (conservative-bias). * DEC-015 compose-then-validate: 'where' is interpolated into the full SELECT then routed through validate_test_sql, mirroring _compile_row_count_between (#169 DEC-005 reuse). Hostile 'where' (stray ';', '--', '/* */', unbalanced parens) returns _InvalidIdentifier rather than raising. * Dispatcher arm added at _compile_test after the CandidateTestRowCountBetween branch. 6 new snapshot fixtures (3 BigQuery + 3 Snowflake) pin the byte-exact emitted SQL across both dialects. The 3 Snowflake fixtures are added to the gated sqlglot Snowflake-dialect parse-guard (tests/prune/test_compiler_fakesnow.py::_ALL_SNOWFLAKE_FIXTURES) per the #121 lesson — snapshot equality certifies shape, not validity; a parser-in-the-loop is what catches reserved-keyword / quoting regressions. Sample-mode routing is out of scope here (US-005b). The engine's source-vs-temp override for unique_combination ships in the sibling bead; this arm consumes table_ref as-is. The dispatcher arm comment points at the load-bearing engine-level pin (test_prune_tests_unique_combination_under_*) that US-005b will add. 12 new compiler tests (6 snapshot equality + adversarial column + hostile where x3 + dispatch-arm uniqueness + safety round-trip). The full default suite (2880 tests) passes; the gated -m snowflake suite (36 offline fakesnow/sqlglot tests) passes; pyright clean; ruff check/format clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-0tq.2: US-002 — Custom __repr__ redaction (mixin + retroactive) Establishes redacted __repr__ on the three text-bearing candidate-test variants — CandidateTestCustomSQL, CandidateTestRowCountBetween, CandidateTestUniqueCombination — so the LLM-emitted free-text fields (sql / where / rationale) never reach log sinks via casual repr() or %s-interpolation. Chosen shape: per-class __repr__ overrides + a small shared helper (_scope_repr) for the column/model-level scope segment. Each variant has a distinct identifying surface (custom_sql's column-scope handling, row_count_between's numeric bounds, unique_combination's columns tuple), so per-class bodies stay clearer than a single mixin. The scope helper keeps the column-vs-model-level convention in one place for any future variant. Surface exposed in repr(): - type (the discriminator Literal) - scope (column=<name> for column-scoped, <model-level> otherwise) - per-variant constraint shape: custom_sql shows only scope; row_count_between shows minimum/maximum bounds; unique_combination shows the columns tuple (column NAMES are not value-bearing) Surface redacted: - sql (CandidateTestCustomSQL) - where (CandidateTestRowCountBetween, CandidateTestUniqueCombination) - rationale (all three) Per the rule (prune-engine.md / grade-layer.md § "Custom __repr__"): Pydantic __str__ is reserved for serialisation and stays untouched — only __repr__ is overridden. model_dump_json() round-trip continues to carry every field (3 positive tests pin this). The four non-text-bearing variants (NotNull, Unique, AcceptedValues, Relationships) are untouched — they carry only column names and structured args, no free-text leak surface. Closes the log-hygiene gap surfaced by Phase 2 Security review (#170 DEC-013). Drive-by: ruff format on tests/draft/test_parser.py (pre-existing format drift from US-004 merge that VALIDATE_CMD would otherwise reject). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-0tq.3: US-003 — Drafter prompt catalogue + _PROMPT_VERSION rotation Add unique_combination as the 7th first-class test primitive in the drafter prompt catalogue (issue #170, DEC-002). Two illustrated forms (no-where and with-where) mirror the #169 row_count_between precedent. New _UNIQUE_COMBINATION_SCOPE_INSTRUCTION block carries cautionary prose steering the LLM away from vacuously-unique tuples like (pk, anything) — the prompt-level prevention complements the grade-side no-redundant criterion calibration (US-008). Catalogue + SCOPE wiring: - _TEST_CATALOGUE_LINES gains the unique_combination entry (after row_count_between), illustrating both no-where and with-where shapes with realistic 2-column examples. - _UNIQUE_COMBINATION_SCOPE_INSTRUCTION mirrors _CUSTOM_SQL_SCOPE_INSTRUCTION: emitted only when unique_combination is allowed, omitted when excluded so the prompt never asks for a type the parser would reject. - _render_system_prompt threads unique_combination_scope through the SCOPE template alongside custom_sql_scope. - _SYSTEM_PROMPT_TEMPLATE gets a {unique_combination_scope} format slot at the end of the SCOPE section. _PROMPT_VERSION rotation (77e9ee8a6ae7d875 → 389c8aa970df86cc): - Constant rotates automatically (computed from _SYSTEM_PROMPT bytes). - _EXPECTED_PROMPT_VERSION in tests/llm/test_prompt_cache_stability.py bumped to the new hex. - Rotation history extended with a #170 entry noting the new catalogue line + SCOPE instruction (cached-block golden unchanged — only the system prompt rotated). exclude_tests test fixes: - Four tests in tests/draft/test_exclude_tests.py added unique_combination to their exclusion tuples. They previously enumerated the standard set exhaustively; the 7th variant joining the catalogue means they need to exclude it too to preserve their original intent (testing how SCOPE renders with only some types remaining). Docstring updates name unique_combination + #170. "Five entries" → "six entries" in the _TEST_CATALOGUE_LINES docstring (custom_sql lives separately in _CUSTOM_SQL_CATALOGUE_LINE per the existing convention). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-0tq.6: US-005b — Engine sample-mode source override + behavioural routing pin Extends the existing row_count_between source-vs-temp conditional in prune/engine.py to include CandidateTestUniqueCombination — composite uniqueness on a sample is semantically approximate (false-negative risk: a duplicate pair may straddle the sampled and unsampled rows), so always-route-to-source mirrors #169 US-007a. Bounded by maximum_bytes_billed. Both sites move in lockstep: * `all_bypass_to_source` short-circuit (CodeRabbit #176 fix) — when EVERY candidate is row_count_between OR unique_combination, skip materialise_sample AND _resolve_sample_bucket pre-work; otherwise a materialisation failure would spuriously route every test to kept-without-evidence. * Per-test `per_test_table_ref` override — when scope="sample" and candidates are mixed, the row_count_between / unique_combination ones still route to source while other variants consume the substituted compile_table_ref. Behavioural routing pin at tests/prune/test_engine.py — mirrors the #169 row_count_between precedent (test_prune_tests_row_count_between_under_materialised_references_source_not_temp_table): * Parametrised across sample_strategy="materialised" AND "oneshot" — the load-bearing pin (snapshot equality from US-005a certifies SQL shape but NOT engine routing per .claude/rules/business-rule-tests.md § "Pin the engine-routing test, not just the compiler snapshot"). * Asserts compiled_sql references the source qualified name AND never references `_SESSION._sf_sample_*`. * Companion scope="full" test as a no-regression belt-and-braces. Traces to #170 DEC-006 (Option (iii): engine override to source via per_test_table_ref). Done when the engine routes unique_combination to source under sample mode, pinned by behavioural test. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-0tq.11: US-010 — Mechanic exhaustiveness gate (6-site dispatch routing test) Adds a targeted dispatch-site routing test (NOT a full AST scan, per DEC-009 of #170) that constructs a minimal instance of every variant in the CandidateTest discriminated union and asserts each routes through every one of the 6 production dispatch sites without raising. The 6 sites (per .claude/rules/business-rule-tests.md § 'The 6 production dispatch sites'): 1. signalforge.prune.compiler._compile_test 2. signalforge._common.artifact_id.model_test_args_hash 3. signalforge.diff._emitter._render_test 4. signalforge.ingest.parser._parse_named_test (external macro recognition) 5. signalforge.draft.parser._validate_anchor_contract 6. signalforge.ingest.anchor.validate_anchor_contract Each variant × site combination is one parametrize iteration; site 4 skips for variants without an external dbt-macro form (only row_count_between and unique_combination have one). Self-checks pin that sites 5 and 6 raise on real violations so the routing arms can't silently mask dispatch bugs with a tautological green. Variant reflection via typing.get_args means an 8th variant added to the union auto-grows the parametrize without a test edit. A cardinality tripwire asserts the union currently holds exactly 7 variants; bumping the union forces the contributor to update _EXPECTED_VARIANT_COUNT, _make_instance, and _EXTERNAL_MACRO_YAML in lockstep — all gated by this test rather than discovered at runtime on the operator's machine. TDD verified: temporarily removing the unique_combination arm from _render_test (site 3) and the model-level exemption from validate_anchor_contract (site 6) each fail one parametrize iteration loudly with a remediation message pointing at the missing arm. 48 parametrize iterations (43 pass + 5 N/A site-4 skips); no production code change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-0tq.14: US-013 — docs SSOT + README + ops paraphrases + CHANGELOG + mkdocs nav Bundles the documentation tail of #170 (unique_combination as 7th first-class CandidateTest variant). Per Phase 1 S2 + DEC-003, the SSOT lives in the operator-facing docs/ tier (not .claude/rules/). - docs/drafter-catalogue.md (new) — SSOT enumerating the seven first-class primitives (not_null, unique, accepted_values, relationships, custom_sql, row_count_between, unique_combination) with YAML examples, structural slots, scope, ingest signatures, semantics; a 'custom_sql is the catch-all' sub-section; a 'What we do NOT generate today' boundary section (column value range, conditional uniqueness beyond simple where, statistical/distributional anomalies, cross-table reconciliation, time-series anomaly detection). - README.md — new 'What tests SignalForge generates' section sitting between 'What it does' and 'How it works' (compact 7-row table + pointer to the SSOT). - docs/draft-ops.md — new 'Composite uniqueness (unique_combination)' section after the row_count_between block, mirroring its shape: what the variant is, when drafted, worked example, exclude_tests short-circuit. - docs/ingest-ops.md — new 'Recognition of dbt_utils.unique_combination_of_columns' subsection mirroring the expect_table_row_count_to_be_between precedent: inbound mapping, skip-recorded shapes, unchanged-other-dbt_utils-macros boundary. - docs/grade-ops.md — new 'Composite-key calibration (unique_combination)' subsection extending the no-redundant criterion narrative; rubric stays at four criteria. - docs/prune-ops.md — callout in the row-count cost model section for the unique_combination engine source-vs-temp routing override. - mkdocs.yml — adds 'Test Catalogue: drafter-catalogue.md' to the nav between 'Claude Code Skill' and 'Pipeline Stages'. - CHANGELOG.md [Unreleased] — Added (unique_combination variant + dbt_utils.unique_combination_of_columns ingest recognition), Docs (drafter-catalogue.md + README section + four ops doc paraphrases), Changed (drafter _PROMPT_VERSION rotation 77e9ee8a6ae7d875 → 389c8aa970df86cc, grade rubric no-redundant extension, grade-side _PROMPT_VERSION snapshot surface 4dae4421972e9c2d established). Validation: uv run ruff check/format/pyright + uv run pytest (2922 passed, 78 deselected, 97.70% coverage) + uv run --only-group docs mkdocs build green. Pre-existing pre-#170 INFO-level link warnings (e.g. draft-ops.md#row-count-tests-row_count_between anchor missing in rendered HTML — a #169 rendering bug from the BUSINESS RULES code fence) are unaffected. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-0tq.13: US-012 — Gated e2e (drafter emits structured unique_combination) New @pytest.mark.e2e-gated test pinning the load-bearing behavioural claim of #170: when the drafter sees a model whose SELECT body advertises a natural composite GROUP BY shape (US-011's engineered fixture model stg_bikeshare_station_pairs), it proposes a structured CandidateTestUniqueCombination candidate — NOT a freeform custom_sql GROUP BY HAVING COUNT(*) > 1. Standard three-env-var gate (mirrors test_e2e_bigquery_smoke.py baseline): SF_RUN_BQ=1 + GOOGLE_CLOUD_PROJECT + ANTHROPIC_API_KEY. Drafter is Anthropic Sonnet 4.6; warehouse is BigQuery; no provider overlay. Engineered determinism via structural (not value-pinning) assertion: AT LEAST ONE PruneDecision must carry test.type == 'unique_combination' with len(test.columns) >= 2. The exact columns tuple is NOT pinned because Sonnet may legitimately propose two- or three-column variants of the fixture's GROUP BY shape; the prune verdict (kept vs. dropped) is NOT pinned because bikeshare data shape is orthogonal to the freeform→structured translation under test. Test reads PruneDecision.test from .signalforge/prune.jsonl via the existing read_prune_decisions helper — the typed CandidateTest discriminated union flows through prune intact (PruneEvent.test: CandidateTest, audit DEC-014). Validation green: ruff + format + pyright + pytest all clean. Test is correctly deselected by default (not e2e) addopts and skips with distinct reason for each missing env var when invoked with -m e2e. AC-1 (drafter proposes structured unique_combination), AC-8 (end-to-end pipeline shape). Maintainer-run command: SF_RUN_BQ=1 ANTHROPIC_API_KEY=sk-... GOOGLE_CLOUD_PROJECT=<project> \ uv run pytest -m e2e -k unique_combination --no-cov Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-0tq.15: US-014 Quality Gate — 6 fixes from 4 reviewer angles Triangulated findings across correctness / conventions / tests / docs+UX reviewers; same finding from 2+ angles upgraded to must-fix per memory qg-diverse-reviewer-angles-catch-cross-surface-drift. Fixes applied: 1. **F1 (Pass 4 must-fix)** — Pre-existing `## BUSINESS RULES` literal inside fenced code block in docs/draft-ops.md:288-300 was parsed as ATX heading by mkdocs anchor generator, breaking 14 of 20 H2 anchors downstream (covers row_count_between + unique_combination — all 8 inbound cross-doc links #170 added go through these). Switched to indented code block (4-space) which defeats the heading scan. 2. **C1 (Pass 1 concern, empirically verified)** — DEC-013 __repr__ redaction was leaking via Pydantic v2's __repr_args__ / __rich_repr__ / __pretty__ hooks (rich.print() / devtools / pprint). Added __repr_args__ overrides on the 3 redacting classes (CandidateTestCustomSQL, CandidateTestRowCountBetween, CandidateTestUniqueCombination). Filters out where / sql / rationale from the structured-debug surface. New test pins the closure across all 3 variants + asserts model_dump_json() still carries the secrets (serialisation contract unchanged). 3. **Pass 3 must-fix (load-bearing routing pin gap)** — US-005b per_test_table_ref override has two conditionals (all_bypass_to_source short-circuit AND per-test override). Single-variant tests only exercised the first; dropping unique_combination from the per-test arm while leaving it in the short-circuit would PASS silently. Added test_prune_tests_mixed_candidates_per_test_override_routes_unique_combination_to_source with not_null + unique_combination mix — exercises the per-test arm directly. Asserts not_null compiles against _SESSION sample, while unique_combination compiles against source — both routings pinned. 4. **F2 (Pass 4 must-fix)** — CHANGELOG Added entry was silent on the dbt-utils install constraint (parallel gap to #169's note on dbt-expectations). Appended the parallel clause. 5. **F3 (Pass 4 should-fix)** — CHANGELOG Changed entries on both _PROMPT_VERSION rotations now name the one-time Anthropic prompt-cache miss operators pay on upgrade. 6. **F4 (Pass 4 should-fix)** — docs/drafter-catalogue.md row_count_between subsection was missing the source-vs-temp routing callout that unique_combination has; added parallel one-paragraph callout naming both variants in lockstep. Deferred to US-015 / follow-up per Pass 2 informational findings: - US-004 #169 pre-existing format drift on tests/draft/test_parser.py (3 workers all "drive-by formatted" the same file; investigate why US-004's own ruff format --check passed) - custom_sql lacks model-level loop exemption in ingest.anchor (benign today, documented gap from US-010 worker) - E501 ignore asymmetry on drafter vs grade snapshot test - _PROMPT_VERSION in grade.prompts __all__ cosmetic - Missing "description" config-key in unique_combination ingest test Validation green across: - uv run ruff check . && uv run ruff format --check . - uv run pyright (0 errors) - uv run pytest (full suite + new tests) - uv run pytest -m cli_subprocess --no-cov (8 pass) - uv run pytest -m wheel_smoke --no-cov (5 pass) - uv run pytest -m snowflake --no-cov (36 pass + 5 live-only skips) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bd_1-scaffolding-0tq.16: US-015 Patterns & Memory — rule updates + 4 memory writes Closes #170 Patterns & Memory tail. ## Rule file updates - **`.claude/rules/business-rule-tests.md`** — bumped "2-instance precedent" → "3-instance precedent" (custom_sql / row_count_between / unique_combination). Reference footnote extended with the #170 plan pointer + the durable conventions section "#170 lessons worth carrying forward" enumerating the 5 patterns the next variant-extension should pre-empt: 1. Two engine conditionals (`all_bypass_to_source` short-circuit AND per-test `per_test_table_ref` override) — both grow in lockstep; mixed-candidate test is load-bearing for the per-test arm. 2. Pydantic v2 `__repr_args__` / `__rich_repr__` / `__pretty__` hooks bypass `__repr__` — override `__repr_args__` for redaction parity across rich.print() / devtools / pprint. 3. SORT a tuple-shaped canonical hash arg (`(a,b) ≡ (b,a)` semantics); the diff EMITTER preserves declared order — split contract. 4. mkdocs ATX heading in fenced code block silently corrupts downstream H2 anchors; use indented (4-space) code blocks for examples that need a literal `##`. 5. Drive-by formatting across multiple workers reveals merge-induced format drift the original PR's gates can't see — add a post-merge `ruff format --check .` to the orchestrator's Step 5. - **`.claude/rules/grade-layer.md`** — documented the grade-side `_PROMPT_VERSION` snapshot surface established by #170 DEC-012, closing the asymmetry where `business-rule-tests.md` had historically claimed "two `_PROMPT_VERSION` constants" but only the drafter had one. Rotation policy mirrors the drafter contract — rotate when the grade `_SYSTEM_PROMPT` text changes OR any of the four DEFAULT_RUBRIC criterion texts changes; the new value is computed and pinned in the same commit (US-008 + US-009 paired). ## Memory writes (4 new files + MEMORY.md index) All in `~/.claude/projects/-home-wesd-Projects-SignalForge/memory/`: 1. `pydantic-v2-repr-args-redaction-required.md` — DEC-013 lesson; custom `__repr__` alone leaks via `__rich_repr__` / `__pretty__`. 2. `prune-engine-two-conditional-routing-pattern.md` — US-005b discovery; mixed-candidate test is load-bearing. 3. `mkdocs-atx-in-fenced-block-breaks-anchors.md` — QG Pass 4 finding; indented code blocks are the fix. 4. `drive-by-format-reveals-merge-induced-drift.md` — observation across US-005a / US-007 / US-002; orchestrator can pre-empt with a post-merge `ruff format --check`. ## Deferred to follow-up issues (not blocking #170) - `ingest.anchor.validate_anchor_contract` lacks model-level loop exemption for `custom_sql` (benign today; documented by US-010 worker in the dispatch-exhaustiveness test docstring). - US-004 of #169 pre-existing format drift on `tests/draft/test_parser.py` (each of US-005a / US-007 / US-002 fixed it independently; root cause is merge-time drift not worker carelessness). - E501 ignore asymmetry on grade vs drafter snapshot tests. - `_PROMPT_VERSION` in `signalforge.grade.prompts.__all__` (cosmetic inconsistency with drafter side). Validation green: ruff / pyright / pytest (2967 passed). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * #170: Address PR review feedback (CodeRabbit, 5 threads) All 5 review threads addressed: **Major (1):** - `src/signalforge/ingest/anchor.py` — the `unique_combination` model-level loop exemption was skipping `test.columns` validation entirely (CodeRabbit MAJOR; triangulated with QG Pass 1 C2 + Pass 2 informational #2 + US-010 worker docstring — four-way agreement). Extended the arm to iterate `test.columns` and surface per-column violations mirroring the draft-parser side. Pinned by new test `test_model_level_unique_combination_with_hallucinated_column_raises_per_column` asserting collect-all (two hallucinated columns → two distinct violations in one error). Used `isinstance(test, CandidateTestUniqueCombination)` for proper pyright narrowing of the discriminated union. **Minor (4):** - `docs/draft-ops.md` link target → direct `unique_combination` anchor in prune-ops.md (was pointing at `#row-count-cost-model`). - `docs/drafter-catalogue.md` link target → same direct anchor fix. - `plans/super/170-unique-combination.md` DEC-002 row — escaped `|` in `str \| None` so the markdown table parses correctly (markdownlint MD056). - `plans/super/170-unique-combination.md` story-dependency-graph fenced block — added `text` language identifier (markdownlint MD040). Validation green: ruff / pyright / pytest (2968 passed, +1 new test).
* Add super plan for #171: row_count_anomaly_by_period * #171: devolve plan to beads (epic + 19 stories) * bd_1-scaffolding-1r7.2: US-002 Dialect 5 new SQL-fragment fields (date arithmetic + percentile) * bd_1-scaffolding-1r7.1: US-001 AnomalyTestStats typed shape + drift mirror + fixture * bd_1-scaffolding-1r7.4: US-015 grade rubric no-redundant extension + _PROMPT_VERSION rotation Extend DEFAULT_RUBRIC no-redundant criterion with calibration prose for per-period anomaly tests (row_count_anomaly_by_period) per #171 DEC-004 — mirrors #169 DEC-009 + #170 DEC-007 verbatim shape (Option A: extend the existing criterion, no 5th, same +25% LLM-cost reason). The prose teaches the judge to score CALIBRATION of the (method, seasonality, threshold) combination: is it tight enough to catch the failure mode (anomalously small/empty period) but loose enough not to fire on legitimate weekday/weekend or seasonal swings? A worked example pins zscore+threshold=3.0 without seasonality='dow' on a weekday-heavy model as a calibration failure (fires every Sat). Rotation lockstep: - src/signalforge/grade/rubric.py — DEFAULT_RUBRIC no-redundant text + rotation-history comment. - tests/grade/test_rubric.py — verbatim-match test updated; _DEFAULT_RUBRIC_GOLDEN_HASH 30a9fda975b6d45c → a4e3ee92cf9ec36f; new test_no_redundant_criterion_carries_row_count_anomaly_ calibration_prose pins the new prose elements; existing other- criteria preservation tests unchanged. - tests/grade/test_prompts.py — prompt_version_template golden 4dae4421972e9c2d → b1e609fae240ac1c; per-criterion no-redundant hash 7b96cfdfe63bc8bc → b24ff0014a5dcb86 (other 3 unchanged). - tests/grade/test_prompt_cache_stability.py — _EXPECTED_PROMPT_VERSION 4dae4421972e9c2d → b1e609fae240ac1c; _RUBRIC_BLOCK_GOLDEN refreshed with the new no-redundant line. The 3-trigger grade degrade taxonomy (DEC-011 of #169) stays locked — weak anomaly calibration routes through a low criterion score → passed: bool threshold → flagged tier, NOT a 4th degrade trigger. The drafter _PROMPT_VERSION is untouched (US-005 territory). Canonical validation passes: ruff check + ruff format + pyright + 2969 pytest passing. * bd_1-scaffolding-1r7.3: US-003 CandidateTestRowCountAnomalyByPeriod variant class + union + drift + fixture + VALID_TEST_TYPES * bd_1-scaffolding-1r7.11: US-012 PruneEvent + PruneDecision + audit schema v2→v3 + serializer + fixture update + drift detector * bd_1-scaffolding-1r7.10: US-009 Prune engine as_of threading + INFO log + variant detection * bd_1-scaffolding-1r7.6: US-005 Drafter prompts catalogue + _PROMPT_VERSION rotation + cache-stability snapshot Extend _TEST_CATALOGUE_LINES with an 8th entry for row_count_anomaly_by_period illustrating three JSON shapes (bare default call; seasonality="dow" for business-calendar grain; explicit method + threshold override). Add _ROW_COUNT_ANOMALY_SCOPE_INSTRUCTION block to the SCOPE section teaching: - propose when projection includes loaded_at / created_at / event_date / partition_date (incremental fact tables) - propose seasonality="dow" for business-calendar grain - default method=mad (median absolute deviation; robust to outliers), default threshold=3.0, lookback_periods=28, min_samples_per_bucket=3 - method-by-method calibration (zscore for outlier sensitivity, percentile for Tukey-style IQR, min_max for zero-margin envelope) Wire _row_count_anomaly_allowed through _render_system_prompt so exclude_tests=("row_count_anomaly_by_period",) drops both the catalogue line AND the SCOPE-instruction block (mirrors #163 / #169 / #170 exclude_tests pattern). _PROMPT_VERSION rotates 389c8aa970df86cc -> a4fea640b3b60f24. Update _EXPECTED_PROMPT_VERSION + the rotation log in tests/llm/test_prompt_cache_stability.py. The cached-block golden (manifest summary) is unchanged — only the system prompt rotated. Update three pre-existing exclude_tests test cases in tests/draft/test_exclude_tests.py to include the new variant in their exclusion tuples so the SCOPE phrase assertions stay valid. Grade-side _PROMPT_VERSION (US-015) untouched. * bd_1-scaffolding-1r7.5: US-004 _common.artifact_id arm + collision rule Add row_count_anomaly_by_period arm to model_test_args_hash in src/signalforge/_common/artifact_id.py. Identifying args are the eight scalar/literal fields (method, seasonality, period, lookback_periods, threshold, min_samples_per_bucket, date_column, where) plus column (always None) for shape-parity. rationale is NOT in the hash domain — it is drafter-emitted prose, not identifying (mirrors precedent on every other variant). Per the code comment: all scalar args, no tuples — no sort needed. The #170 unique_combination sort of its columns tuple addresses a semantic order-invariance specific to composite GROUP BY identity; this variant has only scalars + literals, so sorting is N/A. Collision rule: two anomaly tests on the same model differing only by method (or any identifying arg) get distinct artifact_id suffixes via the existing compute_args_hashes disambiguator — verified by 10 new tests in tests/diff/test_artifact_id.py: * 7 distinct-arg rotations (method/seasonality/period/lookback_periods/ threshold/min_samples_per_bucket/date_column/where) * 1 identical-args same-hash regression (guards against spurious hash inputs) * 1 rationale-not-in-hash regression * 1 collision-disambiguator test exercising compute_args_hashes * 1 cross-stage parity (defence-in-depth alongside function identity) Cross-stage parity (function identity across signalforge._common.artifact_id, signalforge.diff._artifact_id, signalforge.grade.engine) is preserved by construction — diff and grade re-export from _common, no edits needed. Scaffolding update: _VARIANTS_PENDING_DISPATCH_ARMS in tests/test_candidate_test_dispatch_exhaustiveness.py converted from a single frozenset to a per-site dict (site number -> frozenset). Site 2 (_common.artifact_id) entry is now empty — exercising CandidateTestRowCountAnomalyByPeriod through the artifact_id hash parametrize. Sites 1/3/5/6 still pending in sibling beads (US-008/US-014/US-006/US-007); site 4 is N/A (no external dbt-macro form). Per-site granularity is load-bearing: it lets a single dispatch-arm bead land without blocking sibling beads on the same variant. Validation: ruff check, ruff format --check, pyright, and the full pytest suite all green (3064 passed, 6 skipped, 79 deselected; 97.73% coverage). * bd_1-scaffolding-1r7.9: US-008 Prune compiler _compile_stats_query + _compile_violation_query + 8 SQL shapes + partition filter Adds the row_count_anomaly_by_period compile arm to signalforge.prune.compiler: - _compile_anomaly_stats_query + _compile_anomaly_violation_query helpers per DEC-008. Stats: 4 methods (mad/zscore/percentile/min_max) × 2 seasonality (none/dow) = 8 distinct CTE shapes. Violation: one row per row in today's period (the adapter's COUNT-wrap yields today's count; US-011 wires the engine-side band comparison). - Dispatcher arm in _compile_test returns a (stats_sql, violation_sql) tuple ONLY for this variant; signature broadens to str | _RequiresFutureData | _InvalidIdentifier | tuple[str, str]. Other variants still return a single string. - as_of: date | None kwarg threads through _compile_test (keyword-only, default None; non-anomaly variants ignore). Engine (US-009) resolves to date.today() before the compile call. - DEC-011 dialect-driven: every SQL fragment (date_trunc_expr_template, interval_expr_template, extract_dow_expr_template, percentile_cont_expr_template, date_literal_template) is read from Dialect. NEVER branches on dialect.name; the import-guard at tests/prune/test_compiler_import_guard.py stays green. - DEC-012 partition filter present in EVERY emitted shape: stats has >= as_of - INTERVAL <lookback> <unit> AND < as_of (history-only); violation has >= as_of AND < as_of + INTERVAL 1 <unit> (today-only). Pinned via byte-exact snapshot fixtures + a shape-pinning regex test. - DEC-005 compose-then-validate: a hostile where (stray ; / -- / comment / unbalanced parens) routes via _InvalidIdentifier (mirrors row_count_between / unique_combination). date_column safety-checked via validate_identifier as DEC-013 defence-in-depth. Fixtures: 16 BigQuery + 16 Snowflake under tests/fixtures/prune/ compiled_sql/anomaly/{bigquery,snowflake}/ (one file per query per shape). Snowflake fixtures parse cleanly via sqlglot.parse_one(dialect= 'snowflake') under the gated @pytest.mark.snowflake suite. Cross-story scaffolding (#171 US-003 → US-008): - tests/test_candidate_test_dispatch_exhaustiveness.py: _VARIANTS_PENDING_DISPATCH_ARMS frozenset graduates to a PER-SITE dict (_VARIANTS_PENDING_DISPATCH_ARMS_PER_SITE) keyed by site number so each bead can mark its own site as landed without waiting for siblings. site_1 (prune compiler dispatch) is empty post this bead; site_4 is empty (no external macro for this variant); sites 2/3/5/6 still pending US-004/006/007/014. Defensive engine arm: prune.engine grows an isinstance(compile_result, tuple) branch that routes to kept-without-evidence with reason 'row_count_anomaly_by_period two-query split not yet wired in the engine (#171 US-011 pending)'. This keeps the engine type-correct under the broadened compiler return type until US-011 lands the real two-query handling. Mirrors the compiler-side conservative-bias routing pattern. Canonical validation passes: ruff check + format, pyright, full pytest suite (3110 passed, 6 expected skips). Gated Snowflake suite (uv run pytest -m snowflake --no-cov) passes 52 tests including all 16 anomaly Snowflake-fixture parse-guards. * bd_1-scaffolding-1r7.8: US-007 ingest anchor exemption (model-level early-out) Adds the row_count_anomaly_by_period early-continue arm to the model-level loop in signalforge.ingest.anchor.validate_anchor_contract, mirroring the row_count_between and unique_combination exemptions. Without it, the generic 'test.column not in model_columns' check would fire a spurious 'references nonexistent column None' violation on the variant's hardcoded column=None shape. The drafter parser (US-006, sibling bead) remains the anchor-contract authority for date_column shape validation; the ingest anchor's job is column-of-test enforcement only — a defensive test pins this contract boundary so a future refactor cannot inadvertently introduce a cross-layer ingest -> draft coupling. Cross-story scaffolding: removes CandidateTestRowCountAnomalyByPeriod from site 6's _VARIANTS_PENDING_DISPATCH_ARMS entry in tests/test_candidate_test_dispatch_exhaustiveness.py — site 6 now exercises the variant through the routing test. * bd_1-scaffolding-1r7.7: US-006 drafter parser anchor-contract arm (date_column + where type-coherence) * bd_1-scaffolding-1r7.12: US-010 prune engine _test_requires_source_table helper + tighter bypass + DEC-010 behaviour change * bd_1-scaffolding-1r7.15: US-014 diff renderer arm _SKIP route for singular SQL emission Adds the 3rd of 6 dispatch arms for CandidateTestRowCountAnomalyByPeriod (diff emitter site, per .claude/rules/business-rule-tests.md § 'The 6 production dispatch sites'). Mirrors the custom_sql arm: _render_test returns _SKIP, emit_proposed_test_files surfaces the kept test as a standalone tests/*.sql singular test file. Key design: - _render_test arm in src/signalforge/diff/_emitter.py returns _SKIP for the anomaly variant (between the custom_sql and row_count_between arms). - emit_proposed_test_files extended with keyword-only model / dialect / as_of kwargs. The emitter recompiles the violation query via signalforge.prune.compiler._compile_anomaly_violation_query (US-008 helper) against TableRef.from_model(model) + BIGQUERY_DIALECT default + as_of (defaults: kwarg > decision.as_of > date.today()). - render_diff threads as_of through to the emitter; defaults to date.today() via the per-decision fallback. - Filename uses anchor_to_filename with descriptor 'row_count_anomaly_by_period' (model-level only — no column prefix), hashed via the shared _common.artifact_id seam. - The fail-closed _test_file_writer.write_test_file is variant-agnostic; no changes there. - Pinned by snapshot fixture tests/fixtures/diff/proposed_test_files/anomaly/. Cross-story scaffolding (per US-004 convention): removes CandidateTestRowCountAnomalyByPeriod from site 3 in _VARIANTS_PENDING_DISPATCH_ARMS — the routing test now exercises the variant on the diff emitter site. Site 5 (drafter anchor) remains pending in US-006. Plan: plans/super/171-row-count-anomaly.md § US-014 + Phase 1 B.7 (locked: singular SQL only — no dbt-ext macro form for this primitive). Tests: - tests/diff/test_emitter.py: 11 new tests covering _SKIP routing, basic path/marker, body == violation_query, args_hash filename, dropped-decision exclusion, as_of resolution (decision vs kwarg vs today), fail-loud-without-model, custom_sql co-emission, snapshot fixture pin. - tests/test_candidate_test_dispatch_exhaustiveness.py: site 3 pending set emptied; 53 dispatch tests pass. Validation: ruff check / ruff format --check / pyright / pytest all green (3159 passed; coverage 97.72%). * bd_1-scaffolding-1r7.14: US-013 CLI --as-of flag plumbing on generate + prune-existing + 5-surface parity Add --as-of YYYY-MM-DD flag to both 'signalforge generate' and 'signalforge prune-existing' per #171 DEC-001. type=date.fromisoformat parses strict ISO; bad format raises argparse SystemExit(2) which maps cleanly to tier-2 input-validation. Default None lets prune_tests resolve to date.today() at prune time (the resolution belongs in the engine, NOT the CLI, so PruneEvent.as_of records the same resolved value across callers). Thread args.as_of -> prune_tests(as_of=...) in both cmd_generate's _run_single_model AND cmd_prune_existing. In multi-model batch (--select), the same as_of value flows to every per-model call (_run_batch invokes _run_single_model per match with the shared args namespace, so the operator's single --as-of applies uniformly). 5-surface parity per cli-layer.md: 1. argparse help= string (both subcommands) 2. cmd_generate / cmd_prune_existing handler docstrings + add_parser docstrings 3. docs/cli-ops.md Flag reference: bullet under generate, table row under prune-existing 4. test names (per-subcommand) 5. DEC reference: #171 DEC-001 (already in plan) Tests pin: parses cleanly, default None, bad format -> exit 2 + no-traceback floor, threads to engine kwarg, multi-model batch uses one as_of value across all models. test_flag_defaults parametrize grows an entry; help-lists-every-flag grows --as-of. US-010 (engine helper) is in flight in a sibling worktree -- this slice is purely additive on the CLI side; engine.py untouched. * bd_1-scaffolding-1r7.13: US-011 prune engine two-query split + cold-start routing + AnomalyTestStats wiring + DOW degrade Replaces the defensive 'two-query split not yet wired' arm in signalforge.prune.engine with real handling of the (stats_sql, violation_sql) tuple compiler arm landed by US-008 for the row_count_anomaly_by_period variant. Engine wiring per #171 DEC-001 / DEC-003 / DEC-005 / DEC-006 / DEC-008: 1. Stats query (Query 1) runs via adapter.run_stats_query — a new vendor-neutral seam on WarehouseAdapter (ABC default raises StatsQueryNotSupportedError for adapters that have not grown the primitive; BigQuery overrides). Result rows parse into the typed AnomalyTestStats discriminated-union member matching test.method via _parse_anomaly_stats (handles all four methods + seasonal per-DOW shape). 2. Cold-start gate: stats.n_periods < min_samples_per_bucket routes to kept-without-evidence with structured why ('insufficient history: N/M periods'). Query 2 (violation) is SKIPPED — no warehouse call. stats populated on decision + audit. 3. DOW degrade: when seasonality=dow AND any per-DOW bucket below floor, recompile the stats query with seasonality=none, emit ONE operator-actionable WARNING (lazy-format JSON), proceed with the non-seasonal stats. Violation SQL is unchanged (today's bucket is today's bucket regardless of seasonality). 4. Violation query (Query 2) runs via standard adapter.run_test_sql; PruneDecision carries stats + failures per the standard routing matrix (always-passes, failed-on-known-clean-data, kept). All three anomaly routing paths populate PruneDecision.stats and (via _build_prune_event) PruneEvent.stats — audit-of-record per DEC-006 + DEC-013. Conservative-bias contract preserved: a WarehouseError from either query (incl. StatsQueryNotSupportedError from a no-primitive adapter) routes to kept-without-evidence; the DropReason Literal stays 5-valued (never grow). New seams: - WarehouseAdapter.run_stats_query(sql) -> tuple[dict[str, object], ...] + StatsQueryNotSupportedError (tier 3, ABC default raise pattern) - BigQueryAdapter.run_stats_query override (client.query verbatim with the same map_bq_exception path used by run_test_sql). - _parse_anomaly_stats / _decide_anomaly_cold_start / _any_dow_bucket_thin helpers in signalforge.prune.engine. - _decide_from_test_result + _decide_kept_without_evidence_warehouse_error thread stats + as_of through to PruneDecision. Tests (tests/prune/test_engine.py § '#171 US-011' block): - Happy path: both queries run, stats populated, decision routes. - Cold-start: violation query SKIPPED (pinned by fake adapter's assert_all_expectations_met — no unmet violation expectation). - DOW degrade: thin per-DOW bucket triggers recompile + WARNING + non-seasonal proceed; healthy DOW path skips degrade entirely. - stats populated on PruneEvent audit (happy-path drop + cold-start kept-without-evidence both pinned via JSONL readback). - StatsQueryNotSupportedError routes through standard WarehouseError catch surface. - Defensive 'US-011 pending' arm verified replaced (never fires on the happy path). Two pre-existing US-010 routing tests updated to queue stats + violation expectations now that the engine actually issues them (previously they pinned the defensive arm). Standard validation green (3177 tests, 97.35% coverage). Refs #171 US-011, DEC-001, DEC-003, DEC-005, DEC-006, DEC-008, DEC-010, DEC-013. * bd_1-scaffolding-1r7.16: US-016 docs sweep — README + drafter-catalogue + prune-ops + cli-ops + SKILL.md + CHANGELOG * bd_1-scaffolding-1r7.17: US-017 e2e gated test + unit determinism + inject_model_anomaly_rules helper US-017 adds two test surfaces for the row_count_anomaly_by_period variant and the inject_model_anomaly_rules e2e helper: 1. Unit determinism (tests/prune/test_engine.py): runs prune_tests twice with as_of=date(2026, 5, 1) and asserts PruneEvent.compiled_sql is byte-equal across runs; a third run with as_of=date(2026, 5, 2) asserts the compiled SQL differs (proves the as_of threading from engine to compiler is real). Fresh fake adapter + audit path per run per the expect_query consumption model. Pinned via the typed PruneEvent.model_validate path off the audit JSONL. Traces DEC-001. 2. inject_model_anomaly_rules (tests/cli/_e2e_helpers.py): thin specialisation of inject_model_business_rules — same on-disk mutation surface (config.meta.signalforge.business_rules + meta.signalforge.business_rules in lockstep), distinct name so the anomaly e2e reads self-documentingly. Mirrors the precedent the #169 / #170 e2e helpers set for variant steering via prose rules. 3. E2E gated (tests/cli/test_e2e_row_count_anomaly.py): runs signalforge generate --as-of 2023-04-16 against the Austin bikeshare fixture (the documented ~50% volume drop date per #171 plan §Refinement Q9). Belt-and-suspenders gating per testing-signal.md — @pytest.mark.e2e + @pytest.mark.anthropic + @pytest.mark.bigquery + runtime _skip_reason() for the standard three env vars (SF_RUN_BQ, ANTHROPIC_API_KEY, GOOGLE_CLOUD_PROJECT). tmp_path isolation per testing-signal.md — committed fixture untouched; manifest injection writes only to the per-run copy. Asserts: exit 0; diff sidecar present; structured row_count_anomaly_by_period PruneDecision exists; PruneEvent.as_of == 2023-04-16; AnomalyTestStats populated with valid method + n_periods >= 1; decision='kept' (real anomaly caught); no traceback in stderr. Traces DEC-001, DEC-003, DEC-008, DEC-013. Validation: full canonical command passes (ruff check / ruff format check / pyright / pytest); 3205 passed, 6 intentional skips, 96 gated-deselected; coverage 97.36%. Per-task constraints honoured: tests only — no src/ changes; no dispatch-arm work (all six sites already wired); engineered determinism (unit fake stats + hand-picked anomaly date) per testing-signal.md. * bd_1-scaffolding-1r7.19: US-019 Patterns & Memory — update rules + new memories * #171: address CR/Copilot findings #1, #2, #5, #6, #7, #10 - #1, #2 (CodeRabbit MD018): prefix #171 → 'Issue #171' in rule files so markdownlint doesn't parse the bare hash as an ATX heading - #5 (CodeRabbit flake): bracket date.today() resolution within [today_before, today_after] in the as_of-resolves-to-today test to survive midnight crossings - #6, #7 (Copilot doc lies): correct 'IQR multiplier' / 'Tukey-style IQR band' wording for percentile method — actual implementation is half-band width in percentile points (p_lo = threshold/100) - #10 (Copilot CRITICAL): _parse_anomaly_stats now normalises raw dialect-emitted DOW integers to POSIX (Mon=0..Sun=6) via the new _normalize_dow_to_posix(raw, dialect) helper. Without this, BQ's DAYOFWEEK (1..7, Sun=1) and Snowflake's DOW (0..6, Sun=0) would produce mutually-incompatible per_dow dict keys, breaking any consumer that does dict.get(date.weekday()) Drafter _PROMPT_VERSION rotated to c11a73cc95b31614 (the percentile wording fix touched _SYSTEM_PROMPT). Validation: 3205 passed, 0 ruff / pyright errors. Pending fixes from same review pass (next commit): - #3 (CodeRabbit confirmation reply on date_column validation) - #4 (period boundary for week/hour as_of) - #8, #9 (Copilot CRITICAL: emitted singular test SQL needs full band-check shape, not just violation query) * #171: address CR/Copilot findings #4, #8, #9 (singular-test SQL + period boundary) - #4 (CodeRabbit): _render_as_of_literal wraps the as_of date literal in DATE_TRUNC(<lit>, <unit>) for period in {week, hour} so the today-window aligns to the natural period boundary. period=day is a no-op (date literal is already at day-boundary 00:00:00); existing day-period snapshots stay byte-equal. Documented hour-period limit: DATE_TRUNC of a DATE is dialect-divergent for HOUR. - #8, #9 (Copilot CRITICAL): new _compile_anomaly_singular_test_sql helper emits the FULL band-check SQL — history + per-method stats CTEs + today CTE + band-violation WHERE predicate. Returns 0 rows when in-band and >=1 row only when out-of-band (the correct dbt singular-test contract). The prior emitted SQL was just _compile_anomaly_violation_query, which returns ALL rows in the as_of period and fails the dbt test on every non-empty day. Per-method band predicates (all 4 covered): * mad: ABS(0.6745 * (today.cnt - stats.median)) > k * NULLIF(stats.mad, 0) * zscore: ABS(today.cnt - stats.mean) > k * NULLIF(stats.stddev, 0) * percentile: today.cnt < stats.p_lo OR today.cnt > stats.p_hi * min_max: today.cnt < stats.min_cnt OR today.cnt > stats.max_cnt Seasonal (seasonality=dow): today CTE projects MAX(EXTRACT(DOW)) too; JOIN against per-DOW stats so band check fires only for today's DOW. Diff emitter wired to call the new helper instead of the engine-side violation query. Snapshot fixture updated. 13 new tests covering the per-method predicates, seasonal JOIN, period truncation, where-clause symmetry, min_max threshold-ignored, percentile half-band-width semantics, Snowflake dialect fragment usage. Validation: 3218 passed, 0 ruff / pyright errors. Pending: #3 (CodeRabbit confirmation reply — just a comment), summary comment, resolve all 10 review threads. * #171: address CR follow-up findings #11, #12, #13 (closeout round 2) CodeRabbit re-reviewed my first closeout commits and caught 3 real issues with the new singular-test SQL helper + emitter wiring. - #13 (CRITICAL: zero-row seasonal silent-pass): the today CTE computed dow via MAX(EXTRACT(... FROM date_column)) which is NULL on an empty period; the downstream stats.dow = today.dow JOIN then drops every row and the test passes silently — even though a zero-count period IS itself a meaningful anomaly (catastrophic load failure). Fix: derive dow from the anchored as_of LITERAL (compile-time constant per emitted SQL), so the band check fires correctly even when COUNT(*) = 0. Pinned by new regression test test_singular_test_sql_seasonal_dow_is_stable_for_empty_today_period. - #12 (CRITICAL: invalid SQL for period=hour): DATE_TRUNC(DATE, HOUR) is rejected by BigQuery (DATE_TRUNC of DATE only accepts year/month/week/day; HOUR requires DATETIME/TIMESTAMP). With a date-typed as_of the period=hour case cannot emit valid SQL. Fix: return _InvalidIdentifier at compile entry → engine routes to kept-without-evidence per conservative-bias. v0.x ships day/week only; hour-period support is deferred until as_of becomes a datetime. Pinned by test_compile_row_count_anomaly_period_hour_returns_invalid_identifier. - #11 (emitter bypassed compiler safety checks): the diff emitter called _compile_anomaly_singular_test_sql() directly, skipping the validate_identifier(date_column) + validate_test_sql(composed) gates that the engine-side _compile_row_count_anomaly_by_period runs. A kept anomaly with a hostile where clause could land in operator- shipped dbt SQL. Fix: re-run both safety gates in the emitter before writing; on failure, skip emission silently (the engine separately routes the case to kept-without-evidence). Pinned by test_emit_proposed_test_files_anomaly_skips_hostile_where_clause. Validation: 3221 passed, 0 ruff / pyright errors.
Phase A binary (weekly_query_cost.sql) + Phase B aggregate (15 candidates) against the substrate built for the 2026-05-30 baseline. #170 ships well- calibrated (100% match-rate, beats projection); #169 misses the projection by 18.9 pp on a 14-model sample with a named root cause (missing _ROW_COUNT_BETWEEN_SCOPE_INSTRUCTION in prompts.py per DEC-012); #171 has a drafter-calibration bug visible on every model with audit-timestamp columns. One additional ops-class finding: 170-col model blows the safety-layer audit-record size cap. Three follow-ons named for separate filing; Phase C (#171 calibration retest) deferred per epic option 3 — needs maintainer-controlled Snowflake fixture warehouse.
* #186: super-plan for grade-layer asyncio refactor 24 DECs covering provider scope (all 3), seam shape (sibling async), concurrency cap default (10), provider async capability flag, ExceptionGroup defence, CancelledError budget attribution, nested-loop guard, fail-loud fallback on sync-only providers, fsync executor wrap, JSONL arrival-order + test-side sort helper, Linux PIPE_BUF assumption, cache-cost penalty acceptance, AST scan extensions (10 -> 12). 15 stories: 13 implementation + Quality Gate + Patterns & Memory. Dependency graph allows parallel landing of the three per-vendor shims (US-003 / US-004 / US-005) and downstream test stories (US-010 / US-011 / US-012). * #186: bump plan phase to published; link PR #190 * #186: devolve plan into beads (bd_1-scaffolding-v8j epic + 15 tasks); phase=devolved * #186: US-001 pytest-asyncio dev-dep + asyncio marker + _async_sleep alias Add pytest-asyncio>=0.23,<1 to both dependency groups (dual-listing per python-build.md), set asyncio_mode = 'strict', register the asyncio marker, and declare _async_sleep = asyncio.sleep at module scope in signalforge.llm.client (mirrors _sleep / _rand_uniform per llm-drafter.md DEC-004). Two smoke tests pin the identity alias and prove the marker collects + runs an async test body under strict mode. Closes bd_1-scaffolding-v8j.1 * #186: US-002 LLMProvider ABC additions + LLMProviderAsyncUnsupportedError Extends the provider-neutral LLM seam with the async surface declarations needed by the grade-layer asyncio refactor: supports_async ClassVar (default True), abstract make_async_client(), _LLMAsyncMessagesProtocol + _LLMAsyncClientProtocol, and LLMProviderAsyncUnsupportedError mapped at CLI tier 3. Concrete providers (Anthropic/OpenAI/Gemini) carry temporary NotImplementedError stubs pending US-003/004/005. Closes bd_1-scaffolding-v8j.2 * #186: US-007 Fake clients gain dual sync+async surface Add an .aio namespace to FakeAnthropicClient / FakeOpenAIClient / FakeGeminiClient exposing async create / count_tokens methods that drain the SAME _create_queue / _count_queue as the existing sync surfaces. Tests can mix sync and async drains against one fake instance, proving the production sync (call_llm) and async (call_llm_async, US-006) paths share identical expectation logic per DEC-012's one-queue-per-kind rule. Closes bd_1-scaffolding-v8j.7 * #186: US-005 Gemini async adapter (.aio passthrough; no new AST scan) Add _GeminiAsyncMessagesAdapter / _GeminiAsyncClientAdapter alongside the sync siblings in providers.py; both forward through the SDK's .aio.models namespace on the same bare genai.Client. GeminiProvider.make_async_client reuses the existing _make_gemini_client factory — Scan 10 stays green by construction (no new vendor constructor). Closes bd_1-scaffolding-v8j.5 * #186: US-003 Anthropic async shim + AnthropicProvider.make_async_client + AST Scan 3b Extend signalforge.llm._anthropic_client with the async surface: _AnthropicAsyncMessagesProtocol, AsyncAnthropicClientProtocol (@runtime_checkable), and _make_anthropic_async_client(api_key) (lazy AsyncAnthropic import). Replace the NotImplementedError stub in AnthropicProvider.make_async_client with the real implementation delegating to the shim factory. AnthropicProvider.supports_async stays True (set by US-002). Add AST Scan 3b to tests/test_audit_completeness.py pinning anthropic.AsyncAnthropic(...) construction to _anthropic_client.py. Reuses _AttributeCallFinder and ships the planted-violation regression test exercising all four bypass patterns (bare / import-alias / module-attribute / late-import alias), mirroring Scan 9's coverage. Total project AST scans: 10 -> 11. Extend tests/llm/test_client_shim.py with parallel coverage of the async shim: factory returns a protocol-satisfying object, the shared FakeAnthropicClient satisfies AsyncAnthropicClientProtocol via its .aio namespace (DEC-012 of #186), AnthropicProvider.make_async_client returns an async client, and the regex-floor no-construction-outside- shim check for anthropic.AsyncAnthropic(. Closes bd_1-scaffolding-v8j.3 * #186: US-008 GradeConfig.max_concurrent_calls + GradeNestedEventLoopError + entry guards Adds the asyncio concurrency-cap knob and two pre-flight guards that fire at grade_artifacts sync entry — before any iterator / asyncio.run. - GradeConfig.max_concurrent_calls: int = 10 (DEC-003), placed after total_budget_seconds; field validator rejects v<1 or v>100 with "must be in the closed interval [1, 100]". - GradeNestedEventLoopError(GradeError) — typed remediation-bearing signal when grade_artifacts is invoked from inside a running asyncio loop (DEC-009); CLI tier 1 (operator call-site / environment error). - Two guards wired in engine.py between the envelope-breach scan and run-wide derived values: 1. asyncio.get_running_loop() succeeds -> raise GradeNestedEventLoopError 2. provider_for(config.provider).supports_async is False AND config.max_concurrent_calls > 1 -> raise LLMProviderAsyncUnsupportedError (DEC-006, fail loud, no clamp) - Tests pin both directions (guards fire + escape hatches don't). Closes bd_1-scaffolding-v8j.8 * #186: US-004 OpenAI async shim + OpenAIProvider.make_async_client + AST Scan 9b Extend signalforge.llm._openai_client with the async surface: _OpenAIAsyncMessagesProtocol, AsyncOpenAIClientProtocol (@runtime_checkable), _OpenAIAsyncClientAdapter (the .messages.create awaitable forwards to 'await self._raw.chat.completions.create(**kw)'; count_tokens raises NotImplementedError defensively per the supports_token_count=False capability flag), and _make_openai_async_client(api_key) (lazy AsyncOpenAI import). Replace the NotImplementedError stub in OpenAIProvider.make_async_client with the real implementation delegating to the shim factory. OpenAIProvider.supports_async stays True (set by US-002). Add AST Scan 9b to tests/test_audit_completeness.py pinning openai.AsyncOpenAI(...) construction to _openai_client.py. Reuses _AttributeCallFinder and the existing _LLM_OPENAI_EXCLUSIONS set (same shim file hosts both sync + async factories). Ships the planted-violation regression test exercising all four bypass patterns (bare / import-alias / module-attribute / late-import alias), mirroring Scan 3b / Scan 9 / Scan 10 coverage. Total project AST scans: 11 -> 12. Extend tests/llm/test_openai_client_confinement.py with parallel coverage of the async shim: factory returns a protocol-satisfying object, the shared FakeOpenAIClient satisfies AsyncOpenAIClientProtocol via its .aio namespace (DEC-012 of #186), OpenAIProvider.make_async_client returns an async client (env-var gated since the OpenAI SDK enforces credentials at construction), and the async adapter's messages.create delegates to chat.completions.create preserving the JSON-mode response_format kwarg (DEC-006 of #136). Closes bd_1-scaffolding-v8j.4 * #186: US-006 call_llm_async — async sibling orchestrator Sibling of signalforge.llm.client.call_llm with exact signature parity: identical keyword-only surface, same provider strategy dispatch, same retry taxonomy + per-class budgets, same lazy-format JSON WARNING/INFO emission. Backoff awaits _async_sleep(...) instead of _sleep(...); messages.count_tokens and messages.create are both awaited; everything else mirrors call_llm byte-for-byte. Resolves strategy via provider_for(name); raises LLMProviderAsyncUnsupportedError at orchestrator entry if strategy.supports_async is False (BEFORE any client construction); builds client via strategy.make_async_client() (sync factory returning an async-capable client) when client is None. Factors two shared helpers — _map_count_tokens_exception (count-probe exception taxonomy) and _build_result_from_response (finish-reason gate, text/usage extraction, dual-zero cache-anomaly WARNING, LLMResult assembly) — so the sync and async paths stay byte-identical by construction. The sync call_llm path is byte-equivalent to its prior shape (the existing test_client.py / test_client_retries.py suites all pass unchanged). Extends FakeNoCacheProvider with an async make_async_client returning a new _FakeNoCacheAsyncClient (awaitable canned-response surface) and adds FakeSyncOnlyNoCacheProvider — a sync-only subclass with supports_async = False that drives the US-006 capability-gate test. Subclass (not instance flag) because the ABC declares supports_async as ClassVar. tests/llm/test_call_llm_async.py covers: - signature parity vs call_llm (parameter names/kinds/defaults/annotations) - happy path (LLMResult assembly via .aio.messages.create) - 429/5xx/conn retry-then-success - 429/5xx/conn exhausted (proper typed raises + per-class WARNING shape) - _async_sleep alias receives the expected backoff math - sync-only provider (supports_async=False) short-circuits at entry - async no-cache provider drives the make_async_client end-to-end Validation: ruff/ruff-format/pyright/pytest all green (3278 passed, 97.18% coverage). Logger grep gate green (all new log sites use lazy- format JSON). The grep-gate AST test_no_async_anthropic_construction_outside_shim gate stays green by avoiding the literal AsyncAnthropic constructor string in client.py docstrings. Closes bd_1-scaffolding-v8j.6 * #186: US-009 Grade engine asyncio refactor (TaskGroup + Semaphore + budget timeout) Replace the sequential (criterion, artifact) loop in grade_artifacts with an async core orchestrated by asyncio.TaskGroup + asyncio.Semaphore + asyncio.timeout. Public grade_artifacts stays sync; internally invokes asyncio.run(_grade_artifacts_async_core(...)). Each (artifact, criterion) pair runs as a coroutine _grade_one_async that awaits call_llm_async. The per-coroutine try/except catches GradeLLMError / GradeOutputError / GradePromptEnvelopeBreachError and degrades the pair without aborting siblings (DEC-004 retry isolation). CancelledError attribution via the orchestrator-scope _budget_exceeded flag (DEC-008) routes budget-trip cancellation to the locked 'grade budget exceeded (Ns) before evaluation' reasoning. Audit writes happen via loop.run_in_executor(None, ...) so the synchronous fsync doesn't block the event loop (DEC-017). The 6th AST scan (single GradeEvent construction seam) is preserved verbatim — every event still flows through _build_grade_event in signalforge.grade.audit. Budget-trip WARNING field set is locked per DEC-018: {run_id, model_unique_id, completed_count, cancelled_count, degraded_count, total_budget_seconds}. Pinned by test_grade_artifacts_concurrent_budget_warning_shape_locked. Three new tests cover the async-core acceptance criteria: - test_grade_artifacts_concurrent_dispatches_in_parallel — instrumented fake records peak in-flight count; asserts <= max_concurrent_calls=3 - test_grade_artifacts_concurrency_1_byte_equivalent_to_v0_1 — semaphore of 1 preserves JSONL iteration ordering - test_grade_artifacts_concurrent_budget_warning_shape_locked — locks the DEC-018 JSON field set Updated existing budget tests to drive cancellation via a slow monkey-patched _grade_one_async (replaces the old time.monotonic monkey-patch which no longer fits the asyncio.timeout shape). Closes bd_1-scaffolding-v8j.9 * #186: US-010 ExceptionGroup renderer in format_error_to_stderr (defence-in-depth) Add an isinstance(exc, ExceptionGroup) branch to signalforge.cli._helpers.format_error_to_stderr that renders the group as the existing multi-bullet stderr shape: header naming the concurrent- failure count + " - <ExcClass>: <repr(msg)>" bullets, one per inner exception, capped at 10 with a " ... and K more" overflow line. Each inner exception's text routes through repr() so ANSI / control bytes cannot inject terminal-control sequences (defence-in-depth — the print_stderr sink also strips ANSI per #60). This is the CLI-side half of DEC-007 of #186. The grade engine's inner BaseExceptionGroup unwrap re-raises single-exception groups as the inner typed exception (so callers can pattern-match); multi- exception groups bubble unchanged to cmd_<name>'s single try/except Exception boundary. Before US-010, a hostile non-grade-typed exception (e.g. KeyError from buggy worker code) escaping the per-coroutine catch list would leak a default ExceptionGroup repr plus a Python traceback to stderr — the failure mode this branch closes. Tests: - test_exception_group_renders_as_multi_bullet — 3-exception group produces header + 3 bullets, no traceback leak. - test_exception_group_caps_at_10_bullets_with_overflow — 12-exception group renders header + 10 bullets + " ... and 2 more". - test_exception_group_repr_quotes_ansi_and_control_bytes_in_messages — raw \x1b CSI byte does not survive the renderer. - test_exception_group_single_inner_renders_singular_header — pins singular/plural grammar (1 concurrent failure / N concurrent failures). - test_non_group_exceptions_render_unchanged — regression guard for the default single-line ERROR shape. - test_grade_artifacts_hostile_coroutine_no_traceback — drives the engine with a hostile _grade_one_async stub that raises KeyError; asserts format_error_to_stderr of the raised exception contains no "Traceback" substring (the DEC-016 floor invariant). Validation: ruff/format/pyright/pytest all green (3288 tests, 97.28% coverage). Closes bd_1-scaffolding-v8j.10 * #186: US-012 Three per-provider live async smoke tests Add tests/cli/test_e2e_{anthropic,openai,gemini}_async_smoke.py exercising the concurrent grade-dispatch path (grade.max_concurrent_calls=3) against each real provider. Each gated identically to the sync sibling (@pytest.mark.e2e + @pytest.mark.<vendor> + env-var skip helper). Each asserts: (a) signalforge generate exits 0, (b) audit JSONL has expected pair count (read-time sorted by (artifact_id, criterion_id) so the assertion is dispatch-order independent per DEC-015), (c) no 'Traceback' in stderr. Extend tests/cli/_e2e_helpers.py::apply_provider_override with an additive grade_max_concurrent_calls: int | None = None kwarg (None default preserves byte-equality for every existing caller). Update CONTRIBUTING.md § 'Tests in the live e2e suite' to enumerate the three new smokes alongside the five existing ones (5 -> 8 paid e2e tests total). Update tests/test_contributing_e2e_enumeration_parity.py _PAID_E2E_FILES tuple to include the three new basenames. Validation: 3282 passed, 6 skipped, 99 deselected, coverage 97.27%. Closes bd_1-scaffolding-v8j.12 * #186: US-011 JSONL sort helper + retry-isolation test + budget-cancellation test Add tests/grade/_helpers.py::_sort_grade_events keyed by (artifact_id, criterion_id) — restores deterministic ordering to the grade-audit JSONL after the asyncio refactor (US-009) made on-disk arrival order non-deterministic. Per DEC-015 the orchestrator does NOT sort before writing (that would buffer and break per-decision fail-closed durability); tests sort after the fact via this helper when they need to snapshot-compare against an iteration-order baseline. Pin the helper's two load-bearing properties via tests/grade/test_helpers.py: idempotent (calling twice == once) and stable for ties (Python's sorted preserves input order for records sharing the sort key). Plus missing-key defensiveness so a malformed JSONL line doesn't take down every snapshot test via KeyError; on an empty list returns an empty list; does not mutate its input. Document the order-agnostic invariant on the v8j.9 retry-isolation test (test_grade_artifacts_one_criterion_retry_exhausted_does_not_fail_whole_report). Under concurrent dispatch the fake's FIFO matching means whichever pair's coroutine wins the dispatch race consumes the rate-limit expectation; the assertion is count-based (one degraded, N-1 scored) — invariant to which specific pair degrades — so the test is deterministic regardless of dispatch order. This is the pair-identity contract DEC-015 lands at the operator-visible level. The v8j.9 budget-warning test (test_grade_artifacts_concurrent_budget_warning_shape_locked) already pins the DEC-018 WARNING field set verbatim (run_id, model_unique_id, completed_count, cancelled_count, degraded_count, total_budget_seconds); no further changes needed there. Kept v8j.9's _stub_grade_one_async_slow mechanism (monkey-patches _grade_one_async to await asyncio.sleep(60) > total_budget_seconds=1) rather than refactoring to _async_sleep injection. The plan suggested the alias-injection mechanism; v8j.9's slow-coroutine mechanism works identically (asyncio.timeout cancels the slow coroutine, CancelledError routes via the _budget_exceeded flag) and is already pinned by three tests. Working > textbook; the _async_sleep alias remains in the engine module as a forward-compat seam pinned by test_grade_artifacts_module_level_async_sleep_alias_present. Closes bd_1-scaffolding-v8j.11 * #186: US-013 Docs + rule files + CHANGELOG Document the grade-layer asyncio orchestrator across four surfaces: - .claude/rules/grade-layer.md graduates the 'One LLM call per (artifact x criterion); sequential' section to 'parallel via asyncio (DEC-004 of #6, graduated by #186)' with TaskGroup + Semaphore + asyncio.timeout shape, ExceptionGroup defence, Linux PIPE_BUF assumption, cache-cost penalty, supports_async capability flag. - .claude/rules/llm-drafter.md extends the _sleep/_rand_uniform aliases section with _async_sleep sibling + call_llm_async surface + AST scan count 10 -> 12. - docs/grade-ops.md adds max_concurrent_calls to the config block + new 'Concurrency (asyncio orchestrator)' section covering typed errors, cost expectations, Linux atomicity assumption, JSONL arrival ordering. - CHANGELOG.md [Unreleased] gets an Added entry for the asyncio orchestrator + Changed entries for arrival-order JSONL, cache cost penalty, pytest 9->8 pin. - Plan doc phase marker bumped to 'implemented'. Closes bd_1-scaffolding-v8j.13 * #186: US-014 Quality Gate fixes (4-angle code review triage) Four parallel reviewers reported. Triage: BLOCKERS fixed: - Pass 1 (correctness): Race condition on `asyncio.run_in_executor` audit await. When the budget timeout fires DURING the audit write, the executor thread durably writes the record but the coroutine raises CancelledError BEFORE `results_by_index[index] = grading_result` runs. Synthesis pass then writes a SECOND audit record for the same pair (violates one-record- per-pair invariant + breaks counter accounting). Fix: assign the slot BEFORE the audit await; wrap the executor call in `asyncio.shield` so the in-memory state precedes durable state and the executor thread is not cancelled mid-flight. - Pass 4 (docs): "~5-6× speedup on a typical 30-pair model" was internally inconsistent with the plan's "~280s sequential, ~30s concurrent" — typical model is ~280 pairs (~70 artifacts × 4 criteria), speedup is ~9× (close to Amdahl ceiling at concurrency=10). "50× headroom under concurrent dispatch" was wrong; should be ~10×. TRIANGULATED (Pass 1 + Pass 3): The orchestrator's `_budget_exceeded` flag was set in `except TimeoutError` AFTER `TaskGroup`'s `__aexit__` returned — but cancelled children's `except CancelledError` arms fire BEFORE that, so the flag was always False when checked there. The `if budget_state['exceeded']` branch and `counters['cancelled']` were dead code. Fix: remove the `except CancelledError` arm; the synthesis pass is the single source of truth for un-completed pairs. Drop `cancelled_count` from the WARNING JSON (DEC-018 simplifies to `{run_id, model_unique_id, completed_count, degraded_count, total_budget_seconds}`). The simpler invariant `completed_count + degraded_count == total_pairs` holds by construction. HIGH (Pass 1): Sync-only provider + max_concurrent_calls=1 silently degraded every pair (not an escape hatch — the engine consumes call_llm_async exclusively). Drop the `> 1` predicate from the entry guard; raise LLMProviderAsyncUnsupportedError regardless of cap. Remediation rewritten: pick an async-capable provider. MUST-FIX (Pass 3): The three e2e async smokes inlined their own copy of `_sort_grade_events` to dodge cross-dir imports before US-011 landed. US-011 has now landed (`tests/grade/_helpers.py::_sort_grade_events`). Replace the inline copies with a top-level import. NICE-TO-FIX (Pass 3): Drop the `if isinstance(excinfo.value, ExceptionGroup)` conditional in the hostile-coroutine no-traceback test; assert unconditionally so a regression that unwrapped multi-exception groups silently can't pass. Pass 2 (conventions) and Pass 4 minor concern on OpenAI/Gemini cost prose: addressed in the same fixes (docs prose tightening). Validation green on Python 3.11 floor + 3.13 ceiling: - ruff check / format / pyright: clean - pytest: 3294 passed, 6 skipped, 99 deselected - coverage: 94% on engine.py + client.py (project floor 80%) * #186: US-015 Patterns & Memory Rule updates: - .claude/rules/testing-signal.md § "AST single-construction-seam scans": add the post-#186 "total project AST scans: 12" note + the graduation rule (one new scan per new vendor SDK class name, NOT per async-graduating vendor; Gemini's .aio namespace on existing Client needed no new scan). - .claude/rules/testing-signal.md § "Engineered determinism": add the dispatch-order-agnostic assertions section covering _sort_grade_events and pair-identity predicates — the two patterns future asyncio graduations (prune et al.) should copy. Three new memory entries at /home/wesd/.claude/projects/-home-wesd-Projects-SignalForge/memory/: - signalforge-asyncio-orchestrator-pattern — TaskGroup + Semaphore + asyncio.timeout shape, plus three QG-discovered traps (dead _budget_exceeded flag, run_in_executor cancellation-during-audit-write duplicate record, sync make_async_client). - signalforge-dual-sync-async-fake-pattern — one class per vendor with both sync .messages and async .aio.messages surfaces draining the same expect_* queue. - signalforge-async-seam-confinement — AST scan growth rule (per new SDK class name, not per vendor), docstring regex-floor trap. - MEMORY.md updated with three one-line index entries. These memories anchor #186's lessons for the eventual prune asyncio graduation (prune-engine.md DEC-028) and any future per-(X × Y) concurrent stage. Closes bd_1-scaffolding-v8j.15 * #186: Address PR review feedback (CodeRabbit + Copilot) 10 unresolved review threads, triage: CodeRabbit (5): 1. engine.py:688 (Major) — shielded run_in_executor swallowed GradeAuditWriteError when cancelled mid-await. Fix: capture the executor future; on CancelledError, await the future to surface the writer's exception. Preserves fail-closed guarantee under concurrent cancellation. 2. plan markdown table (Minor) — unescaped `|` inside backtick-quoted code in a table cell broke MD column count. Escape as \| . 3. test_call_llm_async.py provider-registry teardown (Minor) — save prior registration; restore-or-pop in finally. Prevents order-dependent failure if another suite registered the name first. 4. test_client_shim.py async-delegation test (Major) — was calling real `anthropic.AsyncAnthropic` constructor; SDK may reject without credentials. Hermetic via monkeypatch of `_make_anthropic_async_client` returning a sentinel, mirroring the sync sibling test. Copilot (5) — all in the same cluster: my QG fix that tightened the sync-only-provider guard (dropping the `max_concurrent_calls > 1` predicate) left stale docstrings claiming the old gate: - src/signalforge/grade/config.py GradeConfig docstring - src/signalforge/llm/providers.py LLMProvider.supports_async docstring - src/signalforge/cli/_helpers.py exit-code map comment - docs/grade-ops.md typed-errors section (LLMProviderAsyncUnsupportedError) - docs/grade-ops.md "Linux atomicity assumption" PIPE_BUF claim: PIPE_BUF is a pipe/FIFO concept; the original wording implied a POSIX guarantee for regular-file writes which doesn't hold. Revised to describe `O_APPEND` per-syscall atomicity + the short-write caveat + the operator escape (`max_concurrent_calls: 1`). Same correction propagated to .claude/rules/grade-layer.md. No false positives — every comment addressed in code. Validation: ruff/format/pyright clean; pytest 3294 passed. ---------
* Add super plan for #183: _ROW_COUNT_BETWEEN_SCOPE_INSTRUCTION Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * #183: mark plan phase published Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * #183: devolve plan to beads (phase=devolved, Beads Manifest) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * SignalForge-jvu.1: add _ROW_COUNT_BETWEEN_SCOPE_INSTRUCTION (#183) Add the missing narrative SCOPE-instruction block for the row_count_between test primitive, mirroring the two existing sibling blocks (_UNIQUE_COMBINATION_SCOPE_INSTRUCTION, _ROW_COUNT_ANOMALY_SCOPE_INSTRUCTION). The primitive itself shipped its catalogue line in #169 but never got a dedicated scope-instruction block. - prompts.py: define _ROW_COUNT_BETWEEN_SCOPE_INSTRUCTION (embeds the DEC-012 worked example), insert {row_count_between_scope} placeholder in primitive order, wire the allowed/scope toggle through _render_system_prompt + the format() call. - test_prompt_cache_stability.py: rotate _EXPECTED_PROMPT_VERSION c11a73cc95b31614 -> e568fb3e4602e465 with a #183 rotation-history entry. _CACHED_BLOCK_GOLDEN untouched (system-prompt edit only). - test_prompts.py: add bounded-aggregation / calibration prose tests, default-render-inclusion test, keep-when-other-types-excluded test, and extend the row_count_between exclude test to assert the scope prose disappears too. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * SignalForge-jvu.2: backfill _UNIQUE_COMBINATION_SCOPE_INSTRUCTION prose tests (#183, #170 parity) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * SignalForge-jvu.3: Quality gate — doc parity for _ROW_COUNT_BETWEEN_SCOPE_INSTRUCTION (#183) 4 code-review passes: plumbing + test-quality + fidelity clean. One finding: docs/draft-ops.md row_count_between section now names the scope-instruction constant for parity with the sibling sections. CodeRabbit skill unavailable in this environment (skipped). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * SignalForge-jvu.4: capture scope-instruction paired-prose-test convention (#183) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * #183: mark plan complete (closeout) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * #183: address PR review — fix dangling line break in draft-ops.md Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* #185: super plan for wide-table audit-record size cap Plans the compress + chunk solution for AuditRecordTooLargeError on 170+ column dbt models (surfaced by #179 retest). 11 stories: 9 implementation (US-001..009) + Quality Gate + Patterns & Memory. Key design locks DEC-001..008: symbol-table-by-reason compression (75% reduction); header-first chunking with deterministic audit_id; drop v3 entirely (library pre-1.0); chunking always-on, no config knob; AuditRecordTooLargeError stays tier 3 with new three-sentence remediation. Refs #185. * #185: mark plan phase devolved (beads manifest) Plan PR #192. Epic bd_1-scaffolding-2i3 + 11 task beads (US-001..009 implementation + US-010 QG + US-011 P&M). US-001 is bd ready; the rest block on their predecessors per the plan's story-dependency graph. * bd_1-scaffolding-2i3.1: AuditEvent v4 shape + chunked-event shape validator Drops v3 redactions tuple; adds redactions_by_reason / column_name_map / audit_id / chunk_index / chunk_count. @model_validator enforces three shape rules (non-chunked / chunk-header / chunk-continuation). __repr__ omits column_name_map per safety-layer.md DEC-022 precedent. RedactionRecord class stays (used by request.py on build path). Refs #185. * bd_1-scaffolding-2i3.2: request.py builds v4 AuditEvent (symbol-table compression) Bumps _AUDIT_SCHEMA_VERSION 3 → 4. build_llm_request folds the per-column RedactionRecord list into redactions_by_reason (sorted tuples, deterministic) + column_name_map. LLMRequest's internal redactions field is unchanged (US-001 left it on LLMRequest). Non-chunked path: audit_id/chunk_index/chunk_count all None — the writer (US-003) decides whether to chunk based on serialised size. Test migration: - test_build_llm_request_audit_carries_schema_version_3 → test_build_llm_request_audit_carries_schema_version_4 (asserts == 4). - Added test_build_llm_request_v4_redactions_by_reason_sorted_deterministic (each tuple sorted; values are tuple, not list). - Added test_build_llm_request_v4_column_name_map_covers_every_hashed_name (every hashed name has a mapback; mapback values are the real names). - Added test_build_llm_request_v4_non_chunked_correlation_triple_all_none (orchestrator emits non-chunked shape; writer owns chunking). - Added autouse _silent_audit_unless_overridden fixture so tests don't hit the v3-shape audit.write (US-003 territory; lands in a parallel bead). Failure-mode tests still override via monkeypatch.setattr. - Marked test_build_llm_request_writes_audit_to_disk_under_default_path as strict xfail with a US-003 unblocker pointer. AST Scan 2 (AuditEvent construction confined to safety.request) still passes. Refs #185. * bd_1-scaffolding-2i3.3: audit.py chunker + multi-line writer + reader helper Adds _chunk_event greedy-fill chunker with deterministic audit_id (blake2b-8 over model_unique_id+timestamp+version). write() pre-open size-checks every chunk, writes header first then continuations with per-chunk fsync inside one Try/finally (Scan 8 invariant preserved). read_audit_events reassembles by audit_id; warns on partial groups. Refs #185. * bd_1-scaffolding-2i3.5: drift detector + fixture regen + v3-test cleanup (v4-only) Surface A — drift detector + fixture (#185 US-005 original scope): StrictAuditEvent (tests/safety/test_drift_detector.py) mirrors US-001's v4 AuditEvent shape with extra="forbid". Every metadata field graduates to `| None`; redactions move from `tuple[RedactionRecord, ...]` to the symbol-table-by-reason `dict[RedactionReason, tuple[str, ...]]` plus the sibling `column_name_map: dict[hashed_name, real_name]`. The chunk-correlation triple (`audit_id` / `chunk_index` / `chunk_count`) lands as three optional fields. A re-implemented `@model_validator(mode="after")` mirrors production's three valid-shape branches verbatim (non-chunked / chunk-header / chunk-continuation). The committed fixture grows from 2 lines to 3 — one per v4 shape. Line 1 is a small non-chunked record carrying a `pattern_match` redaction; line 2 is a chunked header (`chunk_index=0`, `chunk_count=2`, empty redaction maps); line 3 is the matching continuation (`chunk_index=1`, all metadata null, `draft_skip_column_meta` ride). A new `test_audit_event_fixture_exercises_chunked_shape` test pins that the fixture covers all three v4 shapes. regenerate.sh updated to emit the v4 shape; the audit_schema_version history comment carries the 3 → 4 bump rationale. Surface B — v3-test cleanup (expanded scope per the bead): `tests/safety/test_audit.py` — the `_make_event` helper migrated to v4 shape (one source for 16 dependent tests); the two log-line tests now assert `audit_schema_version": 4` and the JSON round-trip asserts the `redactions_by_reason` + `column_name_map` shape instead of the gone `redactions` field. All 16 prior failures pass. `tests/safety/test_request.py` — removed US-002's `xfail(strict=True)` on `test_build_llm_request_writes_audit_to_disk_under_default_path`; US-003's v4-aware writer ships, so the end-to-end disk-write test runs for real. Bug-fix to the test itself: `monkeypatch.setattr` cannot recover the real `audit.write` once the autouse no-op fixture has overwritten the attribute (subsequent `getattr(audit, "write")` returns the replacement). `monkeypatch.undo()` is the load-bearing seam — it rewinds the autouse patch so the original function is restored. Three `policy_flags` assertions gain narrowing `assert ... is not None` guards to satisfy pyright after US-001 graduated the field to `tuple[str, ...] | None`. `tests/_common/test_timestamp.py` — `test_cross_writer_timestamp_byte_parity` migrated to v4 shape (one of the five cross-writer surfaces). Full canonical validation green: uv run ruff check . — All checks passed! uv run ruff format --check . — 315 files formatted uv run pyright — 0 errors uv run pytest --no-cov — 3320 passed, 6 skipped, 99 deselected Refs #185. * bd_1-scaffolding-2i3.4: parametric AuditRecordTooLargeError + three-sentence remediation Adds column_count kwarg; default remediation builds an operator-actionable script naming the column count, suggesting meta.signalforge.skip_draft: true, explicitly closing the issue text's misleading safety.mode: aggregate-only hint, and pointing at the follow-up issue. Stability test pins the verbatim text. audit.py write() passes column_count derived from column_name_map + redactions_by_reason. Tier 3 mapping unchanged. Refs #185. * bd_1-scaffolding-2i3.6: wide-table integration tests (4 tests, new module) test_wide_table.py exercises the v4 compress+chunk path end-to-end: (1) deterministic chunk boundary search; (2) single-line happy path with reader round-trip; (3) chunked path with audit_id correlation + byte-identical reassembly (dict-order-normalised); (4) pathological column-name triggers AuditRecordTooLargeError with no on-disk artefact. _make_wide_model helper builds synthetic manifests in process (no dbt round-trip). The original task description's notional column counts (170 single-line, 500 chunked) rest on the planning-doc compression target. The shipped writer also serialises columns_sent into the header, which materially bloats the per-line size on tag-driven-pii synthetic models. The tests pick the smallest column counts that exhibit each path against the actual implementation (60 single-line; 200 chunked-but-writable), with the chunk boundary pinned to col_count=66 with a ±5 drift tolerance. Refs #185. * bd_1-scaffolding-2i3.7: extend concurrent-write coverage for chunked events Adds test_audit_write_concurrent_threads_mix_small_and_chunked: 10 threads x 50 events alternating small + 170-col chunked. Verifies (a) every line <= 4000 B (PIPE_BUF invariant); (b) all 500 events round-trip through read_audit_events; (c) per-event chunk correlation preserved under concurrent writes; (d) no partial-group WARNINGs emitted (caplog). Refs #185. * bd_1-scaffolding-2i3.8: docs safety-ops.md v4 schema + CHANGELOG [Unreleased] Adds 'Audit JSONL schema (v4)' section to safety-ops.md documenting the three legal v4 record shapes (non-chunked / chunk header / chunk continuation), reader reassembly via read_audit_events, and the real chunking ceilings measured against the shipped _chunk_event: single- line up to ~65 columns, chunked through ~243, AuditRecordTooLargeError beyond that (header chunk's columns_sent tuple overflows). Documents the anti-pattern that 'safety.mode: aggregate-only' does NOT shrink redactions; meta.signalforge.skip_draft: true is the right workaround. Updates the AuditRecordTooLargeError row in the typed-error reference to point at the new section. CHANGELOG [Unreleased] gains: Added — symbol-table compression + chunked records + read_audit_events + AuditRecordTooLargeError.column_count + actionable remediation Changed — audit_schema_version 3 -> 4 (no v3 backward-compat read); AuditEvent.redactions field replaced; remediation rewritten Removed — AuditEvent.redactions: tuple[RedactionRecord, ...] field Plan-design estimate (170-col single-line at 3,865 B) was measured against the redactions-only payload; the full event also carries columns_sent, lowering the practical single-line ceiling to ~65 and the hard ceiling to ~243. Reflected in docs as operator-actionable context; plan document left as a historical record per task brief. Refs #185. * bd_1-scaffolding-2i3.9: safety-layer rule update — v4 chunking + DEC-014 history Adds new § 'Audit chunking (issue #185)' describing the v4 redaction shape (redactions_by_reason + column_name_map), the three legal record shapes (non-chunked / chunk header / chunk continuation), the writer + reader contracts (per-chunk fsync, header-first, partial-group WARNING), and the empirical chunking ceilings (~65 single-line / ~243 hard cap due to columns_sent in header). DEC-011 carries a forward reference to the chunking section. DEC-014 history bumped 3 → 4 with the drop-v3 pre-1.0 simplification note. AuditRecordTooLargeError parametric remediation + tier-3 retention documented. Reference § cross-links the #185 plan + test_wide_table fixture. Refs #185. * bd_1-scaffolding-2i3.10: QG batch fix — column_count, doc parity, reader hardening, coverage backfill QG triangulated findings across 4 reviewer angles (correctness / conventions+parity / tests+coverage / docs+UX): - TRIANGULATED HIGH (Pass 1 + Pass 3): `AuditRecordTooLargeError.column_count` double-counted redacted columns (sum of column_name_map size AND redactions_by_reason values — same columns counted twice by construction). Fix in audit.py: column_count = len(event.column_name_map or {}) — single source of truth; the same columns appear in both surfaces by construction in request.py. Test asserting == 16 for 8 redacted columns corrected to == 8. - TRIANGULATED BLOCKER (Pass 2 + Pass 4): docs/safety-ops.md showed a fabricated remediation example that didn't match the locked code text. Replaced with the verbatim text from test_errors.py. - BLOCKER (Pass 4): docs/safety-ops.md preserved two ## Audit JSONL schema sections — v3 (stale) and v4. Deleted the v3 section entirely (production AuditEvent accepts v4 only post-#185). - HIGH (Pass 4): 5 stale RedactionRecord.column_name references in safety-ops.md replaced with column_name_map. Audit log sensitivity advisory now points at the right field. - HIGH (Pass 4): safety.jsonl filename references in CHANGELOG and the new docs section corrected to audit.jsonl (canonical default path). - HIGH (Pass 4): Dangling 'issue #185 follow-up' reference in locked remediation text softened to 'see the columns_sent roadmap in docs/safety-ops.md' (no follow-up issue filed). Locked test + docs + code + rule all updated in lockstep. - MEDIUM (Pass 4): 'redactions tuple' wording in locked remediation references the gone v3 field — replaced with 'redactions surface' (which the docs already used). Locked test + docs + code + rule in lockstep. - HIGH coverage gap (Pass 3 H1): 5 untested @model_validator rejection branches in models.py — added test_audit_event_non_chunked_missing_metadata_raises, test_audit_event_chunk_index_negative_raises, test_audit_event_chunk_header_missing_metadata_raises, test_audit_event_chunk_header_with_none_redaction_maps_raises, test_audit_event_chunk_continuation_with_none_redaction_maps_raises. - HIGH coverage gap (Pass 3 H2): 3 untested read_audit_events branches in audit.py — added test_read_audit_events_tolerates_empty_lines, test_read_audit_events_skips_chunk_with_missing_audit_id, test_read_audit_events_warns_and_skips_out_of_range_chunk_index. - MEDIUM (Pass 1 #4): Reader silent data loss on out-of-range chunk_index (file corruption / external tampering). Added guard + WARNING + pinned by the out-of-range test above. - MEDIUM (Pass 1 #8): read_audit_events docstring claimed 'does NOT raise' but v3 records raise pydantic.ValidationError. Docstring corrected to reflect the drop-v3 design choice (DEC-005); rule file note added. - MEDIUM (Pass 4): docs/audits.md stale references to audit_schema_version 3 / redactions[] array / RedactionRecord.column_name — updated to v4 shape with cross-reference to safety-ops.md § v4 schema. - MEDIUM (Pass 4): DEBUG log description in safety-ops.md referenced the gone 'redactions' count — updated to 'column_name_map size' plus chunk-correlation triple. 3338 passed (+8 from coverage backfill); coverage 97.34% (up from 97.26%); ruff + pyright clean; mkdocs build green. Refs #185. * #185: address PR #192 review feedback CodeRabbit (2 nitpicks): - docs/safety-ops.md MD040 — added language tag to remediation example fence (``` → ```text) - plans/super/185-wide-table-audit.md MD040 — same fix on the issue-text reproduction fence Copilot (6 actionable findings): - audit.py:_compute_audit_id docstring rewritten to scope the determinism claim correctly: the same AuditEvent VALUE (including timestamp) chunks to the same audit_id; re-running the pipeline correctly mints a new audit_id because timestamp is part of the hash input - audit.py chunker: replaced column_name_map.get(hash, '') with direct indexing column_name_map[hash] at two call sites (the greedy-fit loop + the single-name-over-cap path). Every hashed name in redactions_by_reason has a matching column_name_map entry by construction in request.py; a silent empty-string fallback would corrupt the reviewer mapback contract. KeyError fails loud instead - docs/safety-ops.md audit_id derivation: aligned the inline formula with the actual code, naming the NUL-byte separators (blake2b(model_unique_id + b'\\x00' + timestamp.isoformat() + b'\\x00' + signalforge_version)) - docs/safety-ops.md DEBUG INFO log description: aligned with the actual writer emission (unique_id, mode, columns_sent count, redacted count summed across redactions_by_reason values, audit_schema_version, chunk_count). Previous prose named 'column_name_map size' and a 'chunk-correlation triple' — neither is in the actual payload - tests/safety/test_request.py autouse fixture docstring: removed the outdated v3/v4-migration-in-flight narrative; clarified the real reason for the no-op default is test isolation from disk I/O 3338 passed; ruff + format + pyright + mkdocs build all green. Refs #185, PR #192.
…#193) * Add super plan for #187: faster grade defaults (Haiku + per-provider) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Mark #187 plan published (PR #193) * Devolve #187 plan to beads (epic SignalForge-dpy + 8 tasks) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * SignalForge-dpy.4: US-004 — align DraftConfig.cheap_model to bare SKU claude-haiku-4-5 The dated id claude-haiku-4-5-20251001 is not a key in signalforge.llm.pricing.PRICES (which uses bare SKUs). Change the unused (declared-not-consumed) cheap_model default to the bare SKU so it matches pricing and the new grade default; update the DEC-017 reference comment and field docstring in lockstep. Lockstep consistency fix only — not wired into any code path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * SignalForge-dpy.1: US-001 — PROVIDER_FAST_MODELS + PROVIDER_SKU_PREFIXES constants Add two provider->string mapping constants to signalforge.llm.providers, keyed by the three registered provider names (anthropic/openai/gemini), and export both in __all__: - PROVIDER_FAST_MODELS — cheap/fast judge SKU per provider; every value is an exact key in signalforge.llm.pricing.PRICES (claude-haiku-4-5 / gpt-4o-mini / gemini-2.5-flash) so lookup()/--estimate never raise. - PROVIDER_SKU_PREFIXES — SKU-string prefix per provider. Refactor signalforge.llm.cost._rollup so its SKU-prefix dispatch derives _PROVIDER_PREFIXES by inverting PROVIDER_SKU_PREFIXES instead of duplicating the claude-/gpt-/gemini- literals. Classification behaviour is unchanged; existing rollup tests stay green and a new parametrised test pins the per-SKU classification plus the derived-from-source-of-truth invariant. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * SignalForge-dpy.2: US-002 — GradeConfig per-provider fast-model resolver + compat validator + 1024 cap Make the grade-judge model default a per-provider fast model resolved at config-load, plus a model<->provider compat validator and a raised max_output_tokens cap. - model: str -> str | None = None sentinel. A @model_validator(mode="before") resolves the sentinel to PROVIDER_FAST_MODELS[provider] (frozen-safe; no mutation of the constructed instance). Unknown provider is NOT injected so the existing provider field-validator still raises UnknownProviderError rather than being masked. After construction model is always concrete. - _model_non_empty field-validator now passes None through cleanly (the only None-survival path is the unknown-provider case that raises downstream). - New @model_validator(mode="after") _validate_model_provider_compat rejects a SKU-prefix/provider mismatch (e.g. provider=openai + claude-* model), reusing PROVIDER_SKU_PREFIXES. Only the three known-prefix providers participate; custom/plugin providers (not in the table) may use any model name. - max_output_tokens default 256 -> 1024 (DEC-004) to avoid one-line gemini-2.5-flash grade-JSON truncation. It is a cap, not a target. - Module + class docstrings and the DEC-023..027 locked-defaults list updated. - tests/grade/test_config.py: updated the defaults regression test to the new resolved defaults (haiku, 1024) and added coverage for per-provider resolution, explicit-honour, compat reject/accept, whitespace guard, unknown-provider-not-masked, and the loader round-trips. - grade/engine.py + cli/_estimate.py: read-only assert narrows for the now str|None field (runtime-guaranteed concrete) to keep pyright green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * SignalForge-dpy.3: US-003 — re-baseline estimate goldens + grade fixtures for Haiku default The #187 US-002 grade default flip (anthropic resolves to claude-haiku-4-5, max_output_tokens 256 -> 1024) drifted two estimate fixtures pinned to the old claude-sonnet-4-6 grade default. Re-baseline in lockstep: - tests/fixtures/estimate/anthropic_byte_identity_golden.txt: regenerated. Only diff is the grade-model id (sonnet -> haiku) and the corresponding grade USD figures (per-criterion $0.0292 -> $0.0078, grade $0.1170 -> $0.0312, total $0.1814 -> $0.0956). Verified the draft section, token counts, call counts, artifact counts, and warehouse section are unchanged. - tests/cli/test_estimate_engine.py::test_estimate_total_llm_usd_matches_hand_calculation: the hand calc now keys the grade half on claude-haiku-4-5 pricing ($0.80/MTok input, $4/MTok output) instead of reusing the draft model's pricing. The grade output-token figure stays the fixed _GRADE_OUTPUT_TOKENS_PER_CALL (50) — the 256->1024 max_output_tokens bump is a response cap, not the estimate's per-call output projection. - tests/fixtures/grade/grade_event_v1.jsonl: cosmetic model id sonnet -> haiku for consistency with the new default (drift detector validates shape only; still green). No production source changed. Full suite green: 3329 passed, 8 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * SignalForge-dpy.5: US-005 — Haiku calibration harness + writeup (gated) Build the maintainer-run empirical gate for the #187 Haiku grade default. - tests/research/187-haiku-calibration/_substrate.py: pinned Model + CandidateSchema (engineered-deterministic, spans the rubric calibration space) + curated Sonnet-baseline loader. Artifact-id set derived from the grade engine itself so it can't drift from the formatter. - sonnet_baseline_sample.json: curated 44-verdict Sonnet baseline (11 artifacts x 4 DEFAULT_RUBRIC criteria), NOT the un-committed #179 Phase-B dump; provenance documented. - test_haiku_calibration.py (@pytest.mark.anthropic + runtime skip on ANTHROPIC_API_KEY): re-grades the sample with the resolved Haiku default (claude-haiku-4-5, max_output_tokens=1024), joins to the baseline by (artifact_id, criterion_id), asserts >=85% per-criterion concordance, prints the breakdown for the writeup. - test_gemini_1024_no_truncation.py (@pytest.mark.gemini + runtime skip on SF_RUN_GEMINI/GOOGLE_API_KEY): grades a verbose artifact on gemini-2.5-flash @ 1024 tokens, asserts no score=None truncation degrade (verifies DEC-004). - docs/research/187-haiku-calibration.md: writeup mirroring #179's structure with a clearly-marked "Result (maintainer-filled)" TODO. Gating is belt-and-suspenders (existing anthropic/gemini markers excluded in addopts + runtime skip). No production default changed; src/ untouched; no new marker registered. Default `uv run pytest` stays fully green (3329 passed, gated tests deselected) at 97% coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * SignalForge-dpy.6: US-006 — docs + CHANGELOG + rule lockstep for Haiku grade default Update every non-code surface to agree with the #187 shipped behaviour: grade default model resolves per-provider (anthropic -> claude-haiku-4-5, openai -> gpt-4o-mini, gemini -> gemini-2.5-flash) via PROVIDER_FAST_MODELS; grade max_output_tokens default 256 -> 1024; model<->provider compat validation via PROVIDER_SKU_PREFIXES. - docs/grade-ops.md: model + max_output_tokens defaults; per-provider fast-default cost table (haiku/gpt-4o-mini/gemini-2.5-flash) with pricing.py figures; config example shows model auto-resolve. - docs/llm-providers-ops.md: PROVIDER_FAST_MODELS table + gemini-2.5-flash 1024-cap truncation note; default grader-model matrix row. - docs/draft-ops.md: cheap_model shown as bare SKU claude-haiku-4-5. - .claude/rules/grade-layer.md: locked-defaults section (per-provider sentinel resolution + compat validator). - .claude/rules/llm-drafter.md: PROVIDER_FAST_MODELS / PROVIDER_SKU_PREFIXES source note (reusable by a future draft --cheap). - CHANGELOG.md: Unreleased "Changed" entry. README/CLAUDE.md unchanged (neither enumerates the grade default model; the slim CLAUDE.md delegates API surfaces to CHANGELOG/docs by design). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * SignalForge-dpy.7: Quality gate — fix bugs from 4 code-review passes Real bug (passes 1 & 2): GradeConfig.model could stay None for a registered-but-not-in-PROVIDER_FAST_MODELS provider (the custom/plugin growth path), contradicting the engine/_estimate `assert model is not None`. Now fails loud at config-load requiring an explicit grade.model, restoring the post-construction invariant. Updated the #135 neutrality test contract + the divergent-providers estimate test to supply an explicit model. Accuracy fixes (passes 3 & 4): - Soften the "gemini-2.5-flash JSON is not truncated at 1024" claim across config docstring, grade-ops.md, CHANGELOG — 1024 reduces but doesn't eliminate Gemini truncation at full-fixture scale (floors table wants 4096). - grade-ops.md: gpt-4o example reframed as explicit override (default resolves to gpt-4o-mini). - hand-calc estimate test: pin the resolved literal claude-haiku-4-5 so a resolver regression fails there instead of drifting into a tautology. - calibration gate: reject degraded-dominated runs (comparable >= degraded). - round-trip test docstring + example_config.yml comments corrected (fixture pins explicit overrides, not the new defaults). CodeRabbit: unavailable in this environment (skipped). Full validation green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * SignalForge-dpy.8: Patterns & Memory — capture sentinel-resolver + registry-table-invariant + calibration-gate patterns Distil the durable, generalisable conventions surfaced by epic #187 into the rule files so future work benefits: - .claude/rules/grade-layer.md — new "Reusable conventions distilled from #187" subsection: (1) frozen-config default-from-sibling-field resolves in @model_validator(mode="before") not mode="after" (frozen=True forbids the after-mutation); (2) the load-bearing lesson — a default looked up in a table keyed by a registry-growable field must FAIL LOUD on a registered-but-absent key rather than leak a None/sentinel, citing the #187 QG bug as the cautionary example, generalised for any per-X default table. - .claude/rules/testing-signal.md — two additions cross-referencing the e2e-gated conventions: gated calibration/concordance harness as a research-test pattern (tests/research/187-haiku-calibration/ — pinned baseline + marker + runtime skip + maintainer-run decision gate); and concordance-gate denominator hygiene (exclude degraded score=None pairs but fail loud when degraded dominates: comparable >= degraded). Docs-only; no src/ or tests/ behaviour change. Full suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * #187 calibration: capture a REAL Sonnet baseline from intuit_airflow Replaces the hand-authored US-005 calibration baseline with a genuine one, per request: - capture_sonnet_baseline.py drafts artifacts for the real intuit_airflow model calendar_hour (schema-only, no warehouse), freezes them to real_candidate.json, and grades them with claude-sonnet-4-6. - _substrate.py: build_model() is now the real model (deterministic); build_candidate() loads the frozen drafted artifacts. - sonnet_baseline_sample.json: 80 live Sonnet verdicts (4 degraded excluded). RESULT — the gate FAILS: Haiku concordance 81.8% / 77.0% on two runs, below the 85% DEC-005 bar. Divergence is systematic (Haiku stricter than Sonnet, concentrated on no-redundant + clarity). Per DEC-005 this points to shipping Haiku as opt-in, NOT the default. Full analysis in docs/research/187-haiku-calibration.md § Result / Disposition. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * #187: keep Sonnet the grade default, Haiku opt-in (calibration result) The empirical calibration (real Sonnet baseline from intuit_airflow) found claude-haiku-4-5 grades stricter than Sonnet (~77-82% concordance, below the 85% DEC-005 bar). Per the decision rule, Haiku does NOT ship as the default. - Rename PROVIDER_FAST_MODELS -> PROVIDER_DEFAULT_MODELS (Sonnet isn't "fast"); anthropic -> claude-sonnet-4-6 (was claude-haiku-4-5). OpenAI/Gemini fast defaults unchanged (gpt-4o-mini / gemini-2.5-flash). - GradeConfig() now resolves to claude-sonnet-4-6; claude-haiku-4-5 is the documented opt-in (grade.model: claude-haiku-4-5). - Revert estimate goldens + grade_event fixture + hand-calc to Sonnet (estimate is independent of max_output_tokens, so origin/dev values apply). - Kept: the per-provider resolver, model<->provider compat validator, the 1024 max_output_tokens cap (still justified by the gemini-2.5-flash default). - Calibration gate now selects Haiku explicitly and still asserts >=85% (fails) as the durable record that Haiku is the stricter opt-in. - Docs/rules/CHANGELOG/writeup updated across all surfaces. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * #187: record Gemini 1024-token check PASS (run live) gemini-2.5-flash graded a verbose artifact at the new max_output_tokens=1024 default with no score=None truncation degrade — DEC-004 holds at the single-artifact scale (full-fixture runs may still want 4096 per #158). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * #187: address PR review (CodeRabbit + Copilot) - test_haiku_calibration: lazy _substrate import inside the test (no import-time sys.path mutation when deselected) — Copilot. - docstrings: 'never collects' -> 'deselected'; 'Haiku default' -> 'Haiku opt-in' across the calibration tests — Copilot. - llm-providers-ops: drafter row now honest (no per-provider drafter default; switching provider requires llm.model) with a footnote — Copilot. - plan doc: ```text fence on the dependency graph + blank line before the SKU table (markdownlint) — CodeRabbit; Phase -> complete.
* #184: super plan for drafter anomaly-scope fix * #184: update plan phase to published + link PR #194 * #184: phase devolved + beads manifest * #184 US-001: rewrite _ROW_COUNT_ANOMALY_SCOPE_INSTRUCTION to teach model-level scope * #184 US-002: add ReshapeRecord + LLMResponseEvent.parser_reshaped; bump audit_schema_version 1->2 * #184 US-003: parser re-attach for column-scoped model-only variants + WARNING + ReshapeRecord audit * #184 US-004: wire parser_reshaped from parser through draft_from_request to LLMResponseEvent audit * #184 US-006: docs + rules + CHANGELOG for parser re-attach + audit field bump * #184 US-007: Phase B 3/15 candidate live re-aggregation — followup section * #184 US-009: Quality gate — fix bugs from code review (Pass 1 + Pass 4) Pass 1 (Correctness) fixes: - New _validate_model_only_test_args helper called from the re-attach branch — re-attached tests' date_column / where / unique_combination.columns args are now re-validated against model_columns, preserving the 'fail loud on hallucinations' contract. Without it a re-attached row_count_anomaly_by_period(date_column='phantom') silently degraded to kept-without-evidence at prune instead of surfacing a typed parser violation. (Pass 1 Finding 1, promoted to must-fix per qg-pass-3-defer-defensive-tests-fails-codecov.) - 4 new tests in test_parser.py pinning the new validation path: test_reattach_validates_anomaly_date_column_against_model_columns, test_reattach_validates_row_count_between_where_against_model_columns, test_reattach_validates_unique_combination_columns_against_model_columns, test_reattach_preserves_sibling_tests_on_same_column (Pass 1 Finding 2 — sibling-test preservation in _apply_reattach_actions was previously unpinned). Pass 4 (Docs+UX) fixes: - BLOCKER: docs/draft-ops.md audit-fields table updated audit_schema_version from 'Currently 1' to 'Currently 2' (was stale post-US-002 bump). - MAJOR: Added a row to the audit-fields table for the new parser_reshaped field, with the back-compat default explanation. - MINOR: Added operator-workflow cleanup paragraph to docs/draft-ops.md parser-re-attach section + CHANGELOG Fixed entry instructing operators to remove the pre-fix llm.exclude_tests workaround. - TRIVIAL: Added sort_keys=True to the parser re-attach WARNING json.dumps so the doc sample and the actual emitted bytes share key ordering (alphabetical is also diff-friendly across runs). Pass 2 (Conventions) and Pass 3 (Tests) returned zero findings. Validation: 3368 passed, 6 skipped, 99 deselected; ruff / format / pyright all 0 errors. * #184: address PR review (Copilot) — clarify writeup is partial 3/15 Copilot flagged that the writeup overclaimed DEC-008's acceptance bar. The plan's DEC-008 explicitly committed to all 15 Phase B candidates; the partial 3-of-15 run was an opt-in budget concession, not the bar DEC-008 set. Adjusted to honestly mark the remaining 12 candidates as PENDING and to scope the 3-candidate evidence to 'the bug is fixed on the load-bearing failure shape' (necessary but not sufficient for the full DEC-008 close). No code changes — research-doc clarification only.
* Add super plan for #188: bulk-mode shared cached prefix for --select * Record PR #195 + mark plan published * Devolve #188 plan to beads (epic SignalForge-fzo, 10 tasks) * SignalForge-fzo.1: add DraftConfig.cache_scope field + drift detection Add cache_scope: Literal["per-model", "project"] = "per-model" to DraftConfig under the llm: namespace (DEC-001 of #188). DraftConfig stays extra="forbid" and _DraftConfigFile stays extra="ignore". The docstring names issue #188 and the auto-promote semantics (--select >= 2 models promotes to project) consumed by later stories (US-002..US-008). Establish the StrictDraftConfig drift pair in tests/draft/test_drift_detector.py plus a draft_config_v1.json fixture: field-set parity with production, fixture validation, and a cache_scop typo failing loud. Add config round-trip / Literal-rejection / unknown-llm-sibling-key tests to tests/draft/test_config.py. Foundation field only; renderer / prompt-version / CLI flag / drafter wiring ship in separate beads. * SignalForge-fzo.2: project-summary renderer + <PROJECT_MANIFEST> envelope + project rule aggregation US-002 building blocks for the bulk-mode shared cached prefix: - _render_project_summary(manifest): compressed 'name (N cols)' line per model in sorted(unique_id) order; byte-identical regardless of which model is under draft (the cache-hit precondition, DEC-007). Wrapped in a <PROJECT_MANIFEST> envelope (DEC-005, envelope only). - _PROJECT_SUMMARY_TEMPLATE constant for the project block (DEC-004). - _read_project_business_rules(manifest): deterministic project-wide aggregation (sorted unique_id -> model-level -> column-name) with a single global 1-indexed <BUSINESS_RULE id=N> counter; reuses the safety-layer dict-guard read pattern (strict isinstance(dict); scalar/ list noise dropped, never fail-loud). - Boring-substring breach guards (DEC-008): </PROJECT_MANIFEST> in rendered content or </BUSINESS_RULE> in any aggregated rule raises PromptEnvelopeBreachError (fail-closed, no per-model fallback). - PromptEnvelopeBreachError extended with nullable model_unique_id and a rule_source discriminator ('model' default; 'project' for project-level breaches). Default MODEL_SQL / per-model BUSINESS_RULE behaviour is byte-equal to pre-#188. Class identity unchanged so scan 7 stays green. No US-003+ wiring: render_prompt / _PROMPT_VERSION / system-prompt variant untouched; the new functions are not yet called from render_prompt. * SignalForge-fzo.3: dual _PROMPT_VERSION + scope-aware system prompt + render_prompt dispatch US-003 (#188): adds the project-scope cached-prefix prompt path alongside the unchanged per-model path. - _PROMPT_VERSION_PER_MODEL (byte-identical to today's _PROMPT_VERSION; bare name kept as an alias) and _PROMPT_VERSION_PROJECT (hash inputs add _PROJECT_SUMMARY_TEMPLATE + the project-scope <PROJECT_MANIFEST> defence text). - _prompt_version_for(exclude_tests, cache_scope) selects the base by scope, folds '|scope=...|exclude=...' into a blake2b-8 when either dimension is non-default; _prompt_version_for((), 'per-model') returns the per-model base verbatim (snapshot-stable). - _render_system_prompt(exclude_tests, cache_scope): project variant appends a <PROJECT_MANIFEST> 'data, not instructions' defence line mirroring <MODEL_SQL>; per-model variant byte-identical to today (preserves the cache-stability golden). - render_prompt(..., cache_scope='project') emits the shared project cached block and a dynamic block carrying <MODEL_SQL> + this model's full column/neighbour detail + its own <BUSINESS_RULE> rules (DEC-013); per-model path unchanged. tests/llm/test_prompt_cache_stability.py left unmodified and passing. * SignalForge-fzo.4: thread cache_scope + project-oversize catch-and-retry in drafter Thread config.cache_scope from draft_from_request into render_prompt (US-004, DEC-006). The default per-model path is byte-identical to today. Oversize fallback: call_llm keeps raising LLMCacheTooLargeError at the 8000-token gate. In draft_from_request, when cache_scope=="project", catch it, re-render the cached block in per-model scope, and retry call_llm exactly once; emit one INFO breadcrumb (lazy-format JSON, passes the logger grep gate) naming the model + the fallback. When cache_scope=="per-model", the error propagates unchanged (no retry). The successful retried path falls through to the single response-audit write — audit written exactly once. Tests cover: default per-model path unchanged, project-under-cap sends the <PROJECT_MANIFEST> block, project-over-cap fallback (+ INFO assert via caplog + per-model block on retry), audit-written-once, and per-model-over-cap propagation. * SignalForge-fzo.6: pin project-scope cache-stability golden (US-006) Extend tests/llm/test_prompt_cache_stability.py with the project-scope cache prefix goldens alongside the existing per-model ones: - _EXPECTED_PROMPT_VERSION_PROJECT pins the _PROMPT_VERSION_PROJECT base constant (0d7dbc9e5f69f5fc); _RENDERED_PROMPT_VERSION_PROJECT pins the composed value render_prompt(..., cache_scope="project") actually returns (2cd2df7087ce462d — _prompt_version_for folds the non-default scope into a fresh hash). - _CACHED_BLOCK_GOLDEN_PROJECT pins the rendered <PROJECT_MANIFEST> cached block byte-for-byte (captured via render_prompt, not hand-written). - New tests: base-version pin, composed-version pin, project cached-block byte-stability (unified diff on mismatch), cross-model byte-identity (the DEC-007 cache-hit precondition), and project != per-model version. - Docstring documents the disjoint lockstep rotation policy (per-model rotates on _SYSTEM_PROMPT/_MANIFEST_SUMMARY_TEMPLATE/_DATA_SECTION_TEMPLATES; project on the project _SYSTEM_PROMPT variant/_PROJECT_SUMMARY_TEMPLATE/ _DATA_SECTION_TEMPLATES). Existing per-model golden + version assertions are untouched and green. * SignalForge-fzo.5: CLI --cache-scope flag + auto-promote + draft overlay (US-005) Add the --cache-scope {per-model,project} flag to `signalforge generate`, mirroring the --mode / --scope / --sample-strategy / --format overlay shape. - argparse: --cache-scope, default=None sentinel, choices-validated (exit 2 on a bad value), full help text. - _run_single_model gains a draft_overrides kwarg applied via the canonical DraftConfig.model_validate({**dump, **overrides}) re-validation overlay (DEC-010) — None/empty leaves the loaded config untouched so the single-model positional path stays byte-identical to v0.1. - Single-model dispatch reflects ONLY an explicit --cache-scope flag; never auto-promotes. - _run_batch resolves the overlay once via _resolve_batch_draft_overrides with precedence: explicit flag > YAML llm.cache_scope (non-default) > auto-promote to project when >= 2 models matched (DEC-002 / DEC-003). - Tests across the full matrix (batch>=2 promotes, single-match no-promote, single-model never-promote, flag-wins, force-project-on-single, YAML honoured, overlay re-validates via model_validate, invalid value exit 2), all asserting the no-traceback floor. * SignalForge-fzo.7: document --cache-scope flag + correct sibling-cache caveat + 5-surface parity test (US-007) Add the --cache-scope {per-model,project} flag reference to the signalforge generate flag list in docs/cli-ops.md, covering default per-model, auto-promotion to project on --select >= 2 models, and the precedence (explicit flag > YAML llm.cache_scope > auto-promote). Correct the stale Anthropic prompt-cache caveat in the 'Running across many models' section: under cache_scope=project the shared project prefix is cached once and read across the batch, so the marked cache now DOES amortise across siblings. Per-model scope retains the prior no-amortise behaviour; document the oversize fallback. Ship tests/cli/test_5_surface_parity_cache_scope.py mirroring the --select parity test: hard asserts that --cache-scope, per-model, and project appear in argparse --help, docs/cli-ops.md, and the plan. * SignalForge-fzo.8: document bulk-mode shared cache (US-008) Add a 'Bulk-mode shared cache' section to docs/draft-ops.md covering what the feature is (drafter --select cache hit 0% -> ~95%), how the shared project prefix works, automatic activation on --select >= 2 models plus the --cache-scope override + precedence, the oversize auto-fallback to per-model, and a cost model derived from the verified Sonnet pricing multipliers (write 1.25x input, read 0.1x input). Add a provider & model applicability subsection verified against the code: AnthropicProvider.supports_prompt_caching=True (Sonnet/Opus min 1024 tokens, Haiku 2048 per _MIN_CACHEABLE_TOKENS in llm/client.py); OpenAI and Gemini supports_prompt_caching=False -> no benefit and no penalty (the restructure is a no-op for the cost lever on those providers). Cross-link the new section from README's Supported LLM providers area. SKILL.md left unchanged: it documents single-model generate only, not --select batch runs. * SignalForge-fzo.9: Quality gate — fix _PROMPT_VERSION_PROJECT hash inputs (DEC-014) Code review (4 passes: correctness/security/CLI/tests) found one real MEDIUM: _PROMPT_VERSION_PROJECT folded _MANIFEST_SUMMARY_TEMPLATE into its hash, so the project version rotated on per-model template changes — violating DEC-014 rotation independence. Removed that term; project base version 0d7dbc9e->49e58185, composed 2cd2df70->e3ec5997. Updated the two pinned constants in test_prompt_cache_stability.py + the stale docstring in lockstep. CodeRabbit unavailable in this environment. ruff/format/pyright clean; 3435 passed. * SignalForge-fzo.10: distil #188 bulk-cache patterns into rule files US-010 (Patterns & Memory). Document the project-scope shared cached prefix conventions from #188 across four rule files + plan references: - llm-drafter.md: new "Project-scope cached prefix for --select batches" section — DraftConfig.cache_scope, _render_project_summary + <PROJECT_MANIFEST> envelope, deterministic total order, dual _PROMPT_VERSION + _prompt_version_for dispatch, oversize catch-and-retry vs breach fail-closed, and the DEC-014 rotation-independence lesson (project version hashes _PROJECT_SUMMARY_TEMPLATE, not the per-model one). - cli-layer.md: correct the DEC-15 sibling-cache caveat (now amortises under project scope); document the draft_overrides overlay + --cache-scope flag + auto-promote precedence. - business-rule-tests.md: project-wide rule aggregation (_read_project_business_rules, global 1-indexed counter) + DEC-013 dual-placement rule (cardinality gate reads the Model object). - testing-signal.md: second cache-stability golden + cross-model byte-identity test pattern. Docs-only; no production/test code changed. Validation green (ruff, format, pyright, 3435 pytest passed). * #188: mark super plan complete (all stories + QG merged)
* Add super plan for #189: --no-grade flag + persistent grade cache 20 DECs locked across 10 implementation stories + Quality Gate + Patterns & Memory (12 beads total). Architecture review surfaced three blockers, all resolved via DECs: - DEC-008: GradeEvent.audit_schema_version must be int (not Literal) before the 1→2 bump, or v1 fixture round-trip breaks. - DEC-006: 16KB pre-write size cap on cache records. - DEC-004: cache key includes provider (5-part recipe) to prevent cross-provider SKU collisions. Two Phase-2 concerns bound to dedicated stories: - US-007 ships --no-cache alongside --no-grade for per-run bypass. - US-008 ships nested 'cache clear --grade' subcommand (documented deviation from cli-layer.md DEC-009 for forward-compat). Composes multiplicatively with the already-shipped #186 (asyncio parallel grade) and #187 (per-provider fast defaults). * #189: devolve plan to beads Epic bd_1-scaffolding-63t with 12 children created. Ready at devolve (4 stories startable in parallel): - .1 US-001 (audit_schema_version Literal->int prereq) - .3 US-003 (grade.cache module) - .4 US-004 (cache typed errors) - .5 US-005 (GradeConfig.cache_enabled knob) Blocked stories follow the dependency graph; Quality Gate gates the patch coverage check before Patterns & Memory commits the rule + memory updates. * bd_1-scaffolding-63t.5: US-005 — GradeConfig.cache_enabled knob Adds the cache_enabled: bool = True master switch to GradeConfig (#189 DEC-016). extra="forbid" makes a typo such as cache_enable: (missing the trailing 'd') fail loud at config-load via ValidationError. NO TTL knob in this ticket — the cache key is content-addressed so invalidation is structural; a TTL is a future ticket if demand emerges. docs/grade-ops.md documents grade.cache_enabled under the existing grade: namespace section. * bd_1-scaffolding-63t.1: US-001 — audit_schema_version Literal[1] → int (prereq for #189 cache_hit bump) Widens production `GradeEvent.audit_schema_version` from `Literal[1]` to `int` so older audit JSONLs round-trip across version bumps. The strict drift-detector mirror (`StrictGradeEvent` in `tests/grade/test_drift_detector.py`) deliberately keeps `Literal[1]` — it continues to pin the committed v1 fixture's shape; a future v2 bump (US-002) grows a sibling `StrictGradeEventV2` mirror. Mirrors `safety-layer.md` § "AuditEvent reproducibility fields" (DEC-014): `audit_schema_version: int` is the canonical shape across every fail-closed audit writer in the project (safety / draft / prune / diff already follow this). Grade was the outlier. Mechanical type-flip only; no semantic change. Unblocks US-002 (the `cache_hit` field addition + 1→2 schema bump) without breaking v1 fixture round-trip. Tests: - `test_grade_event_audit_schema_version_defaults_to_one` — default unchanged at 1. - `test_grade_event_audit_schema_version_is_int_typed` — constructing with `audit_schema_version=99` succeeds. - `test_strict_grade_event_still_rejects_non_literal_one` — strict mirror still rejects values other than 1 (v1 fixture pin preserved). Refs DEC-008, plans/super/189-no-grade-cache.md. * bd_1-scaffolding-63t.4: US-004 — cache typed errors + exit-code registration Add four typed errors in `signalforge.grade.errors` per DEC-017 of #189: - `GradeCacheReadError(GradeError)` — tier 3 (external dep, disk I/O); carries `cause=` kwarg; raised in the engine's catch-and-warn read path. - `GradeCacheWriteError(GradeError)` — tier 3; exists for catch-and-warn diagnostics, never propagates from `grade_artifacts` (fail-soft per DEC-005); carries `cause=` kwarg. - `GradeCachePathError(GradeError)` — tier 1 (load-time / parse-layer; symlink-containment gate refusing escape from project_dir). - `GradeCacheRecordTooLargeError(GradeCacheWriteError)` — subclass of Write; inherits tier 3 via MRO; carries (size, limit) per the audit-record-too-large precedent. Register the three primary classes in `_EXCEPTION_TO_EXIT_CODE` in `signalforge.cli._helpers` (Path → tier 1, Read + Write → tier 3). The TooLarge subclass inherits via the MRO walk and is added to `_EXCEPTION_MAPPING_EXCLUDED_BASES` in `tests/test_audit_completeness.py` so the 7th AST scan does not require an explicit table entry. Pin the locked `default_remediation` text for each of the four classes in `tests/grade/test_errors.py` so a drive-by edit cannot silently weaken the operator-facing message. Add per-class exit-code tests in `tests/cli/test_exit_codes.py` for Read/Write/Path/TooLarge. Re-export the four classes from `signalforge.grade.__init__.__all__` so they're available from the public surface for downstream consumers (the cache module / engine surgery in US-006 + US-008). Validation: `uv run ruff check . && uv run ruff format --check . && uv run pyright && uv run pytest` — 3414 passed, 0 errors, 97.31% cov. * bd_1-scaffolding-63t.2: US-002 — GradeEvent.cache_hit field + audit_schema_version 1→2 + v2 fixture Production changes (src/signalforge/grade/models.py): - New cache_hit: bool = False field inserted between response_text_hash and model per DEC-009 of #189 (reproducibility hashes stay adjacent; cache_hit sits at the hash/token boundary as a sentinel). - audit_schema_version default bumped 1 → 2 per DEC-008 of #189. Field stays typed int (US-001) so older v1 audit JSONLs still round-trip. - Custom __repr__ keeps cache_hit visible (non-sensitive bool) while redacting evidence / reasoning per DEC-022 of #6. Construction seam (src/signalforge/grade/audit.py): - _build_grade_event(..., cache_hit: bool = False) per DEC-010 of #189 — keyword-only, threaded into GradeEvent. The 6th AST scan still gates GradeEvent construction to this single seam. Drift detector + v2 fixture: - New StrictGradeEventV2(extra='forbid') mirror pins the v2 shape; StrictGradeEvent (v1) stays as the replay anchor for grade_event_v1.jsonl. Field-set parity test now compares against StrictGradeEventV2 (field-set-current). - tests/fixtures/grade/grade_event_v2.jsonl committed with TWO lines (one cache_hit: true, one cache_hit: false); v1 fixture untouched. - New test_v1_fixture_still_validates_against_production_grade_event asserts a pre-#189 v1 line still loads through production via extra='ignore' + cache_hit default. Test surface adjustments: - _make_event helper grew cache_hit: bool = False parameter (tests/grade/test_models.py + tests/grade/test_audit.py). - Two test_audit.py assertions of audit_schema_version == 1 flipped to == 2 (live writes now produce v2 records). - Three provider-neutrality tests (anthropic / openai / gemini) switched their strict-drift import from StrictGradeEvent → StrictGradeEventV2 so live v2-shaped audit lines validate against the v2 forbid mirror. * bd_1-scaffolding-63t.6: US-010 — cost-rollup zero-cost test for cache_hit=True GradeEvents Adds one test pinning DEC-020 of plans/super/189-no-grade-cache.md: a cache-hit GradeEvent (cache_hit=True, all four token-count fields = 0) contributes $0 to the rollup, but IS walked by _ingest_jsonl (the rollup must NOT skip cache-hit rows — they're GradeEvents like any other, and call_count must reflect the on-disk record count). Two assertions: (a) Total grade USD == miss-only contribution (cache-hit contributes 0). (b) call_count == 2 (both miss AND hit are walked), not 1. Extends the _grade_record() helper with a cache_hit: bool = False kwarg so the test can author the cache-hit row inline; existing tests continue to pass with the default. * bd_1-scaffolding-63t.3: US-003 — signalforge.grade.cache module (keys, CacheRecord, I/O, fail-soft writer) Net-new module implementing the persistent grade cache surface per plans/super/189-no-grade-cache.md DEC-004, DEC-005, DEC-006, DEC-007, DEC-011, DEC-012, DEC-013, DEC-014, DEC-018. Public surface (per DEC-018): * CacheRecord — frozen Pydantic v2 model, flat duplication (DEC-011); score: float (NOT Optional) — DEC-007 refuses degraded results at the model layer; custom __repr__ + __repr_args__ omit evidence/reasoning per the pydantic-v2-repr-args-redaction-required convention. * compute_cache_key — five-part NUL-separated blake2b-8 digest (DEC-004), with provider between artifact_text_hash and model_id for cross-provider SKU collision safety. * lookup_cache — best-effort read; every degenerate path returns None (missing dir, missing file, malformed JSON, schema-version mismatch, Pydantic validation failure incl. score:null poisoning). * write_cache — fail-soft (DEC-005, inverse of audit fail-closed): pre-open size cap (DEC-006: 16_000), 0o600 file mode (DEC-012), O_EXCL concurrent-write safety (DEC-014, EEXIST → DEBUG + return). Any OSError / oversize / EEXIST collapses to a single WARNING via lazy-format JSON logger. * clear_cache — symlink-hardened shutil.rmtree (DEC-015); refuses any canonicalised path whose final two components are not '.signalforge/grade-cache'. Idempotent on missing directory. Drift detector + fixture: * tests/fixtures/grade/grade_cache_record_v1.json * tests/grade/test_drift_detector.py — StrictCacheRecord(extra="forbid") mirror + field-set parity + fixture validation tests. Test surface (32 tests, all passing): * Cache-key determinism + per-axis sensitivity (5 axes parametrized). * Provider-in-recipe cross-collision guard. * CacheRecord rejects None / out-of-range / NaN / inf scores. * __repr__ + __repr_args__ omit PII-bearing evidence/reasoning. * lookup_cache returns None on every degenerate path; round-trip with write_cache. * write_cache lazy mkdir, 0o600 mode, O_EXCL skip-on-existing, fail-soft on OSError, oversize → WARNING + no on-disk artefact. * clear_cache: idempotent missing-dir, removes existing dir, refuses paths not under .signalforge/grade-cache suffix, refuses symlink escape (canary file outside the symlink target survives). Canonical validation: uv run ruff check . && uv run ruff format --check . && uv run pyright (0 errors) && uv run pytest (3459 passed, 97% coverage). * bd_1-scaffolding-63t.8: US-008 — signalforge cache clear --grade subcommand New top-level subcommand 'cache' with a nested 'clear' sub-action and one required flag '--grade'. Removes <project_dir>/.signalforge/grade-cache/ recursively via signalforge.grade.cache.clear_cache (which owns the symlink-hardening + idempotent-on-missing-dir semantics per DEC-015). Per DEC-015 the nested-subparser shape is a documented deviation from cli-layer.md DEC-009 ('flat, per-subcommand modules'); justified by forward-compat for a future cache stats / cache list / cache clear --drafter family. No --confirm flag: scope is bounded by the conventional .signalforge/grade-cache/ suffix and the operator typed --grade explicitly. Behaviour: - Happy path: populated cache dir removed; exit 0. - Missing dir: idempotent no-op; exit 0. - Symlink escape: GradeCachePathError -> tier 1 (exit 1) via the existing US-004 exit-code registration. - Missing --grade: argparse exits 2 (required=True on the flag). Module: src/signalforge/cli/cache.py registers via add_parser(); cmd_cache dispatches on args.cache_subcommand. Re-binds the root logger via setup_logging() so signalforge.grade.cache's INFO line lands on the live stderr rather than a stale handle inherited from a prior in-process test. Skill-parity gate: 'cache' name auto-grows via category-1 (every subcommand) and the QG category-4 (every taught flag) gates. Updated _subparser_flags in tests/cli/test_skill_cli_parity.py to BFS over nested subparser trees so 'signalforge cache clear --grade' in SKILL.md prose validates against the nested --grade flag on the clear sub-action. Minimum SKILL.md mention added in the audit-JSONL paragraph (full prose update is US-009's scope). Files: - src/signalforge/cli/cache.py (new) - src/signalforge/cli/__init__.py (register cache_cmd.add_parser) - src/signalforge/skills/signalforge/SKILL.md (minimum cache clear --grade mention; full prose deferred to US-009) - tests/cli/test_cache.py (new — seven in-process tests) - tests/cli/test_skill_cli_parity.py (extended _subparser_flags to BFS over nested subparser trees) Traces to: DEC-015 in plans/super/189-no-grade-cache.md. * bd_1-scaffolding-63t.7: US-006 — wire cache into grade engine (sync-prefix lookup + post-grade write) Engine surgery in signalforge.grade.engine to wire the grade cache landed in US-002/003/004/005 into the orchestrator: 1. Sync-prefix cache lookup (DEC-013). After path canonicalisation and before asyncio.run(...), canonicalise <project>/.signalforge/ grade-cache/ via signalforge._common.path_safety.canonicalise_path (PathContainmentError → GradeCachePathError). For every (criterion, artifact) pair compute the 5-part cache key, call lookup_cache, and on a hit (record.score is not None — guaranteed by CacheRecord's score: float type per DEC-007) build a GradingResult from the record + a cache_hit=True GradeEvent via the SOLE construction seam _build_grade_event(..., cache_hit=True, ...) with all four token fields == 0. Write the audit event via the existing fail-closed _write_event_or_abort; pre-fill the results_by_index slot. Cache hits NEVER acquire the async semaphore. 2. Async dispatch (DEC-004). _grade_artifacts_async_core gains three keyword-only parameters: prefilled_results (carries cache-hit slots), cache_dir (None when disabled), and artifact_text_hash_by_index. The inner TaskGroup loop skips indices already populated by the sync prefix; the synthesis pass correctly treats them as 'do not budget-degrade'. 3. Post-grade cache write (DEC-005, DEC-007). Inside the per-pair coroutine, AFTER the audit-write shield-await, when cache is enabled AND grading_result.score is not None: build a CacheRecord from the GradeEvent's reproducibility fields (criterion_prompt_ hash, response_text_hash, rubric_hash, prompt_version_template), compute the cache key, and call write_cache. Defence-in-depth try/except surfaces any escaping exception as one WARNING line (write_cache is fail-soft per US-003 but the engine guards in case a future refactor or test stub violates the contract — see test_grade_engine_cache_write_failure_is_fail_soft). Degraded results NEVER reach the cache. 4. cache_enabled=False short-circuit. When the knob is False, the sync prefix skips cache_dir resolution entirely (no canonicalise call), no lookup runs, and the async core sees cache_dir=None so no post-grade write fires. 5. Cache-hit slots count toward counters['completed'] so the budget-trip WARNING faithfully reflects total progress. Tests (TDD-first, 10 new — all 5 invalidation axes covered): test_grade_engine_cache_hit_skips_llm_call test_grade_engine_cache_miss_writes_entry test_grade_engine_cache_disabled_skips_lookup_and_write test_grade_engine_artifact_text_change_invalidates_cache test_grade_engine_model_change_invalidates_cache test_grade_engine_provider_change_invalidates_cache test_grade_engine_degraded_result_not_cached test_grade_engine_cache_write_failure_is_fail_soft test_grade_engine_cache_hit_event_has_zero_tokens test_grade_engine_cache_hit_dispatch_order_preserved_with_async_misses AST scan 6 (every GradeEvent flows through _build_grade_event) stays green — cache-hit events use the same factory. Traces to: DEC-004, DEC-005, DEC-007, DEC-010, DEC-013, DEC-016, DEC-018 in plans/super/189-no-grade-cache.md. Canonical validation passes: ruff check + ruff format --check + pyright (0 errors) + pytest (3470 passed, 97.16% coverage). * bd_1-scaffolding-63t.9: US-007 — --no-grade + --no-cache flags + [N/4] progress UX Adds two bare boolean flags to `signalforge generate`: - `--no-grade` (DEC-001): wraps the grade-stage block in `if not no_grade:` so `grade_artifacts` is never invoked; `grade_report` defaults to `None` and flows through to `render_diff`, which already accepts that path (diff-renderer.md § 'Tier classification with no-grading-report degrade' — the `flagged` tier only fires when `grading_report is not None`). No `grade.jsonl` / `grade.json` side files are produced. - `--no-cache` (DEC-002): flips `grade_config.cache_enabled=False` via `model_copy(update=...)` BEFORE `grade_artifacts` so the engine's sync-prefix cache lookup AND the post-grade write are both no-ops for this run. Existing cache files on disk are NOT touched (operator uses `cache clear --grade` for that). Precedence (DEC-002): `--no-grade` implicitly wins when both are set — the entire grade block (including the `cache_enabled=False` mutation) is skipped, so `load_grade_config` is never called and the cache layer never sees either flag. No explicit mutex. Progress UX (DEC-003): the orchestrator computes `total = 4 if args.no_grade else 5` ONCE at startup and threads it into every `emit_progress_entry`/`emit_progress_done` call. Under `--no-grade` the operator sees `[1/4] safety / [2/4] draft / [3/4] prune / [4/4] diff` — NO `[X/5] grade: skipped` line; the pipeline is honestly four stages. Default behaviour is byte-equal with the prior `[N/5]` shape. Files ----- - `src/signalforge/cli/generate.py` — argparse additions (after `--no-color`), `total`/`_diff_stage_n` resolved once at `_run_single_model` entry, every progress call threaded with `total=total`, grade-stage block wrapped, `--no-cache` mutation, docstring extended. - `tests/cli/test_generate.py` — 11 new tests pinning DEC-001 / DEC-002 / DEC-003 behaviour: `grade_artifacts` skipped under `--no-grade`; `grading_report=None` threaded to `render_diff`; no `[X/5]` substring under `--no-grade`; `load_grade_config` not called; `cache_enabled=False` reaches the engine under `--no-cache`; existing cache files untouched; precedence; help text mentions both flags; flag combines with `--write`. Multi-surface parity -------------------- This story updates surfaces (1) argparse help string and (2) handler docstring per cli-layer.md § 'Multi-surface parity'. Surfaces (3) docs/cli-ops.md, (4) docs/grade-ops.md, (5) SKILL.md and the bespoke 5-surface parity test land in US-009. Traces to DEC-001, DEC-002, DEC-003 in plans/super/189-no-grade-cache.md. * bd_1-scaffolding-63t.10: US-009 — docs + SKILL.md + 5-surface parity test for --no-grade / --no-cache / cache clear --grade Update all five DEC-019 parity surfaces in lockstep for the three new operator-facing tokens #189 ships: * docs/cli-ops.md — three new cookbook sections ("Skip grading for fast iteration", "Bypass the grade cache for one run", "Clear the grade cache"). Document the --no-grade / --no-cache precedence (--no-grade implicitly wins) per DEC-002, the progress UX re-numbering to [N/4] per DEC-003, and the cache-clear behaviour (idempotent on missing dir, symlink-hardened, required --grade flag) per DEC-015. * docs/grade-ops.md — new "## Grade cache" section between the Configuration block and Threshold-fail behaviour. Documents cache layout (<project>/.signalforge/grade-cache/<16-hex>.json, 0o600), the five-part key recipe (DEC-004 invalidation axes), the grade.cache_enabled knob (DEC-016), the conservative degrade-not-cached contract (DEC-007), and the wipe subcommand. * src/signalforge/skills/signalforge/SKILL.md — extend the generate section with the two new flags + their precedence, promote the grade-cache + cache-clear paragraph to its own '## 3a. The .signalforge/ directory and the grade cache' section with operator commands. Skill parity gate (test_skill_cli_parity.py) stays green — "cache" was already named per US-008; this story expands the prose. * tests/cli/test_5_surface_parity_no_grade.py — NEW. Mirrors test_5_surface_parity_select.py shape. Pins the literal tokens --no-grade, --no-cache, cache clear --grade across five surfaces (argparse help blob, plan file DECs, docs/cli-ops.md, this module's __doc__ as surface 4, SKILL.md). Substring match, no whitespace / case normalisation — boring-match philosophy from the envelope-breach guards. Adds an auxiliary assertion that docs/grade-ops.md carries the '## Grade cache' header so the cross-link from docs/cli-ops.md stays meaningful. Traces to: DEC-001, DEC-002, DEC-015, DEC-016, DEC-019 of plans/super/189-no-grade-cache.md. Validation: - uv run ruff check . — clean - uv run ruff format --check . — clean - uv run pyright — 0 errors - uv run pytest — 3492 passed, 6 skipped, 101 deselected (97% cov) - uv run --only-group docs mkdocs build — no new warnings on the new sections (pre-existing repo-internal link warnings stay). - skill-parity gate green (cache subcommand already named per US-008) * bd_1-scaffolding-63t.11: Quality gate — fix bugs from code review Four diverse review passes (correctness / conventions / tests / docs+UX) across the #189 implementation diff. CodeRabbit deferred to PR-comment review on PR #196. CORRECTNESS (Pass 1): - lookup_cache key-verification gate: recompute the 5-part cache key from the loaded record's stored hashes; on mismatch (corrupt or hostile file body), return None + INFO log. Closes the cache-poisoning gap where a file under <key>.json could lie about its forensic hashes and silently rehydrate the wrong verdict. - clear_cache project-anchor containment: accept an optional project_dir kwarg, verify the canonical cache path lives inside the canonical project tree. Closes the symlink-escape gap where .signalforge/grade-cache -> /tmp/.signalforge/grade-cache would pass the old suffix-only check. - CLI handler threads project_dir into clear_cache. - Correct the misleading 'byte-identical' EEXIST comment. TEST COVERAGE (Pass 3 — 8 gaps + 1 weak): - write_cache mkdir OSError branch - write_cache mid-write OSError + partial-file unlink - write_cache os.write returning 0 branch - lookup_cache OSError-on-read branch - clear_cache RuntimeError/ELOOP canonicalisation branch - engine GradeCachePathError raise on cache_dir canonicalisation - cli/cache.py project-dir resolution error paths (2 tests + defensive cmd_cache unknown-sub-action branch) - lookup_cache key-mismatch test (P1 #1 sibling) - clear_cache project-anchor symlink-escape test (P1 #2 sibling) - Strengthen test_grade_engine_cache_write_failure_is_fail_soft to assert the forensic WARNING line fires (P3 #8 weak finding). - prefilled_results misalignment guard deliberately uncovered (unreachable from any public seam; documented trade-off). DOCS (Pass 4 — 4 missing + 4 weak): - CHANGELOG [Unreleased] gets the full #189 feature surface entry. - docs/cli-ops.md ## Subcommands enumeration: six -> seven. - New ### signalforge cache clear --grade reference section. - New ### Grade cache write WARNING section under ## Stderr shapes. - ## Progress lines now mentions the [N/4] re-numbering under --no-grade with a cross-reference. - --no-grade + --estimate composition caveat (estimate still projects grade cost; live run correctly skips). - docs/grade-ops.md cache-read failure shape: docs now match the code (INFO log, not WARNING). Broken self-referential anchor removed. Cache key-verification gate documented. Canonical validation: ruff/format/pyright clean, 3504 pytest passes (+12 from QG fixes), mkdocs build clean (only pre-existing external-link warnings). * bd_1-scaffolding-63t.12: Patterns & Memory — #189 lessons .claude/rules/grade-layer.md — new section 'Persistent grade cache (#189) — content-addressed, fail-soft, inverse posture to audit' captures six load-bearing invariants for the cache (5-part key with provider axis, fail-soft write posture, key-recomputation gate on read, two-layer symlink containment, cache-hit GradeEvent zero-token contract, degraded results never cached) + three reusable conventions (forensic-hash trail load-bearing on read, two-layer containment for destructive lib seams, inverse-posture WARNING contract). The cache is the 4th instance of the audit-vs-derived posture split (existing: safety/draft/prune/ grade audit fail-closed; warehouse cleanup fail-soft). .claude/rules/cli-layer.md — new subsection 'Nested-subaction precedent — cache clear (#189 DEC-015)' documents the first nested-subparser shape in the codebase as a bounded deviation from the flat-per-subcommand convention, justified by forward-compat for cache stats / cache list. Names the skill-parity gate's BFS-extension helper for any future sibling nested subcommand. Memory store (~/.claude/projects/.../memory/): - grade-cache-five-part-key-includes-provider.md - audit-schema-version-must-be-int-not-literal.md - cache-write-fail-soft-vs-audit-fail-closed.md Plus three MEMORY.md pointers. * bd_1-scaffolding-63t: Address PR review feedback Address all 10 inline review comments on PR #196 (4 Copilot, 6 CodeRabbit). Two MAJOR-flagged findings + 8 minor; categorisation: all 10 are real, zero false positives. CORRECTNESS: - engine.py:765-787 (Copilot) — cache write block sources every cache-key axis from resolved_config (live), NOT from event.* — symmetric with the sync-prefix lookup at line 1186. In practice event.model == resolved_config.model today, but a future refactor or test stub that diverges them would silently break the cache. The key-recomputation gate in lookup_cache (added by QG Pass 1) would also reject hits if write used a different model string. - engine.py:1218-1222 (CodeRabbit) — cache-hit GradeEvent records THIS run's authoritative rubric_hash / prompt_version_template / criterion_prompt_hash (live) instead of the stored record's values. rubric_hash is NOT in the 5-part cache key, so a sibling-criterion edit can leave THIS criterion's key axes unchanged while the full rubric_hash differs — using record.rubric_hash would record false provenance. response_text_hash stays sourced from the stored record (forensic trail of the original LLM response, which a cache-hit re-run never produced). - generate.py:1109 (CodeRabbit) — --no-grade now removes any stale .signalforge/grade.json / .signalforge/grade.jsonl from a prior run so operators can't mistake them for current output. The cache-cache directory is NOT touched — cached entries from previous runs remain available to a subsequent default run. - models.py:354 (Copilot) — GradeEvent.__repr_args__ override added. Pydantic v2 structured-repr surfaces (rich.print, devtools.pretty) use __repr_args__; without it, evidence + reasoning would leak through those paths even though __repr__ redacted them. DOCS / CONVENTIONS: - errors.py:497 (Copilot) — GradeCacheReadError docstring corrected from 'surfaces in the engine's catch-and-warn path' to 'reserved-but-currently-inert' per the GradeBudgetExceededError precedent (lookup_cache handles every degenerate path via None+INFO; nothing currently raises this class). - tests/cli/test_skill_cli_parity.py:300 (Copilot) — 'BFS' comment corrected to 'DFS' (queue.pop() is LIFO); variable renamed to 'stack' for accuracy. Traversal order doesn't matter for the resulting set so the behaviour is unchanged. - docs/grade-ops.md:193,215 (CodeRabbit MD040) — fenced blocks now carry 'text' / 'python' language tags. - docs/grade-ops.md:320 (CodeRabbit MD051) — broken heading link '#reproducibility-hash-fields' fixed by renaming heading '## Reproducibility / hash fields' → '## Reproducibility hash fields' (removes the slash that broke anchor generation). - plans/super/189-no-grade-cache.md (CodeRabbit MD037 + MD040) — emphasis spacing on line 21 fixed by wrapping _build_grade_event in backticks; three fenced blocks tagged with 'python' / 'text'. - tests/grade/test_models.py:14 (CodeRabbit) — docstring 'default 1' → 'default 2' to match the post-#189 schema. NEW TESTS: - test_grade_event_repr_args_redacts_evidence_and_reasoning — proves PII redaction holds under the structured-repr surface. - test_cache_hit_grade_event_uses_live_run_hashes_not_stored_values — pre-populates a cache entry with a stale rubric_hash, runs grade_artifacts, asserts the audit row records the LIVE rubric_hash (the false-provenance bug). Validation: ruff/format/pyright clean, 3506 pytest passes (+2 new review-fix tests). * bd_1-scaffolding-63t: Address PR review round 2 Two new CodeRabbit findings on the previous fix commit (adab8d7). Both real; zero false positives. CORRECTNESS / PII: - models.py:130-160 + models.py:259-289 (CodeRabbit MAJOR) — __repr_args__ redaction extended to GradingResult and GradingReport (previous commit only covered GradeEvent). Pydantic v2 structured-repr surfaces (rich.print, devtools.pretty, pprint) use this hook; without it, evidence and reasoning on GradingResult and the nested results tuple on GradingReport would leak through those paths even though their __repr__ was redacted. Belt-and-braces: GradingReport.__repr_args__ shows aggregate fields only (results_count instead of results); the nested GradingResult.__repr_args__ also redacts. - tests/grade/test_models.py:584 (CodeRabbit MINOR) — the response_text_hash sentinel 5555666677778888 triggered an OpenGrep credit-card PAN scanner. Swapped to a non-PAN-shape mixed-alphanumeric value (55a5c6667d7788ef). The pre-existing _make_event helper keeps the original digits-only form for fixture parity with the v1 JSONL. NEW TESTS: - test_grading_result_repr_args_redacts_evidence_and_reasoning - test_grading_report_repr_args_redacts_nested_result_payload Validation: 3508 pytest passes (+2), ruff/pyright clean. * bd_1-scaffolding-63t: Address PR review round 3 One CodeRabbit finding on commit fb034dd: the explanatory comment mentioning 5555666677778888 re-tripped the credit-card PAN scanner even though the actual sentinel value had been swapped. FIXED: - tests/grade/test_models.py:584 (CodeRabbit MINOR) — reworded the comment to describe the rejected literal abstractly ('a digits-only 16-char value') instead of quoting it. Applied CodeRabbit's proposed diff verbatim. No production code touched; no test logic changed. Validation: ruff clean, 36 test_models.py tests pass.
…ine writeup (#200) * #179: per-stage runtime benchmark harness skeleton Drafts the gated maintainer-run harness for the 'Runtime benchmark retest' story on epic #179 — measures per-stage (draft/prune/grade/diff) wall-clock plus the grade-budget-exceeded count so the efficiency work since the 2026-05-30 baseline can be attributed per stage: - #186 grade-layer asyncio refactor (always-on) - #187 Haiku fast-grade defaults (opt-in via SF_BENCH_GRADE_MODEL) - #188 bulk --select cached prefix (measured via CLI extension) Skeleton runs against an inlined substrate model (no intuit_airflow needed); documents the two-checkout before/after A/B and the real-slice swap. Gated by the anthropic marker (deselected by default; clean skip without a key). * #179: retarget runtime benchmark to prod-PyPI-vs-dev CLI script Replace the in-process pytest harness with a standalone stdlib script that shells out to the venv-local signalforge CLI against the local intuit_airflow repo. The A/B is now the released signalforge-dbt PyPI package (before) vs the dev editable checkout (after) — an installed package can't be swapped mid-process, so each side runs in its own venv and the stable cross-version contract is the CLI, not the orchestrator API. - benchmark_runtime.py: pure stdlib; resolves signalforge from the running venv, times one generate run, reads grade/diff duration_seconds from the sidecars (stable since #7/#8), derives draft+overhead, counts budget-exceeded grade degradations. Prod venv needs only pip install signalforge-dbt. - Documents the net-release-delta caveat (dev also adds #169/#170/#171 primitives) and the 90af28b-vs-dev pure-isolation alternative. - Remove the obsolete test_runtime_benchmark.py + _substrate.py. - Doc + issue #179 story updated to the prod-vs-dev protocol. * #179: control for #189 grade cache in benchmark (cold A/B + warm re-run) Merging dev brought #189 (persistent grade cache, default-on at .signalforge/grade-cache/). A warm dev cache would make dev look artificially fast vs the cacheless prod release. Add --cache-mode {bypass,warm}: - bypass (default): version-gate dev's --no-cache so the grade stage runs cold (fair A/B vs prod); prod omits the flag and is cold by definition. - warm: allow read/write so a second run measures the #189 re-run win (dev-only). Doc updated with the #189 row + cache methodology note. * #179: first benchmark measurement (prod 0.5.0 vs dev 0.6.0.dev0) Fix bin resolver to prefer the unresolved venv bin (uv symlinks bin/python to the store interpreter; .resolve() jumped out of the venv). Measured results against intuit_airflow weekly_query_cost (16 cols, Sonnet): - #186 asyncio: grade 303s budget-capped/133-degraded (prod) -> 222s/0-degraded (dev), grading MORE artifacts. Baseline 17/34 failure mode closed. - Net total 342.9s -> 268.7s (-22%) despite dev drafting 8 primitives vs 5. - #189 grade cache gives NO re-run speed-up: key includes artifact_text_hash and the live drafter is non-deterministic, so every generate re-draft misses. Reframe its value as re-grade-of-identical-candidate. (follow-on candidate) - #187 Haiku + #188 batch still pending. * #179: ruff format the benchmark harness
#199) * Add super plan for #198: grade budget scaling + cost ceilings * #198: mark plan phase published (PR #199) * #198: devolve plan to beads (epic SignalForge-xfg + 7 tasks) * SignalForge-xfg.1: #198 US-001 GradeConfig budget + ceiling fields + validator split total_budget_seconds becomes int | None = None (optional absolute hard ceiling; None routes to the scaled formula). Add budget_base_seconds=60, budget_per_pair_seconds=20.0, and the three opt-in soft ceilings max_grade_calls / max_grade_cost_usd / max_grade_tokens (all None=off). Split the _positive field_validator: positive-only for max_output_tokens / budget_base_seconds / budget_per_pair_seconds; allow-None-or-positive for total_budget_seconds and the three max_grade_*. Existing fixtures (300/600 explicit ints) still load; extra=forbid still rejects typos. Traces DEC-001/002/004/006 of plans/super/198-grade-budget-scaling.md. * SignalForge-xfg.2: #198 US-002 _compute_effective_budget pure helper + formula tests * SignalForge-xfg.3: #198 US-003 wire scaled budget + rename WARNING field + 40-col AC test * SignalForge-xfg.4: #198 US-004 grade cost/calls/tokens ceilings + ceiling WARNING * SignalForge-xfg.5: #198 US-005 docs/grade-ops.md + example fixture parity * SignalForge-xfg.6: #198 Quality gate — pin wall-clock degrade reason + tighten config validator/round-trip tests * SignalForge-xfg.6: #198 Quality gate — fix unpriced-SKU cost-ceiling run abort + demo comment parity The cost ceiling looked up pricing per-pair inside the TaskGroup; a prefix-valid- but-unpriced model (e.g. claude-opus-4-8) + max_grade_cost_usd raised EstimateUnknownModelError mid-run (uncaught by the per-pair except), aborting the run after billable calls. Resolve pricing ONCE up front so an unpriced SKU fails fast at orchestrator entry before any LLM call; reuse the resolved pricing per pair. Found by the CodeRabbit-substitute review pass. * SignalForge-xfg.7: #198 Patterns & Memory — grade-layer.md scaled-budget + ceilings contract * #198: mark plan complete (PR #199) * #198: Address PR review feedback (CodeRabbit + Copilot) - engine: math.ceil (not int) on effective budget so a fractional per_pair never rounds the backstop DOWN - engine: route ceiling-degrade audit write through run_in_executor+shield (DEC-017) so the fsync doesn't block the concurrent loop (was sync); set slot first to avoid double-audit - config: reject non-finite floats (.nan/.inf) in budget_per_pair_seconds / max_grade_cost_usd (yaml parses them; nan<=0 is False so they slipped past the positivity guard) - docs: GradeBudgetExceededError marked reserved/not-raised in v0.1 (both error reference + table); fix invalid #logging anchor -> #debugging; drop trailing space in plan code span - tests: NaN/inf rejection, fractional-ceil (no round-down), cap-ceil - grade-layer.md: note updated to shielded-executor ceiling-degrade write
…#201) 40-col weekly_query_cost A/B: prod 0.5.0 degraded 325/408 pairs at the flat 300s ceiling; dev 0.6.0.dev0 (#198 scaled budget) had 0 budget-exceeded degradations (338 scored, ~2.8x throughput). The 70 dev GradeLLMError degradations are a rate-limit artifact (10-way concurrency overshoots the per-minute cap), not #198. Notes the orthogonal draft-width limit (40-col draft truncates at the default 4096 output cap).
…uire-complete (#203) * Add super plan for #202: Grade to 100% (throttle + sweep + require-complete) * Plan #202: add DEC-210 bias-to-completion budget posture * Plan #202: mark phase published (PR #203) * Plan #202: devolved to beads (epic SignalForge-0z5, 11 tasks) * SignalForge-0z5.2: add RateLimitBudget + provider extract_rate_limit_info seam Add the neutral, frozen RateLimitBudget value object (requests/tokens remaining, requests/tokens reset, retry_after — all optional) plus a vendor-neutral header-parse helper in a new src/signalforge/llm/_rate_limiter.py. Add LLMProvider.extract_rate_limit_info(exc, *, response=None) with a base default returning an EMPTY budget (OpenAI/Gemini inherit it, graceful degradation). AnthropicProvider overrides it to populate the budget from retry-after + anthropic-ratelimit-* headers, reaching them purely via duck-typed getattr so no anthropic.*/httpx.* type crosses the provider seam (DEC-012 confinement upheld without an SDK import). Parsing is fully tolerant: present/absent/malformed headers never raise; a malformed numeric leaves that field None. Tests: FakeRateLimitError/FakeResponseWithHeaders test doubles in tests/llm/_fake.py + tests/llm/test_rate_limiter.py covering Anthropic populated, OpenAI/Gemini empty, tolerant parsing, response>exc precedence, and no-vendor-type-leak. Public-surface pins updated. This is US-002 scope only — limiter classes and call_llm wiring are US-003/US-004. * SignalForge-0z5.1: add degrade_reason_type discriminator + bump audit_schema_version to 3 Add a structured degrade_reason_type discriminator (Literal["transient", "budget","ceiling"] | None) to GradingResult and GradeEvent (None when scored; set for every degraded pair). The prose->discriminator mapping is centralised in _build_degraded via _classify_degrade_reason so the later sweep / require_complete logic classifies degrades without string-matching the human-readable reason text. Unknown reasons default conservatively to "transient". Bump GradeEvent.audit_schema_version 2 -> 3. Field defaults safely so pre-#202 (v1/v2) records still load through the extra="ignore" production model; the v2 replay anchor still validates. Refresh fixtures + Strict drift mirrors: add grade_event_v3.jsonl (scored + all three degrade causes), add StrictGradeEventV3 (field-set-current mirror), keep StrictGradeEvent / StrictGradeEventV2 as v1/v2 replay anchors. Add degrade_reason_type to StrictGradingResult. Switch the provider-neutrality end-to-end JSONL checks and the field-set parity gate to the v3 mirror. Update docs/audits.md + docs/grade-ops.md. * SignalForge-0z5.3: shared sync+async rate limiters + client 429 wiring (Stage-1) Implement SyncRateLimiter / AsyncRateLimiter over a shared _RateLimiterState cell with AIMD adaptive concurrency (multiplicative-decrease on 429, additive-increase on headroom, clamped to [1, max_concurrent_calls]). Wire both into the client 429 retry branches: on a 429 the limiter computes the wait from retry-after / reset headers (via strategy.extract_rate_limit_info) to pace AT the rate limit, falling back to the historical blind backoff when the budget is empty (no headers / OpenAI / Gemini). Limiters are threaded via current_sync_rate_limiter / current_async_rate_limiter ContextVars (optional, default None) so signature parity between call_llm and call_llm_async is preserved. _backoff_warn WARNING shape and the _sleep/_async_sleep/_rand_uniform override seams are unchanged. Headline acceptance: a 429+retry-after burst that previously exhausted the 3x429 budget now succeeds (limiter paces per the header). AIMD decrease/increase + bounds, header-vs-blind-fallback, and both client branches are unit-tested. grade/engine.py untouched (US-004 owns the TaskGroup wiring). * SignalForge-0z5.4: share rate limiter across grade async core (Stage-1 complete) Wire the shared async rate limiter into the grade orchestrator so every concurrent grade coroutine paces against ONE shared AIMD budget instead of each blind-backing-off independently and bursting past the rate limit. In `_grade_artifacts_async_core` (the `asyncio.run` core), build a cooperating limiter pair via `make_rate_limiters(resolved_config.max_concurrent_calls)` (seeding initial effective concurrency from `max_concurrent_calls`) and publish the async sibling on the `current_async_rate_limiter` ContextVar so every coroutine dispatched in the TaskGroup (`_one` -> `_grade_one_async` -> `call_llm_async`) resolves the SAME limiter. The ContextVar is set inside the core (which `asyncio.run` runs in a copied context) and reset in a try/finally so it never leaks past the run. The three mechanisms stay orthogonal: the limiter PACES (per-429 wait via `retry-after`, blind-backoff fallback), the `asyncio.Semaphore` CAPS raw concurrency, the `asyncio.timeout(effective_budget)` is the runaway backstop. Limiter waits legitimately count against the budget, so the #198 over-budget degrade path is unchanged. Builds on US-002 (`RateLimitBudget`) and US-003 (`SyncRateLimiter`/ `AsyncRateLimiter` + ContextVars + `call_llm_async` 429 wiring); does not touch `client.py` or `_rate_limiter.py`. Tests (`tests/grade/test_engine_rate_limiter.py`): - a concurrent 429-burst fan-out that previously left N transient `GradeLLMError` degradations now reaches 0 (all pairs scored, `aggregate_complete=True`); - every concurrent coroutine observes the SAME non-None limiter, and the ContextVar resets to None after `grade_artifacts` returns (no leakage); - the budget-timeout degrade path still degrades every pair and resets the ContextVar on the timeout path too (#198 no-regression). Full gate green: ruff check/format, pyright, 3657 passed / 8 skipped, 97% cov. * SignalForge-0z5.5: bounded transient-pair sweep + sweep_round audit field (Stage 2) Add an ALWAYS-ON, bounded transient-recovery sweep (DEC-206) folded into the async core so the GradingReport is built from POST-sweep results — a run reaches 100% scored when transient LLM failures recover on a calmer retry. - engine: after the main pass (and a test-overridable cool-down) re-grade ONLY score=None / degrade_reason_type=="transient" pairs SEQUENTIALLY (concurrency 1) via _grade_one_async, looping until zero transient pairs remain or sweep_max_rounds is reached. budget/ceiling degrades are never swept. results_by_index stays the single source of truth the sidecar + report read, so aggregates reflect the sweep. Each swept attempt appends a NEW sweep_round-tagged audit record (immutable log); a recovered pair is written to the grade cache via the existing fail-soft path. - config: add sweep_max_rounds=3 (non-negative) and sweep_cooldown_seconds=2.0 (non-negative finite; 0 disables the wait, sweep stays always-on). Cool-down routes through the _async_sleep test-override seam. - models/audit: add additive optional GradeEvent.sweep_round (None main pass; 1+ for sweep rounds). No audit_schema_version bump (stays 3). - drift detector / v3 fixture / Strict mirror updated in lockstep. - tests: recovery-to-completion, sweep_max_rounds cap (no infinite loop), budget/ceiling never swept, sweep_round-tagged record + cache reuse on recovery, cool-down seam, sweep_max_rounds=0 no-sweep, config validators. Autouse conftest neutralises the cool-down by default. * SignalForge-0z5.6: GradeIncompleteError + require_complete engine check (Stage 3 core) Add the fail-loud completeness contract (DEC-204 + DEC-207): - New tier-2 GradeIncompleteError (parent GradeError) carrying incomplete_pairs / require_complete / aggregate_complete; the message names the first ~20 ungraded pairs then '… and N more' (full list in the JSONL audit), repr-quoting ids for log-injection safety. - Register GradeIncompleteError at tier 2 in _EXCEPTION_TO_EXIT_CODE (same tier as GradeBelowThresholdError); scan-7 + planted self-check pass. - GradeConfig.require_complete: bool = True (extra='forbid'). - Engine check in grade_artifacts after the sidecar write + INFO log and before the fail_on_below_threshold raise. Branches on degrade_reason_type: 'transient' always trips; 'budget' trips only when total_budget_seconds is None (default-scaled); 'ceiling' and explicit 'budget' are exempt. Raised after the sidecar is durably written. No CLI flag (US-007 owns --require-complete); limiter/sweep untouched. * SignalForge-0z5.7: --require-complete CLI flag + 6-surface parity (Stage 3 CLI) DEC-208. Surface the grade-completeness contract on the generate CLI with --require-complete / --no-require-complete via argparse.BooleanOptionalAction (default=None no-clobber sentinel). Overrides grade.require_complete ONLY when explicitly passed, via GradeConfig.model_validate so validators re-run; a bare run never re-arms a grade.require_complete: false set in signalforge.yml. When armed, the engine (US-006, merged) raises GradeIncompleteError (tier 2, exit 2) after the sidecar write, naming the still-ungraded pairs in stderr. Six parity surfaces in lockstep: argparse help, cmd_generate + _run_single_model docstrings, docs/cli-ops.md (flag ref + exit-code table + Grade-completeness behaviour stderr shape), tests/cli/test_generate.py (override True/False, no-clobber preserve, e2e transient-incomplete exit 2), DEC-208 in plans/super/9-cli-entrypoint.md, and SKILL.md (skill-parity gate). * SignalForge-0z5.8: raise max_retries_429 default + bias-to-completion docs/rules (Stage 4) Raise GradeConfig.max_retries_429 default 3 -> 6 (DEC-209; belt-and-braces over the #202 header-honoring rate limiter, the primary 429 fix). Update its docstring, the defaults-inventory comment, and the config test pinning the default. Add a deterministic guard test (test_default_scaled_budget_is_non_binding_at_representative_scale) that asserts the DEFAULT scaled budget leaves clear headroom (>=1.5x) over a modelled limiter-paced completion at a representative scale, so a default run never passively binds on the budget (DEC-210). Pure arithmetic on _compute_effective_budget — no sleeping or LLM calls. Docs/rules brought into line with the bias-to-completion posture and the transient-recoverable / operator-ceiling / unrecoverable taxonomy: - docs/grade-ops.md: new Bias-to-completion posture section (opt-in limit knobs, default-budget-trip-fails-loud, three degrade classes, require_complete contract); raised max_retries_429 reflected; cost-knob section no longer claims a default-budget trip silently completes. - docs/cli-ops.md: grade-completeness section points to the posture. - .claude/rules/grade-layer.md: new bias-to-completion + degrade-class section; 'partial is acceptable' scoped to operator-ceiling only. - .claude/rules/llm-drafter.md: document the rate-limit-aware limiter seam (header-honoring backoff + shared AIMD limiter via ContextVars) and the concurrency<->rate-limit relationship that caused the original ~70 degradations. Logic in the limiter/sweep/require_complete path is unchanged. Full gate green: ruff check + format, pyright (0 errors), pytest 3682 passed / 8 skipped, coverage 97.26%. * SignalForge-0z5.9: wire #202 knobs into 179 benchmark harness + writeup scaffold (live run pending) Wire the #202 grade-to-completion changes into the standalone #179 runtime benchmark harness (AUTOMATABLE PREP ONLY — the live metered run is operator-only): - benchmark_runtime.py: add a version-gated --require-complete flag (omitted on the prod arm exactly like --no-cache); classify degradations on the #202 degrade_reason_type discriminator (US-001) with a prose fallback for pre-#202 sidecars; report aggregate_complete, the per-reason split, and the ungraded (artifact, criterion) pair list; explain exit code 2 = GradeIncompleteError. - test_benchmark_runtime.py: unit-test the pure sidecar-parsing helpers (no API key, no subprocess) — loaded by file path so collection never mutates sys.path. - docs/research/179-runtime-benchmark.md: add a clearly-marked '#202 grade-to-completion retest' section with EMPTY _pending live run_ placeholder cells alongside the prod / dev #198 baselines, plus exact live-run instructions (command, env vars, 40-col + 16-col models, cold grade cache, fixed rate tier). No benchmark numbers fabricated; every #202 result cell is an explicit placeholder for the operator's live run. Full gate green. * SignalForge-0z5.9: record #202 live retest results (PASS both arms — 408/408 + 212/212 scored, aggregate_complete=True) * SignalForge-0z5.10: wire adaptive concurrency + bound sweep + QG review fixes #202 Quality-Gate review fixes (4-pass review: 2 MAJOR + 3 minor). FIX 1 (MAJOR) — Wire the AIMD adaptive concurrency for real. - New AsyncConcurrencyGate in _rate_limiter.py: an asyncio.Condition + in-flight counter that admits at most the limiter's live effective_concurrency in-flight grade calls (replacing the fixed asyncio.Semaphore(max_concurrent_calls) in the grade engine). Reads its cap from the SAME shared _RateLimiterState cell the limiter pair publishes (one source of truth). acquire() blocks while in_flight >= cap (floored at 1 so a fully-throttled run still makes progress); release() (in __aexit__/finally) notifies all waiters so a freed slot OR a headroom-widened cap wakes a blocked acquirer (no lost wakeup). The threading lock is never held across an await; the in-flight counter mutates only on the event loop under the Condition. - Engine _one now uses `async with gate:` instead of the semaphore. - call_llm_async calls limiter.record_headroom() on a clean (non-429) return, gated on a limiter being present (sync drafter / limiter-free async unaffected), so a 429 narrows the cap and clean completions probe it back toward max_concurrent_calls. Bounds rigorously [1, max_concurrent_calls]. - Banner/docstrings in _rate_limiter.py + engine.py updated to describe the now FUNCTIONAL adaptive concurrency. - Tests: gate admits up-to-cap, blocks the (N+1)th, narrows under a 429 storm, widens back on headroom (wakes waiter), stays in [1,max], releases in finally; engine-level proof that in-flight concurrency narrows below max and widens back. FIX 2 (MAJOR) — Bound the sweep's wall-clock. - New GradeConfig.sweep_budget_seconds (default 300, positive validator). The always-on sweep loop is wrapped in its own asyncio.timeout(sweep_budget_seconds); on TimeoutError the sweep STOPS (does not raise) and leaves remaining transient pairs degraded (fail loud under require_complete, or honest partial otherwise). One forensic WARNING. Test: a never-recovering slow sweep is bounded. FIX 3 (MINOR) — reset-header honesty. Implemented the reset-derived fallback: when a 429 carries no retry-after but a requests/tokens reset instant, _wait_from_budget derives a bounded wait (soonest reset, capped 60s vs clock skew) via an injectable _utcnow clock. Code + comments now agree (reset IS honoured). Deterministic tests via a pinned clock. FIX 4 (MINOR) — burst acceptance baseline now uses the SAME input on both arms (3x429+success, retry-after:30) and asserts the limiter paces every retry at the header value (30,30,30) vs the no-limiter blind backoff (1,2,4); separate test pins that an over-budget burst still exhausts (the limiter paces, it does not enlarge the budget). FIX 5 (TRIVIAL) — grade-layer.md degrade_reason_type cite DEC-203 -> US-001. Design choices: - Adaptive gate = asyncio.Condition + in-flight counter sharing the limiter's state cell (Semaphore can't be resized; Condition gives correct wakeups on cap growth + release-in-finally under cancellation). - Sweep budget = explicit config knob (default 300s) rather than a derived value: simplest, most testable, mirrors the main-pass effective_budget posture. Full canonical gate green: ruff check + format + pyright clean; pytest 3709 passed, 8 skipped; coverage 97.24% (>= 80). * SignalForge-0z5.10: fail-loud assert on unbalanced AsyncConcurrencyGate.release (QG review hardening) * SignalForge-0z5.10: record adaptive-gate live retest (412/412 + 220/220 scored, grade throughput 0.73/s) * SignalForge-0z5.11: capture #202 rate-limit pacing + recovery-taxonomy patterns and the validated throughput lesson * #202: mark plan complete (all stories closed, live retest PASS) * #202: address PR review — escape table pipe in plan doc (CodeRabbit) * #202: address PR review feedback (CodeRabbit) - _rate_limiter._parse_float: reject non-finite (nan/inf) retry-after values (a non-finite wait would poison backoff arithmetic) + tests - audit._build_grade_event: enforce sweep_round >= 1 at the construction seam + tests - test_config: pin the new #202 defaults (sweep_max_rounds/cooldown/budget, require_complete) - benchmark_runtime: correct misleading --require-complete default messaging
…to generate (#204) SignalForge drafts column-level tests from model.columns, which dbt populates from schema .yml files; a model with no schema yml yields zero columns and only model-level tests. Document the prerequisite, how to generate schema files (dbt-codegen generate_model_yaml or by hand), and that dbt docs generate enriches types but does not add columns.
… 5 smoke test uv pins a package name to the first index that contains it (dependency-confusion guard). Now that signalforge-dbt is published on real PyPI, the TestPyPI smoke install fails to resolve the new prerelease version unless uv is told to consider all indexes. Add the flag plus an explanatory note.
) The #189 grade cache is cross-invocation only (lookups run in the sync prefix before any write of the current run, so there is no intra-run reuse — single-run speed is the #186 asyncio fan-out). Its key mixes a hash of the drafted artefact text, and the drafter is a live, non-deterministic LLM, so a full `signalforge generate` re-run rotates the key and misses on every pair (measured: 370 entries written, 0 read back — docs/research/179-runtime-benchmark.md). Left on by default it silently wrote hundreds of never-hit .signalforge/grade-cache/*.json files and implied a 'fast re-run' UX the architecture can't deliver. Flip GradeConfig.cache_enabled default True -> False. The keying is correct (changed text should re-grade), so the cache stays in the code and is opt-in for the narrow cross-run paths where candidate text is identical (pinned-candidate CI, --no-grade draft-then-grade, resumed grade). Re-key (issue Option 3) rejected; delete deferred — it's the already-correct half of a future draft-cache 'fast re-run' feature. - config.py: default flip + rewritten docstring - grade-ops.md / SKILL.md: correct the overclaiming cache section - tests: default-assertion flip; cache tests opt in via _config_cache_on()
…schema prerequisite - CHANGELOG [Unreleased]: add the missing entries — grade-to-100% (#202), grade budget scaling + cost ceilings (#198), bulk --select shared cache (#188), row_count_between scope instruction (#183), schema.yml-prerequisite docs (#204), and grade cache default flipped True->False (#197) + #189 line reconciled. - README roadmap: move shipped v0.5/v0.6 into the Shipped table (v0.6 = the test-primitives + grade-hardening line), renumber Planned (Airflow->v0.7, etc). - Surface the column-level-tests-need-a-schema.yml prerequisite from the README quick-start and cli-ops.md generate section (cross-linking #204's how-to).
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (259)
📝 WalkthroughWalkthroughConsolidates v0.6.0 documentation, rulebooks, CLI ops, research writeups, and engineering plans for new test primitives, async grading, caching, safety/audit v4, parser re-attach behavior, and grade budget controls. ChangesDocumentation and Planning Consolidation for v0.6.0
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related issues
Possibly related PRs
Poem
|
# Conflicts: # README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cuts v0.6.0 to PyPI. Pre-flight passed (ruff/format/pyright/pytest — 3717 passed);
uv build+uvx twine checkPASSED on wheel and sdist. CHANGELOG promoted from[Unreleased]to[0.6.0]; skill metadata (SKILL.md+SKILL.eval.json) bumped to 0.6.0 in lockstep with__version__.Dev-ahead-of-main release: branched from
dev(41 commits ahead of main), so this PR carries the full v0.6 line — three new test primitives (#169/#170/#171), grade asyncio orchestrator (#186), persistent grade cache + flags (#189/#197), grade-to-100% (#202), grade budget scaling + cost ceilings (#198), bulk--selectshared cache (#188), wide-table safety audit (#185), per-provider judge defaults (#187) — plus the release-prep commit.Summary by CodeRabbit
New Features
row_count_between,unique_combination,row_count_anomaly_by_period.--as-offlag for time-bound anomaly detection reproducibility.--require-completeflag and grade completeness enforcement with controlled degradation.--cache-scopefor Anthropic multi-model prompt-cache sharing.signalforge cache clear --gradecommand with symlink-hardened safety.Docs