#184: drafter row_count_anomaly_by_period scope fix

[wjduenow]

Summary

Super plan for #184 — drafter mis-scopes row_count_anomaly_by_period to date columns on models with audit timestamps (reproduced 3/3 on Phase B of the #179 retest).

Phase: detailing (awaiting approval)
Stories: 8 implementation + Quality Gate + Patterns & Memory = 10 total
Decisions: 11 captured (DEC-001 … DEC-011)

What the plan ships

• Primary lever: prose-only rewrite of _ROW_COUNT_ANOMALY_SCOPE_INSTRUCTION to explicitly teach model-level scope (closes the LLM-steering gap).
• Belt-and-braces parser: silent re-attach for column-scoped emissions of all three model-only variants (row_count_anomaly_by_period / row_count_between / unique_combination), with an always-emit _LOGGER.warning AND a durable audit record (parser_reshaped: tuple[ReshapeRecord, ...] on LLMResponseEvent).
• LLMResponseEvent.audit_schema_version: 1 → 2 — new fixture row + drift-detector strict mirror added; v1 fixture stays valid (extra="ignore").
• exclude_tests kill-switch beats re-attach — operator opt-out is always honoured.
• Validation: full 15-candidate Phase B re-aggregation against the intuit_airflow substrate; results land as a "Followup #171 follow-on: Drafter mis-scopes row_count_anomaly_by_period to date columns on models with audit timestamps #184 resolution" section in docs/research/179-test-primitive-expansion-retest.md.

Eight-surface lockstep change (extends #183's six-surface template)

prompts.py + parser.py + audit.py + test_prompts.py + test_prompt_cache_stability.py + test_parser.py + drift detector / fixture + four doc surfaces (draft-ops.md, two .claude/rules/ files, CHANGELOG.md).

Plan document

See plans/super/184-anomaly-scope-fix.md for the full plan — decisions, story breakdown, dependency ordering, and Quality Gate focus areas.

Next steps

• Review the plan in this PR
• Reply "approved" / "looks good" to proceed to devolve (beads creation)

Related

• Add row_count_anomaly_by_period: drafter-proposed historical-trend row-count guard #171 (origin variant)
• Epic: Test Generation Expansion (#169 + #170 + #171) #179 (retest that surfaced it) / PR #179: empirical retest writeup against intuit_airflow #182 (writeup)
• #169 follow-on: Add _ROW_COUNT_BETWEEN_SCOPE_INSTRUCTION (DEC-012 worked example never landed) #183 / PR #183: Add _ROW_COUNT_BETWEEN_SCOPE_INSTRUCTION #191 (parallel row_count_between scope-instruction pattern; merged 2026-06-02)

Summary by CodeRabbit

• Bug Fixes

  • Model-only tests (row_count_anomaly_by_period, row_count_between, unique_combination) incorrectly placed under a column are now automatically moved to model scope; a single operator-visible warning is emitted when this happens.

• Documentation

  • Updated guidance and worked examples to show correct model-level placement and to advise removing prior workarounds.

• Audit

  • Audit trail now records when the parser performs an automatic re-attachment for forensic visibility.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: b6a16a44-aa7d-4564-8362-031910cbad9e

📥 Commits

Reviewing files that changed from the base of the PR and between aa7623f and 7f8401b.

📒 Files selected for processing (1)

• docs/research/179-test-primitive-expansion-retest.md

🚧 Files skipped from review as they are similar to previous changes (1)

• docs/research/179-test-primitive-expansion-retest.md

---

📝 Walkthrough

Walkthrough

This PR rewrites prompt guidance and adds parser recovery that detects model-only tests emitted under column scope, emits a structured warning, re-attaches them to model scope (with argument revalidation), records ReshapeRecord(s) in a v2 audit event, and updates tests/docs end-to-end.

Changes

Model-only variant scope correction

Layer / File(s)	Summary
Audit schema v2 with ReshapeRecord src/signalforge/draft/audit.py, tests/draft/test_audit.py, tests/fixtures/draft/llm_response_with_reshape_v2.jsonl	New frozen ReshapeRecord model and LLMResponseEvent.parser_reshaped: tuple[ReshapeRecord,...] = (); audit_schema_version default bumped to 2; _build_response_event accepts parser_reshaped and exports updated.
Parser re-attach core logic src/signalforge/draft/parser.py, src/signalforge/draft/schema.py, tests/draft/test_parser.py	_validate_anchor_contract detects model-only variants in column scope, emits _LOGGER.warning("parser re-attach: %s", json_payload), optionally appends ReshapeRecord, records (col_idx,test_idx) reattach actions, validates test args at model scope via _validate_model_only_test_args, and _apply_reattach_actions rebuilds the frozen CandidateSchema before raising violations; respects exclude_tests and preserves collect-all/no-dedupe semantics.
Prompt scope guidance update src/signalforge/draft/prompts.py, tests/draft/test_prompts.py, tests/llm/test_prompt_cache_stability.py	Rewrite of _ROW_COUNT_ANOMALY_SCOPE_INSTRUCTION with a worked YAML example teaching model-level placement; prompt-version pin rotated and prompt tests added/updated.
End-to-end wiring through pipeline src/signalforge/draft/schema.py	draft_from_request initializes reshapes_collected: list[ReshapeRecord], passes it and model.unique_id into parse_draft_response, and attaches parser_reshaped=tuple(reshapes_collected) to the persisted audit event.
Integration tests and drift detector tests/draft/test_schema.py, tests/draft/test_drift_detector.py, tests/fixtures/draft/llm_response_with_reshape_v2.jsonl	Integration tests validate re-attach and audit JSONL round-trip; drift detector adds StrictReshapeRecord mirror and v2 fixture checks; tests updated for audit_schema_version==2 and parser_reshaped defaults.
User documentation and reference .claude/rules/business-rule-tests.md, .claude/rules/llm-drafter.md, CHANGELOG.md, docs/draft-ops.md, docs/research/179-test-primitive-expansion-retest.md, plans/super/184-anomaly-scope-fix.md	Business rules, drafter rules, CHANGELOG, ops docs, research notes, and plan updated to describe re-attach conventions (dual prompt+parser defense, exclude_tests precedence, frozen-model rebuild via model_copy(update=...), logger-string formatting constraint, and audit threading).

Sequence Diagram (high-level)

sequenceDiagram
  participant Client as draft_from_request
  participant Parser as parse_draft_response
  participant Anchor as _validate_anchor_contract
  participant Args as _validate_model_only_test_args
  participant Logger as _LOGGER
  participant Rebuilder as _apply_reattach_actions
  participant Audit as _build_response_event

  Client->>Parser: parse candidate (pass reshapes_collected, model_unique_id)
  Parser->>Anchor: validate columns/tests (collect reattach actions)
  Anchor->>Logger: warning("parser re-attach: %s", payload_json)
  Anchor->>Args: re-validate test args at model scope
  Anchor-->>Parser: violations + reattach actions + reshape records
  Parser->>Rebuilder: apply reattach actions (immutable rebuild)
  Rebuilder-->>Parser: rebuilt CandidateSchema
  Parser->>Audit: build event with parser_reshaped

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related issues

• #171 follow-on: Drafter mis-scopes row_count_anomaly_by_period to date columns on models with audit timestamps #184: Directly addressed by this PR — implements parser re-attach, prompt updates, and audit-schema changes.
• Epic: Test Generation Expansion (#169 + #170 + #171) #179: Related epic for test primitives (row_count_anomaly_by_period, row_count_between, unique_combination) corrected here.

Possibly related PRs

• wjduenow/SignalForge#176: Related anchor-contract handling for row_count_between; closely connected to parser changes here.
• wjduenow/SignalForge#83: Prior exclude_tests / anchor-contract work that this change respects.
• wjduenow/SignalForge#181: Earlier row_count_anomaly_by_period validation work extended by this PR.

Poem

🐰 I found a test where it didn't belong,

Moved it gently, sang a warning song.
Audit keeps the tale in tidy rows,
Prompt now shows where the model goes.
Hop, reshape, log — the rabbit knows!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The PR title clearly and concisely summarizes the main change: fixing the scoping of row_count_anomaly_by_period in the drafter to resolve issue #184. It is specific, directly related to the core changeset across all modified files.
Docstring Coverage	✅ Passed	Docstring coverage is 89.74% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[wjduenow]

@CodeRabbit review

[Copilot]

Pull request overview

This PR addresses issue #184 where the LLM drafter incorrectly emits the model-only row_count_anomaly_by_period test at column scope (often on audit timestamp columns), causing the parser’s anchor-contract validation to fail. The change tightens prompt guidance to explicitly teach model-level placement, and adds a parser-side “belt-and-braces” re-attach mechanism with warning + durable audit trail so runs don’t fail when the LLM still mis-scopes.

Changes:

• Update _ROW_COUNT_ANOMALY_SCOPE_INSTRUCTION to explicitly state model-level placement and include a worked YAML example; rotate prompt version and update cache-stability pin.
• Add parser support to re-attach column-scoped emissions of model-only variants (row_count_anomaly_by_period, row_count_between, unique_combination) to model scope, emitting a warning and recording reshapes.
• Bump LLMResponseEvent.audit_schema_version from 1 → 2 and add parser_reshaped audit field, with fixtures + strict drift-detector coverage and end-to-end threading from draft_from_request.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/signalforge/draft/prompts.py	Clarifies anomaly test model-level scope and adds worked example; preserves calibration guidance.
tests/draft/test_prompts.py	Pins the new scope wording and worked example so prompt regressions fail loudly.
tests/llm/test_prompt_cache_stability.py	Updates _EXPECTED_PROMPT_VERSION to match the rotated prompt hash and documents the rotation.
src/signalforge/draft/parser.py	Implements column→model re-attach for model-only variants, warning emission, and reshape collection + rebuild.
tests/draft/test_parser.py	Adds comprehensive re-attach behavior tests (all 3 variants, warning shape, exclude gate, collect-all, arg validation, sibling preservation).
src/signalforge/draft/audit.py	Introduces ReshapeRecord, adds parser_reshaped, and bumps audit_schema_version default to 2.
src/signalforge/draft/schema.py	Threads reshapes_collected into parsing and forwards it into the response audit event.
tests/draft/test_schema.py	Verifies reshape threading into the audit JSONL and v2 defaults on the no-reshape path.
tests/draft/test_audit.py	Updates audit schema default expectations and asserts the new field default.
tests/draft/test_drift_detector.py	Adds strict mirrors for ReshapeRecord + v2 fixture validation while keeping v1 fixture valid.
tests/fixtures/draft/llm_response_with_reshape_v2.jsonl	New v2 audit fixture exercising the reshape path.
docs/draft-ops.md	Documents the new audit field and the parser re-attach behavior (including exclude_tests precedence).
docs/research/179-test-primitive-expansion-retest.md	Adds follow-up writeup section describing the #184 resolution and partial validation results.
CHANGELOG.md	Adds Fixed + Changed entries for the scope fix and audit schema bump.
.claude/rules/llm-drafter.md	Documents the parser re-attach carve-out and invariants (exclude gate, collect-all, warning shape, no dedupe).
.claude/rules/business-rule-tests.md	Captures #184 as a durable pattern for dual-defence prompt+parser fixes with audit visibility.
plans/super/184-anomaly-scope-fix.md	Adds the detailed plan/DEC log and acceptance criteria for the change set.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 1 comment.

[wjduenow]

PR Review Summary

Fixed (1 item)

File	Line	Issue	Commit
docs/research/179-test-primitive-expansion-retest.md	348	Writeup overclaimed that DEC-008's acceptance bar permits a 3/15-candidate run; the plan's DEC-008 actually commits to all 15.	7f8401b

The writeup now honestly marks Status: PARTIAL — 3 of 15 candidates run, names the remaining 12 candidates as PENDING, and scopes the 3-candidate evidence to "the bug is fixed on the load-bearing failure shape" (necessary but not sufficient for the full DEC-008 close). A future maintainer-side pass with a fresh API window (or chunked across sessions to avoid the Anthropic rate-limit pressure that capped this run) will close the gap.

False Positives (0 items)

No false positives in this review pass.