fix(#171): row_count_anomaly_by_period emits valid BigQuery SQL

[wjduenow]

Summary

While pre-release-testing the Airflow epic (#228), the gated live BigQuery e2e (which had not been run since #171 merged) revealed that row_count_anomaly_by_period never produced executable BigQuery SQL — anomaly detection silently degraded to kept-without-evidence on every BigQuery run. Snapshot fixtures pinned the invalid SQL byte-for-byte, so the offline suite was green.

Root causes

1. PERCENTILE_CONT(p) WITHIN GROUP (ORDER BY …) (the mad / percentile methods) is a standard-SQL ordered-set aggregate BigQuery does not support — its PERCENTILE_CONT is window-only and cannot reduce rows in a GROUP BY. BigQuery now uses the GROUP-BY-compatible APPROX_QUANTILES(expr, 100)[OFFSET(round(p*100))] idiom. Snowflake / Postgres keep the WITHIN GROUP form — the Dialect default flipped, so POSTGRES_DIALECT now declares its percentile template explicitly (a BigQuery-only idiom can't be a shared default).
2. TIMESTAMP-column vs DATE-literal bound. The DEC-012 partition-pruning predicate compared a TIMESTAMP date column against a DATE('…') bound → No matching signature for >= : TIMESTAMP, DATE. The bound literal is now type-matched to the date column's warehouse data_type (new Dialect.datetime_literal_template + _date_value_literal, threading the column type from model.columns[date_column] into both the engine-side compiler and the operator-shipped singular-test compiler). CASTing the column would have disabled partition pruning, so the literal is matched, not the column.

Also fixes the read_prune_decisions test helper dropping as_of / stats on read-back.

Certification

• Live BigQuery anomaly e2e passes (tests/cli/test_e2e_row_count_anomaly.py, real Anthropic + BigQuery) — valid SQL, engineered volume drop detected → kept, as_of/stats populated. Each compiled stats + violation query independently validated via bq query --dry_run.
• Offline: ruff ✓, format ✓, pyright (0 errors) ✓, pytest (4125 passed, including 2 new TIMESTAMP-column regression tests for the engine and emitter paths).
• Snowflake offline (56 fakesnow/sqlglot) unaffected; BQ mad/percentile snapshots regenerated; zscore/min_max unchanged.

Durable lesson

A Dialect date/percentile SQL fragment is not validated by snapshot equality — only a live (or sqlglot/executor) run catches invalid-but-byte-stable SQL. Captured in .claude/rules/business-rule-tests.md § "#171 follow-up correction" and warehouse-adapters.md.

Closes the row_count_anomaly bug found pre-release; the Airflow feature (#228) itself is unaffected and green.

Summary by CodeRabbit

Release Notes

• Bug Fixes

  • Fixed BigQuery anomaly SQL generation for row_count_anomaly_by_period to use executable percentile syntax.
  • Corrected partition-pruning predicates to type-match bound date/time literals to the configured column type.
  • Updated prune decision parsing to include as_of and stats when reading back results.

• Tests

  • Refreshed BigQuery SQL fixtures and added regression coverage for percentile rounding and date-literal type matching.

• Documentation

  • Updated dialect/template guidance for percentile and datetime literal handling.

• Chores

  • Renamed the example Airflow drift DAG file and updated its dag_id.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 2b4998f0-576e-4f3c-8ac2-907258a9b5e5

📥 Commits

Reviewing files that changed from the base of the PR and between c5388e6 and 62e73d9.

📒 Files selected for processing (2)

• src/signalforge/prune/compiler.py
• tests/prune/test_compiler.py

🚧 Files skipped from review as they are similar to previous changes (1)

• src/signalforge/prune/compiler.py

---

📝 Walkthrough

Walkthrough

Fixes two BigQuery bugs in row_count_anomaly_by_period: replaces the invalid PERCENTILE_CONT ... WITHIN GROUP form with APPROX_QUANTILES(...)[OFFSET(...)] in the BigQuery dialect, and adds a datetime_literal_template field plus a _date_value_literal helper to type-match partition-pruning bound literals to the date column's warehouse data_type. Both fixes are threaded from Dialect through the prune compiler and diff emitter.

Changes

BigQuery anomaly SQL generation fixes

Layer / File(s)	Summary
Dialect model: datetime_literal_template and percentile defaults src/signalforge/warehouse/models.py	Adds datetime_literal_template field to Dialect (BigQuery and Snowflake supply values). Changes the Dialect default percentile_cont_expr_template to BigQuery's APPROX_QUANTILES({expr}, 100)[OFFSET({offset})] form and explicitly overrides POSTGRES_DIALECT with the standard ordered-set PERCENTILE_CONT template. Updates module docstring.
Compiler: _date_value_literal, percentile offset, and type threading src/signalforge/prune/compiler.py	Introduces _date_value_literal to select DATE/DATETIME/TIMESTAMP literal templates by column type. Adds optional column_type/date_column_type to all CTE and query builders. Updates _percentile_expr to compute and pass an integer offset. _compile_test resolves anomaly_date_column_type from model.columns and threads it through _compile_row_count_anomaly_by_period.
Diff emitter wiring src/signalforge/diff/_emitter.py	When emitting a kept row_count_anomaly_by_period test, looks up data_type for date_column in model.columns and passes date_column_type into _compile_anomaly_singular_test_sql.
Tests, fixtures, and helpers tests/prune/test_compiler.py, tests/warehouse/test_models.py, tests/fixtures/prune/compiled_sql/anomaly/bigquery/*, tests/fixtures/diff/proposed_test_files/anomaly/orders__row_count_anomaly_by_period_8e4d6245.sql, tests/cli/_e2e_helpers.py, CHANGELOG.md, .claude/rules/*.md	Updates four BigQuery SQL fixtures and the diff proposed-test fixture to use APPROX_QUANTILES. Adds parameterized tests for bound-literal type-matching and singular-test type-matching. Updates dialect model unit tests for new field/template defaults. Fixes read_prune_decisions helper to forward as_of and stats. Adds changelog and rules documentation.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

• wjduenow/SignalForge#125: Introduces SNOWFLAKE_DIALECT and Snowflake adapter in src/signalforge/warehouse/models.py, the same file where this PR adds datetime_literal_template and modifies Snowflake's dialect constant.
• wjduenow/SignalForge#127: Modifies both src/signalforge/prune/compiler.py and src/signalforge/warehouse/models.py to make date/timestamp literal rendering dialect-driven — directly related to this PR's type-matched partition-literal work.
• wjduenow/SignalForge#181: Implements the original row_count_anomaly_by_period feature in the same compiler and emitter files that this PR corrects for BigQuery.

Poem

🐇 Hoppity-hop through the SQL land,
Where PERCENTILE_CONT was wrongly planned!
BigQuery said "APPROX is the way,"
So I matched my timestamps without delay.
DATE, DATETIME, TIMESTAMP — all in line,
The partition pruning now works just fine! ✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'fix(#171): row_count_anomaly_by_period emits valid BigQuery SQL' clearly summarizes the main change: fixing issue #171 to generate valid BigQuery SQL for the row_count_anomaly_by_period test method.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Copilot]

Pull request overview

Fixes row_count_anomaly_by_period so it generates executable BigQuery SQL (restoring anomaly detection from silent kept-without-evidence), by moving BigQuery percentile rendering to APPROX_QUANTILES and type-matching DEC-012 partition-pruning bound literals to the date column’s warehouse data_type across both the engine compiler and emitted dbt singular tests.

Changes:

• Switch BigQuery’s GROUP-BY percentile implementation to APPROX_QUANTILES(...)[OFFSET(...)] while keeping Snowflake/Postgres on ordered-set PERCENTILE_CONT ... WITHIN GROUP.
• Type-match anomaly partition bound literals (DATE/DATETIME/TIMESTAMP) using manifest column data_type, threaded through engine compilation and the diff emitter.
• Update fixtures/tests and fix read_prune_decisions helper to preserve as_of/stats.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/warehouse/test_models.py	Updates dialect field expectations (new datetime_literal_template, BigQuery percentile default).
tests/prune/test_compiler.py	Adds regression tests for type-matched bound literals; updates percentile SQL expectations for BigQuery.
tests/fixtures/prune/compiled_sql/anomaly/bigquery/percentile_none_stats.sql	Snapshot update: BigQuery percentile stats use APPROX_QUANTILES.
tests/fixtures/prune/compiled_sql/anomaly/bigquery/percentile_dow_stats.sql	Snapshot update: BigQuery DOW percentile stats use APPROX_QUANTILES.
tests/fixtures/prune/compiled_sql/anomaly/bigquery/mad_none_stats.sql	Snapshot update: BigQuery MAD stats use APPROX_QUANTILES median/MAD.
tests/fixtures/prune/compiled_sql/anomaly/bigquery/mad_dow_stats.sql	Snapshot update: BigQuery seasonal MAD stats use APPROX_QUANTILES.
tests/fixtures/diff/proposed_test_files/anomaly/orders__row_count_anomaly_by_period_8e4d6245.sql	Snapshot update: emitted singular test SQL uses BigQuery-legal percentile form.
tests/cli/_e2e_helpers.py	Preserves as_of and stats when reading prune decisions back from audit events.
src/signalforge/warehouse/models.py	Extends Dialect with datetime_literal_template; flips BigQuery percentile default; makes Postgres declare ordered-set percentile explicitly.
src/signalforge/prune/compiler.py	Adds _date_value_literal and threads date_column_type; updates percentile rendering to include offset.
src/signalforge/diff/_emitter.py	Threads model column data_type into singular-test SQL compilation for type-matched bounds.
CHANGELOG.md	Documents the BigQuery fix and the read-back helper fix.
.claude/rules/warehouse-adapters.md	Documents the dialect fragment corrections and new datetime literal fragment.
.claude/rules/business-rule-tests.md	Records the durable “live cert vs snapshot” lesson and the concrete fixes.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[wjduenow]

Addressed the Copilot review note on _percentile_expr rounding in 62e73d9: switched the APPROX_QUANTILES OFFSET from Python's banker's-rounding round(p*100) to round-half-up int(p*100 + 0.5), so a half-integer bucket like p=0.125 → OFFSET(13) (away from zero) instead of the even 12. All realistic inputs (median p=0.5, integer-threshold percentiles) are exact integers, so no snapshot moved; added a regression test for the half-integer case.