#258: Snowflake column_stats parity with Databricks

CHANGELOG.md

@@ -8,7 +8,9 @@ All notable changes to SignalForge are documented here. The format is loosely ba
 
 - **Databricks warehouse adapter (epic #219 — #221–#227).** The third concrete `WarehouseAdapter`, graduating the ABC + factory seam through a second non-BigQuery vendor (Architectural Commitment #3 — warehouse-agnostic by design). `WarehouseAdapter.from_profile` dispatches `type: databricks` dbt profiles via the unified `DbtProfileTarget` with a conditional required-key set keyed on `auth_type` (PAT → `token`, OAuth-M2M → `client_id` + `client_secret`; PAT is the only v0.x connection path) and empty-string-as-missing credential validation (#222); the `databricks-sql-connector` shim is confined to `adapters/_databricks_client.py` (one-shim-per-vendor) and ships under the `[databricks]` optional extra so the base install stays connector-free (#221). The prune compiler emits valid Databricks/Spark-SQL purely from `DATABRICKS_DIALECT` — backtick `quote_char`, `identifier_case='lower'` (Unity Catalog lower-folds — the opposite of Snowflake), per-component qualified-name quoting, and the sign-bit-masked `(xxhash64(to_json(struct(*))) & 9223372036854775807)` sampling hash — never branching on dialect name; certified by 32 byte-exact fixtures and an **ungated** `sqlglot` `databricks`-dialect parse-guard (Databricks has no offline execution fake, so the parse-guard is the sole automated validity gate; #223). Deterministic `sample_rows` (projection-subquery hash-mod — Spark rejects `struct(*)` in a Sort node, so the masked `xxhash64` hash is computed in an inner projection alias referenced by `WHERE`/`ORDER BY`; corrected from the inline shape by #226's live cert) + `get_row_count` (`SELECT COUNT(*)`, since `DESCRIBE DETAIL` has no reliable `numRows`) + `materialise_sample` (qualified `CREATE OR REPLACE TABLE` colocated with the source — Databricks rejects a qualified temp name — dropped fail-soft at session cleanup, since a real table does not auto-reap) + `run_test_sql` (per-row `to_json(struct(*))` capture, `json.loads`-ed) + `column_stats`, all on a connection-bound session with fail-soft cleanup; the `validate_catalog_or_project` union validator unblocks short Unity Catalog catalogs (`main` / `workspace`) without weakening identifier hygiene (#224). `estimate_query_bytes` runs `EXPLAIN COST` and parses the maximum Spark CBO `Statistics(sizeInBytes=…)` across plan nodes, reusing the typed `EstimateUnavailableError` for both the no-parseable-stats case and the present-but-sentinel `8.0 EiB` (`Long.MaxValue` / `spark.sql.defaultSizeInBytes`) no-stats shape — never reporting the ~9-exabyte sentinel (#225). **The #226 live-certification test harness + ops docs:** a hand-rolled `FakeDatabricksConnection` (with cursor-leak assertions) drives session / cleanup / error-mapping / sampling / capture behaviour offline; a hand-crafted manifest seed (`tests/fixtures/databricks/`, `samples.nyctaxi.trips`-shaped) plus a credential-free loads-only test certify `signalforge.manifest.load(...)` against it; three gated `@pytest.mark.databricks` live e2e tests run against a real Free-Edition `2X-Small` warehouse — `estimate_live` (`EXPLAIN COST` returns a positive int), prune-live `materialised` (always-passes drop against a writable-catalog temp table), and a full-pipeline `generate` smoke against `samples.nyctaxi.trips` — each gated by `SF_RUN_DATABRICKS=1` + connection env vars on top of the marker; and a consolidated `docs/warehouse-adapter-ops.md § Databricks adapter` carries the operator-facing surface (install, profile keys, dialect, connection-bound session + sampling, `EXPLAIN COST` estimate, error taxonomy, Free-Edition cost guidance, and the live-certification ledger), with the Free-Edition setup walkthrough in `docs/research/databricks-test-environment.md`. **The live pass surfaced three real adapter bugs — a qualified-temp-name rejection (`materialise_sample` → `CREATE OR REPLACE TABLE` + explicit DROP), `struct(*)` rejected in a Sort node (sampling → projection-subquery shape), and a cross-vendor `QuerySyntaxError` message (made vendor-neutral) — all fixed inline and re-certified live.**
 
-> **`column_stats` IS available for Databricks** — so `safety: aggregate-only` works, a deliberate divergence from Snowflake (whose `column_stats` is not yet implemented; Snowflake parity tracked as #258, out of scope for epic #219). Both `prune.sample_strategy: materialised` (needs a writable source catalog) and `oneshot` (no CTAS) are functional. The #221–#225 surface was certified for SQL *shape* (fake connection + ungated `sqlglot` parse-guard + a maintainer-captured `EXPLAIN COST` fixture) and #226 added the gated live Free-Edition certification. **#227 (the epic-#219 closer) reconciled the Databricks residual set — nothing is silently dropped:** complex-type `column_stats` `MIN`/`MAX` is now skipped and `None`d for Spark `array` / `struct` / `map` / `binary` / `variant` types, honouring the `ColumnStats` DEC-016 contract like BigQuery (implemented); the `str`-valued partition-filter escape now uses a Spark-correct `_escape_spark_string_literal` (`''` doubling + backslash) instead of the reused BigQuery helper (fixed); and `run_stats_query` now overrides the ABC `StatsQueryNotSupportedError` degrade so `row_count_anomaly_by_period` **evaluates** on Databricks rather than routing to `kept-without-evidence` (implemented) — with the anomaly two-query stats path and the `to_json(struct(*))` failing-row capture branch both now **live-certified** against the real Free-Edition warehouse. The complete gated live suite is green: `SF_RUN_DATABRICKS=1 uv run pytest -m databricks --no-cov` → **5 tests pass** (`estimate_live` via `EXPLAIN COST`, materialised-sample prune, the anomaly two-query cert, the `to_json` capture cert, and the full-pipeline `generate` smoke) — no inline fixes needed.
+> **`column_stats` IS available for Databricks** — so `safety: aggregate-only` works; Snowflake reached the same parity in #258 (also in this release), so this is no longer a divergence. Both `prune.sample_strategy: materialised` (needs a writable source catalog) and `oneshot` (no CTAS) are functional. The #221–#225 surface was certified for SQL *shape* (fake connection + ungated `sqlglot` parse-guard + a maintainer-captured `EXPLAIN COST` fixture) and #226 added the gated live Free-Edition certification. **#227 (the epic-#219 closer) reconciled the Databricks residual set — nothing is silently dropped:** complex-type `column_stats` `MIN`/`MAX` is now skipped and `None`d for Spark `array` / `struct` / `map` / `binary` / `variant` types, honouring the `ColumnStats` DEC-016 contract like BigQuery (implemented); the `str`-valued partition-filter escape now uses a Spark-correct `_escape_spark_string_literal` (`''` doubling + backslash) instead of the reused BigQuery helper (fixed); and `run_stats_query` now overrides the ABC `StatsQueryNotSupportedError` degrade so `row_count_anomaly_by_period` **evaluates** on Databricks rather than routing to `kept-without-evidence` (implemented) — with the anomaly two-query stats path and the `to_json(struct(*))` failing-row capture branch both now **live-certified** against the real Free-Edition warehouse. The complete gated live suite is green: `SF_RUN_DATABRICKS=1 uv run pytest -m databricks --no-cov` → **5 tests pass** (`estimate_live` via `EXPLAIN COST`, materialised-sample prune, the anomaly two-query cert, the `to_json` capture cert, and the full-pipeline `generate` smoke) — no inline fixes needed.
+
+- **Snowflake `column_stats` — `safety: aggregate-only` now works on Snowflake (#258).** The `SnowflakeAdapter` implements `column_stats` (previously a deferred `NotImplementedError`), bringing Snowflake to parity with Databricks and BigQuery for column profiling — so `safety.mode: aggregate-only` runs end-to-end on Snowflake, closing the last major mode/scope/strategy gap (with one documented geospatial exception, below). A single `INFORMATION_SCHEMA.COLUMNS` catalog pre-filter resolves each queued column's declared type, then a full BigQuery-style per-table batched aggregate computes `count` / `distinct` / `nulls` / `min` / `max` in one query, with `MIN`/`MAX` skipped (→ `None`) for unorderable Snowflake types (`ARRAY` / `OBJECT` / `VARIANT` / `GEOGRAPHY` / `GEOMETRY`); `BINARY` / `TIME` (SQL-orderable but returning connector types outside the `ColumnStats` union) have their `min`/`max` nulled on read-back, honouring the `ColumnStats` contract. One documented limitation: a model carrying a `GEOGRAPHY` / `GEOMETRY` column cannot be profiled via `aggregate-only` — Snowflake forbids `COUNT(DISTINCT)` on those types — so use `safety: schema-only` for geospatial models. Also hardens the adapter's cursor hygiene: `_execute` / `_execute_to_dicts` / `run_test_sql` now close their cursor in `try/finally` on both the success and failure paths (mirroring `_execute_scalar` and the Databricks PR #257 fix), so repeated queries no longer leak server-side cursor handles.
 
 ## [0.7.0] — 2026-06-17
 

README.md

@@ -92,7 +92,7 @@ Full reference: [Claude Code skill](docs/skills.md) — covers the install path
 
 ## Supported warehouses
 
-SignalForge ships three production warehouse adapters today: **BigQuery** (the original target — exercised end-to-end by `signalforge init-demo` and the quick start below), **Snowflake** (full sampling, materialised-sample CTAS, and `EXPLAIN`-based bytes estimation; one combination — `safety: aggregate-only` / Snowflake `column_stats` — is not yet implemented, every other mode/scope/strategy combination is functional), and **Databricks** (full sampling, both materialised + oneshot strategies, `EXPLAIN COST`-based bytes estimation, and `column_stats` — so `safety: aggregate-only` works too, unlike Snowflake; shipped across #224–#225 and live-certified against Databricks Free Edition in #226). **Postgres** ships as a typed `NotImplementedError` stub; **Redshift** remains on the roadmap.
+SignalForge ships three production warehouse adapters today: **BigQuery** (the original target — exercised end-to-end by `signalforge init-demo` and the quick start below), **Snowflake** (full sampling, materialised-sample CTAS, `EXPLAIN`-based bytes estimation, and `column_stats` — `safety: aggregate-only` implemented in #258, functional across every mode/scope/strategy combination except that a model with a `GEOGRAPHY`/`GEOMETRY` column can't use `aggregate-only`), and **Databricks** (full sampling, both materialised + oneshot strategies, `EXPLAIN COST`-based bytes estimation, and `column_stats` — so `safety: aggregate-only` works too; shipped across #224–#225 and live-certified against Databricks Free Edition in #226). **Postgres** ships as a typed `NotImplementedError` stub; **Redshift** remains on the roadmap.
 
 The architecture is warehouse-agnostic — adapters plug in behind a thin sampling/profiling interface (`WarehouseAdapter.from_profile`), so new vendors slot in without touching the draft / prune / grade / diff stages. Per-warehouse setup (auth, cost guardrails, profile-field requirements) lives in [Configuration](#configuration).
 
@@ -486,8 +486,11 @@ reference (sampling, session cleanup, `EXPLAIN`-based bytes estimation,
 known limitations) is in
 [docs/warehouse-adapter-ops.md § Snowflake adapter](docs/warehouse-adapter-ops.md).
 
-> **Known limitation:** `safety: aggregate-only` (Snowflake `column_stats`)
-> is not yet implemented. Every other combination is functional.
+> **Aggregate-only support (#258):** `safety: aggregate-only` is supported on
+> Snowflake — `column_stats` is implemented. One documented exception: a model
+> carrying a `GEOGRAPHY` / `GEOMETRY` column can't be profiled via
+> `aggregate-only` (Snowflake forbids `COUNT(DISTINCT)` on those types) — use
+> `safety: schema-only` for geospatial models.
 
 ### Databricks
 
@@ -509,8 +512,9 @@ schema: my_schema
 
 Install the adapter dependency with the `[databricks]` extra
 (`pip install "signalforge-dbt[databricks]"`); the base install never pulls
-`databricks-sql-connector` in. Unlike Snowflake, **`column_stats` is
-available**, so `safety: aggregate-only` is functional. Both
+`databricks-sql-connector` in. **`column_stats` is available**, so
+`safety: aggregate-only` is functional (Snowflake reached the same
+parity in #258). Both
 `prune.sample_strategy` values work — `materialised` (a `CREATE OR REPLACE
 TABLE` in the source catalog — Databricks rejects a qualified temp name — so it
 needs a writable catalog; the table is dropped at session cleanup) and

docs/warehouse-adapter-ops.md

@@ -393,7 +393,8 @@ the v0.2 stop-gap, not a permanent surface.
 > sample-mode prune now works on live Snowflake.** The `oneshot` strategy works
 > too since #140 routed its sample row-count through the vendor-neutral
 > `WarehouseAdapter.get_row_count` seam (it previously reached a BigQuery-only
-> `_get_client`); see "Known limitations on live Snowflake" below.
+> `_get_client`); see "Live Snowflake (v0.2) — all safety modes supported"
+> below.
 
 ## Query-bytes estimation (v0.2, issue #36)
 
@@ -722,17 +723,38 @@ passes through unchanged. No `BytesBilledExceededError` equivalent —
 Snowflake has no bytes-billed cap (cost is governed by warehouse size +
 auto-suspend, see below).
 
-**Known limitations on live Snowflake (v0.2) — use `safety: schema-only`.** One
-deferred path remains after #139 fixed the `HASH(*)`-in-predicate bug and #140
-added the vendor-neutral row-count seam. Both `prune.sample_strategy` values now
-work; the combinations certified green by the gated live e2e are
+**Live Snowflake (v0.2) — all safety modes supported.** After #139 fixed the
+`HASH(*)`-in-predicate bug, #140 added the vendor-neutral row-count seam, and
+issue #258 implemented `column_stats`, every `safety` × `scope` ×
+`sample_strategy` combination is functional. The combinations certified by the
+**maintainer-run gated live e2e** suite (opt-in — deselected from normal CI) are
 `safety: schema-only` + `prune.scope: full`, or `prune.scope: sample` with
-either `prune.sample_strategy: materialised` or `oneshot`:
-
-- **`safety: aggregate-only` — unsupported.** Profiles columns via
-  `adapter.column_stats`, which `SnowflakeAdapter` leaves as a deferred
-  `NotImplementedError` (the one v0.2 method not yet implemented).
-  `generate` with `safety.mode: aggregate-only` fails (exit 1).
+either `prune.sample_strategy: materialised` or `oneshot`; the `aggregate-only`
+`column_stats` path is **live-certified** too (#258, via key-pair auth — see below):
+
+- **`safety: aggregate-only` — supported as of #258.** Profiles columns via
+  `adapter.column_stats`, now implemented on `SnowflakeAdapter` (parity with
+  Databricks): a catalog pre-filter over `INFORMATION_SCHEMA.COLUMNS` resolves
+  each column's declared type, then a full BigQuery-style per-table batched
+  aggregate computes `count` / `distinct` / `nulls` / `min` / `max`, with
+  `MIN`/`MAX` skipped (→ `None`) for unorderable Snowflake types (`ARRAY` /
+  `OBJECT` / `VARIANT` / `GEOGRAPHY` / `GEOMETRY`). Types that are SQL-orderable
+  but whose connector return type is outside the `ColumnStats.min`/`max` union
+  (`BINARY` → `bytearray`, `TIME` → `datetime.time`) have their `min`/`max`
+  nulled on read-back rather than raising. `generate` with
+  `safety.mode: aggregate-only` now runs on Snowflake.
+  - **Known limitation (#258):** the aggregate emits `COUNT(DISTINCT <col>)` for
+    every column (mirroring the BigQuery adapter). Snowflake forbids `DISTINCT`
+    on `GEOGRAPHY` / `GEOMETRY`, so a model carrying such a column cannot be
+    profiled via `aggregate-only` — the aggregate fails with a typed
+    `WarehouseError`. Use `safety: schema-only` for models with geospatial
+    columns. `DISTINCT` on `VARIANT` / `ARRAY` / `OBJECT` does **not** raise —
+    confirmed by the gated live complex-type cert
+    (`tests/warehouse/test_snowflake_columnstats_live.py`), which profiles those
+    three types and asserts `min=max=None` without error against a real
+    warehouse (run 2026-07-03 via key-pair auth). The `GEOGRAPHY`/`GEOMETRY`
+    `COUNT(DISTINCT)` limit is by inspection of Snowflake's documented
+    restriction, not exercised by the cert (no geospatial column in the fixture).
 
 **Fixed by #140:** `prune.scope: sample` + `prune.sample_strategy: oneshot` on a
 non-BigQuery adapter no longer raises at the engine seam. The sample row-count is
@@ -933,12 +955,11 @@ supplementary-source boundary and renders `<unavailable:
 EstimateUnavailableError>`, falling back to a price-only preview.
 
 **`column_stats` is AVAILABLE for Databricks — `safety: aggregate-only`
-works.** This is a **deliberate divergence from Snowflake**, whose
-`column_stats` (and therefore `safety.mode: aggregate-only`) is **not yet
-implemented**. The Databricks adapter ships `column_stats` (issue #224,
+works.** Snowflake reached the same capability in #258, so this is no longer
+a divergence. The Databricks adapter ships `column_stats` (issue #224,
 DEC-011) as a single aggregate query — `count` / `distinct` / `nulls` /
 `min` / `max` / `data_type` (the last via `MAX(typeof(<col>))`) — over the
-fold-then-quoted column. The Snowflake-parity follow-up is tracked as issue #258.
+fold-then-quoted column. Snowflake's `column_stats` parity shipped in issue #258.
 **Complex-type `MIN`/`MAX` parity (#227 US-001):** like BigQuery, Databricks
 now skips `MIN`/`MAX` (→ `None`) for complex Spark types (`array` / `struct` /
 `map` / `binary` / `variant`), honouring the `ColumnStats` DEC-016 contract.
@@ -1020,8 +1041,8 @@ path + the `to_json(struct(*))` capture branch are live-certified (US-004,
 above). The remaining **shape-only** (not live-exercised) path is
 `column_stats` `MIN`/`MAX` on complex-typed columns — the skip-and-`None`
 post-process is unit-tested but the live pass exercised only scalar columns.
-(Snowflake `column_stats` parity is separate — issue #258, out of scope for
-epic #219.)
+(Snowflake `column_stats` parity shipped separately in issue #258 — out of
+scope for epic #219.)
 
 **Cost guidance — read before running any live Databricks test.** The live
 target is **Databricks Free Edition** (serverless-only). Its single SQL