From 961ef7a9a5b1580c995b2b5c9c9ed3022094fd9d Mon Sep 17 00:00:00 2001 From: Wes Duenow Date: Wed, 24 Jun 2026 20:56:17 -0700 Subject: [PATCH 01/10] Add super plan for #223: Databricks prune compiler dialect --- plans/super/223-databricks-prune-compiler.md | 136 +++++++++++++++++++ 1 file changed, 136 insertions(+) create mode 100644 plans/super/223-databricks-prune-compiler.md diff --git a/plans/super/223-databricks-prune-compiler.md b/plans/super/223-databricks-prune-compiler.md new file mode 100644 index 00000000..d05cf718 --- /dev/null +++ b/plans/super/223-databricks-prune-compiler.md @@ -0,0 +1,136 @@ +# 223 — Databricks: prune compiler emits valid Databricks SQL from DATABRICKS_DIALECT + +## Meta + +- **Ticket:** https://github.com/wjduenow/SignalForge/issues/223 +- **Epic:** #219 (Databricks adapter). Depends on #221 (skeleton). Models on #121 (Snowflake compiler dialect). +- **Branch / worktree:** `feat/223-databricks-prune-compiler` @ `/home/wesd/Projects/worktrees/SignalForge/223-databricks-prune-compiler` +- **Phase:** detailing +- **Sessions:** 1 (2026-06-24) +- **Scoping resolved:** Q1 → ungated parse-guard (DEC-002 stands); Q2 → mirror Snowflake 16+16 (DEC-003 stands). + +--- + +## Discovery + +### What / Why / Who + +**What:** Make `signalforge.prune.compiler._compile_test` emit valid Databricks/Spark-SQL for all 8 test primitives **purely from `DATABRICKS_DIALECT`** (zero `dialect.name` branches), and prove it with byte-exact fixtures + a `sqlglot` `databricks`-dialect parse-guard. + +**Why:** Third warehouse on the dialect-driven prune seam (after BigQuery + Snowflake). Architectural Commitment #3 ("warehouse-agnostic by design") — the compiler stays vendor-neutral; a new warehouse drops in via a sibling `Dialect` constant + fixtures, no compiler edits. Validity (not just shape) must be machine-checked per the #121 lesson: *snapshot equality certifies shape, not validity — keep a parser/executor in the loop.* + +**Who:** Operators running SignalForge against Databricks (epic #219). Live execution is #226; this ticket is the offline compile+validate gate. + +### Key codebase findings (de-risking prototype run during discovery) + +The heavy lifting **already landed in #221**: `DATABRICKS_DIALECT` is fully defined (`src/signalforge/warehouse/models.py:308`) and the compiler is already 100% dialect-driven (every `_render_*` / `_compile_*` reads `Dialect` fields; the import-guard already forbids name-branching). A discovery prototype compiled **all 8 primitives × {full, sample} scope** (including the anomaly two-query split across mad/zscore/percentile + dow seasonality) with `DATABRICKS_DIALECT` and parsed every emitted statement through `sqlglot.parse_one(sql, dialect="databricks")`: + +``` +26/26 statements: OK (0 FAILURES) +``` + +**Implications:** +- No `DATABRICKS_DIALECT` change is required — the #221 values are correct as-shipped. The AC's "verify/extend the Dialect fields" reduces to **verify** (+ pin with unit tests). +- The reserved-word CTE-alias risk (the Snowflake `"sample"` bug) **does not bite Databricks**: `SAMPLE` is not a Spark reserved word; unquoted `WITH sample AS …` parses cleanly under sqlglot's `databricks` dialect. `DATABRICKS_DIALECT` keeps the default `sample_cte_alias="sample"` (no override needed). +- The ticket is therefore **test-authoring + fixtures + one import-guard line + docs** — low-risk, well-bounded. + +**Precedent to mirror (Snowflake #121/#171):** +- `tests/fixtures/prune/compiled_sql/snowflake/` — 16 top-level fixtures (4 built-ins + custom_sql×3 + row_count_between×2 + unique_combination×3, each + sample variants where applicable). +- `tests/fixtures/prune/compiled_sql/anomaly/snowflake/` — 16 anomaly fixtures (4 methods × 2 seasonality × {stats, violation}). +- `tests/prune/test_compiler.py` — **ungated** byte-exact snapshot tests (individual funcs for built-ins; parametrized `(method, seasonality)` for anomaly) + per-dialect unit tests (`test_quote_folds_and_quotes_for_snowflake`, `test_qualified_table_name_per_component_for_snowflake`, sample-CTE-uses-HASH, partition-filter cast-form) + belt-and-braces "BigQuery quoting must not leak" assertions. +- `tests/prune/test_compiler_fakesnow.py` — **gated** `@pytest.mark.snowflake` file co-locating fakesnow execution + the sqlglot parse-guard. +- `tests/prune/test_compiler_import_guard.py` — `_FORBIDDEN_PREFIXES = ("snowflake", "google.cloud")` (AST scan; planted-violation self-check asserts 9 hits). + +### Facts that shape the plan + +- `sqlglot>=30,<31` is a **base runtime dependency** (`[project].dependencies`), so `sqlglot.parse_one(..., dialect="databricks")` is importable in the **default** test suite — no marker needed for a pure-sqlglot parse-guard. +- The `databricks` pytest marker already exists and is already in the `addopts` deselection list (default runs exclude it). Its description already says "offline sqlglot/fake validation AND the gated live Free-Edition certification." +- Databricks has **no offline execution fake** (no fakesnow/DuckDB equivalent for Spark SQL). So sqlglot parse is the *only* automated validity check until the #226 live Free-Edition cert. +- Fixtures use the existing `fake_project.dataset.orders` 3-part TableRef. `fake_project` (12 chars) passes `validate_project_id` (6–30), so the Unity-Catalog short-catalog-name gotcha (`main`/`workspace` < 6 chars failing `TableRef.project`) is a **#224 concern, not #223**. Databricks renders the ref per-component-backtick: `` `fake_project`.`dataset`.`orders` ``. + +--- + +## Architecture Review + +| Area | Rating | Finding | +|---|---|---| +| Warehouse-agnostic seam | **pass** | Compiler already dialect-driven; prototype proves valid output for all 8 primitives. No `dialect.name` branch added. | +| Import-guard confinement | **pass** | Add `databricks` to `_FORBIDDEN_PREFIXES` + planted self-check (9→12 hits). No SDK import under `prune/` — there never was one (compiler reads `Dialect` only). | +| Validation tier (#121 lesson) | **concern → resolved** | Snapshot pins *shape*; sqlglot parse-guard pins *syntactic validity*; real-Spark *semantics* deferred to #226. Decision on parse-guard **gating** captured as DEC-002. | +| Reserved-word CTE alias | **pass** | Prototype confirms unquoted `sample` parses under `databricks` dialect; no `sample_cte_alias` override. Belt-and-braces assertion will pin it. | +| BigQuery/Snowflake regression | **pass** | This ticket only **adds** databricks fixtures + tests; compiler/models untouched, so existing fixtures are byte-unchanged by construction. A regression assertion makes it explicit. | +| Reproducibility caveat | **pass** | `xxhash64` is engine/release-stable, not cross-time (same caveat Snowflake's `HASH()` documented). Already noted in the `DATABRICKS_DIALECT` docstring. | +| Testing strategy | **pass** | Mirrors Snowflake exactly: ungated byte-exact snapshots + dialect unit tests + (gated?) sqlglot parse-guard + planted-violation self-check on the import-guard. | + +No blockers. One concern (parse-guard gating) → DEC-002. + +--- + +## Refinement Log + +### DEC-001 — No `DATABRICKS_DIALECT` change; verify-and-pin only +The #221 dialect values are correct as-shipped (prototype: 26/26 statements parse). This ticket **verifies** (unit tests + fixtures + parse-guard) rather than edits the dialect. If any field were found wrong, the fix would land here — but none is. Rationale: avoid touching `models.py` keeps BigQuery/Snowflake byte-unchanged trivially true. + +### DEC-002 — sqlglot parse-guard is **UNGATED** (deviates from issue's literal "gated under the `databricks` marker") +**Decision:** the `sqlglot.parse_one(sql, dialect="databricks")` parse-guard runs in the **default** suite (no marker), in a new `tests/prune/test_compiler_databricks.py`. +**Rationale:** (a) `sqlglot` is a base dependency — always importable in CI, zero added cost; (b) Databricks has **no** offline execution fake, so this parse-guard is the *sole* automated validity gate until #226's live cert — gating it behind a marker CI never runs would mean the validity gate only fires when a maintainer remembers `-m databricks`, exactly the gap the #121 lesson warns against; (c) testing-signal.md's "keep a parser/executor in the loop" is best served by running the parser on **every** CI run. The Snowflake parse-guard is gated only because it is *co-located* with fakesnow execution (which needs the gated dep) — there is no such coupling here. +**Deviation note:** the issue text says "gated under the `databricks` marker." This DEC consciously deviates; the byte-exact snapshot tests are also ungated, so shape + validity are both default-CI-checked. *(Pending user confirmation — see scoping Q1.)* + +### DEC-003 — Fixture set mirrors the **Snowflake** set (16 top-level + 16 anomaly = 32), not the broader BigQuery set +Mirror the most-recent non-BQ dialect for review parity: 4 built-ins (+ sample variants) + custom_sql (full/sample/fullscan) + row_count_between (plain + where) + unique_combination (pair/three/where) = 16 top-level; anomaly = 4 methods × 2 seasonality × {stats, violation} = 16. Covers all 8 primitives + every anomaly shape. (BigQuery's extra `row_count_between_only_min/only_max` are omitted, matching Snowflake.) + +### DEC-004 — Parse-guard lives in a **new** `tests/prune/test_compiler_databricks.py` +No fakesnow/execution to co-locate (Databricks has no in-memory fake), so a dedicated file (not an extension of `test_compiler_fakesnow.py`). Holds the parametrized parse-guard over all 32 fixtures + the import-guard is updated separately. + +### DEC-005 — Belt-and-braces "no foreign-dialect leakage" assertions +Each Databricks snapshot test asserts the distinguishing markers: `xxhash64` present, `FARM_FINGERPRINT` absent (BigQuery), `HASH(*)` absent (Snowflake), `::` cast-literal absent (Snowflake). Backtick presence cannot distinguish BQ vs Databricks (both use `` ` ``), so it is not used as a discriminator. + +--- + +## Detailed Breakdown + +> Validation command (every story): `uv sync --dev && uv run ruff check . && uv run ruff format --check . && uv run pyright && uv run pytest`. Gated databricks tests: `uv run pytest -m databricks --no-cov`. + +### US-001 — Import-guard: confine the `databricks` SDK out of `prune/` +- **Traces to:** DEC (warehouse-agnostic seam), AC "import-guard green". +- **Description:** Add `"databricks"` to `_FORBIDDEN_PREFIXES` in `tests/prune/test_compiler_import_guard.py`; extend the planted-violation self-check (`import databricks`, `from databricks import sql`, `import databricks.sql as dbx`, etc.) and bump the expected hit count. +- **Files:** `tests/prune/test_compiler_import_guard.py`. +- **Done when:** scan covers `databricks` + planted self-check asserts the new count; `test_no_warehouse_sdk_import_under_prune` green (no databricks import exists under `prune/`). +- **Depends on:** none. +- **TDD:** add planted-violation lines first, watch the count assertion fail, then bump. + +### US-002 — Compiler dialect unit tests for Databricks +- **Traces to:** DEC-001, DEC-005, AC "verify the Dialect fields the compiler consumes". +- **Description:** Mirror the Snowflake per-dialect unit tests for `DATABRICKS_DIALECT`: `_quote` folds-lower + backtick-quotes (`` `customer_id` ``); `_qualified_table_name` per-component backtick (`` `fake_project`.`dataset`.`orders` ``, + two-part); sample-CTE uses `xxhash64(...) & 9223372036854775807` not `FARM_FINGERPRINT`; partition-filter renders `TIMESTAMP '…'` / `DATE '…'` typed-literal form; date-arithmetic (`DATE_TRUNC('DAY', …)` arg-order, `INTERVAL n unit` bare, `DAYOFWEEK(...)` Sunday=1). Each reads from `Dialect`, never hard-coded. +- **Files:** `tests/prune/test_compiler.py` (new `*_databricks_*` unit tests). +- **Done when:** unit tests pin every `Dialect` field the compiler consumes for Databricks; all green; full validation passes. +- **Depends on:** none. + +### US-003 — Byte-exact Databricks fixtures + ungated snapshot tests +- **Traces to:** DEC-003, DEC-005, AC "fixtures committed; BigQuery + Snowflake fixtures byte-unchanged". +- **Description:** Generate the 32 fixtures (16 under `tests/fixtures/prune/compiled_sql/databricks/`, 16 under `…/anomaly/databricks/`) from real compiler output with `DATABRICKS_DIALECT`. Add **ungated** byte-exact snapshot tests in `test_compiler.py` mirroring the Snowflake set (individual funcs for built-ins/custom_sql/row_count_between/unique_combination; parametrized `(method, seasonality)` for the anomaly stats/violation pair). Include the DEC-005 leakage assertions. Add a regression assertion that BigQuery + Snowflake fixtures are byte-unchanged (or rely on their existing snapshot tests staying green — make the intent explicit in a comment/test). +- **Files:** `tests/fixtures/prune/compiled_sql/databricks/*.sql`, `tests/fixtures/prune/compiled_sql/anomaly/databricks/*.sql`, `tests/prune/test_compiler.py`. +- **Done when:** 32 fixtures committed; snapshot tests green; existing BQ/SF snapshot tests still green (byte-unchanged). +- **Depends on:** none (US-002 is independent but naturally lands first). + +### US-004 — sqlglot `databricks`-dialect parse-guard (ungated per DEC-002) + marker doc +- **Traces to:** DEC-002, DEC-004, AC "the sqlglot Databricks-dialect parse-guard passes on every fixture". +- **Description:** New `tests/prune/test_compiler_databricks.py` with a parametrized parse-guard over **all 32** fixtures (`sqlglot.parse_one(fixture, dialect="databricks")` raises on invalid syntax). Ungated (DEC-002) — sqlglot is a base dep. Update the `DATABRICKS_DIALECT` docstring note ("certified offline by the #223 sqlglot databricks parse-guard") and `docs/warehouse-adapter-ops.md` / `docs/prune-ops.md` as needed to reflect the shipped offline gate. +- **Files:** `tests/prune/test_compiler_databricks.py`, `src/signalforge/warehouse/models.py` (docstring only), `docs/*-ops.md`. +- **Done when:** parse-guard green over all 32 fixtures in the default suite; docstring/ops reflect reality. +- **Depends on:** US-003 (needs the fixtures). + +### US-005 — Quality Gate +- **Description:** Run code reviewer ×4 across the full changeset (fix all real bugs each pass); run CodeRabbit. Validation must pass after fixes, plus `uv run pytest -m databricks --no-cov`. Re-confirm the discovery prototype's 26/26 parse result is reflected in committed fixtures. +- **Depends on:** US-001..US-004. + +### US-006 — Patterns & Memory (priority 99) +- **Description:** Update `.claude/rules/prune-engine.md` + `warehouse-adapters.md` with the Databricks-compiler-dialect section (3rd dialect instance; the "unquoted `sample` CTE is fine on Spark" finding; the ungated-parse-guard rationale if DEC-002 holds). Add a memory note (Databricks dialect compiler pattern). Update the `databricks` marker description if gating changed. +- **Depends on:** US-005. + +--- + +## Scoping questions (resolved 2026-06-24) + +1. **Parse-guard gating** (DEC-002): **ungated** default-suite parse-guard. ✅ Confirmed. +2. **Fixture-set scope** (DEC-003): **mirror Snowflake's 16+16**. ✅ Confirmed. From 7b3ad88abda605a82bca20d5a346a5947ce6ab1f Mon Sep 17 00:00:00 2001 From: Wes Duenow Date: Wed, 24 Jun 2026 20:56:41 -0700 Subject: [PATCH 02/10] #223: mark plan phase published --- plans/super/223-databricks-prune-compiler.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plans/super/223-databricks-prune-compiler.md b/plans/super/223-databricks-prune-compiler.md index d05cf718..2943133a 100644 --- a/plans/super/223-databricks-prune-compiler.md +++ b/plans/super/223-databricks-prune-compiler.md @@ -5,7 +5,7 @@ - **Ticket:** https://github.com/wjduenow/SignalForge/issues/223 - **Epic:** #219 (Databricks adapter). Depends on #221 (skeleton). Models on #121 (Snowflake compiler dialect). - **Branch / worktree:** `feat/223-databricks-prune-compiler` @ `/home/wesd/Projects/worktrees/SignalForge/223-databricks-prune-compiler` -- **Phase:** detailing +- **Phase:** published - **Sessions:** 1 (2026-06-24) - **Scoping resolved:** Q1 → ungated parse-guard (DEC-002 stands); Q2 → mirror Snowflake 16+16 (DEC-003 stands). From 53c1faf93132038abb6bafe99bbacb11be0709ab Mon Sep 17 00:00:00 2001 From: Wes Duenow Date: Thu, 25 Jun 2026 08:33:25 -0700 Subject: [PATCH 03/10] #223: devolve plan to beads (epic bd_1-scaffolding-129) --- plans/super/223-databricks-prune-compiler.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/plans/super/223-databricks-prune-compiler.md b/plans/super/223-databricks-prune-compiler.md index 2943133a..32407915 100644 --- a/plans/super/223-databricks-prune-compiler.md +++ b/plans/super/223-databricks-prune-compiler.md @@ -5,7 +5,7 @@ - **Ticket:** https://github.com/wjduenow/SignalForge/issues/223 - **Epic:** #219 (Databricks adapter). Depends on #221 (skeleton). Models on #121 (Snowflake compiler dialect). - **Branch / worktree:** `feat/223-databricks-prune-compiler` @ `/home/wesd/Projects/worktrees/SignalForge/223-databricks-prune-compiler` -- **Phase:** published +- **Phase:** devolved - **Sessions:** 1 (2026-06-24) - **Scoping resolved:** Q1 → ungated parse-guard (DEC-002 stands); Q2 → mirror Snowflake 16+16 (DEC-003 stands). @@ -134,3 +134,17 @@ Each Databricks snapshot test asserts the distinguishing markers: `xxhash64` pre 1. **Parse-guard gating** (DEC-002): **ungated** default-suite parse-guard. ✅ Confirmed. 2. **Fixture-set scope** (DEC-003): **mirror Snowflake's 16+16**. ✅ Confirmed. + +--- + +## Beads Manifest (devolved 2026-06-25) + +- **Epic:** `bd_1-scaffolding-129` +- **Worktree:** `/home/wesd/Projects/worktrees/SignalForge/223-databricks-prune-compiler` (`feat/223-databricks-prune-compiler`) +- **Tasks:** + - `bd_1-scaffolding-129.1` — US-001 Import-guard (ready) + - `bd_1-scaffolding-129.2` — US-002 Dialect unit tests (ready) + - `bd_1-scaffolding-129.3` — US-003 Fixtures + snapshot tests (ready) + - `bd_1-scaffolding-129.4` — US-004 sqlglot parse-guard + docs (blocked → .3) + - `bd_1-scaffolding-129.5` — Quality Gate (blocked → .1 .2 .3 .4) + - `bd_1-scaffolding-129.6` — Patterns & Memory (blocked → .5) From d89c9356313303cc02874bbb73f3d840202af069 Mon Sep 17 00:00:00 2001 From: Wes Duenow Date: Thu, 25 Jun 2026 09:06:43 -0700 Subject: [PATCH 04/10] =?UTF-8?q?bd=5F1-scaffolding-129.1:=20#223=20US-001?= =?UTF-8?q?=20=E2=80=94=20confine=20databricks=20SDK=20out=20of=20prune/?= =?UTF-8?q?=20import-guard?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- tests/prune/test_compiler_import_guard.py | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/tests/prune/test_compiler_import_guard.py b/tests/prune/test_compiler_import_guard.py index 31fdc3dd..1e46d46c 100644 --- a/tests/prune/test_compiler_import_guard.py +++ b/tests/prune/test_compiler_import_guard.py @@ -4,9 +4,10 @@ correct SQL purely from the :class:`signalforge.warehouse.models.Dialect` value object, never by importing a warehouse SDK or branching on a dialect *name* (DEC-025). This AST-based scan enforces that no ``import snowflake`` / -``from snowflake`` / ``import google.cloud`` / ``from google.cloud`` -statement appears anywhere under ``src/signalforge/prune/`` — a regression -that reached for a vendor SDK in the compiler would break the seam silently. +``from snowflake`` / ``import google.cloud`` / ``from google.cloud`` / +``import databricks`` / ``from databricks`` statement appears anywhere under +``src/signalforge/prune/`` — a regression that reached for a vendor SDK in the +compiler would break the seam silently. AST-based (not per-line regex) per ``testing-signal.md`` § "Source-scan gates: AST over per-line regex": a multi-line / parenthesised / aliased @@ -26,7 +27,7 @@ # matches when it equals the prefix or begins with ``.`` (so # ``google.cloud.bigquery`` matches ``google.cloud`` but ``google_leftover`` # does not). -_FORBIDDEN_PREFIXES = ("snowflake", "google.cloud") +_FORBIDDEN_PREFIXES = ("snowflake", "google.cloud", "databricks") def _module_matches_forbidden(module: str | None) -> bool: @@ -73,7 +74,7 @@ def _forbidden_imports(source: str) -> list[str]: def test_no_warehouse_sdk_import_under_prune() -> None: - """No ``snowflake`` / ``google.cloud`` import anywhere under prune/.""" + """No ``snowflake`` / ``google.cloud`` / ``databricks`` import under prune/.""" offenders: list[str] = [] for py in sorted(_PRUNE_DIR.rglob("*.py")): source = py.read_text(encoding="utf-8") @@ -103,10 +104,15 @@ def test_detector_flags_planted_violations() -> None: "import google.cloud.bigquery as bq\n" # namespace-split: module='google', imported name completes 'google.cloud' "from google import cloud\n" + "import databricks\n" + "import databricks.sql\n" + "import databricks.sql as dbx\n" + "from databricks import sql\n" + "from databricks.sql import connect\n" ) hits = _forbidden_imports(planted) - # Every line above is a violation — 9 statements. - assert len(hits) == 9, hits + # Every line above is a violation — 14 statements. + assert len(hits) == 14, hits def test_detector_does_not_false_positive() -> None: @@ -117,6 +123,7 @@ def test_detector_does_not_false_positive() -> None: "from pathlib import Path\n" "import googleapiclient\n" # starts with 'google' but not 'google.cloud' "from snowflakeish import thing\n" # starts with 'snowflake' but not 'snowflake.' + "import databricksutils\n" # starts with 'databricks' but not 'databricks.' "from . import compiler\n" # relative import, module is None ) assert _forbidden_imports(innocent) == [] From b0d7ceace72c9c2fe0a0ff7c2c05cfd056b0a6d8 Mon Sep 17 00:00:00 2001 From: Wes Duenow Date: Thu, 25 Jun 2026 09:09:53 -0700 Subject: [PATCH 05/10] =?UTF-8?q?bd=5F1-scaffolding-129.2:=20#223=20US-002?= =?UTF-8?q?=20=E2=80=94=20Databricks=20compiler=20dialect=20unit=20tests?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- tests/prune/test_compiler.py | 128 +++++++++++++++++++++++++++++++++++ 1 file changed, 128 insertions(+) diff --git a/tests/prune/test_compiler.py b/tests/prune/test_compiler.py index ea1f6a4f..ce068f0b 100644 --- a/tests/prune/test_compiler.py +++ b/tests/prune/test_compiler.py @@ -46,6 +46,7 @@ from signalforge.warehouse._sql_safety import validate_test_sql from signalforge.warehouse.models import ( BIGQUERY_DIALECT, + DATABRICKS_DIALECT, POSTGRES_DIALECT, SNOWFLAKE_DIALECT, Dialect, @@ -1205,6 +1206,133 @@ def test_snowflake_relationships_per_component_and_folded() -> None: assert "`" not in sql +# --------------------------------------------------------------------------- +# US-002 (#223): Databricks per-dialect UNIT tests. The prune compiler is +# dialect-driven — it emits warehouse-correct SQL purely from the ``Dialect`` +# value object, never branching on ``dialect.name`` — so these pin every +# ``DATABRICKS_DIALECT`` field the compiler consumes: backtick quote char + +# ``identifier_case='lower'`` fold, ``quote_qualified_per_component=True``, +# the ``xxhash64(...) & Long.MAX`` sign-bit-masked sample expression (NOT +# BigQuery ``FARM_FINGERPRINT``, NOT Snowflake ``HASH(*)``), and the +# ``TIMESTAMP '...'`` / ``DATE '...'`` typed-literal partition-filter form +# (NOT BigQuery's ``TIMESTAMP('...')`` function form, NOT Snowflake's +# ``'...'::TIMESTAMP`` cast form). Backtick presence does NOT distinguish +# BigQuery from Databricks (both backtick), so the leakage assertions key on +# the hash/literal markers, never the quote char. Byte-exact snapshot +# fixtures for the rendered output are US-003's job. +# --------------------------------------------------------------------------- + + +def test_quote_folds_lower_and_backticks_for_databricks() -> None: + """``identifier_case='lower'`` folds the token to lower-case before + wrapping in the Databricks backtick quote char. An already-lower token is + a no-op fold; a mixed-case token exercises the fold rather than a no-op.""" + assert _quote("customer_id", DATABRICKS_DIALECT) == "`customer_id`" + assert _quote("Customer_ID", DATABRICKS_DIALECT) == "`customer_id`" + + +def test_qualified_table_name_per_component_for_databricks() -> None: + """Databricks quotes each component separately with backticks + (``quote_qualified_per_component=True``) and folds to lower so a dotted + path is not read as one literal identifier named ``db.schema.table``.""" + ref = TableRef(project="prod_db", dataset="sch", name="orders") + assert _qualified_table_name(ref, DATABRICKS_DIALECT) == "`prod_db`.`sch`.`orders`" + + +def test_qualified_table_name_per_component_two_part_for_databricks() -> None: + """``project=None`` yields a two-part per-component backtick-quoted name.""" + ref = TableRef(project=None, dataset="sch", name="orders") + assert _qualified_table_name(ref, DATABRICKS_DIALECT) == "`sch`.`orders`" + + +def test_databricks_sample_cte_uses_xxhash64_mask_not_farm_fingerprint() -> None: + """The Databricks sample CTE uses the inline + ``MOD((xxhash64(to_json(struct(*))) & 9223372036854775807), bucket) < 1`` + form (``sample_hash_in_projection=False``, like BigQuery). The BigQuery + ``FARM_FINGERPRINT`` form and the Snowflake ``HASH(*)`` form never + appear.""" + test = CandidateTestNotNull(column="customer_id") + sql = _compile_test( + test, + _make_orders_table_ref(), + DATABRICKS_DIALECT, + _make_manifest(), + scope="sample", + sample_size=100_000, + sample_bucket=10, + ) + assert isinstance(sql, str) + # The sign-bit-masked xxhash64 expression, inline in the WHERE predicate. + assert "MOD((xxhash64(to_json(struct(*))) & 9223372036854775807), 10) < 1" in sql + # Folded + backtick-quoted identifier. + assert "`customer_id`" in sql + # Cross-dialect leakage guards (backtick is NOT a discriminator — both + # BigQuery and Databricks backtick — so key on the hash markers). + assert "FARM_FINGERPRINT" not in sql + assert "HASH(*)" not in sql + + +def test_databricks_datetime_partition_filter_uses_typed_literal_form() -> None: + """A ``datetime`` partition filter under Databricks renders the + ``TIMESTAMP '...'`` typed-literal form, not BigQuery's ``TIMESTAMP('...')`` + function form nor Snowflake's ``'...'::TIMESTAMP`` cast form.""" + from datetime import datetime + + test = CandidateTestNotNull(column="customer_id") + pf = PartitionFilter(column="event_ts", op=">=", value=datetime(2026, 1, 1, 0, 0, 0)) + sql = _compile_test( + test, + _make_orders_table_ref(), + DATABRICKS_DIALECT, + _make_manifest(), + scope="full", + partition_filter=pf, + ) + assert isinstance(sql, str) + assert "TIMESTAMP '2026-01-01T00:00:00'" in sql + # BigQuery function form and Snowflake cast form are both absent. + assert "TIMESTAMP(" not in sql + assert "::" not in sql + # The partition column is folded + backtick-quoted the Databricks way. + assert "`event_ts` >=" in sql + + +def test_databricks_date_partition_filter_uses_typed_literal_form() -> None: + """A ``date`` partition filter under Databricks renders ``DATE '...'``.""" + from datetime import date as _date + + test = CandidateTestNotNull(column="customer_id") + pf = PartitionFilter(column="event_dt", op=">=", value=_date(2026, 1, 1)) + sql = _compile_test( + test, + _make_orders_table_ref(), + DATABRICKS_DIALECT, + _make_manifest(), + scope="full", + partition_filter=pf, + ) + assert isinstance(sql, str) + assert "DATE '2026-01-01'" in sql + # BigQuery function form and Snowflake cast form are both absent. + assert "DATE(" not in sql + assert "::" not in sql + + +def test_databricks_relationships_per_component_and_folded() -> None: + """End-to-end relationships under Databricks: both child and parent + tables are per-component backtick-quoted + lower-folded; columns + folded + backtick-quoted.""" + test = CandidateTestRelationships(column="customer_id", to="customers", field="id") + sql = _compile_test(test, _make_orders_table_ref(), DATABRICKS_DIALECT, _make_manifest()) + assert isinstance(sql, str) + assert "`fake_project`.`dataset`.`orders`" in sql + assert "`fake_project`.`dataset`.`customers`" in sql + assert "child.`customer_id`" in sql + assert "parent.`id`" in sql + # No Snowflake double-quote leakage. + assert '"' not in sql + + # --------------------------------------------------------------------------- # US-003 (#121): byte-exact Snowflake snapshot fixtures + tests for all four # built-in test types (full + sample modes) and custom_sql (single-table full, From 1518a26189a91caed23993033b5ad96b55b17f50 Mon Sep 17 00:00:00 2001 From: Wes Duenow Date: Thu, 25 Jun 2026 09:15:34 -0700 Subject: [PATCH 06/10] =?UTF-8?q?bd=5F1-scaffolding-129.3:=20#223=20US-003?= =?UTF-8?q?=20=E2=80=94=20Databricks=20compiled-SQL=20fixtures=20+=20ungat?= =?UTF-8?q?ed=20snapshot=20tests?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../anomaly/databricks/mad_dow_stats.sql | 1 + .../anomaly/databricks/mad_dow_violation.sql | 1 + .../anomaly/databricks/mad_none_stats.sql | 1 + .../anomaly/databricks/mad_none_violation.sql | 1 + .../anomaly/databricks/min_max_dow_stats.sql | 1 + .../databricks/min_max_dow_violation.sql | 1 + .../anomaly/databricks/min_max_none_stats.sql | 1 + .../databricks/min_max_none_violation.sql | 1 + .../databricks/percentile_dow_stats.sql | 1 + .../databricks/percentile_dow_violation.sql | 1 + .../databricks/percentile_none_stats.sql | 1 + .../databricks/percentile_none_violation.sql | 1 + .../anomaly/databricks/zscore_dow_stats.sql | 1 + .../databricks/zscore_dow_violation.sql | 1 + .../anomaly/databricks/zscore_none_stats.sql | 1 + .../databricks/zscore_none_violation.sql | 1 + .../databricks/accepted_values.sql | 1 + .../databricks/accepted_values_sample.sql | 1 + .../compiled_sql/databricks/custom_sql.sql | 1 + .../databricks/custom_sql_fullscan.sql | 1 + .../databricks/custom_sql_sample.sql | 1 + .../compiled_sql/databricks/not_null.sql | 1 + .../databricks/not_null_sample.sql | 1 + .../compiled_sql/databricks/relationships.sql | 1 + .../databricks/relationships_sample.sql | 1 + .../databricks/row_count_between.sql | 1 + .../databricks/row_count_between_where.sql | 1 + .../prune/compiled_sql/databricks/unique.sql | 1 + .../databricks/unique_combination_pair.sql | 1 + .../unique_combination_three_columns.sql | 1 + .../unique_combination_with_where.sql | 1 + .../compiled_sql/databricks/unique_sample.sql | 1 + tests/prune/test_compiler.py | 327 ++++++++++++++++++ 33 files changed, 359 insertions(+) create mode 100644 tests/fixtures/prune/compiled_sql/anomaly/databricks/mad_dow_stats.sql create mode 100644 tests/fixtures/prune/compiled_sql/anomaly/databricks/mad_dow_violation.sql create mode 100644 tests/fixtures/prune/compiled_sql/anomaly/databricks/mad_none_stats.sql create mode 100644 tests/fixtures/prune/compiled_sql/anomaly/databricks/mad_none_violation.sql create mode 100644 tests/fixtures/prune/compiled_sql/anomaly/databricks/min_max_dow_stats.sql create mode 100644 tests/fixtures/prune/compiled_sql/anomaly/databricks/min_max_dow_violation.sql create mode 100644 tests/fixtures/prune/compiled_sql/anomaly/databricks/min_max_none_stats.sql create mode 100644 tests/fixtures/prune/compiled_sql/anomaly/databricks/min_max_none_violation.sql create mode 100644 tests/fixtures/prune/compiled_sql/anomaly/databricks/percentile_dow_stats.sql create mode 100644 tests/fixtures/prune/compiled_sql/anomaly/databricks/percentile_dow_violation.sql create mode 100644 tests/fixtures/prune/compiled_sql/anomaly/databricks/percentile_none_stats.sql create mode 100644 tests/fixtures/prune/compiled_sql/anomaly/databricks/percentile_none_violation.sql create mode 100644 tests/fixtures/prune/compiled_sql/anomaly/databricks/zscore_dow_stats.sql create mode 100644 tests/fixtures/prune/compiled_sql/anomaly/databricks/zscore_dow_violation.sql create mode 100644 tests/fixtures/prune/compiled_sql/anomaly/databricks/zscore_none_stats.sql create mode 100644 tests/fixtures/prune/compiled_sql/anomaly/databricks/zscore_none_violation.sql create mode 100644 tests/fixtures/prune/compiled_sql/databricks/accepted_values.sql create mode 100644 tests/fixtures/prune/compiled_sql/databricks/accepted_values_sample.sql create mode 100644 tests/fixtures/prune/compiled_sql/databricks/custom_sql.sql create mode 100644 tests/fixtures/prune/compiled_sql/databricks/custom_sql_fullscan.sql create mode 100644 tests/fixtures/prune/compiled_sql/databricks/custom_sql_sample.sql create mode 100644 tests/fixtures/prune/compiled_sql/databricks/not_null.sql create mode 100644 tests/fixtures/prune/compiled_sql/databricks/not_null_sample.sql create mode 100644 tests/fixtures/prune/compiled_sql/databricks/relationships.sql create mode 100644 tests/fixtures/prune/compiled_sql/databricks/relationships_sample.sql create mode 100644 tests/fixtures/prune/compiled_sql/databricks/row_count_between.sql create mode 100644 tests/fixtures/prune/compiled_sql/databricks/row_count_between_where.sql create mode 100644 tests/fixtures/prune/compiled_sql/databricks/unique.sql create mode 100644 tests/fixtures/prune/compiled_sql/databricks/unique_combination_pair.sql create mode 100644 tests/fixtures/prune/compiled_sql/databricks/unique_combination_three_columns.sql create mode 100644 tests/fixtures/prune/compiled_sql/databricks/unique_combination_with_where.sql create mode 100644 tests/fixtures/prune/compiled_sql/databricks/unique_sample.sql diff --git a/tests/fixtures/prune/compiled_sql/anomaly/databricks/mad_dow_stats.sql b/tests/fixtures/prune/compiled_sql/anomaly/databricks/mad_dow_stats.sql new file mode 100644 index 00000000..afaffb1b --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/anomaly/databricks/mad_dow_stats.sql @@ -0,0 +1 @@ +WITH history AS (SELECT DATE_TRUNC('DAY', `event_date`) AS period, DAYOFWEEK(`event_date`) AS dow, COUNT(*) AS cnt FROM `fake_project`.`dataset`.`orders` WHERE `event_date` >= DATE '2026-05-01' - INTERVAL 28 DAY AND `event_date` < DATE '2026-05-01' GROUP BY period, dow), medians AS (SELECT dow, PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY cnt) AS median FROM history GROUP BY dow), mads AS (SELECT history.dow AS dow, PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY ABS(history.cnt - medians.median)) AS mad FROM history JOIN medians ON history.dow = medians.dow GROUP BY history.dow), counts AS (SELECT dow, COUNT(*) AS n FROM history GROUP BY dow) SELECT medians.dow AS dow, medians.median AS median, mads.mad AS mad, counts.n AS n FROM medians JOIN mads ON medians.dow = mads.dow JOIN counts ON medians.dow = counts.dow \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/anomaly/databricks/mad_dow_violation.sql b/tests/fixtures/prune/compiled_sql/anomaly/databricks/mad_dow_violation.sql new file mode 100644 index 00000000..2984a350 --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/anomaly/databricks/mad_dow_violation.sql @@ -0,0 +1 @@ +SELECT 1 FROM `fake_project`.`dataset`.`orders` WHERE `event_date` >= DATE '2026-05-01' AND `event_date` < DATE '2026-05-01' + INTERVAL 1 DAY \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/anomaly/databricks/mad_none_stats.sql b/tests/fixtures/prune/compiled_sql/anomaly/databricks/mad_none_stats.sql new file mode 100644 index 00000000..d5f5f995 --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/anomaly/databricks/mad_none_stats.sql @@ -0,0 +1 @@ +WITH history AS (SELECT DATE_TRUNC('DAY', `event_date`) AS period, COUNT(*) AS cnt FROM `fake_project`.`dataset`.`orders` WHERE `event_date` >= DATE '2026-05-01' - INTERVAL 28 DAY AND `event_date` < DATE '2026-05-01' GROUP BY period), medians AS (SELECT PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY cnt) AS median FROM history) SELECT (SELECT median FROM medians) AS median, PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY ABS(cnt - (SELECT median FROM medians))) AS mad, COUNT(*) AS n FROM history \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/anomaly/databricks/mad_none_violation.sql b/tests/fixtures/prune/compiled_sql/anomaly/databricks/mad_none_violation.sql new file mode 100644 index 00000000..2984a350 --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/anomaly/databricks/mad_none_violation.sql @@ -0,0 +1 @@ +SELECT 1 FROM `fake_project`.`dataset`.`orders` WHERE `event_date` >= DATE '2026-05-01' AND `event_date` < DATE '2026-05-01' + INTERVAL 1 DAY \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/anomaly/databricks/min_max_dow_stats.sql b/tests/fixtures/prune/compiled_sql/anomaly/databricks/min_max_dow_stats.sql new file mode 100644 index 00000000..93825776 --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/anomaly/databricks/min_max_dow_stats.sql @@ -0,0 +1 @@ +WITH history AS (SELECT DATE_TRUNC('DAY', `event_date`) AS period, DAYOFWEEK(`event_date`) AS dow, COUNT(*) AS cnt FROM `fake_project`.`dataset`.`orders` WHERE `event_date` >= DATE '2026-05-01' - INTERVAL 28 DAY AND `event_date` < DATE '2026-05-01' GROUP BY period, dow) SELECT dow, MIN(cnt) AS min_cnt, MAX(cnt) AS max_cnt, COUNT(*) AS n FROM history GROUP BY dow \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/anomaly/databricks/min_max_dow_violation.sql b/tests/fixtures/prune/compiled_sql/anomaly/databricks/min_max_dow_violation.sql new file mode 100644 index 00000000..2984a350 --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/anomaly/databricks/min_max_dow_violation.sql @@ -0,0 +1 @@ +SELECT 1 FROM `fake_project`.`dataset`.`orders` WHERE `event_date` >= DATE '2026-05-01' AND `event_date` < DATE '2026-05-01' + INTERVAL 1 DAY \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/anomaly/databricks/min_max_none_stats.sql b/tests/fixtures/prune/compiled_sql/anomaly/databricks/min_max_none_stats.sql new file mode 100644 index 00000000..167bf82e --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/anomaly/databricks/min_max_none_stats.sql @@ -0,0 +1 @@ +WITH history AS (SELECT DATE_TRUNC('DAY', `event_date`) AS period, COUNT(*) AS cnt FROM `fake_project`.`dataset`.`orders` WHERE `event_date` >= DATE '2026-05-01' - INTERVAL 28 DAY AND `event_date` < DATE '2026-05-01' GROUP BY period) SELECT MIN(cnt) AS min_cnt, MAX(cnt) AS max_cnt, COUNT(*) AS n FROM history \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/anomaly/databricks/min_max_none_violation.sql b/tests/fixtures/prune/compiled_sql/anomaly/databricks/min_max_none_violation.sql new file mode 100644 index 00000000..2984a350 --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/anomaly/databricks/min_max_none_violation.sql @@ -0,0 +1 @@ +SELECT 1 FROM `fake_project`.`dataset`.`orders` WHERE `event_date` >= DATE '2026-05-01' AND `event_date` < DATE '2026-05-01' + INTERVAL 1 DAY \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/anomaly/databricks/percentile_dow_stats.sql b/tests/fixtures/prune/compiled_sql/anomaly/databricks/percentile_dow_stats.sql new file mode 100644 index 00000000..c4a227c2 --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/anomaly/databricks/percentile_dow_stats.sql @@ -0,0 +1 @@ +WITH history AS (SELECT DATE_TRUNC('DAY', `event_date`) AS period, DAYOFWEEK(`event_date`) AS dow, COUNT(*) AS cnt FROM `fake_project`.`dataset`.`orders` WHERE `event_date` >= DATE '2026-05-01' - INTERVAL 28 DAY AND `event_date` < DATE '2026-05-01' GROUP BY period, dow) SELECT dow, PERCENTILE_CONT(0.05) WITHIN GROUP (ORDER BY cnt) AS p_lo, PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY cnt) AS p_hi, COUNT(*) AS n FROM history GROUP BY dow \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/anomaly/databricks/percentile_dow_violation.sql b/tests/fixtures/prune/compiled_sql/anomaly/databricks/percentile_dow_violation.sql new file mode 100644 index 00000000..2984a350 --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/anomaly/databricks/percentile_dow_violation.sql @@ -0,0 +1 @@ +SELECT 1 FROM `fake_project`.`dataset`.`orders` WHERE `event_date` >= DATE '2026-05-01' AND `event_date` < DATE '2026-05-01' + INTERVAL 1 DAY \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/anomaly/databricks/percentile_none_stats.sql b/tests/fixtures/prune/compiled_sql/anomaly/databricks/percentile_none_stats.sql new file mode 100644 index 00000000..3db7a408 --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/anomaly/databricks/percentile_none_stats.sql @@ -0,0 +1 @@ +WITH history AS (SELECT DATE_TRUNC('DAY', `event_date`) AS period, COUNT(*) AS cnt FROM `fake_project`.`dataset`.`orders` WHERE `event_date` >= DATE '2026-05-01' - INTERVAL 28 DAY AND `event_date` < DATE '2026-05-01' GROUP BY period) SELECT PERCENTILE_CONT(0.05) WITHIN GROUP (ORDER BY cnt) AS p_lo, PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY cnt) AS p_hi, COUNT(*) AS n FROM history \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/anomaly/databricks/percentile_none_violation.sql b/tests/fixtures/prune/compiled_sql/anomaly/databricks/percentile_none_violation.sql new file mode 100644 index 00000000..2984a350 --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/anomaly/databricks/percentile_none_violation.sql @@ -0,0 +1 @@ +SELECT 1 FROM `fake_project`.`dataset`.`orders` WHERE `event_date` >= DATE '2026-05-01' AND `event_date` < DATE '2026-05-01' + INTERVAL 1 DAY \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/anomaly/databricks/zscore_dow_stats.sql b/tests/fixtures/prune/compiled_sql/anomaly/databricks/zscore_dow_stats.sql new file mode 100644 index 00000000..44f65d6f --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/anomaly/databricks/zscore_dow_stats.sql @@ -0,0 +1 @@ +WITH history AS (SELECT DATE_TRUNC('DAY', `event_date`) AS period, DAYOFWEEK(`event_date`) AS dow, COUNT(*) AS cnt FROM `fake_project`.`dataset`.`orders` WHERE `event_date` >= DATE '2026-05-01' - INTERVAL 28 DAY AND `event_date` < DATE '2026-05-01' GROUP BY period, dow) SELECT dow, AVG(cnt) AS mean, STDDEV(cnt) AS stddev, COUNT(*) AS n FROM history GROUP BY dow \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/anomaly/databricks/zscore_dow_violation.sql b/tests/fixtures/prune/compiled_sql/anomaly/databricks/zscore_dow_violation.sql new file mode 100644 index 00000000..2984a350 --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/anomaly/databricks/zscore_dow_violation.sql @@ -0,0 +1 @@ +SELECT 1 FROM `fake_project`.`dataset`.`orders` WHERE `event_date` >= DATE '2026-05-01' AND `event_date` < DATE '2026-05-01' + INTERVAL 1 DAY \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/anomaly/databricks/zscore_none_stats.sql b/tests/fixtures/prune/compiled_sql/anomaly/databricks/zscore_none_stats.sql new file mode 100644 index 00000000..01ffbe25 --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/anomaly/databricks/zscore_none_stats.sql @@ -0,0 +1 @@ +WITH history AS (SELECT DATE_TRUNC('DAY', `event_date`) AS period, COUNT(*) AS cnt FROM `fake_project`.`dataset`.`orders` WHERE `event_date` >= DATE '2026-05-01' - INTERVAL 28 DAY AND `event_date` < DATE '2026-05-01' GROUP BY period) SELECT AVG(cnt) AS mean, STDDEV(cnt) AS stddev, COUNT(*) AS n FROM history \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/anomaly/databricks/zscore_none_violation.sql b/tests/fixtures/prune/compiled_sql/anomaly/databricks/zscore_none_violation.sql new file mode 100644 index 00000000..2984a350 --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/anomaly/databricks/zscore_none_violation.sql @@ -0,0 +1 @@ +SELECT 1 FROM `fake_project`.`dataset`.`orders` WHERE `event_date` >= DATE '2026-05-01' AND `event_date` < DATE '2026-05-01' + INTERVAL 1 DAY \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/databricks/accepted_values.sql b/tests/fixtures/prune/compiled_sql/databricks/accepted_values.sql new file mode 100644 index 00000000..7a856acd --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/databricks/accepted_values.sql @@ -0,0 +1 @@ +SELECT `status` FROM `fake_project`.`dataset`.`orders` WHERE `status` IS NOT NULL AND `status` NOT IN ('placed', 'shipped', 'cancelled') \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/databricks/accepted_values_sample.sql b/tests/fixtures/prune/compiled_sql/databricks/accepted_values_sample.sql new file mode 100644 index 00000000..08cd916a --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/databricks/accepted_values_sample.sql @@ -0,0 +1 @@ +WITH sample AS (SELECT * FROM `fake_project`.`dataset`.`orders` AS t WHERE MOD((xxhash64(to_json(struct(*))) & 9223372036854775807), 10) < 1 LIMIT 100000) SELECT `status` FROM sample WHERE `status` IS NOT NULL AND `status` NOT IN ('placed', 'shipped', 'cancelled') \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/databricks/custom_sql.sql b/tests/fixtures/prune/compiled_sql/databricks/custom_sql.sql new file mode 100644 index 00000000..8c3fb195 --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/databricks/custom_sql.sql @@ -0,0 +1 @@ +select order_id from fake_project.dataset.orders where total < 0 \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/databricks/custom_sql_fullscan.sql b/tests/fixtures/prune/compiled_sql/databricks/custom_sql_fullscan.sql new file mode 100644 index 00000000..afb0c74b --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/databricks/custom_sql_fullscan.sql @@ -0,0 +1 @@ +select o.order_id from fake_project.dataset.orders as o join fake_project.dataset.customers as c on o.customer_id = c.id where c.id is null \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/databricks/custom_sql_sample.sql b/tests/fixtures/prune/compiled_sql/databricks/custom_sql_sample.sql new file mode 100644 index 00000000..bb6fd971 --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/databricks/custom_sql_sample.sql @@ -0,0 +1 @@ +WITH sample AS (SELECT * FROM `fake_project`.`dataset`.`orders` AS t WHERE MOD((xxhash64(to_json(struct(*))) & 9223372036854775807), 10) < 1 LIMIT 100000) select order_id from sample where total < 0 \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/databricks/not_null.sql b/tests/fixtures/prune/compiled_sql/databricks/not_null.sql new file mode 100644 index 00000000..3a678f2b --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/databricks/not_null.sql @@ -0,0 +1 @@ +SELECT `customer_id` FROM `fake_project`.`dataset`.`orders` WHERE `customer_id` IS NULL \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/databricks/not_null_sample.sql b/tests/fixtures/prune/compiled_sql/databricks/not_null_sample.sql new file mode 100644 index 00000000..db5440ba --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/databricks/not_null_sample.sql @@ -0,0 +1 @@ +WITH sample AS (SELECT * FROM `fake_project`.`dataset`.`orders` AS t WHERE MOD((xxhash64(to_json(struct(*))) & 9223372036854775807), 10) < 1 LIMIT 100000) SELECT `customer_id` FROM sample WHERE `customer_id` IS NULL \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/databricks/relationships.sql b/tests/fixtures/prune/compiled_sql/databricks/relationships.sql new file mode 100644 index 00000000..efe013c5 --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/databricks/relationships.sql @@ -0,0 +1 @@ +SELECT child.`customer_id` FROM `fake_project`.`dataset`.`orders` AS child LEFT JOIN `fake_project`.`dataset`.`customers` AS parent ON child.`customer_id` = parent.`id` WHERE child.`customer_id` IS NOT NULL AND parent.`id` IS NULL \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/databricks/relationships_sample.sql b/tests/fixtures/prune/compiled_sql/databricks/relationships_sample.sql new file mode 100644 index 00000000..3e33aae7 --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/databricks/relationships_sample.sql @@ -0,0 +1 @@ +WITH sample AS (SELECT * FROM `fake_project`.`dataset`.`orders` AS t WHERE MOD((xxhash64(to_json(struct(*))) & 9223372036854775807), 10) < 1 LIMIT 100000) SELECT child.`customer_id` FROM sample AS child LEFT JOIN `fake_project`.`dataset`.`customers` AS parent ON child.`customer_id` = parent.`id` WHERE child.`customer_id` IS NOT NULL AND parent.`id` IS NULL \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/databricks/row_count_between.sql b/tests/fixtures/prune/compiled_sql/databricks/row_count_between.sql new file mode 100644 index 00000000..1d07eebc --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/databricks/row_count_between.sql @@ -0,0 +1 @@ +SELECT n FROM (SELECT COUNT(*) AS n FROM `fake_project`.`dataset`.`orders`) AS rc WHERE n < 1 OR n > 1000000 \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/databricks/row_count_between_where.sql b/tests/fixtures/prune/compiled_sql/databricks/row_count_between_where.sql new file mode 100644 index 00000000..a5e856da --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/databricks/row_count_between_where.sql @@ -0,0 +1 @@ +SELECT n FROM (SELECT COUNT(*) AS n FROM `fake_project`.`dataset`.`orders` WHERE event_date >= '2024-01-01') AS rc WHERE n < 1 OR n > 1000000 \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/databricks/unique.sql b/tests/fixtures/prune/compiled_sql/databricks/unique.sql new file mode 100644 index 00000000..a1e60274 --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/databricks/unique.sql @@ -0,0 +1 @@ +SELECT `customer_id` FROM `fake_project`.`dataset`.`orders` WHERE `customer_id` IS NOT NULL GROUP BY `customer_id` HAVING COUNT(*) > 1 \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/databricks/unique_combination_pair.sql b/tests/fixtures/prune/compiled_sql/databricks/unique_combination_pair.sql new file mode 100644 index 00000000..63a74bdf --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/databricks/unique_combination_pair.sql @@ -0,0 +1 @@ +SELECT `customer_id`, `order_date` FROM `fake_project`.`dataset`.`orders` GROUP BY `customer_id`, `order_date` HAVING COUNT(*) > 1 \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/databricks/unique_combination_three_columns.sql b/tests/fixtures/prune/compiled_sql/databricks/unique_combination_three_columns.sql new file mode 100644 index 00000000..4f1ea21e --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/databricks/unique_combination_three_columns.sql @@ -0,0 +1 @@ +SELECT `customer_id`, `order_date`, `region` FROM `fake_project`.`dataset`.`orders` GROUP BY `customer_id`, `order_date`, `region` HAVING COUNT(*) > 1 \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/databricks/unique_combination_with_where.sql b/tests/fixtures/prune/compiled_sql/databricks/unique_combination_with_where.sql new file mode 100644 index 00000000..6b7e4fd4 --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/databricks/unique_combination_with_where.sql @@ -0,0 +1 @@ +SELECT `customer_id`, `order_date` FROM `fake_project`.`dataset`.`orders` WHERE status = 'placed' GROUP BY `customer_id`, `order_date` HAVING COUNT(*) > 1 \ No newline at end of file diff --git a/tests/fixtures/prune/compiled_sql/databricks/unique_sample.sql b/tests/fixtures/prune/compiled_sql/databricks/unique_sample.sql new file mode 100644 index 00000000..68a2a55b --- /dev/null +++ b/tests/fixtures/prune/compiled_sql/databricks/unique_sample.sql @@ -0,0 +1 @@ +WITH sample AS (SELECT * FROM `fake_project`.`dataset`.`orders` AS t WHERE MOD((xxhash64(to_json(struct(*))) & 9223372036854775807), 10) < 1 LIMIT 100000) SELECT `customer_id` FROM sample WHERE `customer_id` IS NOT NULL GROUP BY `customer_id` HAVING COUNT(*) > 1 \ No newline at end of file diff --git a/tests/prune/test_compiler.py b/tests/prune/test_compiler.py index ce068f0b..5fd01471 100644 --- a/tests/prune/test_compiler.py +++ b/tests/prune/test_compiler.py @@ -56,8 +56,10 @@ _FIXTURES_DIR = Path(__file__).parent.parent / "fixtures" / "prune" / "compiled_sql" _SNOWFLAKE_FIXTURES_DIR = _FIXTURES_DIR / "snowflake" +_DATABRICKS_FIXTURES_DIR = _FIXTURES_DIR / "databricks" _ANOMALY_BQ_FIXTURES_DIR = _FIXTURES_DIR / "anomaly" / "bigquery" _ANOMALY_SF_FIXTURES_DIR = _FIXTURES_DIR / "anomaly" / "snowflake" +_ANOMALY_DB_FIXTURES_DIR = _FIXTURES_DIR / "anomaly" / "databricks" def _read_fixture(name: str) -> str: @@ -70,6 +72,11 @@ def _read_snowflake_fixture(name: str) -> str: return (_SNOWFLAKE_FIXTURES_DIR / name).read_text(encoding="utf-8") +def _read_databricks_fixture(name: str) -> str: + """Read a Databricks snapshot fixture file as raw text (no normalisation).""" + return (_DATABRICKS_FIXTURES_DIR / name).read_text(encoding="utf-8") + + def _read_anomaly_bq_fixture(name: str) -> str: """Read a row-count-anomaly BigQuery snapshot fixture.""" return (_ANOMALY_BQ_FIXTURES_DIR / name).read_text(encoding="utf-8") @@ -80,6 +87,11 @@ def _read_anomaly_sf_fixture(name: str) -> str: return (_ANOMALY_SF_FIXTURES_DIR / name).read_text(encoding="utf-8") +def _read_anomaly_db_fixture(name: str) -> str: + """Read a row-count-anomaly Databricks snapshot fixture.""" + return (_ANOMALY_DB_FIXTURES_DIR / name).read_text(encoding="utf-8") + + def _make_orders_table_ref() -> TableRef: return TableRef(project="fake_project", dataset="dataset", name="orders") @@ -2758,3 +2770,318 @@ def test_compile_row_count_anomaly_period_hour_returns_invalid_identifier() -> N f"expected _InvalidIdentifier for period='hour'; got {type(result).__name__}" ) assert "period='hour'" in result.reason + + +# --------------------------------------------------------------------------- +# Databricks snapshot tests (#223 US-003) — byte-exact compiled SQL against +# ``tests/fixtures/prune/compiled_sql/databricks/``. Fixtures are captured from +# real compiler output with DATABRICKS_DIALECT, mirroring the Snowflake snapshot +# inputs 1:1. They pin the backtick-quoted, per-component-qualified, +# lower-folded, inline ``xxhash64``-sample shape. UNGATED — they run in the +# default suite exactly like the Snowflake snapshots. +# +# Belt-and-braces leakage guard (DEC-005): every Databricks snapshot asserts the +# BigQuery row-hash marker (``FARM_FINGERPRINT``), the Snowflake sample marker +# (``HASH(*)``), and the Snowflake cast marker (``::``) are all absent. Backtick +# presence is NOT a discriminator — BigQuery and Databricks both backtick. +# --------------------------------------------------------------------------- + + +def _assert_no_dialect_leakage(sql: str) -> None: + """No BigQuery / Snowflake markers leaked into the Databricks output.""" + assert "FARM_FINGERPRINT" not in sql # BigQuery row-hash marker + assert "HASH(*)" not in sql # Snowflake sample marker + assert "::" not in sql # Snowflake cast marker + + +def test_compile_not_null_databricks_matches_snapshot() -> None: + expected = _read_databricks_fixture("not_null.sql") + test = CandidateTestNotNull(column="customer_id") + actual = _compile_test(test, _make_orders_table_ref(), DATABRICKS_DIALECT, _make_manifest()) + assert actual == expected + assert isinstance(actual, str) + _assert_no_dialect_leakage(actual) + + +def test_compile_unique_databricks_matches_snapshot() -> None: + expected = _read_databricks_fixture("unique.sql") + test = CandidateTestUnique(column="customer_id") + actual = _compile_test(test, _make_orders_table_ref(), DATABRICKS_DIALECT, _make_manifest()) + assert actual == expected + assert isinstance(actual, str) + _assert_no_dialect_leakage(actual) + + +def test_compile_accepted_values_databricks_matches_snapshot() -> None: + expected = _read_databricks_fixture("accepted_values.sql") + test = CandidateTestAcceptedValues(column="status", values=("placed", "shipped", "cancelled")) + actual = _compile_test(test, _make_orders_table_ref(), DATABRICKS_DIALECT, _make_manifest()) + assert actual == expected + assert isinstance(actual, str) + _assert_no_dialect_leakage(actual) + + +def test_compile_relationships_databricks_matches_snapshot() -> None: + expected = _read_databricks_fixture("relationships.sql") + test = CandidateTestRelationships(column="customer_id", to="customers", field="id") + actual = _compile_test(test, _make_orders_table_ref(), DATABRICKS_DIALECT, _make_manifest()) + assert actual == expected + assert isinstance(actual, str) + _assert_no_dialect_leakage(actual) + + +def test_compile_not_null_sample_databricks_matches_snapshot() -> None: + expected = _read_databricks_fixture("not_null_sample.sql") + test = CandidateTestNotNull(column="customer_id") + actual = _compile_test( + test, + _make_orders_table_ref(), + DATABRICKS_DIALECT, + _make_manifest(), + scope="sample", + sample_size=100_000, + sample_bucket=10, + ) + assert actual == expected + assert isinstance(actual, str) + _assert_no_dialect_leakage(actual) + + +def test_compile_unique_sample_databricks_matches_snapshot() -> None: + expected = _read_databricks_fixture("unique_sample.sql") + test = CandidateTestUnique(column="customer_id") + actual = _compile_test( + test, + _make_orders_table_ref(), + DATABRICKS_DIALECT, + _make_manifest(), + scope="sample", + sample_size=100_000, + sample_bucket=10, + ) + assert actual == expected + assert isinstance(actual, str) + _assert_no_dialect_leakage(actual) + + +def test_compile_accepted_values_sample_databricks_matches_snapshot() -> None: + expected = _read_databricks_fixture("accepted_values_sample.sql") + test = CandidateTestAcceptedValues(column="status", values=("placed", "shipped", "cancelled")) + actual = _compile_test( + test, + _make_orders_table_ref(), + DATABRICKS_DIALECT, + _make_manifest(), + scope="sample", + sample_size=100_000, + sample_bucket=10, + ) + assert actual == expected + assert isinstance(actual, str) + _assert_no_dialect_leakage(actual) + + +def test_compile_relationships_sample_databricks_matches_snapshot() -> None: + """Sample-mode relationships samples the CHILD table only; the parent + stays at the full per-component-quoted qualified name (backtick form).""" + expected = _read_databricks_fixture("relationships_sample.sql") + test = CandidateTestRelationships(column="customer_id", to="customers", field="id") + actual = _compile_test( + test, + _make_orders_table_ref(), + DATABRICKS_DIALECT, + _make_manifest(), + scope="sample", + sample_size=100_000, + sample_bucket=10, + ) + assert actual == expected + assert isinstance(actual, str) + _assert_no_dialect_leakage(actual) + # Belt-and-braces: the parent table is NOT sampled — the full + # per-component backtick-quoted parent identifier survives the wrap. + assert "LEFT JOIN `fake_project`.`dataset`.`customers` AS parent" in actual + + +def test_compile_custom_sql_single_table_full_databricks_matches_snapshot() -> None: + """Single-table custom_sql, scope=full: the resolved SQL is returned + unchanged (the adapter wraps it with the ``COUNT(*)`` envelope). ``{{ this }}`` + resolves to the unquoted qualified name via the bounded Jinja resolver.""" + expected = _read_databricks_fixture("custom_sql.sql") + test = CandidateTestCustomSQL(sql="select order_id from {{ this }} where total < 0") + actual = _compile_test( + test, + _make_orders_table_ref(), + DATABRICKS_DIALECT, + _make_manifest(), + model=_make_orders_model(), + ) + assert actual == expected + assert isinstance(actual, str) + _assert_no_dialect_leakage(actual) + + +def test_compile_custom_sql_single_table_sample_databricks_matches_snapshot() -> None: + """Single-table custom_sql, scope=sample: the model's own qualified table + name is substituted with the ``sample`` CTE alias and the deterministic- + sample CTE (Databricks-backtick-quoted, inline ``xxhash64``) is prepended. + The #116 materialised-sample substitution invariant under the Databricks + quote char: the body references the ``sample`` CTE alias, NEVER the source + table.""" + expected = _read_databricks_fixture("custom_sql_sample.sql") + test = CandidateTestCustomSQL(sql="select order_id from {{ this }} where total < 0") + actual = _compile_test( + test, + _make_orders_table_ref(), + DATABRICKS_DIALECT, + _make_manifest(), + model=_make_orders_model(), + scope="sample", + sample_size=100_000, + sample_bucket=10, + ) + assert actual == expected + assert isinstance(actual, str) + _assert_no_dialect_leakage(actual) + # The test body after the CTE must read from the ``sample`` CTE alias, + # never re-name the source table. + assert "select order_id from sample where total < 0" in actual + body = actual.split(") select", 1)[1] + assert "orders" not in body + assert "fake_project.dataset.orders" not in body + + +def test_compile_custom_sql_multi_table_full_scan_databricks_matches_snapshot() -> None: + """A custom_sql test with a JOIN runs full-scan (unsampled) even when + scope=sample is requested (DEC-006). Both ``{{ this }}`` and ``{{ ref() }}`` + resolve to qualified names; no sample CTE is emitted.""" + expected = _read_databricks_fixture("custom_sql_fullscan.sql") + test = CandidateTestCustomSQL( + sql=( + "select o.order_id from {{ this }} as o " + "join {{ ref('customers') }} as c on o.customer_id = c.id " + "where c.id is null" + ) + ) + actual = _compile_test( + test, + _make_orders_table_ref(), + DATABRICKS_DIALECT, + _make_manifest(), + model=_make_orders_model(), + scope="sample", + sample_size=100_000, + sample_bucket=10, + ) + assert actual == expected + assert isinstance(actual, str) + _assert_no_dialect_leakage(actual) + assert "WITH sample" not in actual + + +def test_compile_row_count_between_databricks_no_where_matches_snapshot() -> None: + """Databricks dialect: per-component backtick-quoted, lower-folded + qualified name — pinned by the byte-exact snapshot fixture.""" + expected = _read_databricks_fixture("row_count_between.sql") + test = CandidateTestRowCountBetween(minimum=1, maximum=1_000_000) + actual = _compile_test(test, _make_orders_table_ref(), DATABRICKS_DIALECT, _make_manifest()) + assert actual == expected + assert isinstance(actual, str) + _assert_no_dialect_leakage(actual) + + +def test_compile_row_count_between_databricks_with_where_matches_snapshot() -> None: + """Databricks dialect + `where`: per-component quoting on the table, + `where` interpolated verbatim (no fold on operator-supplied SQL).""" + expected = _read_databricks_fixture("row_count_between_where.sql") + test = CandidateTestRowCountBetween( + minimum=1, maximum=1_000_000, where="event_date >= '2024-01-01'" + ) + actual = _compile_test(test, _make_orders_table_ref(), DATABRICKS_DIALECT, _make_manifest()) + assert actual == expected + assert isinstance(actual, str) + _assert_no_dialect_leakage(actual) + + +def test_compile_unique_combination_databricks_pair_matches_snapshot() -> None: + """Databricks dialect: per-component backtick-quoted, lower-folded + qualified name + lower-folded column identifiers — pinned by the + byte-exact snapshot fixture.""" + expected = _read_databricks_fixture("unique_combination_pair.sql") + test = CandidateTestUniqueCombination(columns=("customer_id", "order_date")) + actual = _compile_test(test, _make_orders_table_ref(), DATABRICKS_DIALECT, _make_manifest()) + assert actual == expected + assert isinstance(actual, str) + _assert_no_dialect_leakage(actual) + + +def test_compile_unique_combination_databricks_with_where_matches_snapshot() -> None: + """Databricks dialect + ``where``: per-component quoting + lower fold on + the columns; ``where`` interpolated verbatim (no fold on + operator-supplied SQL — that would corrupt literals).""" + expected = _read_databricks_fixture("unique_combination_with_where.sql") + test = CandidateTestUniqueCombination( + columns=("customer_id", "order_date"), + where="status = 'placed'", + ) + actual = _compile_test(test, _make_orders_table_ref(), DATABRICKS_DIALECT, _make_manifest()) + assert actual == expected + assert isinstance(actual, str) + _assert_no_dialect_leakage(actual) + + +def test_compile_unique_combination_databricks_three_columns_matches_snapshot() -> None: + """Databricks dialect, three-column tuple: pinned by byte-exact snapshot. + Confirms the comma-join + per-column fold/quote scales beyond two.""" + expected = _read_databricks_fixture("unique_combination_three_columns.sql") + test = CandidateTestUniqueCombination(columns=("customer_id", "order_date", "region")) + actual = _compile_test(test, _make_orders_table_ref(), DATABRICKS_DIALECT, _make_manifest()) + assert actual == expected + assert isinstance(actual, str) + _assert_no_dialect_leakage(actual) + + +@pytest.mark.parametrize("method", ["mad", "zscore", "percentile", "min_max"]) +@pytest.mark.parametrize("seasonality", ["none", "dow"]) +def test_compile_row_count_anomaly_databricks_stats_matches_snapshot( + method: str, seasonality: str +) -> None: + """8 Databricks stats-query snapshots — lower-folded per-component + backtick-quoted identifiers, ``DATE_TRUNC('DAY', …)`` argument-order, + ``DATE '…'`` cast literals, ``INTERVAL 28 DAY`` payload form, and the + ``DAYOFWEEK(…)`` DOW extraction — every one read from :class:`Dialect` + (DEC-011), not hard-coded.""" + test = _make_anomaly_test(method=method, seasonality=seasonality) + result = _compile_test( + test, + _make_orders_table_ref(), + DATABRICKS_DIALECT, + _make_manifest(), + as_of=_ANOMALY_AS_OF, + ) + assert isinstance(result, tuple) + stats_sql, _ = result + expected = _read_anomaly_db_fixture(f"{method}_{seasonality}_stats.sql") + assert stats_sql == expected + _assert_no_dialect_leakage(stats_sql) + + +@pytest.mark.parametrize("method", ["mad", "zscore", "percentile", "min_max"]) +@pytest.mark.parametrize("seasonality", ["none", "dow"]) +def test_compile_row_count_anomaly_databricks_violation_matches_snapshot( + method: str, seasonality: str +) -> None: + """8 Databricks violation-query snapshots.""" + test = _make_anomaly_test(method=method, seasonality=seasonality) + result = _compile_test( + test, + _make_orders_table_ref(), + DATABRICKS_DIALECT, + _make_manifest(), + as_of=_ANOMALY_AS_OF, + ) + assert isinstance(result, tuple) + _, violation_sql = result + expected = _read_anomaly_db_fixture(f"{method}_{seasonality}_violation.sql") + assert violation_sql == expected + _assert_no_dialect_leakage(violation_sql) From 5b8b9a948789a7c40ce1b5b494d108a675658013 Mon Sep 17 00:00:00 2001 From: Wes Duenow Date: Thu, 25 Jun 2026 09:20:38 -0700 Subject: [PATCH 07/10] =?UTF-8?q?bd=5F1-scaffolding-129.4:=20#223=20US-004?= =?UTF-8?q?=20=E2=80=94=20ungated=20sqlglot=20databricks=20parse-guard=20+?= =?UTF-8?q?=20docs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/prune-ops.md | 2 +- src/signalforge/warehouse/models.py | 10 +- tests/prune/test_compiler_databricks.py | 131 ++++++++++++++++++++++++ 3 files changed, 138 insertions(+), 5 deletions(-) create mode 100644 tests/prune/test_compiler_databricks.py diff --git a/docs/prune-ops.md b/docs/prune-ops.md index 28c2d297..285fa209 100644 --- a/docs/prune-ops.md +++ b/docs/prune-ops.md @@ -838,7 +838,7 @@ concerns are explicitly deferred: - **Per-decision `bytes_billed` recording (DEC-027).** The adapter does not surface job stats in v0.1; the diagnostic probe (US-003) reads them via `INFORMATION_SCHEMA.JOBS_BY_USER` out-of-band rather than through the adapter API. v0.2 extends the adapter's seam to return job stats so the `PruneDecision` can carry the figure natively. - **Per-test `timeout_ms` threading.** `PruneConfig.test_timeout_seconds` is documented but not yet threaded through `WarehouseAdapter.run_test_sql` per call. The plumbing exists in `make_query_job_config` (DEC-013, AR-B2 of issue #6); surfacing it through the public adapter signature is a v0.2 task. - **Test batching — Q4=B / Q4=C optimisations.** The Phase-1 plan catalogues two cost optimisations (per-column `COUNTIF` batching; temp-table-materialised sample). v0.1 does not adopt either. US-003 produces the data needed to evaluate the temp-table option in v0.2. -- **Multi-warehouse adapters.** Postgres, Databricks, Redshift adapters slot in behind `WarehouseAdapter` without prune changes once their adapters land. The prune compiler is fully dialect-driven (DEC-025), reading all warehouse-specific SQL from the `Dialect` value object, never branching on dialect `name`. **Snowflake compiler support landed in issue #121** (see § "Snowflake compiler dialect" above); its live warehouse harness is #124. A new vendor populates a `Dialect` and the compiler emits correct SQL with no compiler change. +- **Multi-warehouse adapters.** Postgres, Databricks, Redshift adapters slot in behind `WarehouseAdapter` without prune changes once their adapters land. The prune compiler is fully dialect-driven (DEC-025), reading all warehouse-specific SQL from the `Dialect` value object, never branching on dialect `name`. **Snowflake compiler support landed in issue #121** (see § "Snowflake compiler dialect" above); its live warehouse harness is #124. **Databricks compiler support landed in issue #223:** `DATABRICKS_DIALECT` emits Spark/Databricks SQL (backtick quoting, `xxhash64` sign-bit-masked sampling hash, Spark date-arithmetic fragments), pinned by byte-exact snapshot fixtures under `tests/fixtures/prune/compiled_sql/databricks/` and certified for syntactic validity by an **ungated** `sqlglot` `databricks`-dialect parse-guard over every fixture (`tests/prune/test_compiler_databricks.py`, runs in the default suite — no marker — because `sqlglot` is a base dep and Databricks has no offline execution fake); real-Spark execution semantics are deferred to the live harness in #226. A new vendor populates a `Dialect` and the compiler emits correct SQL with no compiler change. - **Confidence intervals on `always-passes`.** Surfacing "less than or equal to 3/N upper-bound failure rate at 95 percent confidence" (rule of three) on the decision record so reviewers can calibrate the always-pass verdict. Also covers great-expectations-style `mostly:` thresholds. - **Historical always-pass evidence.** Running candidate tests against multiple `run_results.json` snapshots to assert "never failed in last N runs." The Phase-1 plan considers this for the `failed-on-known-clean-data` evidence channel and defers to v0.2. - **dbt-utils test types.** `dbt_utils.unique_combination_of_columns`, `dbt_utils.accepted_range`, `dbt_utils.expression_is_true`, etc. The drafter's `CandidateTest` union has six variants — the four generic schema tests plus the `custom_sql` business-rule escape hatch (issue #116) plus `row_count_between` (#169); the prune compiler compiles all six. **One `dbt_expectations` macro graduated in #169:** `dbt_expectations.expect_table_row_count_to_be_between` is recognised by `prune-existing` and promoted to the structured `row_count_between` variant (see `docs/ingest-ops.md` § "Recognition of `expect_table_row_count_to_be_between`"). Other namespaced dbt-utils / dbt-expectations macros remain v0.2+ territory (a `custom_sql` test can express many of them by hand in the meantime). diff --git a/src/signalforge/warehouse/models.py b/src/signalforge/warehouse/models.py index 74f8f2cd..284baeb2 100644 --- a/src/signalforge/warehouse/models.py +++ b/src/signalforge/warehouse/models.py @@ -335,10 +335,12 @@ class Dialect: """Databricks/Spark-SQL :class:`Dialect` for the v0.x adapter (issue #221, epic #219). Decided at the skeleton stage; the values the prune compiler keys on are -**certified offline by the #223 ``sqlglot`` ``databricks``-dialect parse-guard -and live by #226** — at the skeleton stage the prune compiler is never invoked -for a Databricks profile (every op raises ``NotImplementedError`` / inherits the -ABC degrade), so these are provisional-but-grounded, not yet executed. +**certified offline by the #223 ``sqlglot`` ``databricks``-dialect parse-guard** +(``tests/prune/test_compiler_databricks.py`` — ungated, runs in the default +suite) and will be certified **live by #226**. At the skeleton stage the prune +compiler is never invoked for a Databricks profile (every op raises +``NotImplementedError`` / inherits the ABC degrade), so these are +grounded-and-parse-validated but not yet executed against real Spark. * ``quote_char='`'`` — Databricks quotes identifiers with backticks (Spark SQL), unlike Snowflake/Postgres double-quote. diff --git a/tests/prune/test_compiler_databricks.py b/tests/prune/test_compiler_databricks.py new file mode 100644 index 00000000..b49aedbd --- /dev/null +++ b/tests/prune/test_compiler_databricks.py @@ -0,0 +1,131 @@ +"""Ungated ``sqlglot`` ``databricks``-dialect parse-guard over every committed +Databricks compiler fixture (#223 US-004). + +Unlike the Snowflake parse-guard (``tests/prune/test_compiler_fakesnow.py``, +gated behind ``@pytest.mark.snowflake`` alongside its ``fakesnow`` execution +tests), this guard runs **UNGATED** — it is collected and executed by the +default ``uv run pytest`` with NO pytest marker (DEC-002 of +``plans/super/223-databricks-prune-compiler.md``, deliberately deviating from +the issue text's "gated under the ``databricks`` marker"). + +Why ungated: + +* ``sqlglot`` is a **base runtime dependency** (``sqlglot>=30,<31`` in + ``[project].dependencies``), so it is always importable in CI — no + maintainer-only install is required to run this guard. +* Databricks has **no offline execution fake** (the Snowflake guard can lean on + ``fakesnow``/DuckDB; there is no equivalent for Spark SQL), so this + parse-guard is the *sole* automated validity gate for the Databricks compiler + SQL until the live cert lands in #226. Gating it behind a marker CI never runs + would mean validity is only checked when a maintainer remembers ``-m + databricks`` — exactly the gap the #121 lesson warns against + (``.claude/rules/prune-engine.md`` § "Compiler is dialect-driven": + *"a new dialect's SQL needs a parser/executor in the loop, not just snapshot + equality — snapshot equality certifies shape, not validity"*). + +What it certifies (and what it does NOT): + +* The byte-exact snapshot fixtures (``tests/prune/test_compiler.py``) certify + the **shape** of the emitted SQL — that the compiler renders the exact bytes + we expect for a given Databricks dialect. +* This guard certifies **syntactic validity** — that those exact bytes parse as + legal Spark/Databricks SQL under ``sqlglot``'s ``databricks`` dialect. A + snapshot can pin invalid SQL byte-for-byte; only a parser in the loop catches + a mis-shaped ``Dialect`` template (a reserved-word collision, a malformed + date-arithmetic fragment, a bad quote placement). +* Real-**Spark execution semantics** (``xxhash64`` value behaviour, identifier + case-folding against a live Unity Catalog table, ``TABLESAMPLE`` / + projection-placement acceptance) are deferred to the gated live harness in + #226 — ``sqlglot`` parses SQL, it does not run it. + +The two fixture directories are globbed at collection time so a future +Databricks fixture is auto-covered without editing this test (mirrors the +Snowflake guard's discovery style). A pinned floor (``>= 32``: 16 top-level + +16 anomaly) guards against an empty/typo'd glob making the guard vacuously +pass. + +Traces to: plans/super/223-databricks-prune-compiler.md US-004 / DEC-002. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest +import sqlglot +from sqlglot.errors import ParseError + +# ``sqlglot`` is a base runtime dependency — import it directly at module scope. +# Do NOT ``pytest.importorskip`` it: that pattern is only for the gated +# marker/fakesnow deps that may be absent from the default environment. + +_FIXTURES_ROOT = Path(__file__).parent.parent / "fixtures" / "prune" / "compiled_sql" +_DATABRICKS_FIXTURES_DIR = _FIXTURES_ROOT / "databricks" +_ANOMALY_DATABRICKS_FIXTURES_DIR = _FIXTURES_ROOT / "anomaly" / "databricks" + +# Discover every committed Databricks fixture (top-level + anomaly) at +# collection time. Sorted for deterministic parametrize ordering. +_ALL_DATABRICKS_FIXTURES: list[Path] = sorted(_DATABRICKS_FIXTURES_DIR.glob("*.sql")) + sorted( + _ANOMALY_DATABRICKS_FIXTURES_DIR.glob("*.sql") +) + +# Pinned floor: 16 top-level + 16 anomaly (4 methods × 2 seasonality × 2 query +# types) = 32. ``>=`` so a future fixture grows the set without editing this. +_EXPECTED_MIN_FIXTURES = 32 + + +def _fixture_id(path: Path) -> str: + """Readable parametrize id: ``/`` so anomaly + top-level fixtures + of the same stem stay distinguishable in the test report.""" + return f"{path.parent.name}/{path.name}" + + +def test_databricks_fixture_set_is_non_empty() -> None: + """The glob discovered at least the 32 committed fixtures. + + Without this floor an empty or mistyped glob would make the parse-guard + below vacuously pass (zero parametrize cases = zero assertions), silently + disabling the sole automated validity gate for the Databricks compiler SQL. + """ + assert len(_ALL_DATABRICKS_FIXTURES) >= _EXPECTED_MIN_FIXTURES, ( + f"expected >= {_EXPECTED_MIN_FIXTURES} Databricks fixtures, " + f"discovered {len(_ALL_DATABRICKS_FIXTURES)}: " + f"{[_fixture_id(p) for p in _ALL_DATABRICKS_FIXTURES]}" + ) + + +@pytest.mark.parametrize("fixture_path", _ALL_DATABRICKS_FIXTURES, ids=_fixture_id) +def test_every_databricks_fixture_parses_under_databricks_dialect( + fixture_path: Path, +) -> None: + """Every committed Databricks compiler fixture must parse under sqlglot's + ``databricks`` dialect. + + A ``sqlglot.errors.ParseError`` here means the compiler emitted invalid + Spark/Databricks SQL — e.g. a mis-shaped :class:`Dialect` template field + (a backtick-quoting slip, a reserved-word CTE alias, a malformed + ``DATE_TRUNC('unit', date)`` / ``xxhash64(...)`` / ``PERCENTILE_CONT … WITHIN + GROUP`` fragment). This reaches both the row-level fixtures and the + sample-mode + row-count-anomaly fixtures (no execution needed), making it the + load-bearing validity certification for the Databricks compiler SQL until the + live cert in #226. + """ + sql = fixture_path.read_text(encoding="utf-8") + # Raises sqlglot.errors.ParseError on invalid Databricks syntax. + parsed = sqlglot.parse_one(sql, dialect="databricks") + assert parsed is not None + + +def test_parse_guard_rejects_malformed_databricks_sql() -> None: + """Planted-violation self-check: the guard CAN fail. + + Per ``.claude/rules/testing-signal.md`` (the planted-violation philosophy), + a gate is only trustworthy if a deliberately-broken input is rejected. A + parse-guard's "violation" is syntactically-invalid SQL — assert + ``sqlglot.parse_one(..., dialect="databricks")`` raises + ``sqlglot.errors.ParseError`` on a + clearly-malformed string, proving the real guard above would catch a + compiler that started emitting invalid Databricks SQL. + """ + with pytest.raises(ParseError): + sqlglot.parse_one("SELECT FROM WHERE )(", dialect="databricks") From bf69b388e470bb59cae9f983afd6978c12919b89 Mon Sep 17 00:00:00 2001 From: Wes Duenow Date: Thu, 25 Jun 2026 09:31:21 -0700 Subject: [PATCH 08/10] =?UTF-8?q?bd=5F1-scaffolding-129.6:=20#223=20US-006?= =?UTF-8?q?=20=E2=80=94=20Databricks=20compiler-dialect=20patterns=20+=20m?= =?UTF-8?q?arker=20note?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Document the 3rd dialect cert in prune-engine.md (verify-don't-re-derive a skeleton dialect; ungated parse-guard DEC-002; unquoted Spark sample CTE) and warehouse-adapters.md (DATABRICKS_DIALECT offline-certified by #223, live deferred to #226). Clarify the databricks pytest marker now gates only the live cert — the offline sqlglot parse-guard is ungated. --- .claude/rules/prune-engine.md | 9 +++++++++ .claude/rules/warehouse-adapters.md | 2 +- pyproject.toml | 2 +- 3 files changed, 11 insertions(+), 2 deletions(-) diff --git a/.claude/rules/prune-engine.md b/.claude/rules/prune-engine.md index 25048fdf..564dd477 100644 --- a/.claude/rules/prune-engine.md +++ b/.claude/rules/prune-engine.md @@ -175,6 +175,15 @@ The compiler emits valid Snowflake SQL purely from `SNOWFLAKE_DIALECT` — see - **`HASH()` reproducibility caveat.** BigQuery's `FARM_FINGERPRINT` is cross-time stable; Snowflake's `HASH()` is deterministic only *within a Snowflake release*. Sufficient for within-run prune determinism (Architectural Commitment #5); documented in `docs/prune-ops.md`. - **Validation tiers (DEC-005).** Byte-exact Snowflake snapshot fixtures (`tests/fixtures/prune/compiled_sql/snowflake/`) are the authoritative shape gate. A gated `@pytest.mark.snowflake` suite (`tests/prune/test_compiler_fakesnow.py`, run `uv run pytest -m snowflake --no-cov`) executes the four built-ins through `fakesnow` (rule-semantic assertions, never `HASH()` value-equality) AND parses every fixture through `sqlglot`'s Snowflake dialect. Real-Snowflake `HASH(*)` semantics + case-folding + sampling are deferred to #124's live harness. The sqlglot parse-guard is the one that caught the `sample` reserved-word bug — keep a parser/executor in the loop for any new dialect. +### Databricks compiler dialect (issue #223) + +The compiler emits valid Databricks/Spark-SQL purely from `DATABRICKS_DIALECT` — the **third** dialect on the dialect-driven seam (after BigQuery + Snowflake). Four #223-specific points worth keeping: + +- **No `DATABRICKS_DIALECT` change was needed.** The dialect values shipped correct in the #221 skeleton; #223 only **verifies** them (unit tests + 32 byte-exact fixtures + a parse-guard) — `models.py` got a docstring edit, no field-value change. A discovery prototype compiled all 8 primitives × {full, sample} (incl. the anomaly two-query split across mad/zscore/percentile + dow) and parsed every statement under `sqlglot`'s `databricks` dialect: 26/26 OK. **Lesson: when a skeleton ships a dialect, the cert ticket is fixtures + tests, not compiler edits — verify, don't re-derive.** +- **The unquoted `sample` CTE alias is valid Spark — no reserved-word collision** (contrast Snowflake's `"sample"` bug, where `SAMPLE` is reserved). `SAMPLE` is not a Spark reserved word; `DATABRICKS_DIALECT` keeps the default `sample_cte_alias="sample"` and the parse-guard confirms it parses. Don't reflexively quote a new dialect's CTE alias — check the vendor's reserved-word list (the parse-guard catches it either way). +- **The `sqlglot` parse-guard is UNGATED (DEC-002 of #223) — deviates from the Snowflake precedent.** `tests/prune/test_compiler_databricks.py` runs `sqlglot.parse_one(fixture, dialect="databricks")` over all 32 fixtures in the **default** suite (no marker). Rationale: `sqlglot` is a base runtime dep (always importable in CI) AND Databricks has **no offline execution fake** (no fakesnow/DuckDB equivalent for Spark), so this parse-guard is the *sole* automated validity gate until the #226 live Free-Edition cert — gating it behind a marker CI never runs would defeat the #121 "keep a parser in the loop" lesson. The Snowflake parse-guard is gated only because it is *co-located* with fakesnow execution (a gated dep); there is no such coupling here. The guard pins a `>= 32` fixture-count floor (no vacuous empty-glob pass) + a planted-violation self-check (sqlglot raises on malformed SQL). **Decision rule for the next dialect: gate the parse-guard only if it's co-located with a gated execution dep; a pure-sqlglot parse-guard with no execution fake should run ungated.** +- **Validation tiers + import-guard.** 32 byte-exact fixtures (`tests/fixtures/prune/compiled_sql/databricks/` 16 + `…/anomaly/databricks/` 16, mirroring the Snowflake set 1:1) are the shape gate; the ungated parse-guard is the validity gate; real-Spark execution semantics are deferred to #226's live harness. `test_compiler_import_guard.py` `_FORBIDDEN_PREFIXES` grew `"databricks"` (planted-violation count 9 → 14) — defensive only; the compiler reads the `Dialect` object and never imports the SDK. Belt-and-braces snapshot leakage assertions check `FARM_FINGERPRINT` (BigQuery) / `HASH(*)` + `::` (Snowflake) are absent — backtick is deliberately NOT a discriminator (BigQuery and Databricks both backtick), so byte-exact **snapshot equality** is the real full-scan-leakage gate, with the leakage asserts as supplementary defence. + ### Centralised bypass routing helper (issue #171) Issue #171 introduced `_test_requires_source_table(test: CandidateTest, sample_strategy: str | None) -> bool` in `signalforge.prune.engine` as the single source of truth for "does this candidate variant require routing to the source table rather than a sampled/materialised temp?". Replaces inline isinstance checks at the two engine sites (`all_bypass_to_source` short-circuit + per-test `per_test_table_ref` override) — both now call the helper, eliminating the two-conditional drift class #170 QG Pass 3 caught. diff --git a/.claude/rules/warehouse-adapters.md b/.claude/rules/warehouse-adapters.md index 0138197b..4f0fc048 100644 --- a/.claude/rules/warehouse-adapters.md +++ b/.claude/rules/warehouse-adapters.md @@ -32,7 +32,7 @@ The ABC is warehouse-agnostic. v0.2 Snowflake/Postgres slot under `adapters/` wi **Databricks skeleton (issue #221, epic #219).** `adapters/databricks.py` is the **fourth** adapter through the skeleton precedent (after BigQuery, the Postgres stub, and Snowflake), mirroring the #119 Snowflake skeleton verbatim: `DatabricksAdapter` captures the conn surface (`host` / `http_path` / `token` / `catalog` / `schema`) plus forward-compat OAuth-M2M params (`auth_type` / `client_id` / `client_secret` — epic open-decision: **PAT only for v0.x**, so `make_real_client` consumes only `token`); `dialect()` returns `DATABRICKS_DIALECT` (`quote_char='`'` backtick, **`identifier_case='lower'`** — Unity Catalog lower-folds, the *opposite* of Snowflake `'upper'`, like Postgres; `supports_qualify=True`; 64-bit **sign-bit-masked** `(xxhash64(to_json(struct(*))) & 9223372036854775807)` sampling hash — `xxhash64` not Murmur3-**32** (Spark's bare `hash(*)`) for collision stability, and the `& Long.MAX` mask not `ABS` because Spark's `ABS(Long.MIN_VALUE)` stays negative in non-ANSI mode and would skew `MOD(, bucket) < 1` (CodeRabbit catch on PR #254); `quote_qualified_per_component=True` for `` `catalog`.`schema`.`table` ``). `sample_rows` / `column_stats` / `run_test_sql` raise `NotImplementedError("…#219…")`; `materialise_sample` / `estimate_query_bytes` / `get_row_count` / `run_stats_query` **inherit the ABC typed degrade** (the four `*NotSupportedError` defaults). `from_profile` dispatches `profile.type == "databricks"` (lazy import; at the #221 skeleton stage it passed only `catalog=profile.project, schema=profile.dataset` — the real `host`/`http_path`/`token`/`catalog` profile fields + per-type validator were deferred to **#222**, exactly the Snowflake #119→#120 deferral; **#222 has since landed** — see § "Unified multi-warehouse `DbtProfileTarget`" → "Databricks arm"). Same three #119 notes apply: (1) **`__repr__` shows only `host` + `http_path` + `catalog`, never `token` / `schema` / `client_secret`** (repr-redaction, pinned by a secret-substring-absent test); (2) **warehouse-side confinement test** (`tests/warehouse/test_databricks_client_confinement.py`) — every `databricks-sql-connector` type/pyright-ignore lives only in `adapters/_databricks_client.py`; (3) `_DatabricksClientProtocol` (connection: `cursor()` / `close()`) split from `_DatabricksCursorProtocol` (`execute(...)` / `fetchall()` / `description` / `close()`). `databricks-sql-connector` ships under the `[databricks]` optional extra (+ dev group — **mirrored unlike `[airflow]`**, since the connector is lightweight + not constraints-pinned); the SDK import stays lazy inside `make_real_client`. The minimal `map_databricks_exception` stub maps auth-flavoured messages → `WarehouseAuthError`, else passthrough (message-marker based, no SDK import); the full table/column/syntax taxonomy + offline fake + gated live e2e land in **#226**. -**`DATABRICKS_DIALECT` values are provisional at the skeleton stage** — the prune compiler is never invoked for a Databricks profile yet (every op raises / degrades), so the dialect's SQL-fragment fields (the `xxhash64` hash, Spark `DATE_TRUNC('unit', date)` / `DAYOFWEEK(date)` / `PERCENTILE_CONT … WITHIN GROUP` date-arithmetic, `TIMESTAMP '{value}'` literals) are grounded-but-unexecuted. They are **certified by the #223 `sqlglot` `databricks`-dialect parse-guard + the #226 gated live run** — snapshot equality alone certifies shape, not validity (the #121/#171 lesson). `identifier_case='lower'` is ⚠️ load-bearing for #223's anchor-contract column matching; verify against a real `CREATE TABLE` round-trip before the compiler locks on it. +**`DATABRICKS_DIALECT` values are offline-certified as of #223; live execution deferred to #226.** At the #221 skeleton stage the dialect's SQL-fragment fields (the `xxhash64` hash, Spark `DATE_TRUNC('unit', date)` / `DAYOFWEEK(date)` / `PERCENTILE_CONT … WITHIN GROUP` date-arithmetic, `TIMESTAMP '{value}'` literals) were grounded-but-unexecuted. **#223 has since landed** the prune-compiler cert: the compiler emits valid Databricks SQL purely from `DATABRICKS_DIALECT` (no field-value change was needed — the #221 values were correct), pinned by 32 byte-exact fixtures (`tests/fixtures/prune/compiled_sql/databricks/` + `…/anomaly/databricks/`) and an **ungated** `sqlglot` `databricks`-dialect parse-guard (`tests/prune/test_compiler_databricks.py`, DEC-002 — runs in the default suite because `sqlglot` is a base dep and Databricks has no offline execution fake; see `prune-engine.md` § "Databricks compiler dialect"). Snapshot equality certifies *shape*, the parse-guard certifies *syntactic validity*; real-Spark execution semantics + a real `CREATE TABLE` round-trip (the `identifier_case='lower'` lower-fold assumption, load-bearing for anomaly-grain column matching) remain deferred to the **#226** gated live Free-Edition run — snapshot/parse equality alone does not certify the engine *accepts* the SQL (the #121/#171 lesson). **⚠️ `TableRef.project` length blocks Unity Catalog catalog names (gotcha for #224).** `TableRef.project` validates as a GCP project id (`validate_project_id`, 6–30 chars), so a Unity Catalog catalog like `main` (4 chars) **fails `TableRef` construction**. The #221 skeleton tests sidestep with `project=None` (two-part `dataset.name`). #224's sampling path threads catalog names into `TableRef.project` (three-part `catalog.schema.table` via `quote_qualified_per_component=True`) and will hit this — either relax the project-id validation for the Databricks dialect or carry the catalog on a separate field. Surfaced while writing the skeleton stub tests; do NOT assume `TableRef` accepts arbitrary catalog identifiers. diff --git a/pyproject.toml b/pyproject.toml index c380c24b..872e8a11 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -183,7 +183,7 @@ markers = [ "e2e: end-to-end smoke test against real Anthropic + real BigQuery (gated by SF_RUN_BQ=1, ANTHROPIC_API_KEY, GOOGLE_CLOUD_PROJECT; skipped by default)", "wheel_smoke: maintainer-only; builds the wheel via `python -m build --wheel` and inspects the artifact for the demo file set (run with --no-cov)", "snowflake: maintainer-only; offline compiled-SQL validation via fakesnow AND the gated live EXPLAIN-estimate certification (run with --no-cov; live tests self-skip without SF_RUN_SNOWFLAKE + connection env vars)", - "databricks: maintainer-only; offline sqlglot/fake validation AND the gated live Free-Edition certification (run with --no-cov; live tests self-skip without SF_RUN_DATABRICKS + DATABRICKS_SERVER_HOSTNAME/HTTP_PATH/TOKEN). See docs/research/databricks-test-environment.md.", + "databricks: maintainer-only; gates the live Free-Edition certification + any fakes (run with --no-cov; live tests self-skip without SF_RUN_DATABRICKS + DATABRICKS_SERVER_HOSTNAME/HTTP_PATH/TOKEN). NOTE: the offline sqlglot databricks parse-guard (tests/prune/test_compiler_databricks.py, #223 DEC-002) is UNGATED — it runs in the default suite since sqlglot is a base dep and there is no offline execution fake. See docs/research/databricks-test-environment.md.", "openai: real-API smoke test against OpenAI (requires SF_RUN_OPENAI=1 + OPENAI_API_KEY; excluded from default CI)", "gemini: real-API smoke test against Gemini (requires SF_RUN_GEMINI=1 + GOOGLE_API_KEY; excluded from default CI)", "airflow: requires Apache Airflow installed under its constraints file (excluded from default CI; run in the gated `airflow` job or locally with `uv run --no-sync pytest -m airflow --no-cov`; DAG-parse + operator legs run unconditionally, the live-DAG leg self-skips without SF_RUN_AIRFLOW=1 + warehouse/LLM env). See docs/research/airflow-test-environment.md.", From cf63dd460479d36c5c0a2cc38f3ce4fdca97bf68 Mon Sep 17 00:00:00 2001 From: Wes Duenow Date: Thu, 25 Jun 2026 09:32:31 -0700 Subject: [PATCH 09/10] #223: plan run-complete stamp --- plans/super/223-databricks-prune-compiler.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/plans/super/223-databricks-prune-compiler.md b/plans/super/223-databricks-prune-compiler.md index 32407915..96084898 100644 --- a/plans/super/223-databricks-prune-compiler.md +++ b/plans/super/223-databricks-prune-compiler.md @@ -148,3 +148,17 @@ Each Databricks snapshot test asserts the distinguishing markers: `xxhash64` pre - `bd_1-scaffolding-129.4` — US-004 sqlglot parse-guard + docs (blocked → .3) - `bd_1-scaffolding-129.5` — Quality Gate (blocked → .1 .2 .3 .4) - `bd_1-scaffolding-129.6` — Patterns & Memory (blocked → .5) + +--- + +## Run complete (2026-06-25) + +All 6 beads landed on `feat/223-databricks-prune-compiler` (sequential in-place; US-002/003/004 share `tests/prune/test_compiler.py`): +- US-001 `d89c935` — import-guard `databricks` prefix (planted count 9→14) +- US-002 `b0d7cea` — 7 Databricks dialect unit tests +- US-003 `1518a26` — 32 byte-exact fixtures + ungated snapshot tests +- US-004 `5b8b9a9` — ungated sqlglot `databricks` parse-guard (34 tests) + docstring/ops +- US-005 (Quality Gate) — 4 diverse review passes, **no real bugs**; two minor single-angle, self-mitigated test-signal observations (leakage assert is supplementary; snapshot equality is the real gate). CodeRabbit runs async on PR #256. +- US-006 `bf69b38` — `prune-engine.md` + `warehouse-adapters.md` patterns; `databricks` marker note; memory. + +Final: `uv run pytest` → 4264 passed / 6 skipped; pyright 0 errors; ruff + format clean. No `DATABRICKS_DIALECT` field change needed (verified correct from #221). Live execution cert remains #226. From 7596a394fd21f2f2e06ed1120e53bd33186e852c Mon Sep 17 00:00:00 2001 From: Wes Duenow Date: Thu, 25 Jun 2026 10:20:12 -0700 Subject: [PATCH 10/10] =?UTF-8?q?#223:=20address=20PR=20review=20=E2=80=94?= =?UTF-8?q?=20distinguish=20anomaly=20fixtures=20in=20parse-guard=20parame?= =?UTF-8?q?trize=20id?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CodeRabbit: path.parent.name is 'databricks' for both the top-level and the anomaly fixture dirs, so the parametrize id couldn't tell them apart. Key the id off the path relative to the fixtures root instead — anomaly fixtures now report as 'anomaly/databricks/'. --- tests/prune/test_compiler_databricks.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/tests/prune/test_compiler_databricks.py b/tests/prune/test_compiler_databricks.py index b49aedbd..53fb1bd7 100644 --- a/tests/prune/test_compiler_databricks.py +++ b/tests/prune/test_compiler_databricks.py @@ -75,9 +75,11 @@ def _fixture_id(path: Path) -> str: - """Readable parametrize id: ``/`` so anomaly + top-level fixtures - of the same stem stay distinguishable in the test report.""" - return f"{path.parent.name}/{path.name}" + """Readable parametrize id relative to the fixtures root, so anomaly + (``anomaly/databricks/…``) and top-level (``databricks/…``) fixtures stay + distinguishable in the test report — ``path.parent.name`` is ``databricks`` + for BOTH dirs, so it can't tell them apart.""" + return str(path.relative_to(_FIXTURES_ROOT)) def test_databricks_fixture_set_is_non_empty() -> None: