#223: Databricks prune compiler emits valid Databricks SQL from DATABRICKS_DIALECT

[wjduenow]

Summary

Super plan for #223 — make _compile_test emit valid Databricks SQL purely from DATABRICKS_DIALECT, proven by byte-exact fixtures + a sqlglot databricks-dialect parse-guard.

Phase: detailing (awaiting approval)
Stories: 4 implementation + Quality Gate + Patterns & Memory
Decisions: 5 captured (DEC-001…005)

Key discovery finding

A de-risking prototype compiled all 8 primitives × {full, sample} scope (incl. the anomaly two-query split across mad/zscore/percentile + dow) with DATABRICKS_DIALECT and parsed every statement through sqlglot's databricks dialect: 26/26 OK, 0 failures. The #221 dialect + dialect-driven compiler are already correct — this ticket is fixtures + tests + one import-guard line + docs.

• No DATABRICKS_DIALECT change needed (DEC-001).
• Unquoted sample CTE alias is valid Spark — no reserved-word collision (unlike Snowflake).
• sqlglot parse-guard runs ungated (DEC-002): sqlglot is a base dep, no offline execution fake exists, so it is the sole automated validity gate until the Databricks: test harness + gated live e2e + ops docs #226 live cert — keep the parser in the loop on every CI run.
• Fixtures mirror the Snowflake 16+16 set (DEC-003).

Plan document

See plans/super/223-databricks-prune-compiler.md.

Next steps

• Review the plan in this PR
• Approve in Claude Code, then say "devolve" to create beads

Summary by CodeRabbit

• Documentation
  • Added expanded Databricks/Spark-SQL compiler support notes, including validation tiering, sampling behavior (keeps sample unquoted), and verification guidance.
  • Published a full planning document for Databricks prune compiler work and refined adapter wording about offline vs live certification.
• Tests
  • Expanded Databricks compiler snapshot coverage with updated/added fixture SQL and stronger per-dialect expectations.
  • Added an ungated SQL parse-guard that parses all committed Databricks fixtures and detects malformed SQL; added a non-empty fixture safety check.
  • Strengthened “no warehouse SDK imports” checks to also forbid Databricks imports.
• Bug Fixes
  • Improved end-to-end Databricks SQL rendering consistency (quoting/qualification, typed date/timestamp filters, and leakage prevention).

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: ebcab97c-cbd8-488e-ab61-c8bb2cbbf73d

📥 Commits

Reviewing files that changed from the base of the PR and between cf63dd4 and 7596a39.

📒 Files selected for processing (1)

• tests/prune/test_compiler_databricks.py

🚧 Files skipped from review as they are similar to previous changes (1)

• tests/prune/test_compiler_databricks.py

---

📝 Walkthrough

Walkthrough

Databricks prune-compiler docs were updated, the import guard now forbids Databricks imports, compiler snapshots and fixtures were added, and an ungated sqlglot parse guard now checks committed Databricks fixtures.

Changes

Databricks prune compiler validation

Layer / File(s)	Summary
Validation docs and markers plans/super/223-databricks-prune-compiler.md, docs/prune-ops.md, .claude/rules/prune-engine.md, .claude/rules/warehouse-adapters.md, pyproject.toml, src/signalforge/warehouse/models.py	The Databricks compiler plan, docs, marker text, and module docstring now describe DATABRICKS_DIALECT emission, offline parse validation, and deferred live execution.
Import guard expansion tests/prune/test_compiler_import_guard.py	The prune import guard now forbids databricks imports and the planted-violation and false-positive tests cover matching import forms.
Compiler assertions and snapshots tests/prune/test_compiler.py, tests/fixtures/prune/compiled_sql/databricks/*, tests/fixtures/prune/compiled_sql/anomaly/databricks/*	Databricks compiler assertions now cover quoting, qualified names, sampling, and partition filters, and the snapshot fixtures add byte-exact SQL for built-in, sample, and anomaly cases.
Databricks parse guard tests/prune/test_compiler_databricks.py	A dedicated Databricks parse guard discovers committed fixtures, checks them with sqlglot under the Databricks dialect, and includes fixture-count and malformed-SQL self-checks.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related issues

• Databricks: prune compiler emits valid Databricks SQL from DATABRICKS_DIALECT #223: The PR adds the Databricks prune-compiler docs, fixtures, parse guard, and import-guard coverage described by this ticket.

Possibly related PRs

• wjduenow/SignalForge#20: Adds the prune-compiler test scaffolding that this PR extends with Databricks-specific assertions and fixtures.
• wjduenow/SignalForge#127: Introduces the prune import-guard pattern that this PR expands to include Databricks imports.
• wjduenow/SignalForge#254: Shares the Databricks dialect contract in src/signalforge/warehouse/models.py that this PR references in compiler tests and docs.

Poem

I hopped through fixtures, neat and small,
With carrot light on every wall.
Databricks SQL now sings okay,
And parse guards keep the bad bits away.
Hooray, a tidy burrow day! 🐰

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 79.07% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: Databricks prune compiler output from DATABRICKS_DIALECT with valid Databricks SQL.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

---

_{Comment @coderabbitai help to get the list of available commands.}

[wjduenow]

Implementation complete ✅

All 6 epic beads (bd_1-scaffolding-129) landed; this PR moves from plan → implementation.

Bead	Commit	What
US-001	d89c935	import-guard databricks prefix (planted count 9→14)
US-002	b0d7cea	7 Databricks dialect unit tests
US-003	1518a26	32 byte-exact fixtures + ungated snapshot tests
US-004	5b8b9a9	ungated sqlglot databricks parse-guard (34 tests) + docstring/ops
Quality Gate	—	4 diverse review passes, no real bugs
US-006	bf69b38	rules patterns + marker note + memory

Key outcome: no DATABRICKS_DIALECT change was needed — the #221 skeleton values were already correct (a discovery prototype compiled all 8 primitives × {full, sample} and parsed 26/26 under sqlglot's databricks dialect). The parse-guard runs ungated (DEC-002): sqlglot is a base dep and Databricks has no offline execution fake, so it's the sole automated validity gate until the #226 live cert. Unquoted sample CTE is valid Spark (no Snowflake-style reserved-word fix).

Final validation: uv run pytest → 4264 passed / 6 skipped, pyright 0 errors, ruff + format clean. BigQuery + Snowflake fixtures byte-unchanged. Live execution cert tracked in #226.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/prune/test_compiler_databricks.py`:
- Around line 77-80: The _fixture_id helper currently uses path.parent.name,
which collapses both fixture trees to the same parametrization ID. Update
_fixture_id in test_compiler_databricks to build the ID from the path relative
to _FIXTURES_ROOT so the anomaly/databricks branch is included and top-level
fixtures remain distinguishable. Keep the returned ID readable, but ensure it
uniquely reflects the full fixture location instead of only the immediate parent
directory.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 30616643-fef2-4de1-96fd-bb6714cd106b

📥 Commits

Reviewing files that changed from the base of the PR and between c1b638c and cf63dd4.

📒 Files selected for processing (41)

• .claude/rules/prune-engine.md
• .claude/rules/warehouse-adapters.md
• docs/prune-ops.md
• plans/super/223-databricks-prune-compiler.md
• pyproject.toml
• src/signalforge/warehouse/models.py
• tests/fixtures/prune/compiled_sql/anomaly/databricks/mad_dow_stats.sql
• tests/fixtures/prune/compiled_sql/anomaly/databricks/mad_dow_violation.sql
• tests/fixtures/prune/compiled_sql/anomaly/databricks/mad_none_stats.sql
• tests/fixtures/prune/compiled_sql/anomaly/databricks/mad_none_violation.sql
• tests/fixtures/prune/compiled_sql/anomaly/databricks/min_max_dow_stats.sql
• tests/fixtures/prune/compiled_sql/anomaly/databricks/min_max_dow_violation.sql
• tests/fixtures/prune/compiled_sql/anomaly/databricks/min_max_none_stats.sql
• tests/fixtures/prune/compiled_sql/anomaly/databricks/min_max_none_violation.sql
• tests/fixtures/prune/compiled_sql/anomaly/databricks/percentile_dow_stats.sql
• tests/fixtures/prune/compiled_sql/anomaly/databricks/percentile_dow_violation.sql
• tests/fixtures/prune/compiled_sql/anomaly/databricks/percentile_none_stats.sql
• tests/fixtures/prune/compiled_sql/anomaly/databricks/percentile_none_violation.sql
• tests/fixtures/prune/compiled_sql/anomaly/databricks/zscore_dow_stats.sql
• tests/fixtures/prune/compiled_sql/anomaly/databricks/zscore_dow_violation.sql
• tests/fixtures/prune/compiled_sql/anomaly/databricks/zscore_none_stats.sql
• tests/fixtures/prune/compiled_sql/anomaly/databricks/zscore_none_violation.sql
• tests/fixtures/prune/compiled_sql/databricks/accepted_values.sql
• tests/fixtures/prune/compiled_sql/databricks/accepted_values_sample.sql
• tests/fixtures/prune/compiled_sql/databricks/custom_sql.sql
• tests/fixtures/prune/compiled_sql/databricks/custom_sql_fullscan.sql
• tests/fixtures/prune/compiled_sql/databricks/custom_sql_sample.sql
• tests/fixtures/prune/compiled_sql/databricks/not_null.sql
• tests/fixtures/prune/compiled_sql/databricks/not_null_sample.sql
• tests/fixtures/prune/compiled_sql/databricks/relationships.sql
• tests/fixtures/prune/compiled_sql/databricks/relationships_sample.sql
• tests/fixtures/prune/compiled_sql/databricks/row_count_between.sql
• tests/fixtures/prune/compiled_sql/databricks/row_count_between_where.sql
• tests/fixtures/prune/compiled_sql/databricks/unique.sql
• tests/fixtures/prune/compiled_sql/databricks/unique_combination_pair.sql
• tests/fixtures/prune/compiled_sql/databricks/unique_combination_three_columns.sql
• tests/fixtures/prune/compiled_sql/databricks/unique_combination_with_where.sql
• tests/fixtures/prune/compiled_sql/databricks/unique_sample.sql
• tests/prune/test_compiler.py
• tests/prune/test_compiler_databricks.py
• tests/prune/test_compiler_import_guard.py

[wjduenow]

PR Review Summary

Fixed (1 item)

File	Line	Issue	Commit
tests/prune/test_compiler_databricks.py	80	Parametrize id used path.parent.name, which is databricks for both the top-level (compiled_sql/databricks/) and anomaly (compiled_sql/anomaly/databricks/) fixture dirs — so anomaly fixtures were indistinguishable in the test report. Now keyed off the path relative to the fixtures root, so anomaly cases report as anomaly/databricks/<name>.	7596a39

False Positives (0 items)

None.

All CI green (lint-test 3.11/3.12/3.13, CodeRabbit, docs-build); pyright 0 errors; tests/prune/test_compiler_databricks.py → 34 passed.