Skip to content

#122: Snowflake sampling + materialise_sample (plan) - #129

Merged
wjduenow merged 11 commits into
devfrom
feature/122-snowflake-sampling
May 26, 2026
Merged

#122: Snowflake sampling + materialise_sample (plan)#129
wjduenow merged 11 commits into
devfrom
feature/122-snowflake-sampling

Conversation

@wjduenow

@wjduenow wjduenow commented May 26, 2026

Copy link
Copy Markdown
Owner

Summary

Super plan for #122 — Snowflake deterministic sample_rows (HASH-mod) + materialise_sample (session TEMP TABLE). Part of epic #118.

Phase: detailing (awaiting approval)
Stories: 4 implementation + Quality Gate + Patterns & Memory
Decisions: 10 (DEC-001 … DEC-010)

Key decisions

Plan document

See plans/super/122-snowflake-sampling.md.

Next steps

  • Review the plan in this PR.
  • Approve in Claude Code, then say "devolve" to create beads.

Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Deterministic row sampling and materialisation into session-scoped temporary tables; deterministic sample identifiers and SQL rendering for partition filters; test execution with optional failure capture.
  • Bug Fixes

    • Improved Snowflake exception mapping; fail-soft session teardown with redacted session logging and safer cleanup behavior.
  • Documentation

    • Expanded Snowflake sampling/materialisation, session/cleanup semantics, sizing rules, and adapter guidance.
  • Tests

    • New and expanded suites for sampling, materialisation, lifecycle, exception mapping, and determinism.

Review Change Stack

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented May 26, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (4)
  • feature/.*
  • bug/.*
  • hotfix/.*
  • feat/.*

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: f4852634-9e3b-4021-b5b3-074a9de90c31

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

This PR implements Snowflake deterministic sampling and materialisation by introducing shared cross-adapter run-id generation helpers, Snowflake exception mapping, connection-bound session lifecycle management, deterministic sample_rows SQL construction with sizing guards, CTAS-based materialise_sample with temp-table pinning, and run_test_sql with inline test wrapping. Complete test coverage includes a custom FakeSnowflakeConnection fake and comprehensive suites for sampling, materialisation, cleanup, and cross-adapter plumbing.

Changes

Snowflake Sampling + Materialisation Implementation

Layer / File(s) Summary
Shared Sampling-ID Infrastructure
src/signalforge/warehouse/_sample_id.py
New module exports deterministic _hash_session_id, _canonical_partition_filter, and _compute_run_id helpers using blake2b hashing and stable JSON for cross-adapter temp-table naming and session redaction.
Snowflake Shim Exception Mapping & Cursor Protocol
src/signalforge/warehouse/adapters/_snowflake_client.py
Adds map_snowflake_exception lazy-import helper translating Snowflake connector errors to typed WarehouseError subclasses, and extends _SnowflakeCursorProtocol with description property for row-shape metadata.
FakeSnowflakeConnection Test Double
tests/warehouse/_fake_snowflake.py
Hand-written expectation-driven test fake for snowflake.connector with expect_execute, fetchall, close, and session_id seam; used throughout Snowflake adapter tests for deterministic SQL validation.
BigQuery Refactoring to Shared Helpers
src/signalforge/warehouse/adapters/bigquery.py
Imports _compute_run_id and _hash_session_id from relocated _sample_id module instead of defining locally, eliminating duplication while preserving temp-table naming determinism.
SnowflakeAdapter Connection Lifecycle & Cleanup
src/signalforge/warehouse/adapters/snowflake.py, tests/warehouse/test_snowflake_lifecycle.py
Injectable connection seam via __init__(connection=), lazy _get_connection building real client, session pinning for temp-table reachability, fail-soft __exit__ cleanup with hashed session logging and auto-reap warnings, and comprehensive lifecycle tests.
SnowflakeAdapter sample_rows Implementation
src/signalforge/warehouse/adapters/snowflake.py, tests/warehouse/test_snowflake_sampling.py
Deterministic hash-mod sampling with INFORMATION_SCHEMA row-count sizing, fail-loud size errors, partition-filter SQL rendering, dict-shaped results from cursor metadata; tests cover determinism, sizing branches, partition filters, and Snowflake exception mapping.
SnowflakeAdapter materialise_sample + run_test_sql
src/signalforge/warehouse/adapters/snowflake.py, tests/warehouse/test_snowflake_materialise.py
CTAS-based temp-table materialisation with deterministic _sf_sample_<run_id> naming, session-bound lifecycle tracking, inline test validation and COUNT/ARRAY_AGG wrapping, compiler-pinned substitution for materialized self-FROM tables, and complete test suite.
Cross-Adapter Sample-ID & Exception Mapping Tests
tests/warehouse/test_sample_id.py, tests/warehouse/test_snowflake_client.py
Validates deterministic run-id generation, canonical partition-filter stability, BigQuery helper identity equality, cursor protocol satisfaction, and exception-mapping taxonomy for Snowflake programming/auth errors and passthrough behavior.
Documentation, Rules, and Plan Updates
.claude/rules/warehouse-adapters.md, CLAUDE.md, plans/super/122-snowflake-sampling.md
Documents Snowflake sampling/materialisation surface, connection-bound session semantics, fail-soft cleanup divergences from BigQuery, shared helper locations, and complete Phase 4-6 plan execution with Beads Manifest task chain.
SnowflakeAdapter Test Stub Updates
tests/warehouse/test_snowflake_stub.py
Updates skeleton test contract to reflect sample_rows, materialise_sample, and run_test_sql now implemented and tested elsewhere; removes NotImplementedError assertions and adds explicit no-op context-manager test for unopened connections.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related issues

Possibly related PRs

  • wjduenow/SignalForge#31: Refactored deterministic run-id and partition-filter hashing logic that this PR centralises into src/signalforge/warehouse/_sample_id.py.
  • wjduenow/SignalForge#126: Overlaps with constructor/auth surface edits to SnowflakeAdapter.__init__ and related wiring.
  • wjduenow/SignalForge#125: Earlier SnowflakeAdapter skeleton and shim changes that this PR completes with full sampling/materialisation logic and exception mapping.

Poem

🐰 I braided hashes, temp tables, and a snowflake tune,
Session pinned and tidy beneath the moon,
Helpers shared across adapters, tests that hum along,
A rabbit's little script—deterministic and strong.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 66.02% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately reflects the main implementation work: Snowflake sampling via hash-mod and materialise_sample temp-table creation as specified in plan #122.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Comment @coderabbitai help to get the list of available commands and usage tips.

@codecov-commenter

codecov-commenter commented May 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

wjduenow and others added 6 commits May 26, 2026 14:02
…apper (#122 US-001)

Foundations for Snowflake sampling — pure plumbing, no adapter behaviour.

- Relocate _compute_run_id / _canonical_partition_filter / _hash_session_id
  VERBATIM out of adapters/bigquery.py into a new shared
  signalforge/warehouse/_sample_id.py (DEC-008) so BigQuery and Snowflake
  produce byte-identical run_ids. BigQuery imports them; recipe bytes
  unchanged (all existing materialise-sample / prune snapshots byte-identical).
- Add `description` to _SnowflakeCursorProtocol (DEC-010, DB-API 2.0 column
  descriptors) so the future adapter builds dict rows without a DictCursor.
- Add map_snowflake_exception(exc, *, context) to _snowflake_client.py
  (DEC-009): lazy snowflake.connector.errors import (one-shim-per-vendor),
  minimal v0.2 taxonomy — ProgrammingError -> QuerySyntaxError, auth/forbidden
  -> WarehouseAuthError, else passthrough. Mirrors map_bq_exception shape.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…tion lifecycle (#122 US-002)

Wire the SnowflakeAdapter connection seam and fail-soft __exit__ cleanup, plus
a hand-rolled FakeSnowflakeConnection test double. Sampling / materialise /
run_test_sql stay NotImplementedError (US-003/US-004).

- tests/warehouse/_fake_snowflake.py: explicit FakeSnowflakeConnection +
  cursor satisfying the _SnowflakeClientProtocol / _SnowflakeCursorProtocol;
  expect_execute() queues round-trips, close_raises drives the cleanup-failure
  path. No MagicMock (testing-signal.md).
- adapters/snowflake.py: injectable connection= kwarg + lazy _get_connection()
  build (DEC-001), connection-bound _active_session state (DEC-002), and a
  fail-soft _cleanup_active_session() that closes the connection (reaping
  session-scoped temp tables), swallows-and-warns on failure with an
  operator-actionable Snowflake-shaped WARNING naming the raw session_id +
  server-side reap fallback (DEC-003/DEC-014); success logs the hashed id only.
  __repr__ unchanged (account + warehouse only).
- tests: new test_snowflake_lifecycle.py covers injection, lazy build,
  __repr__ redaction regression, single-close + idempotent second __exit__,
  hashed-id INFO, and the WARNING failure path; updated the stub's
  context-manager test for the now-real cleanup.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…le_rows (#122 US-003)

Implements deterministic hash-mod sampling for Snowflake, mirroring
BigQuery's sample_rows semantics (DEC-005/006/009/010):

- _get_num_rows: ROW_COUNT from INFORMATION_SCHEMA.TABLES, case-insensitive,
  escaped string literals, CURRENT_DATABASE() fallback when project is None.
- Fail-loud sizing identical to BigQuery: UnknownTableSizeError /
  SamplingRequiresPartitionFilterError; bucket=1000 fallback on unknown size
  with a filter; bucket=max(num_rows//n,1) otherwise. _LARGE_TABLE_THRESHOLD
  re-declared (100M) to avoid importing the BigQuery adapter.
- SQL reuses SNOWFLAKE_DIALECT.sample_row_hash_expr (ABS(HASH(*))) + the
  timestamp/date literal templates so it stays byte-consistent with the prune
  compiler's sample CTE; per-component double-quoting; ORDER BY for
  deterministic LIMIT truncation.
- Tuple fetchall() rows shaped into dicts via cursor.description (DEC-010);
  SDK exceptions routed through map_snowflake_exception (DEC-009).

column_stats / run_test_sql / materialise_sample stay NotImplementedError.
Removed the now-stale test_sample_rows_raises_not_implemented stub test.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…t_sql (#122 US-004)

Implement the two remaining sampling-consumer surfaces on SnowflakeAdapter:

- materialise_sample: CTAS into a session-scoped TEMPORARY TABLE
  (_sf_sample_<run_id>, run_id from the shared _sample_id recipe so it is
  byte-identical to BigQuery; DEC-008), colocated with + fully-qualified via
  the source DB/schema (DEC-007), deterministic MOD(ABS(HASH(*)),bucket)<1 +
  ORDER BY ABS(HASH(*)) read from SNOWFLAKE_DIALECT (DEC-006), pins the live
  connection as _active_session so a follow-up run_test_sql reaches the temp
  table (DEC-002). SDK failures route through map_snowflake_exception then wrap
  in MaterialisationFailedError (DEC-009); INFO log emits the hashed session id
  only (DEC-003). n<=0 -> ValueError.
- run_test_sql: validate_test_sql -> COUNT(*) wrap (ARRAY_AGG(OBJECT_CONSTRUCT(*))
  sample capture when capture_failures>0) on the active connection, returns a
  typed TestResult; case-insensitive alias resolution for Snowflake's upper-folded
  FAILURES/SAMPLES columns (DEC-004).
- Factor the fail-loud sizing pathway into a shared _resolve_sample_bucket helper
  reused by sample_rows + materialise_sample (DEC-005, no duplicated logic).

column_stats stays NotImplementedError (#118); estimate_query_bytes stays the ABC
not-supported degrade (#123).

Remove the now-stale run_test_sql / materialise_sample NotImplemented stub tests;
add tests/warehouse/test_snowflake_materialise.py pinning CTAS shape, fully-qualified
temp TableRef, _active_session pinning, run_test_sql reachability on one connection,
the #116 substitution AC (compiler emits the temp-table name, NOT the source), and
the failure modes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…US-005)

- BUG: _cleanup_active_session no longer nulls self._connection (discarded
  an injected fake / forced a real lazy-rebuild on re-entry; mirrors BigQuery
  which resets only _active_session, never the client).
- #116 substitution test now exercises the bypassable path: CandidateTestCustomSQL
  with {{ this }} at scope="full" (not_null trivially FROMs table_ref and can
  never bypass substitution).
- Add num_rows==0 sizing test (was untested; mirrors the None pathway).
- Fix stale/misleading docstrings (module/class "still raises", _session_started_at
  "auto-expire" text Snowflake never emits).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ng conventions (#122 US-006)

- warehouse-adapters.md: new "Snowflake sampling + connection-bound session"
  section (connection-as-session, dialect-field reuse, INFORMATION_SCHEMA
  sizing, shared _sample_id hoist, Snowflake-shaped fail-soft cleanup); update
  the cleanup-boundary forward-note + Reference.
- CLAUDE.md: public-API surface note for the implemented sampling methods.
- plan: mark complete.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@wjduenow
wjduenow requested a review from Copilot May 26, 2026 21:36
@wjduenow
wjduenow marked this pull request as ready for review May 26, 2026 21:36
Codecov flagged 15 uncovered lines in adapters/snowflake.py (93%). Add tests
for the previously-untested branches — all now 100%:
- _execute / _execute_to_dicts / run_test_sql SDK-error mapping: both the
  mapped (ProgrammingError -> QuerySyntaxError) and unmapped-passthrough
  (mapped is exc -> raise original) branches, on size + sample + count queries.
- _rows_to_dicts dict-row passthrough (DictCursor-style mapping rows).
- materialise_sample with a PartitionFilter (CTAS WHERE rendering).
- _get_connection lazy real-client build when no connection= is injected.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Implements deterministic Snowflake sampling for SnowflakeAdapter: sample_rows (HASH-mod), materialise_sample (session TEMP TABLE), and run_test_sql (COUNT(*) wrap), plus a fail-soft __exit__ cleanup and a shared cross-adapter _sample_id helper module hoisted from BigQuery.

Changes:

  • Implements sample_rows / materialise_sample / run_test_sql on SnowflakeAdapter, with INFORMATION_SCHEMA-based fail-loud sizing and a connection-bound _active_session.
  • Hoists _compute_run_id / _hash_session_id / _canonical_partition_filter to a shared signalforge.warehouse._sample_id module so BigQuery and Snowflake produce byte-identical run_ids.
  • Extends _SnowflakeCursorProtocol with description and adds a minimal map_snowflake_exception mapper in the SDK shim.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
src/signalforge/warehouse/adapters/snowflake.py Implements sampling/materialise/run_test_sql + connection seam + fail-soft cleanup.
src/signalforge/warehouse/_sample_id.py New shared module with relocated deterministic-sample-id helpers.
src/signalforge/warehouse/adapters/bigquery.py Removes local helpers and imports from _sample_id.
src/signalforge/warehouse/adapters/_snowflake_client.py Adds description to cursor protocol and map_snowflake_exception.
tests/warehouse/_fake_snowflake.py New hand-rolled fake with expect_execute API.
tests/warehouse/test_snowflake_sampling.py Sample_rows behavior, sizing branches, partition rendering, error mapping tests.
tests/warehouse/test_snowflake_materialise.py Materialise + run_test_sql + #116 substitution coverage.
tests/warehouse/test_snowflake_lifecycle.py Connection seam, repr redaction, fail-soft cleanup tests.
tests/warehouse/test_snowflake_stub.py Removes stub assertions for implemented methods.
tests/warehouse/test_sample_id.py Tests for relocated helpers + cursor protocol + error mapper.
tests/warehouse/test_snowflake_client.py Adds description and map_snowflake_exception to fake/exports.
plans/super/122-snowflake-sampling.md New plan document.
CLAUDE.md, .claude/rules/warehouse-adapters.md Documentation updates capturing #122 conventions.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/signalforge/warehouse/adapters/snowflake.py Outdated
@wjduenow

Copy link
Copy Markdown
Owner Author

@CodeRabbit review

@coderabbitai

coderabbitai Bot commented May 26, 2026

Copy link
Copy Markdown
✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai

coderabbitai Bot commented May 26, 2026

Copy link
Copy Markdown

Note

Docstrings generation - SUCCESS
Generated docstrings for this pull request at #131

coderabbitai Bot added a commit that referenced this pull request May 26, 2026
Docstrings generation was requested by @wjduenow.

* #129 (comment)

The following files were modified:

* `src/signalforge/warehouse/_sample_id.py`
* `src/signalforge/warehouse/adapters/_snowflake_client.py`
* `src/signalforge/warehouse/adapters/bigquery.py`
* `src/signalforge/warehouse/adapters/snowflake.py`
* `tests/warehouse/_fake_snowflake.py`
* `tests/warehouse/test_sample_id.py`
* `tests/warehouse/test_snowflake_client.py`
* `tests/warehouse/test_snowflake_lifecycle.py`
* `tests/warehouse/test_snowflake_materialise.py`
* `tests/warehouse/test_snowflake_sampling.py`
* `tests/warehouse/test_snowflake_stub.py`
…lookup

Copilot PR-review catch: CURRENT_DATABASE().INFORMATION_SCHEMA.TABLES is
invalid Snowflake — CURRENT_DATABASE() is a scalar function, not a namespace
qualifier, and would fail on a live account (the hand-rolled fake matched it by
regex; live execution is deferred to #124). When table.project is None, leave
the lookup unqualified (INFORMATION_SCHEMA.TABLES), which Snowflake resolves
against the session's current database. Update the test + rule + plan notes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@wjduenow

Copy link
Copy Markdown
Owner Author

PR Review Summary

Fixed (1 item)

File Line Issue Commit
src/signalforge/warehouse/adapters/snowflake.py 355 CURRENT_DATABASE().INFORMATION_SCHEMA.TABLES is invalid Snowflake — CURRENT_DATABASE() is a scalar function, not a namespace qualifier; would fail on a live account. When table.project is None, the size lookup is now left unqualified (INFORMATION_SCHEMA.TABLES), which Snowflake resolves against the session's current database. Updated test_project_none_size_query_is_unqualified + the rule/plan notes accordingly. 781b846

False Positives (0 items)

Good catch — the hand-rolled fake matched the bad SQL by regex, so the unit tests passed; this would only have surfaced in #124's live-Snowflake run. CI green (lint-test 3.11/3.12/3.13, docs-build); Codecov patch coverage 100%.

@wjduenow
wjduenow merged commit eda7c52 into dev May 26, 2026
6 checks passed
@wjduenow
wjduenow deleted the feature/122-snowflake-sampling branch May 26, 2026 22:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants