From f0c9a920c05d9a4058d37265c22545e22b6d4482 Mon Sep 17 00:00:00 2001 From: wjduenow Date: Tue, 28 Apr 2026 22:19:20 -0700 Subject: [PATCH 01/19] Add super plan for #4: PII safety layer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 1 locks library-only scope (no CLI, no LLM client — those land in #9 and #5). Phase 2 surfaced the load-bearing finding that schema-only mode leaks PII via column NAMES; Phase 3 resolves it with stable blake2b-hash placeholders. 26 decisions captured across config, fail- closed audit semantics, AuditEvent reproducibility (signalforge_version + policy_hash + audit_schema_version), extra=forbid on config-shaped models, and the SafetyPolicy.with_mode() seam #9 will use. Detailed Breakdown is 14 stories: scaffolding -> fixtures -> errors -> models -> SafetyPolicy -> config loader -> audit -> redact -> aggregate -> request builder -> public API + drift detector + AST scan -> docs -> quality gate -> patterns. TDD specified for every story with non-trivial business logic. Co-Authored-By: Claude Opus 4.7 (1M context) --- plans/super/4-pii-safety.md | 737 ++++++++++++++++++++++++++++++++++++ 1 file changed, 737 insertions(+) create mode 100644 plans/super/4-pii-safety.md diff --git a/plans/super/4-pii-safety.md b/plans/super/4-pii-safety.md new file mode 100644 index 00000000..c852f25c --- /dev/null +++ b/plans/super/4-pii-safety.md @@ -0,0 +1,737 @@ +# Issue #4 — PII safety: schema-only default, sample opt-in, redaction patterns + +## Meta + +- **Ticket:** [#4](https://github.com/wjduenow/SignalForge/issues/4) +- **Branch:** `feature/4-pii-safety` (off `dev`) +- **Worktree:** `/home/wesd/dev/worktrees/SignalForge/feature/4-pii-safety` (created via `git worktree add`) +- **Phase:** detailing (Phases 1–3 locked 2026-04-28 "use defaults"; ready to publish PR) +- **Sessions:** 1 (started 2026-04-28) +- **Plan author:** Claude Code (Opus 4.7, 1M context) +- **Milestone:** v0.1 (deployment blocker for any team with PII concerns) +- **Labels:** `safety` + +## Discovery + +### Ticket summary + +Default to the safest data-access posture; require explicit opt-in to expose row-level data to the LLM. Three sampling **modes** govern what the LLM ever sees: + +- `schema-only` (default) — column names + types, never queries data +- `aggregate-only` — `count`, `count(distinct)`, `min`, `max`, null-rate per column; no raw values +- `sample` — row-level data subject to redaction patterns + +Plus: configurable redaction patterns (`*_email`, `*_phone`, `*_ssn`), per-column opt-out via dbt `meta.signalforge.sample: false`, an audit log line per LLM call, and a "Data safety" section in the README. + +This is a **deployment blocker** for any team with PII concerns; v0.1 must ship even if minimal. Reference: `docs/research/dbt-claude-technical-surface.md` Section 4.5 ("PII / safety") explicitly recommends "default to schema-only with explicit opt-in for sampling — the only defensible posture." + +### Acceptance criteria (from ticket) + +1. Three sampling modes: `schema-only` (default), `aggregate-only`, `sample`. +2. CLI flag `--mode {schema-only,aggregate-only,sample}` and config-file equivalent. +3. In `sample` mode, redact columns matching configurable patterns (`*_email`, `*_phone`, `*_ssn`) before sending to the LLM. +4. Per-column opt-out via dbt `meta.signalforge.sample: false`. +5. Audit log line per LLM call describing which mode + which columns were sent. +6. README "Data safety" section documenting the model. + +### Codebase findings (Subagent B equivalent — verified directly) + +- **Warehouse adapter (#3) shipped** with `WarehouseAdapter.sample_rows`, `column_stats`, and `run_test_sql`. The safety layer wraps these — `schema-only` calls neither, `aggregate-only` calls `column_stats`, `sample` calls `sample_rows` then redacts. No new adapter methods needed. +- **`signalforge.manifest.Model.meta` already exists** as `dict[str, Any]` (`src/signalforge/manifest/models.py:78`). Per-column meta also lives on `Column.meta` (line 61). Both are surfaced through `Manifest.get_model(...)` already — `meta.signalforge.sample` is readable from day one. **No manifest-layer changes required.** +- **No LLM client seam exists yet.** `grep -r 'llm\|claude\|anthropic\|prompt' src/` returns only incidental matches in errors/path_safety. Issue [#5 — LLM draft pipeline](https://github.com/wjduenow/SignalForge/issues/5) is the home for the actual `client.complete(...)` call. **#4 must produce a contract #5 can consume without dictating #5's design** (see scoping Q9 below). +- **No CLI exists yet.** Issue [#9 — CLI](https://github.com/wjduenow/SignalForge/issues/9) is the home for `signalforge generate --mode ...`. **#4 must lock the policy/config shape; CLI flag wiring is #9's job** (see scoping Q6). +- **No project-level config file exists yet.** `signalforge.yml` is greenfield. PyYAML is already a runtime dep (added by #3 for `profiles.yml`); reusing it is free. +- **Sibling open issues** confirm scope boundaries: #5 (LLM draft), #6 (prune), #7 (grader), #8 (diff renderer), #9 (CLI), #10 (smoke), #11 (README), #12 (release). #4 is library-only; downstream tickets consume the contract. +- **Validation command** (per CLAUDE.md): `pip install -e ".[dev]" && ruff check . && ruff format --check . && pyright && pytest`. + +### Project rules (`.claude/rules/`) audit + +- **`python-build.md`** — Hatchling + src layout + explicit wheel `packages = ["src/signalforge"]`. New subpackage requires no wheel-target edit (already covers all of `src/signalforge/**`). Editable install via quoted `".[dev]"`. +- **`manifest-readers.md`** — Targets external-format readers. The safety layer is **not** a reader (it consumes already-typed data from `signalforge.manifest` and `signalforge.warehouse`), so the symlink-hardening / Pydantic-frozen rules don't directly bind. **Three carry-over principles do apply:** (a) every typed exception subclasses a module base (`SafetyError`) and accepts a `remediation: str` kwarg; (b) `extra="forbid"` is for one-off drift-detection tests only; production policy models use `extra="ignore"` for forward-compat; (c) `repr()`-quote user input in error messages (DEC-022 from #3). +- **`testing-signal.md`** — Hard-applies. No `assert True`-shaped tests. Strict markers (both settings — pytest-9 quirk). No `tests/__init__.py`. **`unittest.mock.MagicMock` is implicitly forbidden** for the LLM-call audit shim — use explicit fakes that fail loudly. The redaction layer is exactly the kind of code where a `MagicMock`-style fake would silently auto-pass while real code leaks PII; explicit `FakeLLMClient` with `expect_*` helpers (mirrors #3's `FakeBigQueryClient`). +- **`ci-supply-chain.md`** — No new CI workflow needed (no integration tests against external services). Single Python 3.11 holds. +- **`warehouse-adapters.md`** — The safety layer **calls** the warehouse adapter, never bypasses it. Path-safety duplication precedent (US-014 from #3) suggests: if the safety layer needs path canonicalisation for `signalforge.yml`, it copies the helper from `signalforge.warehouse._path_safety` rather than imports — keeps each layer's exception surface homogeneous. (Q2 below decides whether we need this.) +- **No `workflow-project.md`** — baseline review areas only in Phase 2. + +### CLAUDE.md commitments that bite this ticket + +- **#1 — Signal over volume.** Audit log is the user-facing receipt that explains *why* the LLM saw what it saw. Every redacted column, every skipped column (per-column opt-out), every mode-driven omission must surface in the audit line; black-box "the LLM saw something" violates the commitment. +- **#3 — Warehouse-agnostic by design.** The safety layer **must not** call BigQuery-specific code. It calls `WarehouseAdapter` only — through the ABC. Tests use the adapter ABC, not `BigQueryAdapter` (use a `FakeAdapter` in tests; v0.2 Snowflake/Postgres get safety for free). +- **#4 — OSS-first / Core-friendly.** No dbt Cloud. Reads dbt `meta` directly from the typed `Manifest.Model.meta` and `Column.meta` dicts (already loaded by `signalforge.manifest`). No `dbt-core` runtime call. +- **#5 — Explainable diffs.** The audit log is the explainable-diff vehicle for *what data the model saw.* Every record carries: mode, columns sent, columns redacted (and why — pattern match? per-column opt-out? model-level opt-out?), row count if `sample` mode. The prune layer (#6) gets a separate audit; this one covers the LLM input boundary. +- **Roadmap anchor.** v0.1 = single-model draft + warehouse prune. The safety layer ships **before** the LLM-drafting pipeline (#5) so the contract is locked when #5 lands. + +### Section 4.5 of the design doc — recommendations already on the table + +The clauditor-repo design doc's PII section (`docs/research/dbt-claude-technical-surface.md` §4.5) lays out the exact posture the ticket asks for, plus several knobs we should explicitly accept or reject for v0.1: + +- **Mode hierarchy: schema-only → aggregate-only → sample** (matches the ticket). +- **`tags: ["pii"]` and `meta.contains_pii: true`** are existing dbt conventions worth honouring as implicit opt-outs (Q3 below). +- **Server-side redaction (Snowflake `AI_REDACT`)** — out of scope; client-side regex only for v0.1. +- **Explicit allowlist (`models/marts/safe_for_review/`)** — interesting, but not in the ticket. Out of scope for v0.1; revisit if users ask. +- **Token budget framing** — "100 rows × 20 cols × ~5 tokens = 10k tokens" — informs the default sample size. Default to 100 rows, configurable. + +### Out of scope (explicit) + +- **The LLM client itself** — issue #5. This ticket builds the contract #5 will consume, not the client. +- **The CLI flag** — issue #9. This ticket locks the policy shape; #9 wires `argparse`. +- **Server-side redaction** (Snowflake `AI_REDACT`, BQ Cortex) — v0.2+; client-side regex only for v0.1. +- **Path-allowlist policy** (`models/marts/safe_for_review/` style) — design-doc inspiration but not in ticket; out of scope. +- **Statistical aggregates beyond `ColumnStats`** (top-N, percentiles, value-distribution histograms) — `aggregate-only` mode delivers what `column_stats` already produces. Histograms land in v0.2 if users ask. +- **Differential-privacy noise** on aggregates — out of scope; far beyond ticket. +- **PII detection on actual *values*** (regex on row contents, ML classifiers) — column-name pattern matching only for v0.1. The design doc explicitly notes this trade-off. +- **Multi-language redaction** (Unicode lookalikes for `email`, etc.) — pattern matching is ASCII-pragmatic for v0.1. +- **The prune-step audit log** — separate from the LLM-call audit; lives with the prune ticket (#6). + +### Phase 1 housekeeping defaults (set unless flagged in Phase 2/3) + +- New subpackage layout TBD by Q1. +- New `signalforge.yml` config file TBD by Q2; default location is project root. +- Per-column opt-out via `meta.signalforge.sample: false` (Q3 expands the alias set). +- Audit log emits via `logging.getLogger("signalforge.safety")` AND a JSONL file persisted per run (Q5). +- Default sample size: 100 rows (matches design doc's token-budget framing). Configurable. +- Default redaction patterns: `*_email`, `*_phone`, `*_ssn` (Q7 decides override semantics). +- No CLI code in this ticket. No LLM client code. Only the contract. +- Unit tests use `FakeAdapter` (a tiny in-memory `WarehouseAdapter` impl) and `FakeLLMClient` with `expect_*` helpers. + +### Scoping questions (Phase 1 — awaiting answers) + +**Q1. Subpackage placement** + +A) New `signalforge.safety` subpackage (sibling to `manifest` / `warehouse`). +B) Inside `signalforge.warehouse` as `safety.py` (data-access policy lives next to data access). +C) New top-level `signalforge.policy` (broader name; room for non-PII policy later — quotas, caching). + +*Lean: A. Mirrors the precedent of one-subpackage-per-stage (`manifest`, `warehouse`); the safety layer sits between them and #5; co-locating in `warehouse` blurs the warehouse-agnostic seam.* + +**Q2. Config-file shape (no config file exists today)** + +A) `signalforge.yml` at project root. YAML, parsed by PyYAML (already a dep). Loaded via new `signalforge.config` module. +B) Piggyback dbt's `dbt_project.yml` — read keys from its `vars:` block. No new file. +C) Defer config-file decision entirely to CLI ticket (#9). Ship typed `SafetyPolicy` ADT only. + +*Lean: A. #9 needs a config file regardless; ship the loader now. Piggybacking dbt is cute but couples us to dbt's loader semantics for non-dbt config.* + +**Q3. Per-column opt-out granularity** + +A) Read column-level `meta.signalforge.sample: false` from `Column.meta` only. +B) Also read MODEL-level `meta.signalforge.sample: false` ("schema-only for this model regardless of mode"). +C) Also honour `tags: ["pii"]` and `meta.contains_pii: true` (existing dbt conventions per design doc) as implicit opt-outs. +D) All of A + B + C. + +*Lean: D. Cheap to implement, generous to users who already use existing dbt conventions; audit log records *which* signal triggered the redaction so behaviour stays explainable.* + +**Q4. Redaction strategy in `sample` mode** + +A) Drop redacted columns entirely from the LLM payload (column never appears). +B) Replace values with `""` constant (column appears, values masked). +C) Per-type placeholder (e.g. `""`) — column appears, values masked, type hint preserved. +D) Hash values (deterministic SHA-256 truncated) — uniqueness signal preserved without content. + +*Lean: B. Drops violate "the LLM should know the column exists" — schema-yml drafting needs to mention the column. Hashing leaks cardinality which the LLM can't usefully act on for redacted columns. Constant `""` is the simplest safe default; ops doc notes the trade-off.* + +**Q5. Audit log destination + format** + +A) Standard `logging.getLogger("signalforge.safety").info(...)` line; structured `extra={...}` per call. No file. +B) JSONL file at a configured path (default `.signalforge/audit.jsonl`); one record per LLM call. Logger is secondary. +C) Both: human-readable WARNING/INFO log line AND structured JSONL record persisted next to the diff output. + +*Lean: C. The "explainable diffs" commitment requires a durable, machine-readable record. The log line is for the developer in the moment; the JSONL file is for the reviewer / compliance auditor / regression test.* + +**Q6. CLI flag wiring scope (CLI doesn't exist yet — #9)** + +A) Lock the `SafetyPolicy` API + config-file shape now; #9 wires `--mode` to load the policy. **No CLI code in this ticket.** +B) Build a thin `signalforge.config` loader with both `from_file()` and `from_args(argv)` entrypoints, so #9 just calls `load_config(argv)`. Some plumbing in this ticket. +C) Defer entirely to #9; ship only the redactor + policy primitives. + +*Lean: A. Mirrors how #3 handled "no CLI yet" — locks the contract, leaves CLI plumbing to #9. Building `from_args` here without a real CLI is design-on-spec.* + +**Q7. Default redaction patterns + override semantics** + +A) Hard-coded built-ins `*_email`, `*_phone`, `*_ssn`; config can `extend` only (never replace). +B) Hard-coded built-ins as defaults; config supports both `extend: [...]` and `replace: [...]`. +C) No built-ins; config MUST list patterns explicitly. Empty list allowed but emits WARNING. + +*Lean: B. Gives users escape hatch for false positives (`my_phone_number_format` column) without forcing every project to re-list the obvious patterns. `replace` is rare but should exist for users with strict allowlist regimes.* + +**Q8. Aggregate-only mode implementation** + +A) Reuse `WarehouseAdapter.column_stats` directly; aggregate-only mode delivers `ColumnStats` to the LLM. +B) Add a convenience `aggregate_columns(table, columns) -> dict[str, ColumnStats]` on the safety layer that wraps `column_stats` AND respects per-column opt-out. +C) Aggregate-only also adds value-distribution buckets (top-N, percentiles) beyond `column_stats`. Bigger scope. + +*Lean: B. (A) leaks per-column-opt-out enforcement to every caller. (C) is v0.2 scope creep — the design doc's "value distribution buckets" suggestion isn't in the ticket.* + +**Q9. LLM-call seam (LLM client doesn't exist yet — #5)** + +A) Define a typed `LLMRequest` ADT in this ticket — `LLMRequest(model_unique_id, mode, columns_sent, redacted_columns, sampled_rows, ...)` — that the safety layer produces. #5 consumes this shape; the actual `client.complete(request)` call lands in #5. +B) Define only an `audit_record(...)` helper that emits the audit log given the inputs. #5 builds its own request shape and calls `audit_record` at its boundary. +C) Build a stub `LLMClient` Protocol with one `complete(request) -> str` method, plus an audit shim that wraps every call. #5 implements the Protocol. + +*Lean: A. (B) leaves the audit-line shape unfixed and lets #5 forget to call it. (C) over-specifies #5's design (e.g. forces sync; #5 may want streaming / batching). The typed `LLMRequest` is small, exercised by tests in this ticket, and #5 inherits a stable contract.* + +### Scoping decisions (Phase 1 — locked 2026-04-28, "use defaults") + +- **DEC-001 — Subpackage placement: `signalforge.safety`** (Q1=A). Sibling to `manifest` and `warehouse`. Public modules: `signalforge.safety.{__init__, policy, redact, audit, request, errors, config}`. *Why:* mirrors one-subpackage-per-stage precedent; co-locating in `warehouse` blurs the warehouse-agnostic seam (Architectural Commitment #3); `policy` as top-level is too broad (room for non-PII concerns we don't yet have). *How to apply:* `__init__.py` re-exports the public surface (`SafetyPolicy`, `SamplingMode`, `LLMRequest`, `RedactionRecord`, `AuditEvent`, errors); `_`-prefixed helpers stay reachable via dotted import only (DEC-017 from #2). +- **DEC-002 — Config file: `signalforge.yml` at project root + new `signalforge.safety.config` module** (Q2=A). PyYAML `safe_load` only. Resolution order: explicit path arg → `/signalforge.yml` → defaults (no error if absent). *Why:* CLI ticket #9 will need a config file regardless; piggybacking `dbt_project.yml`'s `vars:` couples non-dbt config to dbt's loader; PyYAML is already a runtime dep from #3 so cost is zero. *How to apply:* `load_safety_config(project_dir, path=None) -> SafetyPolicy`; symlink-hardened path canonicalisation copied (not imported) from `signalforge.warehouse._path_safety` per the duplication precedent in `warehouse-adapters.md`. +- **DEC-003 — Per-column opt-out: column meta + model meta + `tags:[pii]` + `meta.contains_pii`** (Q3=D). Triggering signals (any one redacts the column): (a) `Column.meta.signalforge.sample == false`; (b) `Model.meta.signalforge.sample == false` (forces schema-only for the entire model regardless of policy mode); (c) `"pii" in Column.tags` or `"pii" in Model.tags`; (d) `Column.meta.contains_pii == true` or `Model.meta.contains_pii == true`. *Why:* honours existing dbt conventions per design doc §4.5; cheap to implement; audit log records *which* signal fired so behaviour stays explainable (Commitment #5). *How to apply:* a `_classify_column(column, model, policy) -> RedactionDecision` pure function returns `(redact: bool, reason: Literal["column_meta_optout", "model_meta_optout", "tag_pii", "meta_contains_pii", "pattern_match", None])`; tests parametrise across the matrix. +- **DEC-004 — Redaction renders `""` constant in `sample` mode** (Q4=B). Column appears in the LLM payload (so schema.yml drafting can mention it) but every value is the literal string `""`. No type-tagged variant for v0.1; no hashing. *Why:* dropping the column violates "the LLM should know the column exists for schema drafting"; type-tagged placeholders are scope creep with no demonstrated value; hashing leaks cardinality but the LLM can't usefully act on cardinality of a redacted column. *How to apply:* `_REDACTED_VALUE: Final = ""` module constant; `redact_rows(rows, redacted_columns) -> list[dict]` returns new dicts with values replaced (never mutates input). Ops doc records the trade-off so future debate references it. +- **DEC-005 — Audit log: BOTH human-readable logger AND structured JSONL file** (Q5=C). Logger: `signalforge.safety` at `INFO` level emits a one-line summary per LLM call (`"LLM call for {unique_id}: mode={mode}, columns_sent={n}, redacted={m}"`). File: JSONL at `/.signalforge/audit.jsonl` (path configurable); one record per LLM call, schema-stable, machine-readable. *Why:* the developer-in-the-moment uses the log line; the reviewer / compliance auditor / regression test uses the JSONL. Single channel forces a trade-off neither audience tolerates. *How to apply:* `AuditEvent` is a frozen Pydantic v2 model with `timestamp`, `model_unique_id`, `mode`, `columns_sent: list[str]`, `redactions: list[RedactionRecord]`, `row_count: int | None`; `audit.write(event)` appends to JSONL and emits the log line. Atomic-append via `O_APPEND` open mode (no locking — JSONL appends are atomic on POSIX up to PIPE_BUF). +- **DEC-006 — CLI scope: lock policy + config shape; no CLI code in this ticket** (Q6=A). `signalforge.safety.config.load_safety_config(project_dir, path=None) -> SafetyPolicy` is the loader; #9's CLI will pass `--mode` to override `SafetyPolicy.mode` after loading from file. No `from_args` plumbing here. *Why:* mirrors how #3 handled "no CLI yet"; building `from_args` without a real CLI is design-on-spec; #9 has freedom to choose `argparse` / `click` / `typer`. *How to apply:* add a "Used by #9" comment on `load_safety_config` so the next implementer knows the seam is intentional. +- **DEC-007 — Built-in redaction patterns + `extend`/`replace` config semantics** (Q7=B). Built-ins: `*_email`, `email`, `*_phone`, `phone`, `*_ssn`, `ssn` (six patterns; covers both prefixed `customer_email` and bare `email`). Config schema: `redact: { extend: [...], replace: [...] }`; `extend` and `replace` are mutually exclusive — `extend` appends to built-ins, `replace` substitutes them entirely. Empty `replace: []` is allowed (disables all redaction) but emits a `WARNING` log line on policy load. *Why:* gives users escape from false positives without forcing every project to re-list defaults; `replace` is rare but exists for strict-allowlist regimes; the warning on empty-replace is signal-not-noise (it fires only when someone explicitly opts out of all PII redaction). *How to apply:* patterns matched via `fnmatch.fnmatchcase` (case-sensitive glob) against column names; `SafetyPolicy.redact_patterns: tuple[str, ...]` is the resolved final list (frozen at construction); a `policy.matches_redaction_pattern(column_name) -> bool` method. +- **DEC-008 — Aggregate-only mode: `aggregate_columns(adapter, model, columns)` wrapper enforces opt-out** (Q8=B). Public: `aggregate_columns(adapter: WarehouseAdapter, model: Model, columns: list[str], policy: SafetyPolicy) -> dict[str, ColumnStats | None]`. Returns `None` for columns that the policy redacts (per-column opt-out signals). Internally it calls `adapter.column_stats(table, column)` for each non-redacted column. *Why:* (A) leaks per-column-opt-out enforcement to every caller and risks noise in the audit log; (C)'s top-N / percentiles aren't in the ticket and would require new adapter methods. *How to apply:* the `dict` value is `None` when redacted (so the caller knows the column was acknowledged but skipped); the audit log captures both the columns sent (`ColumnStats` non-None) and the redacted column names with their reason. +- **DEC-009 — LLM-call seam: typed `LLMRequest` ADT** (Q9=A). Frozen Pydantic v2 model: `LLMRequest(model_unique_id: str, mode: SamplingMode, columns_sent: list[str], redactions: list[RedactionRecord], sampled_rows: list[dict] | None, aggregates: dict[str, ColumnStats | None] | None, schema: list[tuple[str, str]])`. The safety layer produces the request; #5 consumes it (`client.complete(request) -> str`). The `audit.write(event)` call is invoked by the **request-builder** (not by #5), so #5 cannot accidentally skip the audit. *Why:* (B) leaves the audit-line shape unfixed and trusts #5 to remember; (C) over-specifies #5 (forces sync, may want streaming/batching); typed `LLMRequest` is small, exercised by this ticket's tests, and gives #5 a stable contract. *How to apply:* `build_llm_request(model, adapter, policy) -> LLMRequest` is the single entry point; it calls `audit.write` before returning. #5 receives the request, calls its LLM client, and never touches the safety layer's internals. + +--- + +## Architecture Review + +Reviewed 2026-04-28 by seven parallel subagents (privacy/compliance, security, performance, data model, API design, observability, testing strategy) against the locked Phase 1 shape (DEC-001 … DEC-009). Result: **10 unique blockers, 16 concerns** spanning fail-closed semantics, data-model gaps, and API contract precision. Privacy review surfaced the load-bearing finding (B1 below) — column *names* leak PII even when no values are sent. + +### Findings table + +| Area | Rating | Notes | +| --- | --- | --- | +| Privacy — `schema-only` mode leaks PII via column NAMES | **blocker** | Default mode sends column names to LLM. A column named `customer_ssn` or `john_smith_email` leaks PII even when no values are sampled. Resolution: in schema-only AND aggregate-only modes, redact column **names** that match patterns/tags/meta — replace with `""` (drops field) or `""` (stable hash for the LLM to reference). | +| Privacy/Obs — Audit write failure semantics undefined | **blocker** | If `.signalforge/audit.jsonl` append fails (disk full, perms, missing dir), DEC-005 doesn't say what happens. Must be fail-closed: any audit-write exception aborts `build_llm_request`, raising `AuditWriteError`. The LLM call **never** proceeds without a durable audit record. | +| Privacy — Default-mode regression not enforced at every layer | **blocker** | If a future bug makes `load_safety_config` return `mode=sample` instead of `schema-only`, every team's PII silently leaves the warehouse. Need (a) `SafetyPolicy.mode: SamplingMode = SamplingMode.SCHEMA_ONLY` default at the field level, (b) explicit regression test for "no config → schema-only", (c) explicit regression test for "no config + adapter calls inspected → zero warehouse calls in default policy". | +| Security — `audit_path` path-traversal | **blocker** | `audit_path` is config-supplied. Setting `audit_path: /etc/passwd` or `audit_path: ../../../escape.jsonl` is currently unconstrained. Must canonicalise via the `_path_safety` helper, reject `..` segments, and require the resolved path stay inside `project_dir`. | +| Data model — `AuditEvent` missing reproducibility fields | **blocker** | DEC-005's `AuditEvent` lacks `signalforge_version: str`, `policy_hash: str`, and `audit_schema_version: int`. Without these: (a) v0.2 cannot evolve the audit schema without breaking consumers, (b) reviewers cannot verify "all records in this run came from the same policy," (c) #5 cannot deterministically reproduce a redaction decision on re-run. All three are mandatory. | +| Data model / API — `SafetyPolicy` lacks `extra="forbid"` | **blocker** | `signalforge.yml` is user-authored. A typo like `redacts:` (vs `redact:`) currently silently no-ops with `extra="ignore"` — exactly the kind of fail-open bug this ticket exists to prevent. Override the manifest-readers default and use `extra="forbid"` on every config-shaped model (`SafetyPolicy`, the inner `redact:` block). Reader-shaped models (`AuditEvent` deserialised from old JSONL records) keep `extra="ignore"`. | +| API — `load_safety_config` error semantics ambiguous | **blocker** | DEC-002 says "no error if absent" but doesn't disambiguate four cases: (a) explicit `path=` arg points at missing file → raise `ConfigNotFoundError`; (b) implicit project-dir lookup misses → silent fallback to defaults; (c) file exists but is empty → silent fallback to defaults; (d) file exists but YAML is malformed or schema-invalid → raise `InvalidConfigError` / `InvalidSamplingModeError`. Lock the contract. | +| API — `extend` / `replace` resolution mechanism undefined | **blocker** | DEC-007 specifies the YAML shape `redact: { extend: [...], replace: [...] }` but doesn't say how Pydantic deserialises it into `redact_patterns: tuple[str, ...]`. Use a Pydantic v2 `@model_validator(mode="before")` that pops `redact`, resolves `extend` (append to built-ins) or `replace` (substitute), and assigns `redact_patterns`. Mutual exclusion is enforced in the same validator. | +| API — `policy.with_mode()` factory missing | **blocker** | `SafetyPolicy` is frozen, so `--mode` from #9 cannot mutate it. DEC-006 says "wires `--mode` to load the policy" but provides no method. Add `policy.with_mode(mode: SamplingMode) -> SafetyPolicy` using `self.model_copy(update={"mode": mode})`. Document as the canonical override path; #9 has no other route. | +| Obs — `docs/safety-ops.md` not committed | **blocker** | `docs/manifest-loader-ops.md` and `docs/warehouse-adapter-ops.md` are precedent — every public-API subpackage gets an ops guide. Must commit to the doc in this ticket: mode semantics, redaction-pattern syntax, audit JSONL schema with `audit_schema_version`, opt-out mechanisms (column meta, model meta, tags, `meta.contains_pii`), debugging, the "Data safety" README cross-link. | +| Privacy — Single-entry audit not type-enforced | concern | DEC-009 says `build_llm_request` is the single entry point that calls `audit.write` internally. Convention only; nothing prevents future contributors from constructing `LLMRequest` directly. Mitigation: add a module-level test `test_llm_request_construction_requires_audit` that scans for direct `LLMRequest(...)` calls outside `request.py` (lightweight AST check), or document the convention in the `LLMRequest` docstring with `# private constructor` discipline. Lean: docstring + AST test. | +| Privacy — `tags`/`meta.contains_pii` semantics | concern | What if `tags: [PII]` (uppercase)? `meta.contains_pii: "yes"` (string)? `meta.contains_pii: 1` (int)? Lean: tags case-insensitive (normalise to lowercase on read; emit DEBUG when normalisation kicks in); `contains_pii` accepts any truthy value (DEBUG-log when non-bool). Document precedence: column-level beats model-level in case of conflict. | +| Privacy/Security — Pattern matching case-insensitive | concern | `fnmatch.fnmatchcase` is case-sensitive; `*_email` misses `Customer_Email` and `EMAIL`. Lean: lowercase both column name and pattern at match time (`fnmatch.fnmatchcase(name.lower(), pat.lower())`). Built-in patterns become `*email`, `email`, `*phone`, `phone`, `*ssn`, `ssn` (still case-insensitive in matching). Add a "suspicious unmatched columns" WARNING heuristic — flag columns whose name contains substrings like `email`/`phone`/`ssn`/`password`/`token`/`secret` but didn't match any pattern. | +| Privacy — Audit JSONL contains plaintext column names | concern | A column literally named `customer_ssn` is a PII metadata leak even when its values are redacted. Two postures: (A) document the audit JSONL itself as sensitive (Gitignore, treat at-rest as PII); (B) hash the column name in `RedactionRecord` (loses debuggability). Lean: A. The audit log's purpose is human review for compliance — hashing defeats it. Document loudly in `safety-ops.md`. | +| Privacy — `sample` mode guard rails | concern | No env-var override for `mode` (config + CLI flag only). On `SafetyPolicy` load, emit a single WARNING when `mode == SAMPLE`: `"Sample mode enabled — raw row data will be sent to the LLM. Verify column tags/meta opt-outs."`. Persist a corresponding `policy_flags: ["sample_mode_enabled"]` field on every `AuditEvent` from that run so reviewers can scan. | +| Privacy — `LLMRequest.sampled_rows` deep immutability | concern | Pydantic `frozen=True` blocks attribute reassignment but not list-mutation (`request.sampled_rows.append(...)`). Switch the type to `tuple[dict[str, Any], ...] | None` (and `tuple[tuple[str, str], ...]` for `schema`) so #5 cannot accidentally mutate the audit-pinned payload. | +| Security — Pattern-injection rejection | concern | An empty pattern `""` matches everything via `fnmatch`; `*` matches all column names, defeating the redactor by appearing to over-redact (which an LLM-context-builder may then "helpfully" un-redact). Validate each user-supplied pattern at policy-load time: reject empty strings; reject patterns equal to `*` or `?`. Raise `InvalidPatternError(value, reason)`. | +| Security — Atomic-append size bound | concern | POSIX `O_APPEND` is atomic only up to PIPE_BUF (4096 bytes Linux, 512 macOS). A typical `AuditEvent` is ~200–500 bytes; pathological cases (huge column lists, many redactions) could exceed. Add a runtime assertion in `audit.write`: `if len(line) > 4000: raise AuditRecordTooLargeError(size, limit)`. Documents the constraint and catches feature creep. | +| Security — ANSI/log-injection in logger line | concern | `_LOGGER.info(f"LLM call for {model_unique_id}: ...")` with a model-id containing ANSI escapes (`\x1b[31mFAKE\x1b[0m`) injects into log viewers. JSON encoding handles this for the JSONL; the logger line must use lazy-format `_LOGGER.info("...: %s", json.dumps(...))` or repr-quote user-controlled values. | +| Performance — Audit JSONL unbounded growth | concern | One record per LLM call, no rotation. After 10 000 runs the file is megabytes. Lean: document user-side rotation in `safety-ops.md` (logrotate, manual archive); do not implement rotation in v0.1. | +| Data model — `SamplingMode` as `StrEnum` + case-insensitive load | concern | DEC-001 implies a string-typed mode. Use `class SamplingMode(StrEnum): SCHEMA_ONLY = "schema-only"; ...` (Python 3.11+, project minimum is 3.10 — verify, or fall back to a `str, Enum` mixin on 3.10). Add a `@field_validator("mode", mode="before")` on `SafetyPolicy` that normalises `"Schema-Only"` / `"schema_only"` / `"SCHEMA-ONLY"` → `SamplingMode.SCHEMA_ONLY`; raises `InvalidSamplingModeError` on unknown. | +| Data model — `RedactionRecord.reason` as `Literal[...]` | concern | DEC-003 lists the seven reasons informally. Encode as `Literal["column_meta_optout", "model_meta_optout", "tag_pii_column", "tag_pii_model", "meta_contains_pii_column", "meta_contains_pii_model", "pattern_match"]` so audit-log consumers can pattern-match exhaustively. | +| Data model — `signalforge.yml` top-level namespace | concern | Use `{ safety: { mode: ..., redact: ..., ... } }` rather than flat-at-top. Reserves room for future stages (`llm:`, `prune:`, `grade:`) without a v2-config migration. The config loader extracts the `safety:` key. | +| Obs — `policy_flags` and empty-redaction persistence | concern | If `redact: { replace: [] }` disables all redaction, the WARNING fires once on load — but each `AuditEvent` in that run should also carry `policy_flags: ["redaction_disabled"]` so a reviewer scanning the JSONL doesn't conclude "no redactions = no PII columns" (could be "all PII columns intentionally un-redacted"). | +| API — Error hierarchy enumeration | concern | Plan implies but doesn't list typed errors. Lock the set: `SafetyError` (base), `ConfigNotFoundError`, `InvalidConfigError` (parent), `InvalidSamplingModeError`, `InvalidPatternError`, `ColumnNotInModelError`, `AuditWriteError`, `AuditRecordTooLargeError`, `PolicyValidationError`. Each subclasses `SafetyError`, ships a class-level `default_remediation`, and renders user-supplied strings via `repr()` (DEC-022 from #3). Total: ~9 typed subclasses + 1 base. | +| Testing — Drift-detector fixture for `AuditEvent` | concern | Per `testing-signal.md`, production `AuditEvent` uses `extra="ignore"`; pair it with a one-off `StrictAuditEvent` (`extra="forbid"`) test against a committed JSONL fixture (`tests/fixtures/safety/audit_events_sample.jsonl`). Adding a field to production without updating the fixture or the strict model breaks the test loudly. | + +### Blockers (must resolve in Phase 3) + +1. **B1** — Schema-only & aggregate-only modes redact column NAMES (not just values) when names match patterns/tags/meta. Decide: drop column entirely vs. stable-hash placeholder vs. `""` literal. +2. **B2** — Fail-closed audit semantics: any `audit.write` failure aborts `build_llm_request` with `AuditWriteError`. LLM call never proceeds without a durable audit record. +3. **B3** — Default-mode regression: `SafetyPolicy.mode: SamplingMode = SamplingMode.SCHEMA_ONLY` default at field level + dedicated regression tests at policy, config, and request layers. +4. **B4** — `audit_path` security: canonicalise via copied `_path_safety`, reject `..` segments, require resolved path inside `project_dir`. +5. **B5** — `AuditEvent` adds `signalforge_version: str`, `policy_hash: str`, `audit_schema_version: int = 1` as mandatory fields. +6. **B6** — `SafetyPolicy` (and inner config-shaped models) use `extra="forbid"`. `AuditEvent` (read-back path) keeps `extra="ignore"` + drift-detector test. +7. **B7** — `load_safety_config` error contract: explicit-path miss → `ConfigNotFoundError`; implicit-path miss → silent defaults; empty file → silent defaults; malformed YAML → `InvalidConfigError`; schema-invalid → `InvalidSamplingModeError` / `InvalidPatternError`. +8. **B8** — `@model_validator(mode="before")` resolves `redact: {extend, replace}` into `redact_patterns: tuple[str, ...]`. Mutual exclusion enforced; empty `replace: []` allowed but emits WARNING. +9. **B9** — `policy.with_mode(mode) -> SafetyPolicy` via `model_copy(update={"mode": mode})`. Documented as #9's CLI override seam. +10. **B10** — Commit to `docs/safety-ops.md` (parallel to manifest-/warehouse-ops); cross-linked from README "Data safety" section. + +### Concerns to resolve in Phase 3 + +C1 — Single-entry audit enforced via `LLMRequest` docstring discipline + an AST-scan test that catches direct construction outside `request.py`. +C2 — `tags` case-insensitive normalisation; `meta.contains_pii` truthy coercion; column-level beats model-level on conflict; document precedence. +C3 — Pattern matching case-insensitive (lowercase both sides); built-ins become `*email`/`email`/`*phone`/`phone`/`*ssn`/`ssn`; "suspicious unmatched column" WARNING heuristic. +C4 — Audit JSONL contains plaintext column names; documented as sensitive in `safety-ops.md` rather than hashed. +C5 — No env-var override for `mode`; WARNING on `mode == SAMPLE` policy load; `policy_flags: ["sample_mode_enabled"]` on every `AuditEvent` from that run. +C6 — `LLMRequest.sampled_rows: tuple[dict[str, Any], ...] | None`; `schema: tuple[tuple[str, str], ...]`; deep immutability. +C7 — Pattern-injection rejection: empty / `*` / `?` patterns rejected with `InvalidPatternError`. +C8 — `audit.write` asserts `len(line) <= 4000` (PIPE_BUF margin); raises `AuditRecordTooLargeError` otherwise. +C9 — Logger lines use lazy-format `_LOGGER.info("...: %s", json.dumps(value))` for any user-controlled string; no f-string interpolation of user input. +C10 — Audit JSONL rotation documented as user responsibility in `safety-ops.md`; no rotation logic in v0.1. +C11 — `SamplingMode(StrEnum)`; `@field_validator("mode", mode="before")` accepts `"schema-only"` / `"schema_only"` / `"SCHEMA-ONLY"` etc. +C12 — `RedactionRecord.reason` is `Literal[...]` of seven explicit values. +C13 — Top-level `signalforge.yml` namespace: `{ safety: { ... } }`; loader extracts the `safety:` key; other top-level keys reserved for future stages. +C14 — `policy_flags: list[str]` on `AuditEvent`; populated with `"redaction_disabled"`, `"sample_mode_enabled"` etc. as policy state warrants. +C15 — Error hierarchy: `SafetyError` base + 9 subclasses; each has class-level `default_remediation` and `repr()`-quotes user input (DEC-022). +C16 — Drift-detector test: one-off `StrictAuditEvent(extra="forbid")` validates committed `tests/fixtures/safety/audit_events_sample.jsonl` fixture. + +## Refinement Log + +### Phase 3 decisions (resolved 2026-04-28, "all defaults" — every blocker and every concern) + +Seventeen decisions consolidating the ten blockers and sixteen concerns from the Architecture Review. + +- **DEC-010 — Column-name redaction in `schema-only` and `aggregate-only` modes via stable hash** (resolves B1). When a column is classified as redacted (any of the seven `RedactionRecord.reason` signals fires), the column's NAME is replaced in the LLM-bound payload with `f"col_{hashlib.blake2b(name.encode(), digest_size=4).hexdigest()}"` (e.g. `customer_ssn` → `col_a3f29c61`). Schema-only mode sends `[(hashed_name, type)]`; aggregate-only mode keys `aggregates` dict on the hashed name; sample mode redacts both NAME (hashed) AND VALUES (`""` constant). The `(real_name, hashed_name)` mapping is recorded in `RedactionRecord.hashed_name`, persisted to the JSONL audit log, and **not** sent to the LLM. *Why:* drops defeat the point of schema-only (LLM can't draft a `schema.yml` entry for a column it doesn't know exists); bare `""` collides for every redacted column in the same model; blake2b-4 (8 hex chars) gives stable, collision-resistant identifiers and lets downstream tooling (in #5 / #6) map back to real names via the audit log. *How to apply:* `signalforge.safety.redact.hash_column_name(name) -> str` is the single helper; tests assert determinism (same name → same hash) and stability across runs. +- **DEC-011 — Fail-closed audit semantics** (resolves B2). Any exception raised inside `audit.write(event)` (`OSError`, `PermissionError`, encoding failure, size-cap breach) propagates as `AuditWriteError` from `build_llm_request`. The function never returns an `LLMRequest` whose audit record didn't durably hit disk. *Why:* an unaudited LLM call is, by definition, PII leaving the warehouse without a receipt — the entire point of this ticket is to prevent that. *How to apply:* `audit.write` opens the file with `O_APPEND | O_CREAT`, writes one `json.dumps(event.model_dump(mode="json")) + "\n"`, calls `fsync()` before close, and catches *no* exceptions internally. `build_llm_request` calls `audit.write(event)` AFTER constructing the request but BEFORE returning it; on exception, the partial request is dropped. Tests inject `IOError` via a `chmod 000` parent dir under `tmp_path` and assert `AuditWriteError`. +- **DEC-012 — Default-mode regression enforced at field + three layers** (resolves B3). `class SafetyPolicy(BaseModel): mode: SamplingMode = SamplingMode.SCHEMA_ONLY`. Three dedicated regression tests: (a) `test_safety_policy_no_args_is_schema_only` — `SafetyPolicy().mode is SamplingMode.SCHEMA_ONLY`. (b) `test_load_safety_config_no_file_is_schema_only` — `load_safety_config(empty_tmp_path).mode is SamplingMode.SCHEMA_ONLY`. (c) `test_build_llm_request_default_policy_zero_warehouse_calls` — `FakeAdapter` records calls; default policy + `build_llm_request` issues zero `column_stats` / `sample_rows` calls. *Why:* a future bug that flips the default to `sample` silently leaks PII for every team that hasn't authored a `signalforge.yml`. Three layers of defence make a single regression catastrophic-test-fail-loud, not silent-data-leak. *How to apply:* the three tests live in `tests/safety/test_default_mode_regression.py` (a single dedicated file makes the regression cluster greppable for future-you). +- **DEC-013 — `audit_path` path-traversal hardening** (resolves B4). `audit_path: Path` defaults to `/.signalforge/audit.jsonl`. Override via `signalforge.yml`'s `safety.audit_path` is supported but constrained: (a) reject any path containing `..` segments at policy-load time (`InvalidConfigError` with remediation); (b) canonicalise the path via `_path_safety.canonicalise_path` (copied from `signalforge.warehouse._path_safety`, per `warehouse-adapters.md`'s duplication precedent); (c) require the canonicalised path to be within `project_dir` (`is_relative_to`). Symlinks are followed to their target before the containment check. *Why:* `audit_path: /etc/passwd` and `audit_path: ../../escape.jsonl` are both currently unconstrained; both must fail loudly. *How to apply:* the helper lives in `signalforge.safety._path_safety`; the three traps from `manifest-readers.md` (resolve before containment, catch `RuntimeError` on cycles, gate the default path through the same helper) all apply. +- **DEC-014 — `AuditEvent` reproducibility fields** (resolves B5). Mandatory fields beyond DEC-005's baseline: `signalforge_version: str` (from `signalforge.__version__` at write time); `policy_hash: str` (SHA-256 of `SafetyPolicy.model_dump_json()` with sorted keys, hex-encoded, truncated to 16 chars); `audit_schema_version: int = 1` (frozen at the constant — bump in v0.2). *Why:* without `signalforge_version` an audit record can't be reproduced under the same code; without `policy_hash` reviewers can't verify "all records in this run came from the same policy"; without `audit_schema_version` v0.2 can't add fields without breaking JSONL consumers. *How to apply:* `AuditEvent` adds three frozen fields; a `_compute_policy_hash(policy: SafetyPolicy) -> str` helper lives in `signalforge.safety.policy`; tests assert that two policies with identical fields produce the same hash and two semantically-different policies produce different hashes. +- **DEC-015 — `extra="forbid"` on config-shaped models; `extra="ignore"` on read-back models** (resolves B6). `SafetyPolicy`, the inner `_RedactConfig` (the `redact: { extend, replace }` block), and the top-level `_SafetyConfigFile` (the deserialised `signalforge.yml`) all use `ConfigDict(frozen=True, extra="forbid", populate_by_name=True)`. `AuditEvent` (which is *deserialised back* from old JSONL files for drift detection and external audit-log tooling) keeps `ConfigDict(frozen=True, extra="ignore", populate_by_name=True)` and is paired with a `StrictAuditEvent(extra="forbid")` drift-detector test (DEC-026). *Why:* a typo in `signalforge.yml` (`redacts:` vs `redact:`) silently no-ops with `extra="ignore"` — exactly the failure mode this ticket exists to prevent. The read-back path needs forward-compat. *How to apply:* explicit `model_config` declaration on each class; tests parametrise typo cases and assert `ValidationError`. +- **DEC-016 — `load_safety_config` error contract** (resolves B7). Resolution: (1) explicit `path=` arg → file MUST exist; missing → `ConfigNotFoundError(path)`. (2) implicit `/signalforge.yml` → missing is fine; fall through to defaults. (3) Empty file (zero bytes or whitespace-only) → fall through to defaults; emit DEBUG `"signalforge.yml is empty; using defaults"`. (4) File parses but `safe_load` returns non-dict → `InvalidConfigError("expected mapping at top level")`. (5) File parses but the `safety:` key is absent → fall through to defaults (other top-level keys reserved for future stages, per DEC-025). (6) Schema-invalid contents → typed errors per validator (`InvalidSamplingModeError`, `InvalidPatternError`, `PolicyValidationError`). *Why:* explicit-path failures must fail loud (user said they wanted that file); implicit failures must fall through silently to defaults (otherwise every project must author `signalforge.yml` to use the tool); empty files behave like missing files (least-surprise). *How to apply:* `tests/safety/test_config.py` covers all six branches with parametrised fixtures. +- **DEC-017 — `@model_validator(mode="before")` resolves `extend` / `replace`** (resolves B8). On `SafetyPolicy`, a `@model_validator(mode="before")` named `_resolve_redact_patterns` consumes the inbound `redact: {extend?, replace?}` dict, resolves it into `redact_patterns: tuple[str, ...]`, and removes the original `redact` key. Mutual exclusion: `extend` and `replace` simultaneously present → `ValidationError`. Empty `replace: []` → resolved to empty tuple, but emit module-level WARNING `"signalforge.yml: redact.replace=[] disables all redaction patterns"` once at policy-load time. Built-in defaults (after DEC-020 case-insensitivity): `("*email", "email", "*phone", "phone", "*ssn", "ssn")`. *Why:* Pydantic v2's `@model_validator(mode="before")` is the idiomatic place to translate user-facing config shape into typed-field shape; it runs before field validation so downstream invariants (DEC-024 case-insensitive normalisation) see the resolved tuple. *How to apply:* `_resolve_redact_patterns` lives on `SafetyPolicy`; tests cover extend, replace, both-error, neither-error (defaults), and empty-replace warning. +- **DEC-018 — `policy.with_mode()` override factory** (resolves B9). `SafetyPolicy.with_mode(self, mode: SamplingMode) -> SafetyPolicy` returns `self.model_copy(update={"mode": mode})`. Documented in the docstring as the **canonical** path for CLI / programmatic mode override; #9 calls it after `load_safety_config` to apply `--mode`. *Why:* `SafetyPolicy` is frozen; without an explicit override path, callers either reach for hacks (mutating private dicts) or re-construct the entire policy (losing fields). `model_copy` is cheap and Pydantic-idiomatic. *How to apply:* one-line method on `SafetyPolicy`; tests assert frozen-ness of original and equality of all other fields after override. +- **DEC-019 — `docs/safety-ops.md` + README "Data safety" cross-link** (resolves B10). New file `docs/safety-ops.md` (parallel to `docs/manifest-loader-ops.md` and `docs/warehouse-adapter-ops.md`). Sections: mode semantics + when to use each; redaction-pattern syntax (case-insensitive `fnmatch` glob, built-ins, `extend` vs `replace`); per-column opt-out (the four signals from DEC-003); audit JSONL schema with `audit_schema_version` reference; `audit_path` security constraints; debugging (logger names, levels); typed-error reference cross-linked to `signalforge.safety.errors`; the at-rest-sensitivity caveat for the audit log itself (DEC-020 C4); rotation guidance (DEC-026). README gains a "Data safety" section between "Configuration" and "Roadmap" that links to the ops doc and summarises the schema-only-default posture in three sentences. *How to apply:* doc lands in US-012; cross-link in README in same story. +- **DEC-020 — Audit-completeness, semantic-coercion, and audit-log sensitivity** (resolves C1 + C2 + C3 + C4). (a) **Audit completeness:** `LLMRequest`'s docstring states `"Construct only via signalforge.safety.request.build_llm_request — direct construction bypasses the audit log."`; a test in `test_public_api.py` AST-scans the `signalforge.safety` package and asserts `LLMRequest(...)` does not appear outside `request.py`. (b) **`tags`/`meta.contains_pii` coercion:** tag matching is case-insensitive (compare lowercased `column.tags + model.tags` against lowercased `"pii"`); `meta.contains_pii` accepts any truthy value (bool / non-empty string / non-zero int) — non-bool values trigger a DEBUG log noting the coercion; column-level signals beat model-level on conflict and the precedence is asserted by parametrised tests. (c) **Pattern matching:** `_matches_redaction_pattern(name, pattern) := fnmatch.fnmatchcase(name.lower(), pattern.lower())`; built-ins become `("*email", "email", "*phone", "phone", "*ssn", "ssn")`; a "suspicious unmatched column" WARNING fires when a column's lowercased name contains any of `{"email", "phone", "ssn", "password", "token", "secret", "api_key"}` AND no pattern matched. (d) **Audit log sensitivity:** `safety-ops.md` documents the audit JSONL itself as sensitive (Gitignore precedent in `.signalforge/`); column names are stored plaintext in `RedactionRecord.column_name` for debuggability — hashing them defeats the audit's review purpose. *How to apply:* `_classify_column` is the central pure function; `tests/safety/test_classify.py` parametrises across the matrix. +- **DEC-021 — `sample` mode guard rails + `policy_flags` on every audit event** (resolves C5 + C14). On `SafetyPolicy` load, if `mode == SamplingMode.SAMPLE`, emit one WARNING: `"Sample mode enabled — raw row data will be sent to the LLM. Verify column tags/meta opt-outs."`. No env-var override path for `mode`; only file + `with_mode()` programmatic override (which #9's CLI uses). `AuditEvent` gains `policy_flags: tuple[str, ...]`; populated by the request builder from policy state with values from a closed set: `"sample_mode_enabled"` (when `mode == SAMPLE`), `"redaction_disabled"` (when resolved `redact_patterns` is empty), `"audit_path_overridden"` (when `audit_path` differs from the default). Tests parametrise across the flag combinations. *Why:* a reviewer scanning the JSONL needs to distinguish "no redactions because no PII columns" from "no redactions because policy disabled them" — the flags make policy state legible at the per-record level. +- **DEC-022 — `LLMRequest` deep immutability + audit-write size cap + ANSI-safe logger** (resolves C6 + C8 + C9). (a) **Immutability:** `LLMRequest` field types switch to tuples — `columns_sent: tuple[str, ...]`, `redactions: tuple[RedactionRecord, ...]`, `sampled_rows: tuple[dict[str, Any], ...] | None`, `aggregates: dict[str, ColumnStats | None] | None` (dict is fine; `ColumnStats` is already frozen), `schema: tuple[tuple[str, str], ...]`. The Pydantic `frozen=True` plus tuple-typed sequences make the request transitively-immutable for downstream consumers (#5 cannot accidentally append a row to a request that's already been audited). (b) **Size cap:** `audit.write` asserts `len(line.encode("utf-8")) <= 4000` (PIPE_BUF margin on Linux's 4096); breach raises `AuditRecordTooLargeError(size, limit)` so feature creep can't silently break atomic appends. (c) **ANSI / log-injection guard:** every logger call in `signalforge.safety` uses lazy-format with json-encoded user-controlled values — `_LOGGER.info("LLM call: %s", json.dumps({"unique_id": uid, "mode": mode.value, ...}))` — never f-string interpolation of strings that came from manifests/configs/columns. Tests inject `\x1b[31mFAKE\x1b[0m` into a column name and assert the logger output contains the JSON-escape sequence (``), not the raw escape. +- **DEC-023 — Pattern-injection rejection at policy-load time** (resolves C7). `SafetyPolicy._resolve_redact_patterns` validates each pattern after `extend`/`replace` resolution: empty string → `InvalidPatternError(value="", reason="empty pattern")`; `*` alone → `InvalidPatternError(value="*", reason="matches all column names; use redact: {replace: []} to disable redaction explicitly")`; `?` alone → `InvalidPatternError(value="?", reason="matches every single-character column name")`. All other `fnmatch` glob expressions are accepted — including `*` as a sub-pattern (e.g. `*_email` is fine). *Why:* a config-injection that broadens redaction may seem safe but actually defeats it — an over-broad pattern can prompt downstream code to "helpfully" un-redact ("everything is PII? must be a misconfiguration"). +- **DEC-024 — `SamplingMode` as `StrEnum` + `RedactionRecord.reason` as `Literal[...]`** (resolves C11 + C12). `SamplingMode(StrEnum)` with members `SCHEMA_ONLY = "schema-only"`, `AGGREGATE_ONLY = "aggregate-only"`, `SAMPLE = "sample"`. (Python 3.11+ — verify project's runtime floor; v0.1 currently targets 3.10 per pyproject.toml DEC-001 from #1, so use `class SamplingMode(str, Enum)` mixin if 3.11 isn't the floor yet.) `@field_validator("mode", mode="before")` on `SafetyPolicy` accepts `"schema-only"` / `"schema_only"` / `"SCHEMA-ONLY"` / `"Schema-Only"` (lowercase + replace `_` with `-`); unknown values raise `InvalidSamplingModeError(value, allowed=tuple(SamplingMode))`. `RedactionRecord.reason: Literal["column_meta_optout", "model_meta_optout", "tag_pii_column", "tag_pii_model", "meta_contains_pii_column", "meta_contains_pii_model", "pattern_match"]`. *Why:* StrEnum gives type-safe `is`-comparison and clean YAML round-trip; `Literal` makes audit-log consumers exhaustively pattern-match. *How to apply:* `tests/safety/test_models.py` parametrises across all enum members and the seven literal reasons. +- **DEC-025 — Top-level `signalforge.yml` namespace: `{ safety: { ... } }`** (resolves C13). The config file's top-level shape: `{ safety: { mode, redact, audit_path, sample_size, ... } }`. `load_safety_config` extracts the `safety:` key and validates only that subtree against `SafetyPolicy`. Other top-level keys (`llm:`, `prune:`, `grade:`, ...) are reserved for future stages and silently ignored by the safety loader (each future stage validates its own subtree). The `_SafetyConfigFile` pydantic model has `safety: _SafetyPolicyContent` and `extra="ignore"` at the top level (so future top-level keys don't fail today's loads); `_SafetyPolicyContent` has `extra="forbid"` (so typos *inside* `safety:` fail loud, per DEC-015). *Why:* the warehouse adapter's `dbt profiles.yml` reader (#3) lived alongside dbt's existing keys without claiming the whole file; `signalforge.yml` is greenfield, but reserving namespaces upfront avoids a v2-config migration. *How to apply:* documented in `safety-ops.md` and an inline comment on `_SafetyConfigFile`. +- **DEC-026 — Error hierarchy + drift-detector + JSONL rotation as user responsibility** (resolves C15 + C16 + C10). (a) **Errors:** `signalforge.safety.errors` ships `SafetyError` (base; mirrors `WarehouseError` / `ManifestError` patterns from #3 + #2: `default_remediation` ClassVar, `↳ Remediation:` rendering in `__str__`, `_format_value(v) := repr(v)` helper for user-input quoting per DEC-022 from #3) plus 9 subclasses: `ConfigNotFoundError(path)`, `InvalidConfigError(message)`, `InvalidSamplingModeError(value, allowed)`, `InvalidPatternError(value, reason)`, `ColumnNotInModelError(model, column)`, `AuditWriteError(path, cause)`, `AuditRecordTooLargeError(size, limit)`, `PolicyValidationError(field, value, reason)`, `UnknownConfigKeyError(key, scope)` (raised by the `extra="forbid"` validators). Total: 9 typed subclasses + 1 base. (b) **Drift detector:** `tests/fixtures/safety/audit_events_sample.jsonl` commits one canonical `AuditEvent` JSONL line. `tests/safety/test_drift_detector.py` defines a one-off `class StrictAuditEvent(BaseModel): model_config = ConfigDict(extra="forbid"); ...` mirroring production `AuditEvent`'s field set, and validates the fixture against it. Adding a field to production `AuditEvent` without updating the fixture or the strict model breaks the test loudly. The fixture has a regeneration script `tests/fixtures/safety/regenerate.sh` that rebuilds it from a small in-process construction (no external tool needed). (c) **JSONL rotation:** documented in `safety-ops.md` as user responsibility (logrotate, manual archive, or external log shipper); no rotation logic in v0.1 — adding it would force a code path that surfaces at the worst moment (audit write) and conflicts with fail-closed semantics. *How to apply:* errors module is the first implementation story after fixtures (US-003); drift detector lands in US-011; JSONL rotation is a doc-only paragraph in US-012. + +## Detailed Breakdown + +Thirteen stories. Architecture order: deps-and-config → fixtures → errors → models → policy → config loader → audit → redact → aggregate → request builder → public API → docs → quality gate → patterns. Validation command (run after every story): `pip install -e ".[dev]" && ruff check . && ruff format --check . && pyright && pytest`. The `docs/research/dbt-claude-technical-surface.md` §4.5 is the design-doc reference for posture decisions throughout. + +### US-001 — Subpackage scaffolding + pytest config + +**Description:** Create the empty `signalforge.safety` subpackage skeleton, verify `pyproject.toml`'s wheel target + pytest config still cover it without changes (the existing `packages = ["src/signalforge"]` already includes everything under `src/signalforge/**`), and add the `safety` pytest marker. + +**Traces to:** DEC-001, `python-build.md`, `testing-signal.md`. + +**Acceptance criteria:** +- `src/signalforge/safety/__init__.py` exists (empty placeholder; full re-exports in US-011). +- `[tool.pytest.ini_options].markers` adds `"safety: tests for the PII safety layer"`. +- `pip install -e ".[dev]"` still succeeds; `pytest --collect-only` returns the existing test set with no marker errors. +- Validation command passes. + +**Done when:** `from signalforge import safety` works (no error); `pytest -m safety --collect-only` returns 0 tests (none exist yet, no error). + +**Files:** `src/signalforge/safety/__init__.py` (new), `pyproject.toml` (markers). + +**Depends on:** none. + +**TDD:** N/A (scaffolding only). + +--- + +### US-002 — Test fixtures: `signalforge.yml` variants + manifest with PII meta + audit JSONL sample + +**Description:** Hand-author the YAML/JSON fixtures consumed by US-005 through US-011. The manifest fixture is hand-derived (no `dbt parse` needed — only the `meta`/`tags` shape matters for these tests); the JSONL audit sample is regenerated via a tiny in-process script. + +**Traces to:** DEC-016 (config branches), DEC-017 (`extend`/`replace`), DEC-021 (sample-mode flag), DEC-026 (drift-detector fixture). + +**Acceptance criteria:** +- `tests/fixtures/safety/signalforge_minimal.yml` — `{ safety: { mode: schema-only } }` (happy path; everything else defaults). +- `tests/fixtures/safety/signalforge_extend.yml` — `{ safety: { redact: { extend: ["*custom_*"] } } }`. +- `tests/fixtures/safety/signalforge_replace_empty.yml` — `{ safety: { redact: { replace: [] } } }` (warning case). +- `tests/fixtures/safety/signalforge_extend_replace_conflict.yml` — both keys present. +- `tests/fixtures/safety/signalforge_unknown_mode.yml` — `mode: phantom`. +- `tests/fixtures/safety/signalforge_typo.yml` — `redacts: ...` (typo) for `extra="forbid"` test. +- `tests/fixtures/safety/signalforge_audit_path_traversal.yml` — `audit_path: ../../escape.jsonl` for DEC-013 test. +- `tests/fixtures/safety/signalforge_unknown_top_level.yml` — `{ safety: {...}, llm: {...} }` (unknown top-level key, must be ignored per DEC-025). +- `tests/fixtures/safety/manifest_with_pii_meta.json` — small manifest snippet with one model exhibiting all four opt-out signals: a column with `meta.signalforge.sample: false`, a column with `tags: ["pii"]`, a column with `meta.contains_pii: true`, and a column matching `*_email` pattern. Plus a model-level `tags: ["pii"]` variant on a sibling model. +- `tests/fixtures/safety/audit_events_sample.jsonl` — one canonical `AuditEvent` line for the drift detector. +- `tests/fixtures/safety/regenerate.sh` — emits `audit_events_sample.jsonl` via `python -c "from signalforge.safety.models import AuditEvent; ..."`. +- `tests/fixtures/README.md` updated with a new "Safety" section (regeneration trigger, hand-author note for the `.yml` files). + +**Done when:** all eight YAML files load via `yaml.safe_load` without errors; the manifest JSON parses; the JSONL has exactly one record. + +**Files:** `tests/fixtures/safety/*.yml`, `tests/fixtures/safety/manifest_with_pii_meta.json`, `tests/fixtures/safety/audit_events_sample.jsonl`, `tests/fixtures/safety/regenerate.sh`, `tests/fixtures/README.md` (modified). + +**Depends on:** US-001. + +**TDD:** N/A (fixtures only; consumed by US-006, US-008, US-011). + +--- + +### US-003 — Errors module + +**Description:** Implement `signalforge.safety.errors` with the full 10-class hierarchy from DEC-026. + +**Traces to:** DEC-026, DEC-022 (user-input quoting via `_format_value`), `manifest-readers.md` (remediation pattern). + +**Acceptance criteria:** +- `signalforge/safety/errors.py` defines `SafetyError(Exception)` with `default_remediation: ClassVar[str]`, `message`, `remediation` instance attrs, `__str__` rendering `"{message}\n ↳ Remediation: {remediation}"`, and a `_format_value(v) -> str := repr(v)` helper. +- All 9 subclasses listed in DEC-026 implemented with class-level `default_remediation` and discriminating attributes. +- User-supplied strings rendered via `_format_value` in messages: e.g. `InvalidPatternError(value="\x1b[31m", reason="empty")` renders the value repr-quoted with control chars visible. +- `signalforge.safety.errors.__all__` lists all 10 classes. +- Validation command passes. + +**Done when:** `from signalforge.safety.errors import SafetyError, AuditWriteError, ...` works; `tests/safety/test_errors.py` covers each class for remediation rendering and adversarial-input quoting. + +**Files:** `src/signalforge/safety/errors.py` (new), `tests/safety/test_errors.py` (new). + +**Depends on:** US-001. + +**TDD:** Yes. Test cases first: +- `test_safety_error_renders_remediation` — base class `__str__` includes `↳ Remediation:` +- `test_each_subclass_has_default_remediation` — parametrised over the 9 subclasses +- `test_invalid_pattern_error_quotes_user_input` — adversarial value with control chars +- `test_audit_write_error_carries_cause` — `cause: BaseException | None` exposed for chaining +- `test_audit_record_too_large_error_includes_size_and_limit` + +--- + +### US-004 — Typed models: `SamplingMode`, `RedactionRecord`, `AuditEvent`, `LLMRequest` + +**Description:** Implement the read-back-stable typed shapes in `signalforge.safety.models`. `SafetyPolicy` lands separately in US-005 because it carries the `@model_validator` for `extend`/`replace` resolution. + +**Traces to:** DEC-014 (audit reproducibility), DEC-015 (`extra="ignore"` on read-back), DEC-022 (immutability via tuples), DEC-024 (StrEnum + Literal). + +**Acceptance criteria:** +- `SamplingMode` is `StrEnum` (or `str, Enum` mixin if Python floor is 3.10) with `SCHEMA_ONLY = "schema-only"`, `AGGREGATE_ONLY = "aggregate-only"`, `SAMPLE = "sample"`. +- `RedactionRecord` is frozen Pydantic v2 with `column_name: str`, `hashed_name: str`, `redacted: bool`, `reason: Literal[...]` (the seven values from DEC-024). +- `AuditEvent` is frozen Pydantic v2 with: `timestamp: datetime`, `model_unique_id: str`, `mode: SamplingMode`, `columns_sent: tuple[str, ...]`, `redactions: tuple[RedactionRecord, ...]`, `row_count: int | None`, `signalforge_version: str`, `policy_hash: str`, `audit_schema_version: int = 1`, `policy_flags: tuple[str, ...] = ()`. `extra="ignore"`. +- `LLMRequest` is frozen Pydantic v2 with: `model_unique_id: str`, `mode: SamplingMode`, `columns_sent: tuple[str, ...]`, `redactions: tuple[RedactionRecord, ...]`, `sampled_rows: tuple[dict[str, Any], ...] | None`, `aggregates: dict[str, ColumnStats | None] | None`, `schema: tuple[tuple[str, str], ...]`. Imports `ColumnStats` from `signalforge.warehouse.models`. +- `LLMRequest` docstring includes: `"Construct only via signalforge.safety.request.build_llm_request — direct construction bypasses the audit log."` +- Validation command passes. + +**Done when:** `from signalforge.safety.models import SamplingMode, RedactionRecord, AuditEvent, LLMRequest` works; round-trip JSON serialisation preserves field order and types; mutating `request.columns_sent` raises (tuple, not list). + +**Files:** `src/signalforge/safety/models.py` (new), `tests/safety/test_models.py` (new). + +**Depends on:** US-003. + +**TDD:** Yes. Test cases first: +- `test_sampling_mode_enum_values_exact_strings` +- `test_redaction_record_reason_literal_rejects_unknown` — `RedactionRecord(reason="phantom", ...)` raises `ValidationError` +- `test_audit_event_schema_version_default_is_1` +- `test_audit_event_extra_ignore_drops_unknown_field` +- `test_llm_request_columns_sent_immutable` — `request.columns_sent.__class__ is tuple`; `request.columns_sent[0] = "x"` raises `TypeError` +- `test_llm_request_sampled_rows_immutable_when_none` and `test_llm_request_sampled_rows_immutable_when_present` + +--- + +### US-005 — `SafetyPolicy` + `_resolve_redact_patterns` + `_compute_policy_hash` + +**Description:** Implement `signalforge.safety.policy.SafetyPolicy` — the user-facing config-shaped model — with `@model_validator(mode="before")` for `extend`/`replace`, the `with_mode` factory, and the `policy_hash` helper. + +**Traces to:** DEC-007 (built-ins + override semantics), DEC-014 (`policy_hash`), DEC-017 (`@model_validator`), DEC-018 (`with_mode`), DEC-021 (sample-mode warning), DEC-023 (pattern-injection rejection), DEC-024 (`@field_validator` for mode case-insensitive load). + +**Acceptance criteria:** +- `SafetyPolicy` is frozen Pydantic v2 with `mode: SamplingMode = SamplingMode.SCHEMA_ONLY`, `redact_patterns: tuple[str, ...]`, `sample_size: int = 100`, `audit_path: Path` (default `Path(".signalforge/audit.jsonl")`). +- `model_config = ConfigDict(frozen=True, extra="forbid", populate_by_name=True)`. +- `@model_validator(mode="before")` named `_resolve_redact_patterns` consumes `redact: {extend?, replace?}` and resolves into `redact_patterns`. Mutual exclusion enforced. Empty `replace: []` allowed but emits one WARNING. +- `@field_validator("mode", mode="before")` accepts case-insensitive variants (`"schema-only"`, `"Schema-Only"`, `"SCHEMA_ONLY"`, etc.); unknown raises `InvalidSamplingModeError`. +- `@field_validator("redact_patterns")` rejects empty / `*` / `?` patterns with `InvalidPatternError`. +- `with_mode(self, mode: SamplingMode) -> SafetyPolicy` via `self.model_copy(update={"mode": mode})`. Documented as #9's CLI override seam. +- Module emits one WARNING on construction when `mode == SAMPLE`: `"Sample mode enabled — raw row data will be sent to the LLM. Verify column tags/meta opt-outs."` +- `_compute_policy_hash(policy: SafetyPolicy) -> str` — SHA-256 of `policy.model_dump_json(by_alias=True)` with sorted keys, hex-truncated to 16 chars. +- Validation command passes. + +**Done when:** `SafetyPolicy()` constructs with defaults; `SafetyPolicy.model_validate({"mode": "Schema-Only", "redact": {"extend": ["*custom"]}})` resolves correctly; `_compute_policy_hash` is deterministic across two equal policies. + +**Files:** `src/signalforge/safety/policy.py` (new), `tests/safety/test_policy.py` (new). + +**Depends on:** US-003, US-004. + +**TDD:** Yes. Test cases first: +- `test_safety_policy_no_args_is_schema_only` *(default-mode regression — DEC-012(a))* +- `test_safety_policy_extra_forbid_rejects_typo` — `redacts:` typo raises `ValidationError` +- `test_safety_policy_redact_extend_appends_to_builtins` +- `test_safety_policy_redact_replace_substitutes` +- `test_safety_policy_redact_extend_and_replace_simultaneously_errors` +- `test_safety_policy_redact_replace_empty_warns_once` +- `test_safety_policy_mode_case_insensitive_load` — parametrised across 6 case variants +- `test_safety_policy_mode_unknown_raises_invalid_sampling_mode_error` +- `test_safety_policy_pattern_empty_raises_invalid_pattern_error` +- `test_safety_policy_pattern_star_alone_raises_invalid_pattern_error` +- `test_safety_policy_pattern_question_mark_alone_raises` +- `test_safety_policy_with_mode_returns_new_frozen_policy` +- `test_safety_policy_with_mode_preserves_other_fields` +- `test_safety_policy_sample_mode_emits_warning_on_construction` +- `test_compute_policy_hash_deterministic_for_equal_policies` +- `test_compute_policy_hash_differs_for_semantically_different_policies` + +--- + +### US-006 — Config loader + path safety + +**Description:** Implement `signalforge.safety.config.load_safety_config(project_dir, path=None) -> SafetyPolicy` with the full DEC-016 error contract and the audit-path traversal hardening (DEC-013). + +**Traces to:** DEC-002, DEC-013 (`audit_path` security), DEC-016 (error contract), DEC-025 (top-level namespace). + +**Acceptance criteria:** +- `load_safety_config(project_dir: Path, path: Path | None = None) -> SafetyPolicy`. Resolution: explicit-path miss → `ConfigNotFoundError`; implicit-path miss → defaults; empty file → defaults (DEBUG log); non-mapping top level → `InvalidConfigError`; missing `safety:` key → defaults; schema-invalid contents → typed errors. +- `_SafetyConfigFile` Pydantic model has `safety: SafetyPolicyContent` with `extra="ignore"` at top level (other top-level keys reserved for future stages); `SafetyPolicyContent` is `SafetyPolicy` itself (re-exported under that internal name for clarity). +- `audit_path` security: reject `..` segments at validation time (`InvalidConfigError`); canonicalise via `signalforge.safety._path_safety.canonicalise_path`; require resolved path inside `project_dir`. +- `signalforge/safety/_path_safety.py` is copied (not imported) from `signalforge.warehouse._path_safety`, with safety-layer-specific error class `InvalidConfigError` for traversal failures (per `warehouse-adapters.md`'s duplication precedent). +- `yaml.safe_load` only — never `yaml.load`. +- Validation command passes. + +**Done when:** all six DEC-016 branches covered by tests; traversal attempts fail loudly. + +**Files:** `src/signalforge/safety/config.py` (new), `src/signalforge/safety/_path_safety.py` (new, copied from warehouse), `tests/safety/test_config.py` (new). + +**Depends on:** US-002, US-003, US-005. + +**TDD:** Yes. Test cases first: +- `test_load_safety_config_no_file_is_schema_only` *(default-mode regression — DEC-012(b))* +- `test_load_safety_config_empty_file_returns_defaults` +- `test_load_safety_config_missing_safety_key_returns_defaults` +- `test_load_safety_config_unknown_top_level_key_ignored` +- `test_load_safety_config_explicit_path_missing_raises_config_not_found` +- `test_load_safety_config_implicit_path_missing_returns_defaults` +- `test_load_safety_config_malformed_yaml_raises_invalid_config` +- `test_load_safety_config_non_mapping_top_level_raises_invalid_config` +- `test_load_safety_config_unknown_mode_raises_invalid_sampling_mode` +- `test_load_safety_config_typo_in_safety_block_raises_validation_error` +- `test_load_safety_config_audit_path_with_dotdot_raises` +- `test_load_safety_config_audit_path_outside_project_raises` +- `test_load_safety_config_audit_path_symlink_to_outside_raises` +- `test_load_safety_config_extend_resolution_via_yaml` +- `test_load_safety_config_replace_empty_warns_once` + +--- + +### US-007 — Audit module: write + fail-closed + size cap + ANSI-safe logger + +**Description:** Implement `signalforge.safety.audit.write(event, audit_path)` with O_APPEND atomic write, fsync, size assertion, and the lazy-format ANSI-safe logger pattern. + +**Traces to:** DEC-005, DEC-011 (fail-closed), DEC-022 (size cap + ANSI guard). + +**Acceptance criteria:** +- `audit.write(event: AuditEvent, audit_path: Path) -> None` opens with `O_APPEND | O_CREAT | 0o600`, writes `json.dumps(event.model_dump(mode="json"), separators=(",", ":")) + "\n"`, calls `os.fsync(fd)`, closes. +- Catches NO exceptions internally; any `OSError` / `PermissionError` / `IOError` / encoding error propagates as `AuditWriteError(path, cause)`. +- Size assertion: `if len(line.encode("utf-8")) > 4000: raise AuditRecordTooLargeError(size=..., limit=4000)`. +- Parent directory created via `audit_path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)` before open; mkdir failures also propagate as `AuditWriteError`. +- Logger emits one INFO line per write: `_LOGGER.info("audit event: %s", json.dumps({"unique_id": event.model_unique_id, "mode": event.mode.value, "columns_sent": len(event.columns_sent), "redacted": len(event.redactions)}))`. Lazy-format with json.dumps; never f-string interpolation of user-controlled strings. +- Validation command passes. + +**Done when:** concurrent-thread test produces a JSONL whose every line round-trips through `json.loads`; failure-injection test (`chmod 000` on parent dir under `tmp_path`) raises `AuditWriteError`. + +**Files:** `src/signalforge/safety/audit.py` (new), `tests/safety/test_audit.py` (new). + +**Depends on:** US-003, US-004. + +**TDD:** Yes. Test cases first: +- `test_audit_write_appends_one_jsonl_line` +- `test_audit_write_fsyncs_before_returning` *(use `mock` to assert fsync is called — but only for fsync, not the whole open/write chain)* +- `test_audit_write_round_trips_through_json_loads` +- `test_audit_write_creates_parent_dir` +- `test_audit_write_failure_raises_audit_write_error` *(chmod 000 on parent under tmp_path)* +- `test_audit_write_oversize_record_raises_too_large` *(craft an event with a 5000-byte field via the `model_unique_id` string)* +- `test_audit_write_concurrent_threads_no_interleave` *(spawn 10 threads, 100 records each; assert exactly 1000 valid JSONL lines)* +- `test_audit_logger_line_escapes_ansi_in_user_input` *(model_unique_id = `"\x1b[31mFAKE\x1b[0m"`; log capture asserts `` appears, raw escape does not)* + +--- + +### US-008 — Redaction: `_classify_column`, `redact_rows`, `redact_column_names`, `hash_column_name` + +**Description:** Implement the pure redaction helpers in `signalforge.safety.redact`. `_classify_column` is the central function called by both the request builder and the aggregate wrapper. + +**Traces to:** DEC-003 (the four opt-out signals + pattern), DEC-010 (column-name redaction via blake2b hash), DEC-020 (case-insensitivity + suspicious-column heuristic), DEC-024 (Literal reasons). + +**Acceptance criteria:** +- `hash_column_name(name: str) -> str` returns `"col_" + blake2b(name.encode(), digest_size=4).hexdigest()` (8 hex chars). Deterministic. +- `_classify_column(column: Column, model: Model, policy: SafetyPolicy) -> RedactionRecord` (pure). Precedence: column-level signals beat model-level on conflict; first-match-wins among the seven reasons. +- Tag matching is case-insensitive (lowercase both sides); `meta.contains_pii` accepts truthy values (DEBUG-log on coercion); `tags: [PII]` normalises to lowercase. +- Pattern matching: `fnmatch.fnmatchcase(name.lower(), pattern.lower())`. +- "Suspicious unmatched column" WARNING heuristic: if a column's lowercased name contains any of `{"email", "phone", "ssn", "password", "token", "secret", "api_key"}` AND `_classify_column` returned `redacted=False`, emit WARNING once per (model, column) pair. +- `redact_rows(rows: tuple[dict[str, Any], ...], hashed_to_real: dict[str, str]) -> tuple[dict[str, Any], ...]` — replaces values for keys whose name is being redacted with `""`. Does not mutate input. Missing-key-in-row is silently ignored. +- `redact_column_names(columns: tuple[tuple[str, str], ...], records: tuple[RedactionRecord, ...]) -> tuple[tuple[str, str], ...]` — substitutes hashed names for redacted columns in a `(name, type)` tuple sequence. +- Validation command passes. + +**Done when:** `_classify_column` parametrised matrix passes (column meta opt-out, model meta opt-out, column tag pii, model tag pii, column meta.contains_pii, model meta.contains_pii, pattern match, none-of-the-above); precedence assertions pass. + +**Files:** `src/signalforge/safety/redact.py` (new), `tests/safety/test_redact.py` (new), `tests/safety/test_classify.py` (new). + +**Depends on:** US-004, US-005. + +**TDD:** Yes. Test cases first: +- `test_hash_column_name_deterministic` and `test_hash_column_name_distinct_for_distinct_inputs` +- `test_classify_column_matrix` — parametrised seven reasons +- `test_classify_column_precedence_column_over_model` +- `test_classify_tag_pii_uppercase_normalised` +- `test_classify_meta_contains_pii_truthy_string` +- `test_classify_meta_contains_pii_truthy_int` +- `test_classify_meta_contains_pii_falsy_zero_does_not_redact` +- `test_classify_pattern_case_insensitive` +- `test_classify_suspicious_unmatched_column_warns` +- `test_redact_rows_replaces_values` +- `test_redact_rows_does_not_mutate_input` +- `test_redact_rows_missing_column_in_row_silent` +- `test_redact_column_names_substitutes_hashed` + +--- + +### US-009 — Aggregate wrapper: `aggregate_columns` + +**Description:** Implement `signalforge.safety.aggregate.aggregate_columns(adapter, model, columns, policy)`. Calls `_classify_column` for each requested column; for non-redacted columns, calls `adapter.column_stats` inside the adapter's `with` context (per #3's DEC-008 batching). For redacted columns, returns `None` and records a `RedactionRecord`. + +**Traces to:** DEC-008. + +**Acceptance criteria:** +- `aggregate_columns(adapter: WarehouseAdapter, model: Model, columns: list[str], policy: SafetyPolicy) -> tuple[dict[str, ColumnStats | None], tuple[RedactionRecord, ...]]`. +- Non-redacted columns invoke `adapter.column_stats(table=TableRef.from_model(model), column=name)` inside `with adapter:` context. +- Redacted columns yield `None` in the returned dict (keyed by hashed name, not real name) and one `RedactionRecord` in the returned tuple. +- Tests use a `FakeAdapter` (US-009.5 file `tests/safety/_fake_adapter.py`) that mirrors `tests/warehouse/_fake.py`'s `expect_*` API. NOT `MagicMock`. +- Validation command passes. + +**Done when:** 50-column model with 20 redacted + 30 non-redacted produces one batched query (verified by `FakeAdapter` expecting exactly one `column_stats` call per non-redacted column inside the context); redacted columns produce `None` values. + +**Files:** `src/signalforge/safety/aggregate.py` (new), `tests/safety/_fake_adapter.py` (new), `tests/safety/test_aggregate.py` (new), `tests/safety/test_fake_adapter.py` (new — verifies the fake satisfies the ABC). + +**Depends on:** US-005, US-008. + +**TDD:** Yes. Test cases first: +- `test_fake_adapter_satisfies_warehouse_adapter_abc` +- `test_fake_adapter_unexpected_call_raises_assertion_error` +- `test_aggregate_columns_redacted_returns_none` +- `test_aggregate_columns_calls_adapter_for_non_redacted` +- `test_aggregate_columns_does_not_call_adapter_for_redacted` +- `test_aggregate_columns_uses_with_adapter_context` +- `test_aggregate_columns_returns_redaction_records` + +--- + +### US-010 — Request builder: `build_llm_request` + +**Description:** Implement `signalforge.safety.request.build_llm_request(model, adapter, policy)` — the single entry point that produces an `LLMRequest`, calls `audit.write` before returning, and orchestrates the per-mode behaviour. + +**Traces to:** DEC-009 (single entry), DEC-010 (column-name hashing), DEC-011 (fail-closed audit), DEC-012(c) (zero adapter calls in schema-only). + +**Acceptance criteria:** +- `build_llm_request(model: Model, adapter: WarehouseAdapter, policy: SafetyPolicy) -> LLMRequest`. +- Per-mode behaviour: + - `SCHEMA_ONLY`: zero `adapter.column_stats` and zero `adapter.sample_rows` calls. `LLMRequest.sampled_rows = None`, `aggregates = None`. `schema` carries `(hashed_name, type)` for redacted columns and `(real_name, type)` for non-redacted. + - `AGGREGATE_ONLY`: calls `aggregate_columns`. `sampled_rows = None`. `schema` as above. `aggregates` populated (with `None` for redacted-column keys). + - `SAMPLE`: calls `adapter.sample_rows(TableRef.from_model(model), n=policy.sample_size)`; redacts values via `redact_rows`; redacts NAMES in the row dicts (replaces real key with hashed key) so the LLM sees the same hashed identifiers as `schema`. +- Constructs `AuditEvent` with `signalforge_version`, `policy_hash` (via `_compute_policy_hash`), `audit_schema_version=1`, `policy_flags` (per DEC-021 closed set). +- Calls `audit.write(event, policy.audit_path)` AFTER constructing the request, BEFORE returning. Any audit-write exception aborts (no `LLMRequest` returned). +- Validation command passes. + +**Done when:** all three modes produce correct `LLMRequest` shapes verified against `FakeAdapter`; `test_build_llm_request_default_policy_zero_warehouse_calls` passes (default-mode regression DEC-012(c)). + +**Files:** `src/signalforge/safety/request.py` (new), `tests/safety/test_request.py` (new), `tests/safety/test_default_mode_regression.py` (new — clusters DEC-012's three regression tests). + +**Depends on:** US-007, US-008, US-009. + +**TDD:** Yes. Test cases first: +- `test_build_llm_request_schema_only_zero_warehouse_calls` *(DEC-012(c))* +- `test_build_llm_request_aggregate_only_calls_column_stats_per_non_redacted_column` +- `test_build_llm_request_sample_redacts_values_to_redacted_constant` +- `test_build_llm_request_sample_redacts_names_in_rows_to_hashed` +- `test_build_llm_request_schema_uses_hashed_names_for_redacted` +- `test_build_llm_request_audit_emitted_exactly_once` +- `test_build_llm_request_audit_carries_signalforge_version` +- `test_build_llm_request_audit_carries_policy_hash` +- `test_build_llm_request_audit_carries_schema_version_1` +- `test_build_llm_request_audit_policy_flags_sample_mode_enabled` +- `test_build_llm_request_audit_policy_flags_redaction_disabled` +- `test_build_llm_request_audit_write_failure_raises_audit_write_error_no_request_returned` *(DEC-011)* +- `test_build_llm_request_returned_request_is_transitively_immutable` *(DEC-022)* + +--- + +### US-011 — Public API + drift detector + AST audit-completeness check + +**Description:** Wire `signalforge/safety/__init__.py` with the documented public surface; add the `tests/safety/test_drift_detector.py` (DEC-026) and `tests/safety/test_public_api.py` (which includes the AST-scan for direct `LLMRequest(...)` construction outside `request.py`). + +**Traces to:** DEC-001 (re-export discipline), DEC-020(a) (audit-completeness AST scan), DEC-026 (drift detector). + +**Acceptance criteria:** +- `signalforge/safety/__init__.py` re-exports: `SamplingMode`, `SafetyPolicy`, `LLMRequest`, `RedactionRecord`, `AuditEvent`, `load_safety_config`, `build_llm_request`, `aggregate_columns`, `redact_rows`, `SafetyError` + 9 subclasses. +- `__all__` matches the documented surface; private helpers (`_classify_column`, `_compute_policy_hash`, `_resolve_redact_patterns`, `_path_safety`, etc.) reachable via dotted import only. +- `tests/safety/test_drift_detector.py` defines `StrictAuditEvent(extra="forbid")` mirroring production fields; validates `tests/fixtures/safety/audit_events_sample.jsonl` against it. +- `tests/safety/test_public_api.py`: + - `test_documented_surface_importable_from_package_root` — every name in the README's documented surface imports from `signalforge.safety`. + - `test_private_helpers_not_in_dir` — `_classify_column`, `_compute_policy_hash` not in `dir(signalforge.safety)`. + - `test_llm_request_construction_only_in_request_module` — AST-scans `src/signalforge/safety/*.py` (excluding `request.py`) for `Call(func=Name(id="LLMRequest"))`; asserts zero matches. +- Validation command passes. + +**Done when:** `from signalforge.safety import SamplingMode, ...` works for every documented name; the AST scan catches a planted violation in a test fixture (negative test). + +**Files:** `src/signalforge/safety/__init__.py` (full re-exports; replaces US-001's placeholder), `tests/safety/test_public_api.py` (new), `tests/safety/test_drift_detector.py` (new). + +**Depends on:** US-003, US-004, US-005, US-006, US-007, US-008, US-009, US-010. + +**TDD:** N/A (declarative public API). + +--- + +### US-012 — `docs/safety-ops.md` + README "Data safety" section + +**Description:** Author the operational reference matching the precedent set by `docs/manifest-loader-ops.md` and `docs/warehouse-adapter-ops.md`; add a "Data safety" section to README between "Configuration" and "Roadmap". + +**Traces to:** DEC-019 (ops doc commitment), DEC-020(d) (audit log sensitivity), DEC-026(c) (rotation as user responsibility). + +**Acceptance criteria:** +- `docs/safety-ops.md` sections (in order): + 1. **Default posture** — schema-only, the fail-closed model, the "explicit opt-in for sampling" framing. + 2. **Modes** — schema-only / aggregate-only / sample with concrete examples. + 3. **`signalforge.yml` reference** — top-level `safety:` namespace; mode; `redact: { extend, replace }`; `sample_size`; `audit_path`. + 4. **Redaction patterns** — case-insensitive `fnmatch` glob; built-ins; override semantics; the suspicious-column WARNING heuristic. + 5. **Per-column opt-out** — the four signals (column meta, model meta, tags, `meta.contains_pii`); precedence rules; case-insensitive `tags` matching; `contains_pii` truthy coercion. + 6. **Column-name redaction** — DEC-010's blake2b hash; how the audit log maps `(real, hashed)` back. + 7. **Audit JSONL schema** — every field with type and meaning; `audit_schema_version` reference; `policy_flags` closed set. + 8. **Audit log sensitivity** — the JSONL contains plaintext column names; treat at-rest as sensitive (Gitignore in `.signalforge/`). + 9. **Audit log rotation** — user responsibility (logrotate / archive); no built-in rotation in v0.1. + 10. **Debugging** — logger names, levels (`signalforge.safety` at INFO/WARNING/DEBUG); how to read a fail-closed `AuditWriteError`. + 11. **Typed-error reference** — table cross-linked to `signalforge.safety.errors` with each subclass's discriminating fields. + 12. **CLI integration note** — pointer to #9 ("`--mode` flag wires through `policy.with_mode()`"); explicit "no env-var override for mode" callout. +- `README.md` adds a "Data safety" section (3 paragraphs): the schema-only-default posture, link to `docs/safety-ops.md`, callout that `.signalforge/audit.jsonl` should be Gitignored. +- `.gitignore` adds `.signalforge/` (the audit-log directory). +- Validation command passes (no Python changes; ruff/pyright unaffected). + +**Done when:** `docs/safety-ops.md` is committed; README cross-link works; `.gitignore` includes `.signalforge/`. + +**Files:** `docs/safety-ops.md` (new), `README.md` (modified), `.gitignore` (modified). + +**Depends on:** US-001 through US-011 (doc references the implemented behaviour). + +**TDD:** N/A (docs only). + +--- + +### US-013 — Quality Gate + +**Description:** Run code-reviewer four times across the full changeset; address each pass's real bugs. Run CodeRabbit if available. Validation command must pass after all fixes. + +**Traces to:** all DECs. + +**Acceptance criteria:** +- 4 code-reviewer passes; each pass's blockers / concerns fixed before the next. +- CodeRabbit review if MCP/CLI available; non-blocking suggestions logged. +- `pip install -e ".[dev]" && ruff check . && ruff format --check . && pyright && pytest` passes from a clean checkout. +- No `MagicMock` in any new test (grep enforces). +- No raw `yaml.load` (grep enforces). +- No f-string interpolation of user-controlled strings in any logger call within `src/signalforge/safety/` (grep `_LOGGER.\w+\(f"` and audit any hits). +- The three default-mode regression tests (DEC-012) pass. +- The drift detector (US-011) passes against the committed fixture. +- The AST audit-completeness scan (US-011) catches a planted violation. + +**Done when:** all four passes' findings are addressed; validation is clean; no anti-pattern greps return hits. + +**Files:** any file the reviewer flags. + +**Depends on:** US-001 through US-012. + +**TDD:** N/A (review pass). + +--- + +### US-014 — Patterns & Memory + +**Description:** Update `.claude/rules/`, `docs/`, and beads memory with new patterns established by this ticket. + +**Traces to:** all DECs. + +**Acceptance criteria:** +- New `.claude/rules/safety-layer.md` distilling the load-bearing rules: fail-closed audit semantics, column-name redaction via stable hash, audit-event reproducibility fields (signalforge_version + policy_hash + audit_schema_version), `extra="forbid"` on config-shaped models vs. `extra="ignore"` on read-back, the four opt-out signals + precedence, ANSI-safe lazy-format logger, AST audit-completeness scan, drift-detector pattern. +- `bd remember` entries for: (a) "schema-only mode redacts column names too — names alone leak PII"; (b) "audit writes are fail-closed — any write failure aborts the LLM call"; (c) "config-shaped Pydantic models use extra=forbid; read-back models use extra=ignore + drift detector". +- `CLAUDE.md` "Public API surface (v0.1)" section updated with `signalforge.safety` entries (`SamplingMode`, `SafetyPolicy`, `LLMRequest`, `load_safety_config`, `build_llm_request`, `SafetyError` hierarchy). +- `CLAUDE.md` "Repository status" bullet added for issue #4. +- `MEMORY.md` index entry pointing at `.claude/rules/safety-layer.md` (if convention applies — verify against existing memory layout). +- Validation command passes. + +**Done when:** future-you opening this repo cold sees the safety-layer rules immediately on rule-discovery. + +**Files:** `.claude/rules/safety-layer.md` (new), `CLAUDE.md` (modified), `bd remember` invocations. + +**Depends on:** US-013. + +**TDD:** N/A (documentation / rules). + +## Beads Manifest + +*(Phase 7 — populated on devolve.)* + +## Refinement Log + +*(Phase 3 — populated after Architecture Review.)* + +## Detailed Breakdown + +*(Phase 4 — populated after Refinement.)* + +## Beads Manifest + +*(Phase 7 — populated on devolve.)* From 725ccacf303e3a20e8f7c951e488f57b5f90685c Mon Sep 17 00:00:00 2001 From: wjduenow Date: Tue, 28 Apr 2026 22:19:58 -0700 Subject: [PATCH 02/19] Update phase to published with PR #18 link Co-Authored-By: Claude Opus 4.7 (1M context) --- plans/super/4-pii-safety.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plans/super/4-pii-safety.md b/plans/super/4-pii-safety.md index c852f25c..9b426aa7 100644 --- a/plans/super/4-pii-safety.md +++ b/plans/super/4-pii-safety.md @@ -5,7 +5,7 @@ - **Ticket:** [#4](https://github.com/wjduenow/SignalForge/issues/4) - **Branch:** `feature/4-pii-safety` (off `dev`) - **Worktree:** `/home/wesd/dev/worktrees/SignalForge/feature/4-pii-safety` (created via `git worktree add`) -- **Phase:** detailing (Phases 1–3 locked 2026-04-28 "use defaults"; ready to publish PR) +- **Phase:** published (PR [#18](https://github.com/wjduenow/SignalForge/pull/18) draft; awaiting approval before devolve) - **Sessions:** 1 (started 2026-04-28) - **Plan author:** Claude Code (Opus 4.7, 1M context) - **Milestone:** v0.1 (deployment blocker for any team with PII concerns) From 7058f28a4cd8178fe608d60a259eac53f0211426 Mon Sep 17 00:00:00 2001 From: wjduenow Date: Tue, 28 Apr 2026 22:23:57 -0700 Subject: [PATCH 03/19] Devolve plan to beads (epic + 14 tasks) Phase: devolved. Epic bd_1-scaffolding-o6f live; 14 tasks wired into the architecture-order DAG; bd ready confirms US-001 is the single entry point. Co-Authored-By: Claude Opus 4.7 (1M context) --- plans/super/4-pii-safety.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plans/super/4-pii-safety.md b/plans/super/4-pii-safety.md index 9b426aa7..2d88f909 100644 --- a/plans/super/4-pii-safety.md +++ b/plans/super/4-pii-safety.md @@ -5,7 +5,7 @@ - **Ticket:** [#4](https://github.com/wjduenow/SignalForge/issues/4) - **Branch:** `feature/4-pii-safety` (off `dev`) - **Worktree:** `/home/wesd/dev/worktrees/SignalForge/feature/4-pii-safety` (created via `git worktree add`) -- **Phase:** published (PR [#18](https://github.com/wjduenow/SignalForge/pull/18) draft; awaiting approval before devolve) +- **Phase:** devolved (epic `bd_1-scaffolding-o6f` + 14 tasks live; PR [#18](https://github.com/wjduenow/SignalForge/pull/18) draft) - **Sessions:** 1 (started 2026-04-28) - **Plan author:** Claude Code (Opus 4.7, 1M context) - **Milestone:** v0.1 (deployment blocker for any team with PII concerns) From 457b00fb175ab8a234fe992ee1081e3740482ffe Mon Sep 17 00:00:00 2001 From: wjduenow Date: Tue, 28 Apr 2026 22:24:26 -0700 Subject: [PATCH 04/19] Populate Beads Manifest section; remove stale placeholder duplicates Co-Authored-By: Claude Opus 4.7 (1M context) --- plans/super/4-pii-safety.md | 31 +++++++++++++++++++++---------- 1 file changed, 21 insertions(+), 10 deletions(-) diff --git a/plans/super/4-pii-safety.md b/plans/super/4-pii-safety.md index 2d88f909..8e76c2d1 100644 --- a/plans/super/4-pii-safety.md +++ b/plans/super/4-pii-safety.md @@ -722,16 +722,27 @@ Thirteen stories. Architecture order: deps-and-config → fixtures → errors ## Beads Manifest -*(Phase 7 — populated on devolve.)* +Devolved 2026-04-28. Worktree: `/home/wesd/dev/worktrees/SignalForge/feature/4-pii-safety`. -## Refinement Log - -*(Phase 3 — populated after Architecture Review.)* - -## Detailed Breakdown +**Epic:** `bd_1-scaffolding-o6f` — 4: PII safety layer. -*(Phase 4 — populated after Refinement.)* +**Tasks** (in architecture order; each carries the full acceptance criteria from its US-NNN story above): -## Beads Manifest - -*(Phase 7 — populated on devolve.)* +| ID | Story | Depends on | +| --- | --- | --- | +| `bd_1-scaffolding-0ix` | US-001 — Subpackage scaffolding + pytest config | (none — entry point) | +| `bd_1-scaffolding-o64` | US-002 — Test fixtures | US-001 | +| `bd_1-scaffolding-rix` | US-003 — Errors module | US-001 | +| `bd_1-scaffolding-agc` | US-004 — Typed models | US-003 | +| `bd_1-scaffolding-pfj` | US-005 — SafetyPolicy + helpers | US-003, US-004 | +| `bd_1-scaffolding-ly6` | US-006 — Config loader + path safety | US-002, US-003, US-005 | +| `bd_1-scaffolding-mg2` | US-007 — Audit module | US-003, US-004 | +| `bd_1-scaffolding-y47` | US-008 — Redaction helpers | US-004, US-005 | +| `bd_1-scaffolding-3z5` | US-009 — Aggregate wrapper + FakeAdapter | US-005, US-008 | +| `bd_1-scaffolding-969` | US-010 — Request builder + regression suite | US-007, US-008, US-009 | +| `bd_1-scaffolding-2fb` | US-011 — Public API + drift + AST scan | US-003 .. US-010 | +| `bd_1-scaffolding-gx3` | US-012 — `docs/safety-ops.md` + README | US-001 .. US-011 | +| `bd_1-scaffolding-8av` | US-013 — Quality Gate | US-001 .. US-012 | +| `bd_1-scaffolding-51d` | US-014 — Patterns & Memory | US-013 | + +`bd ready` confirms `bd_1-scaffolding-0ix` (US-001) is the single unblocked entry point. The DAG fans out from there. Cross-story `task → epic` parent-edges were rejected by beads' epic semantics ("tasks can only block other tasks, not epics"); all inter-task dependencies are wired and that is what Ralph traverses. From c82bcd8bc4874f83f147c0364e84f12f073a1172 Mon Sep 17 00:00:00 2001 From: wjduenow Date: Tue, 28 Apr 2026 22:28:21 -0700 Subject: [PATCH 05/19] bd_1-scaffolding-0ix: Scaffold signalforge.safety subpackage + add safety pytest marker Empty __init__.py placeholder; full public re-exports land in US-011. New pytest marker enables `pytest -m safety` once tests exist. Co-Authored-By: Claude Opus 4.7 (1M context) --- pyproject.toml | 1 + src/signalforge/safety/__init__.py | 1 + 2 files changed, 2 insertions(+) create mode 100644 src/signalforge/safety/__init__.py diff --git a/pyproject.toml b/pyproject.toml index 6a4538b0..337601bc 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -54,4 +54,5 @@ markers = [ "integration: tests exercising the full load() round-trip", "error: tests asserting an exception with its remediation", "bigquery: tests requiring BigQuery credentials (gated by SF_RUN_BQ=1)", + "safety: tests for the PII safety layer", ] diff --git a/src/signalforge/safety/__init__.py b/src/signalforge/safety/__init__.py new file mode 100644 index 00000000..0b37c15f --- /dev/null +++ b/src/signalforge/safety/__init__.py @@ -0,0 +1 @@ +"""PII safety layer for SignalForge (placeholder; public re-exports land in US-011).""" From c9dc5a1d1c43b3ea78074851c7abc5b0115eef3f Mon Sep 17 00:00:00 2001 From: wjduenow Date: Tue, 28 Apr 2026 22:32:24 -0700 Subject: [PATCH 06/19] bd_1-scaffolding-o64: Add safety-layer test fixtures (signalforge.yml variants, manifest with PII meta, audit JSONL sample) Hand-author the fixture corpora consumed by the upcoming PII-safety stories (US-005..US-011): eight signalforge.yml variants exercising the locked safety: top-level shape (DEC-025), the redact extend/replace mutual exclusion (DEC-017), the sampling-mode flag (DEC-021), and the DEC-013 path-traversal guard; a hand-derived manifest fixture carrying all four column-level PII opt-out signals plus a model-level signal (DEC-026); a one-line audit JSONL sample matching the locked AuditEvent shape (DEC-005 + DEC-014); and a deterministic regeneration script that will swap to the typed AuditEvent model once US-004 lands. Co-Authored-By: Claude Opus 4.7 (1M context) --- tests/fixtures/README.md | 65 +++++++++ .../fixtures/safety/audit_events_sample.jsonl | 1 + .../safety/manifest_with_pii_meta.json | 123 ++++++++++++++++++ tests/fixtures/safety/regenerate.sh | 30 +++++ .../signalforge_audit_path_traversal.yml | 2 + tests/fixtures/safety/signalforge_extend.yml | 4 + .../signalforge_extend_replace_conflict.yml | 4 + tests/fixtures/safety/signalforge_minimal.yml | 2 + .../safety/signalforge_replace_empty.yml | 3 + tests/fixtures/safety/signalforge_typo.yml | 3 + .../safety/signalforge_unknown_mode.yml | 2 + .../safety/signalforge_unknown_top_level.yml | 4 + 12 files changed, 243 insertions(+) create mode 100644 tests/fixtures/safety/audit_events_sample.jsonl create mode 100644 tests/fixtures/safety/manifest_with_pii_meta.json create mode 100755 tests/fixtures/safety/regenerate.sh create mode 100644 tests/fixtures/safety/signalforge_audit_path_traversal.yml create mode 100644 tests/fixtures/safety/signalforge_extend.yml create mode 100644 tests/fixtures/safety/signalforge_extend_replace_conflict.yml create mode 100644 tests/fixtures/safety/signalforge_minimal.yml create mode 100644 tests/fixtures/safety/signalforge_replace_empty.yml create mode 100644 tests/fixtures/safety/signalforge_typo.yml create mode 100644 tests/fixtures/safety/signalforge_unknown_mode.yml create mode 100644 tests/fixtures/safety/signalforge_unknown_top_level.yml diff --git a/tests/fixtures/README.md b/tests/fixtures/README.md index 062bdf00..aa6d711d 100644 --- a/tests/fixtures/README.md +++ b/tests/fixtures/README.md @@ -171,6 +171,71 @@ YAML in the wild — drift-from-tool-output isn't the failure mode (unlike [`testing-signal.md`](../../.claude/rules/testing-signal.md): regenerate via ephemeral `uvx` when the tool emits the artefact, hand-author when humans do. +# Safety + +The fixtures under `tests/fixtures/safety/` back the PII-safety layer +(US-005..US-011) and track DEC-005, DEC-013, DEC-014, DEC-016, DEC-017, +DEC-021, DEC-025, and DEC-026 from +[`plans/super/4-pii-safety.md`](../../plans/super/4-pii-safety.md). + +## Layout + +``` +tests/fixtures/safety/ +├── signalforge_minimal.yml # happy path; mode=schema-only +├── signalforge_extend.yml # redact.extend (DEC-017) +├── signalforge_replace_empty.yml # redact.replace=[]; WARN on load +├── signalforge_extend_replace_conflict.yml # both keys → mutual-exclusion error +├── signalforge_unknown_mode.yml # InvalidSamplingModeError (DEC-021) +├── signalforge_typo.yml # `redacts:` → extra="forbid" trip +├── signalforge_audit_path_traversal.yml # DEC-013 path-traversal guard +├── signalforge_unknown_top_level.yml # unknown top-level keys ignored +├── manifest_with_pii_meta.json # PII signals at column + model level +├── audit_events_sample.jsonl # one canonical AuditEvent line +└── regenerate.sh # regenerates the JSONL only +``` + +## `signalforge.yml` variants — hand-authored + +The eight YAML files are hand-authored, not regenerated. They mirror the +`signalforge.yml` schema locked in DEC-025 (`safety:` is the only top-level +key SignalForge consumes in v0.1; everything else is left for future stages). +Bump them when the `signalforge.yml` schema evolves — there is no tool to +regenerate against, since users author these by hand in the wild. + +Hand-authoring is appropriate here for the same reason as the +[`profiles/`](#profiles) fixtures: drift-from-tool-output is not the failure +mode. The trade-off is documented in +[`testing-signal.md`](../../.claude/rules/testing-signal.md). + +## `manifest_with_pii_meta.json` — hand-derived + +This manifest is hand-derived, not produced by `dbt parse`. It carries the +minimum shape required to construct `signalforge.manifest.models.Manifest` +plus PII metadata exercising all four opt-out signals at column level +(`*email` pattern match, `meta.signalforge.sample: false`, `tags: ["pii"]`, +`meta.contains_pii: true`) and one model-level signal (`tags: ["pii"]` on +the model itself, cascading to every column). + +`metadata.dbt_schema_version` points at v12 — the latest version in the +loader's tolerated v9–v12 range (issue #2). Bump this fixture (and its +schema URL) when the loader's tolerated range narrows or shifts. + +## `audit_events_sample.jsonl` — regenerated + +Regenerate with: + +```bash +bash tests/fixtures/safety/regenerate.sh +``` + +The script is the source of truth for the audit-event shape until US-004 +lands the typed `AuditEvent` model. The committed JSONL is exactly one +line and the regen is deterministic — `git diff` after re-running must be +empty. When US-004 lands, swap the inline dict construction for an +`AuditEvent(...).model_dump_json()` call so drift in the typed model +propagates here. + ## See also - [`docs/manifest-loader-ops.md`](../../docs/manifest-loader-ops.md) — operational diff --git a/tests/fixtures/safety/audit_events_sample.jsonl b/tests/fixtures/safety/audit_events_sample.jsonl new file mode 100644 index 00000000..7c41a3e9 --- /dev/null +++ b/tests/fixtures/safety/audit_events_sample.jsonl @@ -0,0 +1 @@ +{"timestamp":"2026-04-28T22:30:00+00:00","model_unique_id":"model.sf_demo.customers","mode":"schema-only","columns_sent":["id","col_a3f29c61"],"redactions":[{"column_name":"email","hashed_name":"col_a3f29c61","redacted":true,"reason":"pattern_match"}],"row_count":null,"signalforge_version":"0.1.0","policy_hash":"abc123def456789a","audit_schema_version":1,"policy_flags":[]} diff --git a/tests/fixtures/safety/manifest_with_pii_meta.json b/tests/fixtures/safety/manifest_with_pii_meta.json new file mode 100644 index 00000000..53a52c2c --- /dev/null +++ b/tests/fixtures/safety/manifest_with_pii_meta.json @@ -0,0 +1,123 @@ +{ + "metadata": { + "dbt_schema_version": "https://schemas.getdbt.com/dbt/manifest/v12.json", + "project_name": "sf_demo", + "generated_at": null, + "invocation_id": null, + "user_id": null, + "send_anonymous_usage_stats": null, + "adapter_type": null, + "env": {} + }, + "nodes": { + "model.sf_demo.customers": { + "database": "dev", + "schema": "main", + "name": "customers", + "resource_type": "model", + "package_name": "sf_demo", + "path": "marts/customers.sql", + "original_file_path": "models/marts/customers.sql", + "unique_id": "model.sf_demo.customers", + "alias": "customers", + "config": { + "materialized": "table", + "tags": [], + "meta": {} + }, + "tags": [], + "description": "", + "meta": {}, + "columns": { + "id": { + "name": "id", + "description": "", + "meta": {}, + "tags": [] + }, + "email": { + "name": "email", + "description": "", + "meta": {}, + "tags": [] + }, + "customer_ssn_optout": { + "name": "customer_ssn_optout", + "description": "", + "meta": { + "signalforge": { + "sample": false + } + }, + "tags": [] + }, + "taxpayer_id": { + "name": "taxpayer_id", + "description": "", + "meta": {}, + "tags": ["pii"] + }, + "birth_date": { + "name": "birth_date", + "description": "", + "meta": { + "contains_pii": true + }, + "tags": [] + } + }, + "depends_on": { + "macros": [], + "nodes": [] + }, + "refs": [], + "sources": [], + "raw_code": "select 1 as id, 'a@b.c' as email, '000-00-0000' as customer_ssn_optout, '12345' as taxpayer_id, current_date as birth_date", + "language": "sql", + "access": "protected", + "version": null, + "latest_version": null, + "primary_key": [] + }, + "model.sf_demo.orders_pii_at_model": { + "database": "dev", + "schema": "main", + "name": "orders_pii_at_model", + "resource_type": "model", + "package_name": "sf_demo", + "path": "marts/orders_pii_at_model.sql", + "original_file_path": "models/marts/orders_pii_at_model.sql", + "unique_id": "model.sf_demo.orders_pii_at_model", + "alias": "orders_pii_at_model", + "config": { + "materialized": "table", + "tags": ["pii"], + "meta": {} + }, + "tags": ["pii"], + "description": "", + "meta": {}, + "columns": { + "order_id": { + "name": "order_id", + "description": "", + "meta": {}, + "tags": [] + } + }, + "depends_on": { + "macros": [], + "nodes": [] + }, + "refs": [], + "sources": [], + "raw_code": "select 1 as order_id", + "language": "sql", + "access": "protected", + "version": null, + "latest_version": null, + "primary_key": [] + } + }, + "disabled": {} +} diff --git a/tests/fixtures/safety/regenerate.sh b/tests/fixtures/safety/regenerate.sh new file mode 100755 index 00000000..6437ede6 --- /dev/null +++ b/tests/fixtures/safety/regenerate.sh @@ -0,0 +1,30 @@ +#!/usr/bin/env bash +# Regenerate audit_events_sample.jsonl deterministically. +# Until US-004 lands AuditEvent, the schema is hand-authored here and must +# match the documented shape in plans/super/4-pii-safety.md (DEC-005 + DEC-014). +set -euo pipefail +cd "$(dirname "$0")" + +python - <<'PY' > audit_events_sample.jsonl +import json +record = { + "timestamp": "2026-04-28T22:30:00+00:00", + "model_unique_id": "model.sf_demo.customers", + "mode": "schema-only", + "columns_sent": ["id", "col_a3f29c61"], + "redactions": [ + { + "column_name": "email", + "hashed_name": "col_a3f29c61", + "redacted": True, + "reason": "pattern_match", + } + ], + "row_count": None, + "signalforge_version": "0.1.0", + "policy_hash": "abc123def456789a", + "audit_schema_version": 1, + "policy_flags": [], +} +print(json.dumps(record, separators=(",", ":"))) +PY diff --git a/tests/fixtures/safety/signalforge_audit_path_traversal.yml b/tests/fixtures/safety/signalforge_audit_path_traversal.yml new file mode 100644 index 00000000..43397ef1 --- /dev/null +++ b/tests/fixtures/safety/signalforge_audit_path_traversal.yml @@ -0,0 +1,2 @@ +safety: + audit_path: ../../escape.jsonl diff --git a/tests/fixtures/safety/signalforge_extend.yml b/tests/fixtures/safety/signalforge_extend.yml new file mode 100644 index 00000000..e88148ea --- /dev/null +++ b/tests/fixtures/safety/signalforge_extend.yml @@ -0,0 +1,4 @@ +safety: + redact: + extend: + - "*custom_*" diff --git a/tests/fixtures/safety/signalforge_extend_replace_conflict.yml b/tests/fixtures/safety/signalforge_extend_replace_conflict.yml new file mode 100644 index 00000000..8b696562 --- /dev/null +++ b/tests/fixtures/safety/signalforge_extend_replace_conflict.yml @@ -0,0 +1,4 @@ +safety: + redact: + extend: ["*custom"] + replace: ["*specific"] diff --git a/tests/fixtures/safety/signalforge_minimal.yml b/tests/fixtures/safety/signalforge_minimal.yml new file mode 100644 index 00000000..4dca54d0 --- /dev/null +++ b/tests/fixtures/safety/signalforge_minimal.yml @@ -0,0 +1,2 @@ +safety: + mode: schema-only diff --git a/tests/fixtures/safety/signalforge_replace_empty.yml b/tests/fixtures/safety/signalforge_replace_empty.yml new file mode 100644 index 00000000..459c3620 --- /dev/null +++ b/tests/fixtures/safety/signalforge_replace_empty.yml @@ -0,0 +1,3 @@ +safety: + redact: + replace: [] diff --git a/tests/fixtures/safety/signalforge_typo.yml b/tests/fixtures/safety/signalforge_typo.yml new file mode 100644 index 00000000..d0dcd498 --- /dev/null +++ b/tests/fixtures/safety/signalforge_typo.yml @@ -0,0 +1,3 @@ +safety: + redacts: + extend: ["*foo"] diff --git a/tests/fixtures/safety/signalforge_unknown_mode.yml b/tests/fixtures/safety/signalforge_unknown_mode.yml new file mode 100644 index 00000000..ea65be58 --- /dev/null +++ b/tests/fixtures/safety/signalforge_unknown_mode.yml @@ -0,0 +1,2 @@ +safety: + mode: phantom diff --git a/tests/fixtures/safety/signalforge_unknown_top_level.yml b/tests/fixtures/safety/signalforge_unknown_top_level.yml new file mode 100644 index 00000000..f456b282 --- /dev/null +++ b/tests/fixtures/safety/signalforge_unknown_top_level.yml @@ -0,0 +1,4 @@ +safety: + mode: schema-only +llm: + model: claude-opus-4-7 From 962a19b5fe16d685d8db357fc51453a5f80e6895 Mon Sep 17 00:00:00 2001 From: wjduenow Date: Tue, 28 Apr 2026 22:33:36 -0700 Subject: [PATCH 07/19] bd_1-scaffolding-rix: Add signalforge.safety.errors with 10-class hierarchy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit SafetyError base + 9 typed subclasses; mirrors WarehouseError/ManifestError patterns (default_remediation ClassVar, ↳ Remediation rendering, repr-quoted user input via _format_value helper). Co-Authored-By: Claude Opus 4.7 (1M context) --- src/signalforge/safety/errors.py | 285 +++++++++++++++++++++++++++++++ tests/safety/test_errors.py | 245 ++++++++++++++++++++++++++ 2 files changed, 530 insertions(+) create mode 100644 src/signalforge/safety/errors.py create mode 100644 tests/safety/test_errors.py diff --git a/src/signalforge/safety/errors.py b/src/signalforge/safety/errors.py new file mode 100644 index 00000000..079d2514 --- /dev/null +++ b/src/signalforge/safety/errors.py @@ -0,0 +1,285 @@ +"""Typed exception hierarchy for the safety / PII layer. + +Implements DEC-026 (10-class hierarchy rooted at :class:`SafetyError`) and +DEC-022 (user-supplied strings rendered via ``repr()`` so adversarial input — +embedded quotes, control chars, ANSI escapes — cannot smuggle special +characters into log viewers or error messages). Mirrors the style established +by :mod:`signalforge.warehouse.errors` and :mod:`signalforge.manifest.errors`: +every error carries a class-level ``default_remediation`` that the base +``__str__`` renders on a separate ``↳ Remediation:`` line. + +The remediation pattern operationalises the README's "explainable diffs" +commitment at the safety layer's failure surface; every distinct failure mode +the safety machinery can produce gets a typed exception so the CLI / audit +layer can pattern-match without sniffing message text. +""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any, ClassVar + + +def _format_value(v: object) -> str: + """Quote a user-supplied value via ``repr()`` for safe inclusion in + error messages (DEC-022). + + Embedding raw user input in error strings is a log-injection seam: a + crafted pattern like ``"foo'\\nINFO: spoofed log line"`` (or an ANSI + escape such as ``"\\x1b[31m"``) could pollute log viewers or stack + traces. Routing every user-controlled value through ``repr()`` quotes + the string, escapes control characters, and makes whitespace visible. + """ + return repr(v) + + +class SafetyError(Exception): + """Base class for all safety-layer errors. + + Subclasses set a class-level ``default_remediation`` string; instances + may override it via the ``remediation=`` keyword argument. ``__str__`` + renders the message and the remediation on separate lines so log output + and CLI output both read cleanly. + """ + + default_remediation: ClassVar[str] = "(no remediation set — this is the base class)" + + def __init__(self, message: str, *, remediation: str | None = None) -> None: + super().__init__(message) + self.message = message + self.remediation = ( + remediation if remediation is not None else type(self).default_remediation + ) + + def __str__(self) -> str: + return f"{self.message}\n ↳ Remediation: {self.remediation}" + + +class ConfigNotFoundError(SafetyError): + """An explicit ``path=`` argument pointed at a missing safety config file. + + Raised only when the caller passed a path explicitly; the implicit + default-discovery path (``/signalforge.yml``) is allowed to + be absent and falls back to built-in defaults. + """ + + default_remediation: ClassVar[str] = ( + "Verify the path is correct, or pass path=None to fall back to " + "/signalforge.yml or built-in defaults." + ) + + def __init__(self, path: Path, *, remediation: str | None = None) -> None: + self.path = path + message = f"Safety config not found at {_format_value(str(path))}." + super().__init__(message, remediation=remediation) + + +class InvalidConfigError(SafetyError): + """Parent for parse / schema failures in ``signalforge.yml``. + + Free-form message; subclasses (e.g. :class:`InvalidSamplingModeError`, + :class:`InvalidPatternError`) refine it with structured fields. Catching + ``InvalidConfigError`` covers every shape-of-config failure. + """ + + default_remediation: ClassVar[str] = ( + "Check signalforge.yml against the documented schema in docs/safety-ops.md." + ) + + +class InvalidSamplingModeError(InvalidConfigError): + """The ``mode:`` field of the safety config is not one of the allowed + sampling-mode literals (``schema-only`` / ``aggregate-only`` / ``sample``).""" + + default_remediation: ClassVar[str] = ( + "Set safety.mode to one of the documented sampling modes; see docs/safety-ops.md." + ) + + def __init__( + self, + value: Any, + *, + allowed: tuple[str, ...], + remediation: str | None = None, + ) -> None: + self.value = value + self.allowed = allowed + message = f"Invalid sampling mode {_format_value(value)}; allowed values: {allowed}." + if remediation is None: + remediation = f"mode must be one of {allowed}; got {_format_value(value)}." + super().__init__(message, remediation=remediation) + + +class InvalidPatternError(InvalidConfigError): + """A redact / allowlist pattern failed validation. + + Patterns must be non-empty fnmatch globs and may not be the bare + wildcards ``"*"`` / ``"?"`` (which would disable the safety layer + silently). + """ + + default_remediation: ClassVar[str] = ( + "Patterns must be non-empty fnmatch globs and may not be the bare wildcards '*' or '?'." + ) + + def __init__(self, value: str, *, reason: str, remediation: str | None = None) -> None: + self.value = value + self.reason = reason + message = f"Invalid pattern {_format_value(value)}: {_format_value(reason)}." + super().__init__(message, remediation=remediation) + + +class ColumnNotInModelError(SafetyError): + """The caller asked for a column the manifest model does not expose. + + Raised by safety-layer helpers that look up columns by name on a + :class:`signalforge.manifest.Model` and find no match in + ``manifest.nodes[model].columns``. + """ + + default_remediation: ClassVar[str] = ( + "Verify the column exists in manifest.nodes[model].columns." + ) + + def __init__( + self, + model_unique_id: str, + column_name: str, + *, + remediation: str | None = None, + ) -> None: + self.model_unique_id = model_unique_id + self.column_name = column_name + message = ( + f"Column {_format_value(column_name)} is not declared on model " + f"{_format_value(model_unique_id)}." + ) + super().__init__(message, remediation=remediation) + + +class AuditWriteError(SafetyError): + """Appending a record to the JSONL audit log failed (any I/O error). + + The audit log is fail-closed: when a write fails, the LLM call that + triggered it is aborted rather than allowed to proceed without an + audit trail. + """ + + default_remediation: ClassVar[str] = ( + "Check that /.signalforge/ exists and is writable; the " + "audit log is fail-closed — the LLM call is aborted on write failure." + ) + + def __init__( + self, + path: Path, + cause: BaseException | None = None, + *, + remediation: str | None = None, + ) -> None: + self.path = path + self.cause = cause + if cause is not None: + message = ( + f"Audit log write failed at {_format_value(str(path))}: {_format_value(cause)!s}" + ) + else: + message = f"Audit log write failed at {_format_value(str(path))}." + super().__init__(message, remediation=remediation) + + +class AuditRecordTooLargeError(SafetyError): + """An audit JSONL record would exceed the POSIX atomic-append size cap. + + POSIX guarantees ``write(2)`` is atomic only for payloads up to ``PIPE_BUF`` + bytes (typically 4 KiB on Linux). The audit writer enforces a size cap + to keep concurrent appends from interleaving partial records. + """ + + default_remediation: ClassVar[str] = ( + "Audit records must stay under the configured byte limit for atomic " + "concurrent appends; reduce columns_sent or redactions count." + ) + + def __init__(self, size: int, limit: int, *, remediation: str | None = None) -> None: + self.size = size + self.limit = limit + message = f"Audit record size {size} exceeds atomic-append limit {limit}." + if remediation is None: + remediation = ( + f"Audit records must stay under {limit} bytes for atomic " + "concurrent appends; reduce columns_sent or redactions count." + ) + super().__init__(message, remediation=remediation) + + +class PolicyValidationError(SafetyError): + """Generic policy-shape validation failure. + + Raised when a :class:`SafetyPolicy` field has the wrong type or otherwise + fails its post-construction invariants. The discriminating triple + (``field``, ``value``, ``reason``) lets callers report the exact field + that tripped without sniffing message text. + """ + + default_remediation: ClassVar[str] = ( + "Verify SafetyPolicy fields match the documented types in docs/safety-ops.md." + ) + + def __init__( + self, + field: str, + value: Any, + reason: str, + *, + remediation: str | None = None, + ) -> None: + self.field = field + self.value = value + self.reason = reason + message = ( + f"Invalid SafetyPolicy field {_format_value(field)}=" + f"{_format_value(value)}: {_format_value(reason)}." + ) + if remediation is None: + remediation = f"Verify SafetyPolicy field {field!r} matches the documented type." + super().__init__(message, remediation=remediation) + + +class UnknownConfigKeyError(SafetyError): + """A typo'd or unsupported key was found under a known config scope. + + Raised by ``extra="forbid"`` validators on the safety config models + (e.g. ``redacts:`` instead of ``redact:``). Surfacing typos loudly + keeps the safety config from silently doing nothing. + """ + + default_remediation: ClassVar[str] = ( + "Remove or rename the unknown key; see docs/safety-ops.md for the supported schema." + ) + + def __init__(self, key: str, scope: str, *, remediation: str | None = None) -> None: + self.key = key + self.scope = scope + message = f"Unknown config key {_format_value(key)} under scope {_format_value(scope)}." + if remediation is None: + remediation = ( + f"Remove or rename the unknown key {key!r} under {scope!r}; " + "see docs/safety-ops.md for the supported schema." + ) + super().__init__(message, remediation=remediation) + + +# Sorted alphabetically (verified by tests/safety/test_errors.py). +__all__ = [ + "AuditRecordTooLargeError", + "AuditWriteError", + "ColumnNotInModelError", + "ConfigNotFoundError", + "InvalidConfigError", + "InvalidPatternError", + "InvalidSamplingModeError", + "PolicyValidationError", + "SafetyError", + "UnknownConfigKeyError", +] diff --git a/tests/safety/test_errors.py b/tests/safety/test_errors.py new file mode 100644 index 00000000..66b4ab08 --- /dev/null +++ b/tests/safety/test_errors.py @@ -0,0 +1,245 @@ +"""Unit tests for the safety errors module (DEC-026, DEC-022). + +Mirrors :mod:`tests.warehouse.test_errors` and :mod:`tests.manifest.test_errors`. +Every test is capable of failing: no ``assert True``-shaped placeholders +(``testing-signal.md``). +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from signalforge.safety import errors as errors_module +from signalforge.safety.errors import ( + AuditRecordTooLargeError, + AuditWriteError, + ColumnNotInModelError, + ConfigNotFoundError, + InvalidConfigError, + InvalidPatternError, + InvalidSamplingModeError, + PolicyValidationError, + SafetyError, + UnknownConfigKeyError, + _format_value, +) + +# Subclasses (excluding the base) — kept in alphabetical order. +_SUBCLASSES: tuple[type[SafetyError], ...] = ( + AuditRecordTooLargeError, + AuditWriteError, + ColumnNotInModelError, + ConfigNotFoundError, + InvalidConfigError, + InvalidPatternError, + InvalidSamplingModeError, + PolicyValidationError, + UnknownConfigKeyError, +) + + +@pytest.mark.unit +@pytest.mark.safety +def test_safety_error_renders_message_and_remediation() -> None: + """Base ``__str__`` includes both the message and the + ``↳ Remediation:`` marker line.""" + rendered = str(SafetyError("boom", remediation="fix it")) + assert "boom" in rendered + assert "↳ Remediation: fix it" in rendered + + +@pytest.mark.unit +@pytest.mark.safety +def test_safety_error_default_remediation_used_when_remediation_kwarg_omitted() -> None: + """When ``remediation=`` is omitted, the class-level default is used.""" + + class _Sub(SafetyError): + default_remediation = "subclass default hint" + + err = _Sub("something went wrong") + rendered = str(err) + assert "subclass default hint" in rendered + assert err.remediation == "subclass default hint" + + +@pytest.mark.unit +@pytest.mark.safety +def test_safety_error_explicit_remediation_overrides_default() -> None: + """An explicit ``remediation=`` kwarg overrides the class default.""" + + class _Sub(SafetyError): + default_remediation = "subclass default hint" + + err = _Sub("something went wrong", remediation="custom") + rendered = str(err) + assert "custom" in rendered + assert "subclass default hint" not in rendered + + +@pytest.mark.unit +@pytest.mark.safety +@pytest.mark.parametrize("cls", _SUBCLASSES, ids=lambda c: c.__name__) +def test_each_subclass_has_default_remediation(cls: type[SafetyError]) -> None: + """Every concrete subclass declares a non-empty ``default_remediation``.""" + remediation = cls.default_remediation + assert isinstance(remediation, str) + assert remediation.strip(), f"{cls.__name__}.default_remediation must be non-empty" + + +@pytest.mark.unit +@pytest.mark.safety +@pytest.mark.parametrize("cls", _SUBCLASSES, ids=lambda c: c.__name__) +def test_each_subclass_inherits_from_safety_error(cls: type[SafetyError]) -> None: + """Every concrete subclass is a subclass of :class:`SafetyError`.""" + assert issubclass(cls, SafetyError) + + +@pytest.mark.unit +@pytest.mark.safety +def test_invalid_sampling_mode_error_renders_value_and_allowed() -> None: + """``InvalidSamplingModeError`` renders both the bad value and the + allowed tuple in its rendered output.""" + err = InvalidSamplingModeError( + value="phantom", + allowed=("schema-only", "aggregate-only", "sample"), + ) + rendered = str(err) + assert repr("phantom") in rendered + # The allowed tuple's contents appear in the remediation. + assert "schema-only" in rendered + assert "aggregate-only" in rendered + assert "sample" in rendered + + +@pytest.mark.unit +@pytest.mark.safety +@pytest.mark.error +def test_invalid_pattern_error_quotes_user_input_with_control_chars() -> None: + """DEC-022: control characters in user-supplied patterns are rendered via + ``repr()`` so they cannot smuggle ANSI escapes into log viewers.""" + adversarial = "\x1b[31m" + err = InvalidPatternError(value=adversarial, reason="empty") + rendered = str(err) + # repr() of the value MUST appear verbatim somewhere in the message; the + # raw escape sequence must NOT appear unescaped (we check via the repr + # form which contains the literal backslash-x escape). + assert repr(adversarial) in rendered + # Sanity: attribute is preserved for programmatic access. + assert err.value == adversarial + assert err.reason == "empty" + + +@pytest.mark.unit +@pytest.mark.safety +def test_audit_write_error_carries_path_and_cause() -> None: + """``AuditWriteError`` exposes both ``path`` and ``cause`` attributes; + ``cause`` defaults to ``None`` when omitted.""" + p = Path("/tmp/.signalforge/audit.jsonl") + err = AuditWriteError(path=p) + assert err.path == p + assert err.cause is None + + cause = OSError("disk full") + err2 = AuditWriteError(path=p, cause=cause) + assert err2.path == p + assert err2.cause is cause + + +@pytest.mark.unit +@pytest.mark.safety +def test_audit_write_error_str_includes_cause_repr_when_present() -> None: + """When a ``cause`` is supplied, its ``repr()`` appears in the rendered + message so log viewers can see what failed underneath.""" + p = Path("/tmp/.signalforge/audit.jsonl") + cause = OSError("disk full") + err = AuditWriteError(path=p, cause=cause) + rendered = str(err) + assert repr(cause) in rendered + assert repr(str(p)) in rendered + + +@pytest.mark.unit +@pytest.mark.safety +def test_audit_record_too_large_error_includes_size_and_limit() -> None: + """The rendered message contains both the actual size and the limit, and + the remediation substitutes the limit value (not the literal ``{limit}``).""" + err = AuditRecordTooLargeError(size=8192, limit=4096) + rendered = str(err) + assert "8192" in rendered + assert "4096" in rendered + # The format-string placeholder must have been substituted away. + assert "{limit}" not in rendered + assert err.size == 8192 + assert err.limit == 4096 + + +@pytest.mark.unit +@pytest.mark.safety +def test_unknown_config_key_error_includes_key_and_scope() -> None: + """The rendered message contains both the unknown key and its scope.""" + err = UnknownConfigKeyError(key="redacts", scope="safety") + rendered = str(err) + assert repr("redacts") in rendered + assert repr("safety") in rendered + assert err.key == "redacts" + assert err.scope == "safety" + + +@pytest.mark.unit +@pytest.mark.safety +def test_invalid_sampling_mode_extends_invalid_config_error() -> None: + """``InvalidSamplingModeError`` is catchable via ``except InvalidConfigError``.""" + caught: InvalidConfigError | None = None + try: + raise InvalidSamplingModeError( + value="phantom", + allowed=("schema-only", "aggregate-only", "sample"), + ) + except InvalidConfigError as exc: + caught = exc + assert isinstance(caught, InvalidSamplingModeError) + + +@pytest.mark.unit +@pytest.mark.safety +def test_invalid_pattern_extends_invalid_config_error() -> None: + """``InvalidPatternError`` is catchable via ``except InvalidConfigError``.""" + caught: InvalidConfigError | None = None + try: + raise InvalidPatternError(value="*", reason="bare wildcard") + except InvalidConfigError as exc: + caught = exc + assert isinstance(caught, InvalidPatternError) + + +@pytest.mark.unit +@pytest.mark.safety +def test_format_value_helper_uses_repr() -> None: + """``_format_value(v)`` delegates to ``repr(v)`` — DEC-022.""" + s = "foo'bar" + assert _format_value(s) == repr(s) + # And for non-strings too: the helper must be a thin ``repr()`` wrapper. + assert _format_value(42) == repr(42) + assert _format_value(None) == repr(None) + + +@pytest.mark.unit +@pytest.mark.safety +def test_module_all_lists_all_classes() -> None: + """``__all__`` lists all 10 class names (1 base + 9 concrete subclasses).""" + expected = { + "SafetyError", + "ConfigNotFoundError", + "InvalidConfigError", + "InvalidSamplingModeError", + "InvalidPatternError", + "ColumnNotInModelError", + "AuditWriteError", + "AuditRecordTooLargeError", + "PolicyValidationError", + "UnknownConfigKeyError", + } + assert set(errors_module.__all__) == expected + assert len(errors_module.__all__) == 10 From 70e0e10164c8093a352bcb110ed853456900214d Mon Sep 17 00:00:00 2001 From: wjduenow Date: Tue, 28 Apr 2026 22:37:53 -0700 Subject: [PATCH 08/19] bd_1-scaffolding-agc: Add signalforge.safety.models (SamplingMode, RedactionRecord, AuditEvent, LLMRequest) Frozen Pydantic v2 models with deep-immutable tuple sequences (DEC-022). AuditEvent carries reproducibility fields (signalforge_version, policy_hash, audit_schema_version=1) per DEC-014. LLMRequest docstring warns against direct construction (audit-completeness convention; AST scan in US-011). Co-Authored-By: Claude Opus 4.7 (1M context) --- src/signalforge/safety/models.py | 143 +++++++++++++++++++ tests/safety/test_models.py | 232 +++++++++++++++++++++++++++++++ 2 files changed, 375 insertions(+) create mode 100644 src/signalforge/safety/models.py create mode 100644 tests/safety/test_models.py diff --git a/src/signalforge/safety/models.py b/src/signalforge/safety/models.py new file mode 100644 index 00000000..2b025f56 --- /dev/null +++ b/src/signalforge/safety/models.py @@ -0,0 +1,143 @@ +"""Typed models for the PII safety layer (US-004). + +Defines the read-back-stable shapes consumed by every other safety-layer +module: :class:`SamplingMode`, :class:`RedactionRecord`, :class:`AuditEvent`, +and :class:`LLMRequest`. The companion :class:`SafetyPolicy` lands separately +in US-005 (it carries config-validation logic and is policy-shaped, not +data-shaped). + +Design commitments operationalised here: + +* **DEC-014** — :class:`AuditEvent` carries every field needed to reproduce a + draft run: ``signalforge_version``, ``policy_hash``, ``audit_schema_version``, + and ``policy_flags``. Audits without these are unreproducible by definition. +* **DEC-015** — Every model uses ``extra="ignore"`` so audit logs written by + newer SignalForge versions read back cleanly on older ones. The matching + ``extra="forbid"`` drift detector lives in tests (US-011), per the + ``manifest-readers.md`` rule. +* **DEC-022** — Sequences are :class:`tuple` rather than :class:`list`. The + request object is handed to the LLM-drafting layer (issue #5) *after* the + audit event has been written; making the sequences immutable closes the + window where a mutation could desync the request from its audit record. +* **DEC-024** — :class:`SamplingMode` uses ``str + Enum`` (not :class:`StrEnum`, + which is 3.11+) to preserve the project's 3.10 floor. This still gives + type-safe ``is``-comparison plus string-equality and YAML round-trip. +""" + +from __future__ import annotations + +from datetime import datetime +from enum import Enum +from typing import Any, Literal + +from pydantic import BaseModel, ConfigDict + +from signalforge.warehouse.models import ColumnStats + +_BASE_MODEL_CONFIG = ConfigDict(frozen=True, extra="ignore", populate_by_name=True) + + +class SamplingMode(str, Enum): + """Sampling-mode enum for the safety layer (DEC-024). + + Implemented as a ``str + Enum`` mixin rather than :class:`enum.StrEnum` + because the project's Python floor is 3.10. Members compare equal to + their string values (``SamplingMode.SCHEMA_ONLY == "schema-only"``) and + round-trip cleanly through YAML / JSON. + """ + + SCHEMA_ONLY = "schema-only" + AGGREGATE_ONLY = "aggregate-only" + SAMPLE = "sample" + + +RedactionReason = Literal[ + "column_meta_optout", + "model_meta_optout", + "tag_pii_column", + "tag_pii_model", + "meta_contains_pii_column", + "meta_contains_pii_model", + "pattern_match", +] + + +class RedactionRecord(BaseModel): + """One column's redaction outcome. + + Emitted by the redactor for every column considered (whether kept or + dropped) so the audit log records the *full* decision surface, not just + the redactions actually applied. + """ + + model_config = _BASE_MODEL_CONFIG + + column_name: str + hashed_name: str + redacted: bool + reason: RedactionReason + + +class AuditEvent(BaseModel): + """One row in the JSONL audit log (DEC-014). + + Carries every field needed to reproduce a draft run from the audit log + alone: SignalForge version, the policy hash that gated the request, the + audit schema version (so future readers can branch on shape changes), + and any policy flags that were active. ``row_count`` is ``None`` when + the run was schema-only. + """ + + model_config = _BASE_MODEL_CONFIG + + timestamp: datetime + model_unique_id: str + mode: SamplingMode + columns_sent: tuple[str, ...] + redactions: tuple[RedactionRecord, ...] + row_count: int | None = None + signalforge_version: str + policy_hash: str + audit_schema_version: int = 1 + policy_flags: tuple[str, ...] = () + + +class LLMRequest(BaseModel): + """The request payload handed to issue #5's LLM-drafting layer. + + Construct only via :func:`signalforge.safety.request.build_llm_request` — + direct construction bypasses the audit log and breaks the reproducibility + contract documented in DEC-014. The AST scan in US-011 enforces this + convention at lint time; this docstring is the human-readable companion. + + Sequences are :class:`tuple` (DEC-022) so the request cannot be mutated + after the audit event has been written. + """ + + model_config = ConfigDict( + frozen=True, + extra="ignore", + populate_by_name=True, + arbitrary_types_allowed=True, + ) + + model_unique_id: str + mode: SamplingMode + columns_sent: tuple[str, ...] + redactions: tuple[RedactionRecord, ...] + sampled_rows: tuple[dict[str, Any], ...] | None = None + aggregates: dict[str, ColumnStats | None] | None = None + # ``schema`` overrides Pydantic v1's deprecated :meth:`BaseModel.schema` + # method on this subclass. The override is intentional — the field name is + # part of the documented LLMRequest contract — so pyright's structural + # complaint is silenced here rather than renamed. + schema: tuple[tuple[str, str], ...] # pyright: ignore[reportIncompatibleMethodOverride] + + +__all__ = [ + "SamplingMode", + "RedactionReason", + "RedactionRecord", + "AuditEvent", + "LLMRequest", +] diff --git a/tests/safety/test_models.py b/tests/safety/test_models.py new file mode 100644 index 00000000..df892635 --- /dev/null +++ b/tests/safety/test_models.py @@ -0,0 +1,232 @@ +"""Tests for ``signalforge.safety.models`` (US-004). + +Covers the four typed shapes added by this story: + +* :class:`SamplingMode` — ``str + Enum`` mixin (Python 3.10-compatible). +* :class:`RedactionRecord` — frozen Pydantic v2 model with ``Literal`` reason. +* :class:`AuditEvent` — frozen, reproducibility-carrying audit record (DEC-014). +* :class:`LLMRequest` — frozen, deep-immutable request payload (DEC-022). + +The drift-detection ``extra="forbid"`` test lands separately in US-011; this +file only validates the production shapes' behaviour. +""" + +from __future__ import annotations + +from datetime import datetime, timezone +from pathlib import Path + +import pytest +from pydantic import ValidationError + +from signalforge.safety.models import ( + AuditEvent, + LLMRequest, + RedactionRecord, + SamplingMode, +) + +pytestmark = pytest.mark.safety + + +# --------------------------------------------------------------------------- +# SamplingMode +# --------------------------------------------------------------------------- + + +def test_sampling_mode_enum_values_exact_strings() -> None: + assert SamplingMode.SCHEMA_ONLY.value == "schema-only" + assert SamplingMode.AGGREGATE_ONLY.value == "aggregate-only" + assert SamplingMode.SAMPLE.value == "sample" + assert len(SamplingMode) == 3 + + +def test_sampling_mode_is_str_subclass() -> None: + assert isinstance(SamplingMode.SCHEMA_ONLY, str) + # str-equality works: critical for YAML round-trip compatibility. + assert SamplingMode.SCHEMA_ONLY == "schema-only" + assert SamplingMode.AGGREGATE_ONLY == "aggregate-only" + assert SamplingMode.SAMPLE == "sample" + + +def test_sampling_mode_iteration() -> None: + assert tuple(SamplingMode) == ( + SamplingMode.SCHEMA_ONLY, + SamplingMode.AGGREGATE_ONLY, + SamplingMode.SAMPLE, + ) + + +# --------------------------------------------------------------------------- +# RedactionRecord +# --------------------------------------------------------------------------- + + +def _valid_record() -> RedactionRecord: + return RedactionRecord( + column_name="email", + hashed_name="col_a3f29c61", + redacted=True, + reason="pattern_match", + ) + + +def test_redaction_record_construction_happy_path() -> None: + record = _valid_record() + assert record.column_name == "email" + assert record.hashed_name == "col_a3f29c61" + assert record.redacted is True + assert record.reason == "pattern_match" + + +def test_redaction_record_reason_literal_rejects_unknown() -> None: + with pytest.raises(ValidationError): + RedactionRecord( + column_name="x", + hashed_name="col_y", + redacted=True, + reason="phantom", # type: ignore[arg-type] + ) + + +def test_redaction_record_is_frozen() -> None: + record = _valid_record() + with pytest.raises(ValidationError): + record.column_name = "z" # type: ignore[misc] + + +# --------------------------------------------------------------------------- +# AuditEvent +# --------------------------------------------------------------------------- + + +def _valid_audit_event(**overrides: object) -> AuditEvent: + base: dict[str, object] = { + "timestamp": datetime(2026, 4, 28, 22, 30, tzinfo=timezone.utc), + "model_unique_id": "model.sf_demo.customers", + "mode": SamplingMode.SCHEMA_ONLY, + "columns_sent": ("id", "col_a3f29c61"), + "redactions": (_valid_record(),), + "signalforge_version": "0.1.0", + "policy_hash": "abc123def456789a", + } + base.update(overrides) + return AuditEvent(**base) # type: ignore[arg-type] + + +def test_audit_event_schema_version_default_is_1() -> None: + event = _valid_audit_event() + assert event.audit_schema_version == 1 + + +def test_audit_event_extra_ignore_drops_unknown_field() -> None: + event = _valid_audit_event(unknown_field="x") # extra="ignore" + dumped = event.model_dump() + assert "unknown_field" not in dumped + + +def test_audit_event_round_trips_through_json_dumps() -> None: + fixture_path = ( + Path(__file__).resolve().parents[1] / "fixtures" / "safety" / "audit_events_sample.jsonl" + ) + line = fixture_path.read_text(encoding="utf-8").splitlines()[0] + event = AuditEvent.model_validate_json(line) + assert event.model_unique_id == "model.sf_demo.customers" + assert event.mode is SamplingMode.SCHEMA_ONLY + assert event.columns_sent == ("id", "col_a3f29c61") + assert event.row_count is None + assert event.signalforge_version == "0.1.0" + assert event.policy_hash == "abc123def456789a" + assert event.audit_schema_version == 1 + assert event.policy_flags == () + assert len(event.redactions) == 1 + assert event.redactions[0].reason == "pattern_match" + + # Round-trip back through JSON and reconstruct an equal record. + redumped = event.model_dump_json() + event2 = AuditEvent.model_validate_json(redumped) + assert event2 == event + + +def test_audit_event_columns_sent_immutable() -> None: + event = _valid_audit_event() + assert event.columns_sent.__class__ is tuple + # Concatenation works (returns a new tuple); mutation is not available. + assert event.columns_sent + ("x",) == ("id", "col_a3f29c61", "x") + assert not hasattr(event.columns_sent, "append") + + +# --------------------------------------------------------------------------- +# LLMRequest +# --------------------------------------------------------------------------- + + +def _valid_llm_request(**overrides: object) -> LLMRequest: + base: dict[str, object] = { + "model_unique_id": "model.sf_demo.customers", + "mode": SamplingMode.SCHEMA_ONLY, + "columns_sent": ("id", "col_a3f29c61"), + "redactions": (_valid_record(),), + "schema": (("id", "INT64"), ("col_a3f29c61", "STRING")), + } + base.update(overrides) + return LLMRequest(**base) # type: ignore[arg-type] + + +def test_llm_request_columns_sent_is_tuple() -> None: + request = _valid_llm_request() + assert request.columns_sent.__class__ is tuple + + +def test_llm_request_redactions_is_tuple_of_records() -> None: + request = _valid_llm_request() + assert request.redactions.__class__ is tuple + assert all(isinstance(r, RedactionRecord) for r in request.redactions) + + +def test_llm_request_sampled_rows_immutable_when_none() -> None: + request = _valid_llm_request(sampled_rows=None) + assert request.sampled_rows is None + + +def test_llm_request_sampled_rows_immutable_when_present() -> None: + request = _valid_llm_request( + sampled_rows=({"id": 1, "col_a3f29c61": "abc"},), + ) + assert request.sampled_rows is not None + assert request.sampled_rows.__class__ is tuple + with pytest.raises(ValidationError): + request.sampled_rows = None # type: ignore[misc] + + +def test_llm_request_schema_field_is_tuple_of_tuples() -> None: + request = _valid_llm_request() + assert request.schema.__class__ is tuple + for entry in request.schema: + assert entry.__class__ is tuple + assert len(entry) == 2 + assert isinstance(entry[0], str) + assert isinstance(entry[1], str) + + +def test_llm_request_docstring_warns_about_direct_construction() -> None: + assert LLMRequest.__doc__ is not None + assert "build_llm_request" in LLMRequest.__doc__ + assert "audit log" in LLMRequest.__doc__ + + +# --------------------------------------------------------------------------- +# Module surface +# --------------------------------------------------------------------------- + + +def test_module_all_lists_documented_classes() -> None: + from signalforge.safety import models as safety_models + + assert tuple(safety_models.__all__) == ( + "SamplingMode", + "RedactionReason", + "RedactionRecord", + "AuditEvent", + "LLMRequest", + ) From bc5f1d8c6952a75007355e9aac5f88304eafb33b Mon Sep 17 00:00:00 2001 From: wjduenow Date: Tue, 28 Apr 2026 22:42:30 -0700 Subject: [PATCH 09/19] bd_1-scaffolding-pfj: Add SafetyPolicy + _resolve_redact_patterns + _compute_policy_hash Frozen Pydantic v2 with extra=forbid; default mode=SCHEMA_ONLY; case-insensitive mode load; @model_validator resolves redact.extend/replace mutual exclusion; pattern-injection rejection (empty/*/?); with_mode() factory for #9's CLI; sample-mode WARNING; deterministic 16-hex policy hash for AuditEvent. Co-Authored-By: Claude Opus 4.7 (1M context) --- src/signalforge/safety/policy.py | 257 ++++++++++++++++++++++++++ tests/safety/test_policy.py | 297 +++++++++++++++++++++++++++++++ 2 files changed, 554 insertions(+) create mode 100644 src/signalforge/safety/policy.py create mode 100644 tests/safety/test_policy.py diff --git a/src/signalforge/safety/policy.py b/src/signalforge/safety/policy.py new file mode 100644 index 00000000..4082db6f --- /dev/null +++ b/src/signalforge/safety/policy.py @@ -0,0 +1,257 @@ +"""User-facing safety-policy model and helpers (US-005). + +Defines :class:`SafetyPolicy` — the config-shaped Pydantic v2 model that +mirrors the ``safety:`` block of ``signalforge.yml`` — plus two +underscore-prefixed helpers exercised by other safety-layer modules: + +* :func:`_resolve_redact_patterns` — applies ``redact: {extend|replace: [...]}`` + semantics on top of :data:`DEFAULT_REDACT_PATTERNS` (DEC-007). +* :func:`_compute_policy_hash` — deterministic 16-hex SHA-256 of the policy + used as :class:`signalforge.safety.models.AuditEvent.policy_hash` (DEC-014). + +Design commitments operationalised here: + +* **DEC-015** — The policy uses ``extra="forbid"`` (config-shaped: typos + must fail loud). Contrast with :mod:`signalforge.safety.models`, which + uses ``extra="ignore"`` (data-shaped: forward-compat for audit replay). +* **DEC-017** — A ``@model_validator(mode="before")`` resolves the + ``redact:`` block into ``redact_patterns`` *before* field validation, + so the typo-rejection in the ``redact:`` sub-keys uses the typed + :class:`signalforge.safety.errors.UnknownConfigKeyError` rather than + Pydantic's own ``extra="forbid"`` (which would only fire at the top + level). +* **DEC-018** — :meth:`SafetyPolicy.with_mode` is the canonical override + path for the CLI's ``--mode`` flag. Frozen Pydantic models cannot be + mutated; this helper hands back a fresh ``SafetyPolicy`` with the new + mode and every other field preserved. +* **DEC-021** — Constructing a policy with ``mode=SAMPLE`` emits a + WARNING via :data:`_LOGGER`. This fires once per construction; the + CLI's quiet flag is the user's escape hatch. +* **DEC-023** — Patterns ``""``, ``"*"`` and ``"?"`` are rejected at + construction time. ``"*"`` would silently disable the redactor; ``""`` + would never match anything; ``"?"`` would match every single-character + column. All three are footgun-shaped, so we raise + :class:`signalforge.safety.errors.InvalidPatternError` rather than + accept them. +* **DEC-024** — A ``@field_validator(mode="before")`` on ``mode`` accepts + any case + ``-`` / ``_`` mix (``"schema-only"`` / + ``"schema_only"`` / ``"Schema-Only"`` / …). Unknown values raise + :class:`signalforge.safety.errors.InvalidSamplingModeError`. +""" + +from __future__ import annotations + +import hashlib +import json +import logging +from pathlib import Path +from typing import Any, Final + +from pydantic import BaseModel, ConfigDict, field_validator, model_validator + +from signalforge.safety.errors import ( + InvalidConfigError, + InvalidPatternError, + InvalidSamplingModeError, + UnknownConfigKeyError, +) +from signalforge.safety.models import SamplingMode + +_LOGGER: Final = logging.getLogger("signalforge.safety") + + +DEFAULT_REDACT_PATTERNS: Final[tuple[str, ...]] = ( + "*email", + "email", + "*phone", + "phone", + "*ssn", + "ssn", +) +"""Six built-in redaction patterns. Case-insensitive ``fnmatch`` globs +matched lowercased in US-008's ``_matches_redaction_pattern``. Each PII +class is covered by both a prefixed form (``"*email"``) and a bare form +(``"email"``) so columns like ``user_email`` and ``email`` both match.""" + + +DEFAULT_AUDIT_PATH: Final[Path] = Path(".signalforge/audit.jsonl") +"""Default audit-log path, relative to ``project_dir``.""" + + +def _resolve_redact_patterns(block: Any) -> tuple[str, ...]: + """Resolve a raw ``redact:`` config block into a pattern tuple. + + The contract (DEC-007): + + * ``None`` or ``{}`` → built-in defaults. + * ``{"extend": [...]}`` → built-ins plus the user's additions. + * ``{"replace": [...]}`` → user's list verbatim. Empty list emits a + WARNING and disables redaction entirely. + * Both ``extend`` and ``replace`` set → :class:`InvalidConfigError`. + * Any other key → :class:`UnknownConfigKeyError` on the first + offender (sorted) so the message is deterministic. + + Module-level helper (not a method) so the same logic is reachable + from the ``model_validator(mode="before")`` and from ad-hoc tests. + """ + if block is None or block == {}: + return DEFAULT_REDACT_PATTERNS + if not isinstance(block, dict): + raise InvalidConfigError( + message=f"redact must be a mapping; got {type(block).__name__}", + ) + + has_extend = "extend" in block + has_replace = "replace" in block + if has_extend and has_replace: + raise InvalidConfigError( + message="redact cannot specify both extend and replace; choose one", + remediation=( + "Use redact.extend to append patterns to the built-ins; " + "use redact.replace to substitute them entirely." + ), + ) + + unknown = set(block) - {"extend", "replace"} + if unknown: + first = sorted(unknown)[0] + raise UnknownConfigKeyError(key=first, scope="safety.redact") + + if has_extend: + extra = block["extend"] + if not isinstance(extra, list): + raise InvalidConfigError( + message=f"redact.extend must be a list; got {type(extra).__name__}", + ) + return DEFAULT_REDACT_PATTERNS + tuple(extra) + + # has_replace branch (mutual exclusion enforced above). + replacements = block["replace"] + if not isinstance(replacements, list): + raise InvalidConfigError( + message=f"redact.replace must be a list; got {type(replacements).__name__}", + ) + if len(replacements) == 0: + _LOGGER.warning( + "redaction disabled: %s", + '{"message":"signalforge.yml: redact.replace=[] disables all redaction patterns"}', + ) + return tuple(replacements) + + +class SafetyPolicy(BaseModel): + """User-facing safety-policy config (DEC-015 + DEC-017 + DEC-018). + + Frozen Pydantic v2 model with ``extra="forbid"`` so typos at the + top level (``redacts:`` instead of ``redact:``) raise loudly. Field + defaults match SignalForge's "secure by default" posture: schema-only + mode, the six built-in redaction patterns, and a project-relative + audit path. + """ + + model_config = ConfigDict(frozen=True, extra="forbid", populate_by_name=True) + + mode: SamplingMode = SamplingMode.SCHEMA_ONLY + redact_patterns: tuple[str, ...] = DEFAULT_REDACT_PATTERNS + sample_size: int = 100 + audit_path: Path = DEFAULT_AUDIT_PATH + + @model_validator(mode="before") + @classmethod + def _resolve_redact_patterns_validator(cls, data: Any) -> Any: + """Translate ``redact: {extend|replace: [...]}`` into + ``redact_patterns`` before per-field validation runs. + + Only handles dict input. Already-built ``SafetyPolicy`` instances + and other shapes pass through untouched. + """ + if not isinstance(data, dict): + return data + if "redact" in data: + redact_block = data.pop("redact") + data["redact_patterns"] = _resolve_redact_patterns(redact_block) + return data + + @field_validator("mode", mode="before") + @classmethod + def _normalise_mode(cls, value: Any) -> Any: + """Accept any case + ``-`` / ``_`` mix; reject unknown values.""" + if isinstance(value, SamplingMode): + return value + if isinstance(value, str): + normalised = value.lower().replace("_", "-") + for member in SamplingMode: + if member.value == normalised: + return member + raise InvalidSamplingModeError( + value=value, + allowed=tuple(m.value for m in SamplingMode), + ) + return value + + @field_validator("redact_patterns") + @classmethod + def _validate_patterns(cls, patterns: tuple[str, ...]) -> tuple[str, ...]: + for p in patterns: + if p == "": + raise InvalidPatternError(value=p, reason="empty pattern") + if p == "*": + raise InvalidPatternError( + value=p, + reason=( + "matches all column names; use redact: {replace: []} " + "to disable redaction explicitly" + ), + ) + if p == "?": + raise InvalidPatternError( + value=p, + reason="matches every single-character column name", + ) + return patterns + + @model_validator(mode="after") + def _warn_on_sample_mode(self) -> SafetyPolicy: + if self.mode is SamplingMode.SAMPLE: + _LOGGER.warning( + "sample mode enabled: %s", + ( + '{"message":"Sample mode enabled — raw row data will be ' + 'sent to the LLM. Verify column tags/meta opt-outs."}' + ), + ) + return self + + def with_mode(self, mode: SamplingMode) -> SafetyPolicy: + """Return a new :class:`SafetyPolicy` with ``mode`` overridden. + + Used by issue #9's CLI to apply ``--mode`` after loading from + ``signalforge.yml``. Frozen Pydantic models cannot be mutated; + this is the canonical override path (DEC-018). + """ + return self.model_copy(update={"mode": mode}) + + +def _compute_policy_hash(policy: SafetyPolicy) -> str: + """Deterministic 16-hex-char SHA-256 of the policy (DEC-014). + + The hash is what :class:`signalforge.safety.models.AuditEvent.policy_hash` + carries for every audit row, letting future readers tell whether two + audits ran under the same policy without re-loading + ``signalforge.yml``. The double-serialise (``model_dump_json`` → + ``json.dumps(sort_keys=True)``) forces canonical key ordering since + Pydantic does not guarantee it. + + ``audit_path`` is dumped as a string by Pydantic; the hash is stable + across runs as long as the policy serialises identically. + """ + payload = policy.model_dump_json(by_alias=True, exclude_none=False) + canonical = json.dumps(json.loads(payload), sort_keys=True, separators=(",", ":")) + return hashlib.sha256(canonical.encode("utf-8")).hexdigest()[:16] + + +__all__ = [ + "DEFAULT_REDACT_PATTERNS", + "DEFAULT_AUDIT_PATH", + "SafetyPolicy", +] diff --git a/tests/safety/test_policy.py b/tests/safety/test_policy.py new file mode 100644 index 00000000..449ee993 --- /dev/null +++ b/tests/safety/test_policy.py @@ -0,0 +1,297 @@ +"""Tests for ``signalforge.safety.policy`` (US-005). + +Covers :class:`SafetyPolicy`, :func:`_resolve_redact_patterns`, and +:func:`_compute_policy_hash`. Traces to DEC-007 (built-ins + +override semantics), DEC-014 (``policy_hash``), DEC-017 +(``@model_validator``), DEC-018 (``with_mode``), DEC-021 (sample-mode +warning), DEC-023 (pattern-injection rejection), DEC-024 +(case-insensitive mode load). +""" + +from __future__ import annotations + +import logging +import re +from pathlib import Path + +import pytest +from pydantic import ValidationError + +from signalforge.safety.errors import ( + InvalidConfigError, + InvalidPatternError, + InvalidSamplingModeError, + UnknownConfigKeyError, +) +from signalforge.safety.models import SamplingMode +from signalforge.safety.policy import ( + DEFAULT_AUDIT_PATH, + DEFAULT_REDACT_PATTERNS, + SafetyPolicy, + _compute_policy_hash, + _resolve_redact_patterns, +) + +pytestmark = pytest.mark.safety + + +# --------------------------------------------------------------------------- +# Defaults +# --------------------------------------------------------------------------- + + +def test_safety_policy_no_args_is_schema_only() -> None: + """DEC-012(a) regression: default mode must be schema-only.""" + assert SafetyPolicy().mode is SamplingMode.SCHEMA_ONLY + + +def test_safety_policy_default_redact_patterns_are_six_builtins() -> None: + policy = SafetyPolicy() + assert policy.redact_patterns == DEFAULT_REDACT_PATTERNS + assert len(policy.redact_patterns) == 6 + + +def test_safety_policy_default_sample_size_is_100() -> None: + assert SafetyPolicy().sample_size == 100 + + +def test_safety_policy_default_audit_path() -> None: + assert SafetyPolicy().audit_path == Path(".signalforge/audit.jsonl") + assert DEFAULT_AUDIT_PATH == Path(".signalforge/audit.jsonl") # noqa: SIM300 + + +# --------------------------------------------------------------------------- +# extra="forbid" +# --------------------------------------------------------------------------- + + +def test_safety_policy_extra_forbid_rejects_typo() -> None: + """DEC-015: typos must fail loud at the config-shaped surface.""" + with pytest.raises(ValidationError): + SafetyPolicy.model_validate({"redacts": {"extend": ["*foo"]}}) + + +# --------------------------------------------------------------------------- +# redact: extend / replace +# --------------------------------------------------------------------------- + + +def test_safety_policy_redact_extend_appends_to_builtins() -> None: + policy = SafetyPolicy.model_validate({"redact": {"extend": ["*custom"]}}) + assert policy.redact_patterns[-1] == "*custom" + for builtin in DEFAULT_REDACT_PATTERNS: + assert builtin in policy.redact_patterns + assert len(policy.redact_patterns) == len(DEFAULT_REDACT_PATTERNS) + 1 + + +def test_safety_policy_redact_replace_substitutes() -> None: + policy = SafetyPolicy.model_validate({"redact": {"replace": ["*specific"]}}) + assert policy.redact_patterns == ("*specific",) + + +def test_safety_policy_redact_extend_and_replace_simultaneously_errors() -> None: + with pytest.raises(InvalidConfigError): + SafetyPolicy.model_validate({"redact": {"extend": ["*x"], "replace": ["*y"]}}) + + +def test_safety_policy_redact_replace_empty_warns_once_and_returns_empty( + caplog: pytest.LogCaptureFixture, +) -> None: + with caplog.at_level(logging.WARNING, logger="signalforge.safety"): + policy = SafetyPolicy.model_validate({"redact": {"replace": []}}) + assert policy.redact_patterns == () + warning_records = [ + r for r in caplog.records if r.name == "signalforge.safety" and r.levelno == logging.WARNING + ] + assert len(warning_records) == 1 + assert "redact.replace=[]" in warning_records[0].getMessage() + + +def test_safety_policy_redact_unknown_key_raises_unknown_config_key_error() -> None: + with pytest.raises(UnknownConfigKeyError): + SafetyPolicy.model_validate({"redact": {"phantom": ["x"]}}) + + +# --------------------------------------------------------------------------- +# Mode case-insensitive load +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "raw", + [ + "schema-only", + "Schema-Only", + "SCHEMA-ONLY", + "schema_only", + "SCHEMA_ONLY", + "Schema_Only", + ], +) +def test_safety_policy_mode_case_insensitive_load(raw: str) -> None: + policy = SafetyPolicy.model_validate({"mode": raw}) + assert policy.mode is SamplingMode.SCHEMA_ONLY + + +def test_safety_policy_mode_unknown_raises_invalid_sampling_mode_error() -> None: + with pytest.raises(InvalidSamplingModeError): + SafetyPolicy.model_validate({"mode": "phantom"}) + + +# --------------------------------------------------------------------------- +# Pattern-injection rejection (DEC-023) +# --------------------------------------------------------------------------- + + +def test_safety_policy_pattern_empty_raises_invalid_pattern_error() -> None: + with pytest.raises(InvalidPatternError): + SafetyPolicy(redact_patterns=("",)) + + +def test_safety_policy_pattern_star_alone_raises_invalid_pattern_error() -> None: + with pytest.raises(InvalidPatternError): + SafetyPolicy(redact_patterns=("*",)) + + +def test_safety_policy_pattern_question_mark_alone_raises_invalid_pattern_error() -> None: + with pytest.raises(InvalidPatternError): + SafetyPolicy(redact_patterns=("?",)) + + +def test_safety_policy_pattern_mixed_invalid_first_raises() -> None: + """Empty pattern must be caught even when not at index 0.""" + with pytest.raises(InvalidPatternError): + SafetyPolicy(redact_patterns=("foo", "")) + + +# --------------------------------------------------------------------------- +# with_mode (DEC-018) +# --------------------------------------------------------------------------- + + +def test_safety_policy_with_mode_returns_new_frozen_policy() -> None: + policy = SafetyPolicy() + overridden = policy.with_mode(SamplingMode.SAMPLE) + assert overridden is not policy + assert overridden.mode is SamplingMode.SAMPLE + assert policy.mode is SamplingMode.SCHEMA_ONLY + + +def test_safety_policy_with_mode_preserves_other_fields() -> None: + policy = SafetyPolicy( + redact_patterns=("*custom",), + sample_size=42, + audit_path=Path("custom/audit.jsonl"), + ) + overridden = policy.with_mode(SamplingMode.AGGREGATE_ONLY) + assert overridden.redact_patterns == ("*custom",) + assert overridden.sample_size == 42 + assert overridden.audit_path == Path("custom/audit.jsonl") + assert overridden.mode is SamplingMode.AGGREGATE_ONLY + + +# --------------------------------------------------------------------------- +# Sample-mode warning (DEC-021) +# --------------------------------------------------------------------------- + + +def test_safety_policy_sample_mode_emits_warning_on_construction( + caplog: pytest.LogCaptureFixture, +) -> None: + with caplog.at_level(logging.WARNING, logger="signalforge.safety"): + SafetyPolicy(mode=SamplingMode.SAMPLE) + warning_records = [ + r for r in caplog.records if r.name == "signalforge.safety" and r.levelno == logging.WARNING + ] + assert len(warning_records) == 1 + assert "Sample mode enabled" in warning_records[0].getMessage() + + +def test_safety_policy_schema_only_mode_no_warning( + caplog: pytest.LogCaptureFixture, +) -> None: + with caplog.at_level(logging.WARNING, logger="signalforge.safety"): + SafetyPolicy(mode=SamplingMode.SCHEMA_ONLY) + warning_records = [ + r for r in caplog.records if r.name == "signalforge.safety" and r.levelno == logging.WARNING + ] + assert warning_records == [] + + +# --------------------------------------------------------------------------- +# Frozen +# --------------------------------------------------------------------------- + + +def test_safety_policy_is_frozen() -> None: + policy = SafetyPolicy() + with pytest.raises(ValidationError): + policy.mode = SamplingMode.SAMPLE # type: ignore[misc] + + +# --------------------------------------------------------------------------- +# _compute_policy_hash +# --------------------------------------------------------------------------- + + +def test_compute_policy_hash_deterministic_for_equal_policies() -> None: + policy_a = SafetyPolicy() + policy_b = SafetyPolicy() + assert _compute_policy_hash(policy_a) == _compute_policy_hash(policy_b) + assert _compute_policy_hash(policy_a) == _compute_policy_hash(policy_a) + + +def test_compute_policy_hash_differs_for_semantically_different_policies() -> None: + policy_a = SafetyPolicy() + policy_b = SafetyPolicy(mode=SamplingMode.AGGREGATE_ONLY) + assert _compute_policy_hash(policy_a) != _compute_policy_hash(policy_b) + + +def test_compute_policy_hash_differs_when_redact_patterns_differ() -> None: + policy_a = SafetyPolicy() + policy_b = SafetyPolicy(redact_patterns=("*foo",)) + assert _compute_policy_hash(policy_a) != _compute_policy_hash(policy_b) + + +def test_compute_policy_hash_returns_16_hex_chars() -> None: + digest = _compute_policy_hash(SafetyPolicy()) + assert len(digest) == 16 + assert re.fullmatch(r"[0-9a-f]{16}", digest) is not None + + +# --------------------------------------------------------------------------- +# _resolve_redact_patterns +# --------------------------------------------------------------------------- + + +def test_resolve_redact_patterns_none_returns_defaults() -> None: + assert _resolve_redact_patterns(None) == DEFAULT_REDACT_PATTERNS + + +def test_resolve_redact_patterns_empty_dict_returns_defaults() -> None: + assert _resolve_redact_patterns({}) == DEFAULT_REDACT_PATTERNS + + +def test_resolve_redact_patterns_extend_with_non_list_raises() -> None: + with pytest.raises(InvalidConfigError): + _resolve_redact_patterns({"extend": "*foo"}) + + +def test_resolve_redact_patterns_replace_with_non_list_raises() -> None: + with pytest.raises(InvalidConfigError): + _resolve_redact_patterns({"replace": "*foo"}) + + +# --------------------------------------------------------------------------- +# __all__ +# --------------------------------------------------------------------------- + + +def test_module_all_lists_documented_names() -> None: + from signalforge.safety import policy as policy_module + + assert set(policy_module.__all__) == { + "DEFAULT_REDACT_PATTERNS", + "DEFAULT_AUDIT_PATH", + "SafetyPolicy", + } From 01fc6fb333e694d32255f94150960839955269ad Mon Sep 17 00:00:00 2001 From: wjduenow Date: Tue, 28 Apr 2026 22:43:41 -0700 Subject: [PATCH 10/19] bd_1-scaffolding-mg2: Add signalforge.safety.audit (fail-closed JSONL writer) O_APPEND atomic-append with fsync; mkdir parent at 0o700; file at 0o600. PIPE_BUF size cap (4000 bytes) raises AuditRecordTooLargeError. Any I/O exception propagates as AuditWriteError (DEC-011 fail-closed). Logger uses lazy-format with json.dumps to avoid ANSI/log-injection (DEC-022). Co-Authored-By: Claude Opus 4.7 (1M context) --- src/signalforge/safety/audit.py | 131 ++++++++++++++++ tests/safety/test_audit.py | 268 ++++++++++++++++++++++++++++++++ 2 files changed, 399 insertions(+) create mode 100644 src/signalforge/safety/audit.py create mode 100644 tests/safety/test_audit.py diff --git a/src/signalforge/safety/audit.py b/src/signalforge/safety/audit.py new file mode 100644 index 00000000..bf3e7753 --- /dev/null +++ b/src/signalforge/safety/audit.py @@ -0,0 +1,131 @@ +"""Fail-closed JSONL audit-log writer for the safety layer (US-007). + +This module is the safety layer's single observability seam. Every LLM call +the request builder makes lands here as exactly one JSONL line; any I/O +failure aborts the call (DEC-011 fail-closed) so the system never proceeds +without an audit trail. + +Three load-bearing properties: + +* **Atomic concurrent appends** (DEC-005). The writer uses ``os.open`` with + ``O_APPEND`` and writes the full record in a single ``os.write`` call. POSIX + guarantees ``write(2)`` is atomic up to ``PIPE_BUF`` (4 KiB on Linux); the + module-level :data:`_AUDIT_RECORD_LIMIT_BYTES` enforces a 4000-byte cap with + a 96-byte margin so concurrent writers cannot interleave partial records. +* **Fail-closed on every error** (DEC-011). Serialisation errors, ``mkdir`` + failures, ``open`` failures, ``write`` / ``fsync`` failures all propagate + as :class:`AuditWriteError`. Oversize records propagate as + :class:`AuditRecordTooLargeError`. Callers (the request builder in US-008) + must abort on either, never swallow. +* **ANSI-safe lazy-format logger** (DEC-022). The summary line is logged via + ``%s`` lazy-format with ``json.dumps`` of the user-controlled fields. f-string + interpolation here would let a crafted ``model_unique_id`` containing raw + ANSI escapes pollute the log viewer; routing through ``json.dumps`` escapes + control characters as ``\\uXXXX`` and quotes the strings, closing the + log-injection seam. + +The "no logging in stage-0 modules" rule from ``manifest-readers.md`` does NOT +apply here — this module *is* the observability stage. INFO-level logging is +its job, not noise. +""" + +from __future__ import annotations + +import contextlib +import json +import logging +import os +from pathlib import Path +from typing import TYPE_CHECKING, Final + +if TYPE_CHECKING: + from signalforge.safety.models import AuditEvent + +_LOGGER: Final = logging.getLogger("signalforge.safety") + +# POSIX guarantees ``write(2)`` is atomic only up to ``PIPE_BUF`` bytes +# (typically 4096 on Linux). The 96-byte margin leaves room for trailing +# newline plus any line-buffering / kernel overhead so the module's atomic- +# concurrent-append contract holds even at the size cap. +_AUDIT_RECORD_LIMIT_BYTES: Final[int] = 4000 + + +def write(event: AuditEvent, audit_path: Path) -> None: + """Append a single JSONL record to ``audit_path``. Fail-closed. + + Args: + event: the :class:`~signalforge.safety.models.AuditEvent` to persist. + audit_path: absolute or project-relative path; the parent directory + is created with mode ``0o700`` if missing, and the audit file + itself is created with mode ``0o600`` on first call. + + Raises: + AuditWriteError: any underlying ``OSError`` / ``PermissionError`` / + ``IOError``, or a JSON-encoding failure. DEC-011 fail-closed: + callers (the request builder) abort the LLM call rather than + proceed without an audit record. + AuditRecordTooLargeError: the serialised line exceeds the + POSIX-atomic-append size cap (DEC-022); reduce ``columns_sent`` + or ``redactions`` count. + """ + # Local import keeps ``audit`` importable without forcing the errors module + # at module-eval time and matches the style used elsewhere in the package. + from signalforge.safety.errors import AuditRecordTooLargeError, AuditWriteError + + # Serialise the event. Any encoding error (e.g. unserialisable custom + # type smuggled through ``model_construct``) becomes ``AuditWriteError``. + try: + payload = event.model_dump(mode="json") + line = json.dumps(payload, separators=(",", ":")) + "\n" + except Exception as exc: + raise AuditWriteError(path=audit_path, cause=exc) from exc + + encoded = line.encode("utf-8") + if len(encoded) > _AUDIT_RECORD_LIMIT_BYTES: + raise AuditRecordTooLargeError(size=len(encoded), limit=_AUDIT_RECORD_LIMIT_BYTES) + + # Ensure parent dir exists with private permissions. ``mode=0o700`` is the + # umask-respecting permission used at *creation* time; an existing dir is + # left alone (``exist_ok=True``). + try: + audit_path.parent.mkdir(parents=True, exist_ok=True, mode=0o700) + except OSError as exc: + raise AuditWriteError(path=audit_path, cause=exc) from exc + + # ``O_APPEND`` gives atomic concurrent appends; ``O_CREAT`` handles the + # first call; ``0o600`` keeps the file owner-only. + fd = -1 + try: + fd = os.open(str(audit_path), os.O_WRONLY | os.O_APPEND | os.O_CREAT, 0o600) + os.write(fd, encoded) + os.fsync(fd) + except OSError as exc: + raise AuditWriteError(path=audit_path, cause=exc) from exc + finally: + if fd >= 0: + # Best-effort close; the write/fsync above already succeeded + # (or raised), so a close failure here would only mask the + # real outcome. + with contextlib.suppress(OSError): + os.close(fd) + + # ANSI-safe lazy-format summary. The summary fields are user-controlled + # (``model_unique_id`` ultimately comes from a dbt manifest) so they + # MUST go through ``json.dumps`` rather than f-string interpolation — + # ``json.dumps`` escapes ANSI / control bytes as ``\uXXXX`` so a crafted + # value cannot smuggle terminal escape sequences into a log viewer. + _LOGGER.info( + "audit event: %s", + json.dumps( + { + "unique_id": event.model_unique_id, + "mode": event.mode.value, + "columns_sent": len(event.columns_sent), + "redacted": len(event.redactions), + "audit_schema_version": event.audit_schema_version, + } + ), + ) + + +__all__ = ["write", "_AUDIT_RECORD_LIMIT_BYTES"] diff --git a/tests/safety/test_audit.py b/tests/safety/test_audit.py new file mode 100644 index 00000000..5480c19f --- /dev/null +++ b/tests/safety/test_audit.py @@ -0,0 +1,268 @@ +"""Tests for ``signalforge.safety.audit`` (US-007). + +The audit module is the safety layer's single observability seam (DEC-005, +DEC-011, DEC-022): it appends one JSONL record per LLM call, fail-closed, with +a POSIX-atomic-append size cap and an ANSI-safe lazy-format logger. These +tests exercise real I/O on ``tmp_path`` because, per the testing-strategy +review, mocks of ``open`` hide buffering bugs that the real syscall surface +exposes. +""" + +from __future__ import annotations + +import json +import logging +import os +import stat +import sys +from concurrent.futures import ThreadPoolExecutor +from datetime import datetime, timezone +from pathlib import Path +from typing import Any + +import pytest + +from signalforge.safety.audit import write +from signalforge.safety.errors import AuditRecordTooLargeError, AuditWriteError +from signalforge.safety.models import AuditEvent, SamplingMode + +pytestmark = pytest.mark.safety + + +def _make_event(**overrides: Any) -> AuditEvent: + base: dict[str, Any] = dict( + timestamp=datetime(2026, 1, 1, tzinfo=timezone.utc), + model_unique_id="model.test.x", + mode=SamplingMode.SCHEMA_ONLY, + columns_sent=("id", "name"), + redactions=(), + row_count=None, + signalforge_version="0.1.0", + policy_hash="abc123def456789a", + audit_schema_version=1, + policy_flags=(), + ) + base.update(overrides) + return AuditEvent(**base) + + +def test_audit_write_appends_one_jsonl_line(tmp_path: Path) -> None: + audit_path = tmp_path / "audit.jsonl" + write(_make_event(), audit_path) + assert audit_path.exists() + contents = audit_path.read_text(encoding="utf-8") + assert contents.endswith("\n") + lines = contents.splitlines() + assert len(lines) == 1 + json.loads(lines[0]) # parses + + +def test_audit_write_round_trips_through_json_loads(tmp_path: Path) -> None: + audit_path = tmp_path / "audit.jsonl" + write(_make_event(), audit_path) + payload = json.loads(audit_path.read_text(encoding="utf-8").splitlines()[0]) + assert payload["model_unique_id"] == "model.test.x" + assert payload["mode"] == SamplingMode.SCHEMA_ONLY.value + assert payload["columns_sent"] == ["id", "name"] + assert payload["redactions"] == [] + assert payload["audit_schema_version"] == 1 + assert payload["signalforge_version"] == "0.1.0" + assert payload["policy_hash"] == "abc123def456789a" + + +def test_audit_write_creates_parent_dir_with_mode_0o700(tmp_path: Path) -> None: + audit_path = tmp_path / ".signalforge" / "audit.jsonl" + assert not audit_path.parent.exists() + write(_make_event(), audit_path) + assert audit_path.parent.is_dir() + mode = audit_path.parent.stat().st_mode & 0o777 + # Be lenient: assert group/other bits are zero. + assert mode & 0o077 == 0 + # And owner has read/write/exec at minimum. + assert mode & 0o700 == 0o700 + + +def test_audit_write_two_calls_two_lines(tmp_path: Path) -> None: + audit_path = tmp_path / "audit.jsonl" + write(_make_event(model_unique_id="model.test.a"), audit_path) + write(_make_event(model_unique_id="model.test.b"), audit_path) + lines = audit_path.read_text(encoding="utf-8").splitlines() + assert len(lines) == 2 + parsed = [json.loads(line) for line in lines] + assert [p["model_unique_id"] for p in parsed] == ["model.test.a", "model.test.b"] + + +def test_audit_write_emits_logger_info_line( + tmp_path: Path, + caplog: pytest.LogCaptureFixture, +) -> None: + audit_path = tmp_path / "audit.jsonl" + with caplog.at_level(logging.INFO, logger="signalforge.safety"): + write(_make_event(), audit_path) + + records = [r for r in caplog.records if r.name == "signalforge.safety"] + assert len(records) == 1 + msg = records[0].getMessage() + # Summary JSON should embed the key fields. The summary uses ``unique_id`` + # rather than the full ``model_unique_id`` field name to keep the line + # short — both name and value are present. + assert "model.test.x" in msg + assert f'"mode": "{SamplingMode.SCHEMA_ONLY.value}"' in msg + assert '"columns_sent": 2' in msg + assert '"redacted": 0' in msg + assert '"audit_schema_version": 1' in msg + + +def test_audit_write_logger_message_escapes_ansi_in_user_input( + tmp_path: Path, + caplog: pytest.LogCaptureFixture, +) -> None: + audit_path = tmp_path / "audit.jsonl" + nasty = "\x1b[31mFAKE\x1b[0m" + with caplog.at_level(logging.INFO, logger="signalforge.safety"): + write(_make_event(model_unique_id=nasty), audit_path) + + records = [r for r in caplog.records if r.name == "signalforge.safety"] + assert len(records) == 1 + raw_msg = records[0].getMessage() + # The raw ANSI escape byte (ESC, 0x1b) must NOT appear in the rendered + # log message — json.dumps escapes it as . + assert "\x1b" not in raw_msg + assert "\\u001b" in raw_msg + + +@pytest.mark.skipif(sys.platform == "win32", reason="POSIX-only permission semantics") +def test_audit_write_failure_on_unwritable_parent_raises_audit_write_error( + tmp_path: Path, +) -> None: + # Skip when running as root because root bypasses permission checks. + if hasattr(os, "geteuid") and os.geteuid() == 0: + pytest.skip("root bypasses POSIX permission checks") + + locked = tmp_path / "locked" + locked.mkdir() + audit_path = locked / "denied" / "audit.jsonl" + locked.chmod(0o000) + try: + with pytest.raises(AuditWriteError) as excinfo: + write(_make_event(), audit_path) + assert isinstance(excinfo.value.cause, OSError) + finally: + # Restore so tmp_path cleanup can succeed. + locked.chmod(0o700) + + +def test_audit_write_oversize_record_raises_too_large( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr("signalforge.safety.audit._AUDIT_RECORD_LIMIT_BYTES", 50) + audit_path = tmp_path / "audit.jsonl" + with pytest.raises(AuditRecordTooLargeError) as excinfo: + write(_make_event(), audit_path) + assert excinfo.value.limit == 50 + assert excinfo.value.size > 50 + + +def test_audit_write_oversize_does_not_create_file( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr("signalforge.safety.audit._AUDIT_RECORD_LIMIT_BYTES", 50) + audit_path = tmp_path / "audit.jsonl" + with pytest.raises(AuditRecordTooLargeError): + write(_make_event(), audit_path) + assert not audit_path.exists() + + +def test_audit_write_concurrent_threads_no_interleave(tmp_path: Path) -> None: + audit_path = tmp_path / "audit.jsonl" + + def writer(thread_idx: int) -> None: + for i in range(50): + write( + _make_event(model_unique_id=f"thread.{thread_idx}.row.{i}"), + audit_path, + ) + + with ThreadPoolExecutor(max_workers=10) as ex: + list(ex.map(writer, range(10))) + + lines = audit_path.read_text(encoding="utf-8").splitlines() + assert len(lines) == 500 + parsed = [json.loads(line) for line in lines] + unique_ids = {p["model_unique_id"] for p in parsed} + assert len(unique_ids) == 500 + + +def test_audit_write_does_not_swallow_exceptions( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + audit_path = tmp_path / "audit.jsonl" + + def boom(fd: int, data: bytes) -> int: # pragma: no cover - patched out + raise OSError("simulated write failure") + + monkeypatch.setattr("signalforge.safety.audit.os.write", boom) + with pytest.raises(AuditWriteError) as excinfo: + write(_make_event(), audit_path) + assert isinstance(excinfo.value.cause, OSError) + + +def test_audit_write_serialisation_failure_propagates_as_audit_write_error( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + audit_path = tmp_path / "audit.jsonl" + + def boom(*_args: Any, **_kwargs: Any) -> str: + raise TypeError("simulated json failure") + + # Force the json.dumps call inside audit.write to fail. + monkeypatch.setattr("signalforge.safety.audit.json.dumps", boom) + with pytest.raises(AuditWriteError) as excinfo: + write(_make_event(), audit_path) + assert isinstance(excinfo.value.cause, TypeError) + + +def test_audit_write_fsyncs_before_close( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + audit_path = tmp_path / "audit.jsonl" + calls: list[int] = [] + real_fsync = os.fsync + + def record_fsync(fd: int) -> None: + calls.append(fd) + real_fsync(fd) + + monkeypatch.setattr("signalforge.safety.audit.os.fsync", record_fsync) + write(_make_event(), audit_path) + assert len(calls) == 1 + assert calls[0] >= 0 + + +def test_audit_write_logger_includes_audit_schema_version( + tmp_path: Path, + caplog: pytest.LogCaptureFixture, +) -> None: + audit_path = tmp_path / "audit.jsonl" + with caplog.at_level(logging.INFO, logger="signalforge.safety"): + write(_make_event(), audit_path) + records = [r for r in caplog.records if r.name == "signalforge.safety"] + assert len(records) == 1 + assert '"audit_schema_version": 1' in records[0].getMessage() + + +@pytest.mark.skipif(sys.platform == "win32", reason="POSIX-only permission semantics") +def test_audit_write_file_perms_0o600_when_newly_created(tmp_path: Path) -> None: + audit_path = tmp_path / "audit.jsonl" + write(_make_event(), audit_path) + mode = audit_path.stat().st_mode + # Lenient assertion (umask interactions): no group/other bits. + assert mode & 0o077 == 0 + # And it is a regular file with owner read/write. + assert stat.S_ISREG(mode) + assert mode & 0o600 == 0o600 From 1f850c72a78f579af83c53fe4dbac4212796063b Mon Sep 17 00:00:00 2001 From: wjduenow Date: Tue, 28 Apr 2026 22:47:07 -0700 Subject: [PATCH 11/19] bd_1-scaffolding-ly6: Add signalforge.safety.config (load_safety_config + path safety) Full DEC-016 error contract: explicit-path miss raises ConfigNotFoundError; implicit miss / empty / missing-safety-key falls through to defaults; malformed YAML / non-mapping / schema-invalid raise typed errors. yaml.safe_load only. audit_path canonicalised + reject .. segments + require inside project_dir (DEC-013). _path_safety.py copied from warehouse layer per duplication precedent. --- src/signalforge/safety/_path_safety.py | 83 ++++++++ src/signalforge/safety/config.py | 184 ++++++++++++++++ tests/safety/test_config.py | 277 +++++++++++++++++++++++++ 3 files changed, 544 insertions(+) create mode 100644 src/signalforge/safety/_path_safety.py create mode 100644 src/signalforge/safety/config.py create mode 100644 tests/safety/test_config.py diff --git a/src/signalforge/safety/_path_safety.py b/src/signalforge/safety/_path_safety.py new file mode 100644 index 00000000..9dcc3a59 --- /dev/null +++ b/src/signalforge/safety/_path_safety.py @@ -0,0 +1,83 @@ +"""Symlink-hardened path canonicalisation for the safety subpackage. + +This is the safety-side counterpart to +:func:`signalforge.warehouse._path_safety.canonicalise_path` and +:func:`signalforge.manifest.loader._canonicalise_path`. Per the +``warehouse-adapters.md`` rule "Path safety: duplicated, not extracted", +each layer keeps its own copy so the layer's catch surface stays +homogeneous: every "we couldn't load the safety config" condition raises +:class:`signalforge.safety.errors.InvalidConfigError`. + +The three traps from ``manifest-readers.md`` apply: + +1. Resolve symlinks before checking containment + (:meth:`pathlib.Path.relative_to` does **not** follow symlinks). +2. Catch :class:`RuntimeError` from :meth:`pathlib.Path.resolve` — it raises + on symlink cycles regardless of the ``strict=`` flag. +3. Apply the same gate to the *default* path the loader chooses, not just to + user-supplied overrides — convention is not a security boundary. +""" + +from __future__ import annotations + +from pathlib import Path + +from signalforge.safety.errors import InvalidConfigError + + +def canonicalise_path(input_path: Path | str, project_dir: Path) -> Path: + """Resolve ``input_path`` to an absolute path inside ``project_dir``'s tree. + + Both the input path and ``project_dir`` are run through + :meth:`pathlib.Path.resolve` so symlinks are followed before the + containment check. The default-path argument the caller supplies must + flow through this helper too — DEC-013's hardening only holds when both + user-supplied and default paths use the same gate. + + Raises: + InvalidConfigError: when the resolved path escapes ``project_dir`` + (so a symlink pointing at ``/etc/passwd`` cannot be reached via a + relative input), when either path traverses a symlink cycle + (``Path.resolve`` raises :class:`RuntimeError` on cycles + regardless of ``strict=``), or when ``project_dir`` itself does + not exist or is not a directory. + """ + p = Path(input_path) + try: + project_resolved = project_dir.resolve(strict=True) + except RuntimeError as exc: + raise InvalidConfigError( + message=f"project_dir {project_dir} contains a symlink loop", + remediation=( + f"project_dir {project_dir} contains a symlink loop; resolve the " + "loop or pass a different project directory." + ), + ) from exc + except (FileNotFoundError, NotADirectoryError) as exc: + raise InvalidConfigError( + message=f"project_dir {project_dir} does not exist or is not a directory", + remediation=(f"project_dir {project_dir} does not exist or is not a directory."), + ) from exc + + if not p.is_absolute(): + p = project_resolved / p + try: + resolved = p.resolve(strict=False) + except RuntimeError as exc: + raise InvalidConfigError( + message=f"Path {p} contains a symlink loop", + remediation=( + f"Path {p} contains a symlink loop; resolve the loop or remove " + "the offending symlink." + ), + ) from exc + if not resolved.is_relative_to(project_resolved): + raise InvalidConfigError( + message=f"Path {resolved} escapes project_dir {project_resolved}", + remediation=( + f"Path {resolved} escapes project_dir {project_resolved}; " + "audit_path must point inside the project tree " + "(default: .signalforge/audit.jsonl)." + ), + ) + return resolved diff --git a/src/signalforge/safety/config.py b/src/signalforge/safety/config.py new file mode 100644 index 00000000..f114b5dd --- /dev/null +++ b/src/signalforge/safety/config.py @@ -0,0 +1,184 @@ +"""Safety-config loader for ``signalforge.yml`` (US-006). + +Implements DEC-016 (the full error contract for resolution / parsing) and +DEC-013 (symlink-hardened ``audit_path`` containment check) on top of the +:class:`signalforge.safety.policy.SafetyPolicy` model. + +Resolution order (DEC-016): + +1. If ``path=`` is explicit, the file MUST exist; missing → + :class:`signalforge.safety.errors.ConfigNotFoundError`. +2. Else ``/signalforge.yml``. Missing → defaults silently. +3. Empty file (zero bytes / whitespace-only / only YAML comments) → + defaults (DEBUG log). +4. Non-mapping top-level (YAML list, scalar, …) → + :class:`signalforge.safety.errors.InvalidConfigError`. +5. Missing ``safety:`` key → defaults (other top-level keys reserved per + DEC-025 namespace). +6. Schema-invalid contents → typed errors + (:class:`signalforge.safety.errors.InvalidSamplingModeError`, + :class:`signalforge.safety.errors.InvalidPatternError`, + :class:`signalforge.safety.errors.UnknownConfigKeyError`, + :class:`signalforge.safety.errors.PolicyValidationError`). + +``audit_path`` is pre-validated (reject ``..`` segments) and routed through +:func:`signalforge.safety._path_safety.canonicalise_path` before being +handed to :class:`SafetyPolicy`. Both user-supplied and default paths flow +through the same gate. + +``yaml.safe_load`` only — ``yaml.load`` accepts arbitrary Python object +construction tags and is unsafe for any input we don't fully control. +""" + +from __future__ import annotations + +import logging +from pathlib import Path +from typing import Final + +import yaml +from pydantic import ValidationError + +from signalforge.safety._path_safety import canonicalise_path +from signalforge.safety.errors import ( + ConfigNotFoundError, + InvalidConfigError, + InvalidPatternError, + InvalidSamplingModeError, + PolicyValidationError, + UnknownConfigKeyError, +) +from signalforge.safety.policy import SafetyPolicy + +_LOGGER: Final = logging.getLogger("signalforge.safety") +_DEFAULT_CONFIG_FILENAME: Final = "signalforge.yml" + + +def load_safety_config(project_dir: Path, path: Path | None = None) -> SafetyPolicy: + """Load a :class:`SafetyPolicy` from ``signalforge.yml``. + + See module docstring for the full resolution order and error contract. + + Args: + project_dir: Project root used both as the base for the default + config-file lookup (``/signalforge.yml``) and as + the containment boundary for ``audit_path``. + path: Optional explicit config path. When given the file must + exist; missing raises :class:`ConfigNotFoundError`. + + Returns: + A fully-validated :class:`SafetyPolicy`. + + Raises: + ConfigNotFoundError: Explicit ``path=`` was given and the file does + not exist. + InvalidConfigError: The file is not valid YAML, its top level is + not a mapping, the ``safety:`` block is not a mapping, or + ``audit_path`` contains ``..`` segments / escapes + ``project_dir`` / traverses a symlink loop. + InvalidSamplingModeError: ``safety.mode`` is not a known sampling + mode. + InvalidPatternError: A redact pattern is empty or one of the bare + wildcards ``"*"`` / ``"?"``. + UnknownConfigKeyError: An unknown key was found under a known + scope (e.g. ``safety.redacts:`` instead of + ``safety.redact:``, or top-level ``safety.foo:``). + PolicyValidationError: Generic Pydantic validation failure not + covered by the more specific exceptions above. + """ + if path is not None: + config_file = path + if not config_file.exists(): + raise ConfigNotFoundError(path=config_file) + else: + config_file = project_dir / _DEFAULT_CONFIG_FILENAME + if not config_file.exists(): + return SafetyPolicy() + + raw_text = config_file.read_text(encoding="utf-8").strip() + if not raw_text: + _LOGGER.debug("safety config file %r is empty; using defaults", str(config_file)) + return SafetyPolicy() + + try: + loaded = yaml.safe_load(raw_text) + except yaml.YAMLError as exc: + raise InvalidConfigError( + message=f"signalforge.yml is not valid YAML: {exc}", + ) from exc + + if loaded is None: + # File parses to None (e.g. only comments) — same as empty. + return SafetyPolicy() + + if not isinstance(loaded, dict): + raise InvalidConfigError( + message=(f"signalforge.yml top level must be a mapping; got {type(loaded).__name__}"), + ) + + safety_block = loaded.get("safety") + if safety_block is None: + # Missing safety: key — other top-level keys reserved per DEC-025. + return SafetyPolicy() + + if not isinstance(safety_block, dict): + raise InvalidConfigError( + message=( + f"signalforge.yml: 'safety' must be a mapping; got {type(safety_block).__name__}" + ), + ) + + # Pre-validate audit_path (DEC-013): reject `..` segments outright, + # then canonicalise + containment-check via the symlink-hardened helper. + audit_path_raw = safety_block.get("audit_path") + if audit_path_raw is not None: + candidate = Path(audit_path_raw) + if any(part == ".." for part in candidate.parts): + raise InvalidConfigError( + message=(f"audit_path may not contain '..' segments; got {audit_path_raw!r}"), + remediation=( + "Use a path inside the project directory (default: .signalforge/audit.jsonl)." + ), + ) + resolved = canonicalise_path(audit_path_raw, project_dir) + # Replace the raw value with the resolved Path so SafetyPolicy + # stores the canonical form. + safety_block = {**safety_block, "audit_path": resolved} + + try: + return SafetyPolicy.model_validate(safety_block) + except ( + InvalidSamplingModeError, + InvalidPatternError, + InvalidConfigError, + UnknownConfigKeyError, + ): + # Validators in SafetyPolicy raise our typed exceptions directly; + # let those propagate without being wrapped. + raise + except ValidationError as exc: + # Walk the error list and surface the most specific safety-layer + # exception attached to a Pydantic error context, if any. + for err in exc.errors(): + ctx = err.get("ctx", {}) or {} + inner = ctx.get("error") if isinstance(ctx, dict) else None + if isinstance( + inner, + ( + InvalidSamplingModeError, + InvalidPatternError, + InvalidConfigError, + UnknownConfigKeyError, + ), + ): + raise inner from exc + # Last-resort wrap: surface the Pydantic failure as a typed + # safety-layer error so callers can pattern-match. + raise PolicyValidationError( + field="", + value=safety_block, + reason=str(exc), + ) from exc + + +__all__ = ["load_safety_config"] diff --git a/tests/safety/test_config.py b/tests/safety/test_config.py new file mode 100644 index 00000000..fcbe17e1 --- /dev/null +++ b/tests/safety/test_config.py @@ -0,0 +1,277 @@ +"""Tests for ``signalforge.safety.config`` (US-006). + +Covers :func:`load_safety_config` — full DEC-016 resolution / error contract +plus DEC-013 ``audit_path`` traversal hardening. The 24 tests below exercise +every documented branch of the loader, including the symlink-hardened +``_path_safety.canonicalise_path`` copy. +""" + +from __future__ import annotations + +import logging +from pathlib import Path + +import pytest + +from signalforge.safety.config import load_safety_config +from signalforge.safety.errors import ( + ConfigNotFoundError, + InvalidConfigError, + InvalidSamplingModeError, + SafetyError, +) +from signalforge.safety.models import SamplingMode +from signalforge.safety.policy import DEFAULT_REDACT_PATTERNS + +pytestmark = pytest.mark.safety + + +_FIXTURES_DIR = Path(__file__).resolve().parent.parent / "fixtures" / "safety" + + +def _copy_fixture(fixture_name: str, dest_dir: Path) -> Path: + """Copy a fixture YAML into ``dest_dir/signalforge.yml``.""" + src = _FIXTURES_DIR / fixture_name + dest = dest_dir / "signalforge.yml" + dest.write_text(src.read_text(encoding="utf-8"), encoding="utf-8") + return dest + + +# --------------------------------------------------------------------------- +# Defaults / no file / empty file +# --------------------------------------------------------------------------- + + +def test_load_safety_config_no_file_returns_defaults(tmp_path: Path) -> None: + """DEC-012(b): missing default config falls back to schema-only.""" + assert load_safety_config(tmp_path).mode is SamplingMode.SCHEMA_ONLY + + +def test_load_safety_config_no_file_default_redact_patterns(tmp_path: Path) -> None: + assert load_safety_config(tmp_path).redact_patterns == DEFAULT_REDACT_PATTERNS + + +def test_load_safety_config_empty_file_returns_defaults(tmp_path: Path) -> None: + (tmp_path / "signalforge.yml").write_bytes(b"") + policy = load_safety_config(tmp_path) + assert policy.mode is SamplingMode.SCHEMA_ONLY + assert policy.redact_patterns == DEFAULT_REDACT_PATTERNS + + +def test_load_safety_config_whitespace_only_file_returns_defaults(tmp_path: Path) -> None: + (tmp_path / "signalforge.yml").write_text(" \n \n", encoding="utf-8") + policy = load_safety_config(tmp_path) + assert policy.mode is SamplingMode.SCHEMA_ONLY + + +def test_load_safety_config_yaml_with_only_comments_returns_defaults(tmp_path: Path) -> None: + (tmp_path / "signalforge.yml").write_text("# just a comment\n", encoding="utf-8") + policy = load_safety_config(tmp_path) + assert policy.mode is SamplingMode.SCHEMA_ONLY + + +def test_load_safety_config_missing_safety_key_returns_defaults(tmp_path: Path) -> None: + (tmp_path / "signalforge.yml").write_text("llm:\n model: claude-opus-4-7\n", encoding="utf-8") + policy = load_safety_config(tmp_path) + assert policy.mode is SamplingMode.SCHEMA_ONLY + assert policy.redact_patterns == DEFAULT_REDACT_PATTERNS + + +def test_load_safety_config_unknown_top_level_key_ignored(tmp_path: Path) -> None: + """DEC-025 namespace: unknown top-level keys (other than ``safety:``) + are silently ignored — they belong to other reserved top-level scopes. + """ + _copy_fixture("signalforge_unknown_top_level.yml", tmp_path) + policy = load_safety_config(tmp_path) + assert policy.mode is SamplingMode.SCHEMA_ONLY + + +# --------------------------------------------------------------------------- +# Path resolution / explicit path +# --------------------------------------------------------------------------- + + +def test_load_safety_config_explicit_path_missing_raises(tmp_path: Path) -> None: + missing = tmp_path / "missing.yml" + with pytest.raises(ConfigNotFoundError): + load_safety_config(tmp_path, path=missing) + + +def test_load_safety_config_implicit_path_missing_returns_defaults(tmp_path: Path) -> None: + """Companion to test #1 — explicit name for the implicit-default branch.""" + assert load_safety_config(tmp_path, path=None).mode is SamplingMode.SCHEMA_ONLY + + +# --------------------------------------------------------------------------- +# Malformed / wrong-shape input +# --------------------------------------------------------------------------- + + +def test_load_safety_config_malformed_yaml_raises_invalid_config(tmp_path: Path) -> None: + (tmp_path / "signalforge.yml").write_text(": : :\n", encoding="utf-8") + with pytest.raises(InvalidConfigError): + load_safety_config(tmp_path) + + +def test_load_safety_config_non_mapping_top_level_raises_invalid_config( + tmp_path: Path, +) -> None: + (tmp_path / "signalforge.yml").write_text( + "- a list at top level\n- like this\n", encoding="utf-8" + ) + with pytest.raises(InvalidConfigError): + load_safety_config(tmp_path) + + +# --------------------------------------------------------------------------- +# Fixture-driven happy paths +# --------------------------------------------------------------------------- + + +def test_load_safety_config_minimal_fixture(tmp_path: Path) -> None: + _copy_fixture("signalforge_minimal.yml", tmp_path) + policy = load_safety_config(tmp_path) + assert policy.mode is SamplingMode.SCHEMA_ONLY + assert policy.redact_patterns == DEFAULT_REDACT_PATTERNS + + +def test_load_safety_config_extend_fixture(tmp_path: Path) -> None: + _copy_fixture("signalforge_extend.yml", tmp_path) + policy = load_safety_config(tmp_path) + assert policy.redact_patterns[-1] == "*custom_*" + for builtin in DEFAULT_REDACT_PATTERNS: + assert builtin in policy.redact_patterns + + +def test_load_safety_config_replace_empty_fixture_warns( + tmp_path: Path, caplog: pytest.LogCaptureFixture +) -> None: + _copy_fixture("signalforge_replace_empty.yml", tmp_path) + with caplog.at_level(logging.WARNING, logger="signalforge.safety"): + policy = load_safety_config(tmp_path) + warnings = [r for r in caplog.records if r.levelno == logging.WARNING] + assert len(warnings) == 1 + assert policy.redact_patterns == () + + +# --------------------------------------------------------------------------- +# Fixture-driven error paths +# --------------------------------------------------------------------------- + + +def test_load_safety_config_extend_replace_conflict_fixture_raises(tmp_path: Path) -> None: + _copy_fixture("signalforge_extend_replace_conflict.yml", tmp_path) + with pytest.raises(InvalidConfigError): + load_safety_config(tmp_path) + + +def test_load_safety_config_unknown_mode_fixture_raises(tmp_path: Path) -> None: + _copy_fixture("signalforge_unknown_mode.yml", tmp_path) + with pytest.raises(InvalidSamplingModeError): + load_safety_config(tmp_path) + + +def test_load_safety_config_typo_fixture_raises(tmp_path: Path) -> None: + """The ``redacts:`` typo (rather than ``redact:``) under ``safety:`` — + the loader must surface a typed safety-layer error. The exact subclass + depends on whether the validator can map it to + :class:`UnknownConfigKeyError` or whether it falls through to the + Pydantic generic path; either way it must inherit from + :class:`SafetyError`. + """ + _copy_fixture("signalforge_typo.yml", tmp_path) + with pytest.raises(SafetyError): + load_safety_config(tmp_path) + + +# --------------------------------------------------------------------------- +# audit_path traversal hardening (DEC-013) +# --------------------------------------------------------------------------- + + +def test_load_safety_config_audit_path_with_dotdot_raises(tmp_path: Path) -> None: + _copy_fixture("signalforge_audit_path_traversal.yml", tmp_path) + with pytest.raises(InvalidConfigError): + load_safety_config(tmp_path) + + +def test_load_safety_config_audit_path_outside_project_raises(tmp_path: Path) -> None: + (tmp_path / "signalforge.yml").write_text( + "safety:\n audit_path: /tmp/escape.jsonl\n", encoding="utf-8" + ) + with pytest.raises(SafetyError): + load_safety_config(tmp_path) + + +def test_load_safety_config_audit_path_relative_resolves_inside_project( + tmp_path: Path, +) -> None: + (tmp_path / "signalforge.yml").write_text( + "safety:\n audit_path: .signalforge/audit.jsonl\n", encoding="utf-8" + ) + policy = load_safety_config(tmp_path) + expected = (tmp_path / ".signalforge" / "audit.jsonl").resolve() + assert policy.audit_path == expected + assert policy.audit_path.is_absolute() + assert policy.audit_path.is_relative_to(tmp_path.resolve()) + + +def test_load_safety_config_audit_path_symlink_to_outside_raises(tmp_path: Path) -> None: + """A symlink at ``/.signalforge/escape.jsonl`` pointing at + ``/tmp/escape.jsonl`` must be rejected — the resolved path falls outside + the project tree, so the containment check fires. + """ + sigdir = tmp_path / ".signalforge" + sigdir.mkdir() + target = Path("/tmp") / f"sf_escape_{tmp_path.name}.jsonl" + link = sigdir / "escape.jsonl" + try: + link.symlink_to(target) + except OSError: + pytest.skip("symlink creation not supported on this platform") + (tmp_path / "signalforge.yml").write_text( + "safety:\n audit_path: .signalforge/escape.jsonl\n", encoding="utf-8" + ) + with pytest.raises(SafetyError): + load_safety_config(tmp_path) + + +# --------------------------------------------------------------------------- +# YAML-driven redact resolution + safety +# --------------------------------------------------------------------------- + + +def test_load_safety_config_extend_resolution_via_yaml(tmp_path: Path) -> None: + (tmp_path / "signalforge.yml").write_text( + 'safety:\n redact:\n extend: ["*foo"]\n', encoding="utf-8" + ) + policy = load_safety_config(tmp_path) + assert policy.redact_patterns[-1] == "*foo" + for builtin in DEFAULT_REDACT_PATTERNS: + assert builtin in policy.redact_patterns + + +def test_load_safety_config_replace_empty_resolution_via_yaml( + tmp_path: Path, caplog: pytest.LogCaptureFixture +) -> None: + (tmp_path / "signalforge.yml").write_text( + "safety:\n redact:\n replace: []\n", encoding="utf-8" + ) + with caplog.at_level(logging.WARNING, logger="signalforge.safety"): + policy = load_safety_config(tmp_path) + assert policy.redact_patterns == () + warnings = [r for r in caplog.records if r.levelno == logging.WARNING] + assert len(warnings) == 1 + + +def test_load_safety_config_uses_yaml_safe_load_not_load(tmp_path: Path) -> None: + """Defensive regression: ``yaml.safe_load`` rejects arbitrary Python + object construction tags. If the loader ever regressed to ``yaml.load`` + this fixture would silently invoke ``os.system("ls")`` instead of + raising. We expect :class:`InvalidConfigError`. + """ + (tmp_path / "signalforge.yml").write_text( + 'safety: !!python/object/apply:os.system ["ls"]\n', encoding="utf-8" + ) + with pytest.raises(InvalidConfigError): + load_safety_config(tmp_path) From c3195c54289b44e608f16217166498aafee7979d Mon Sep 17 00:00:00 2001 From: wjduenow Date: Tue, 28 Apr 2026 22:49:57 -0700 Subject: [PATCH 12/19] bd_1-scaffolding-y47: Add signalforge.safety.redact (classify + redact_rows + hash_column_name) _classify_column returns RedactionRecord | None; precedence column>model; case-insensitive tag matching; meta.contains_pii truthy coercion (DEBUG-log). Pattern match case-insensitive (lowercase both sides). hash_column_name uses blake2b-4 for stable 8-hex-char placeholders (DEC-010). redact_rows replaces values with '' constant (no mutation). redact_column_names substitutes hashed names. Suspicious-unmatched-column WARNING heuristic. Co-Authored-By: Claude Opus 4.7 (1M context) --- src/signalforge/safety/redact.py | 303 +++++++++++++++++++++++++ tests/safety/test_classify.py | 378 +++++++++++++++++++++++++++++++ tests/safety/test_redact.py | 144 ++++++++++++ 3 files changed, 825 insertions(+) create mode 100644 src/signalforge/safety/redact.py create mode 100644 tests/safety/test_classify.py create mode 100644 tests/safety/test_redact.py diff --git a/src/signalforge/safety/redact.py b/src/signalforge/safety/redact.py new file mode 100644 index 00000000..22a1b64f --- /dev/null +++ b/src/signalforge/safety/redact.py @@ -0,0 +1,303 @@ +"""Pure redaction helpers for the PII safety layer (US-008). + +This module is the central place where SignalForge decides whether a column +should be sent to the LLM at all (and, when redacted, what placeholder name +to use). Three pieces of public surface plus one underscore-prefixed helper: + +* :func:`hash_column_name` — stable per-name ``col_<8 hex>`` placeholder + (DEC-010). Used by ``schema-only`` and ``aggregate-only`` modes so the + LLM can still reference a column without the real name leaking PII. +* :func:`redact_rows` — replace values for redacted columns with the + ``""`` constant. Pure, non-mutating, returns a tuple. +* :func:`redact_column_names` — substitute hashed names into a + ``(name, type)`` schema tuple for redacted columns. +* :func:`_classify_column` — pure precedence-resolved classifier with the + seven :data:`signalforge.safety.models.RedactionReason` outcomes. + Underscore-prefixed because callers should usually go through + :mod:`signalforge.safety.request` (US-009) rather than calling this + directly; tests import it explicitly. + +Design commitments operationalised here: + +* **DEC-003** — The four opt-out signals (column meta ``signalforge.sample``, + column tag ``pii``, column ``meta.contains_pii``, plus their model-level + twins) take precedence over the pattern matcher. Column-level signals + beat model-level on conflict. +* **DEC-010** — Column-name redaction via ``blake2b`` (digest_size=4) so + the LLM can still reference the column by its hashed placeholder. +* **DEC-020** — Tag matching is case-insensitive (``["PII"]`` triggers). + ``meta.contains_pii`` accepts truthy values; falsy values do not. Pattern + matching lowercases both sides before ``fnmatch.fnmatchcase``. A column + whose name contains a suspicious substring (``email``, ``phone``, ``ssn``, + ``password``, ``token``, ``secret``, ``api_key``) but matches nothing + emits a one-line WARNING — operator-noticeable but non-fatal. +* **DEC-024** — :data:`signalforge.safety.models.RedactionReason` is a + ``Literal`` of seven values; ``_classify_column`` returns + ``RedactionRecord | None`` (``None`` = pass-through, no record needed). +""" + +from __future__ import annotations + +import fnmatch +import hashlib +import json +import logging +from collections.abc import Iterable +from typing import Any, Final + +from signalforge.manifest.models import Column, Model +from signalforge.safety.models import RedactionRecord +from signalforge.safety.policy import SafetyPolicy + +_LOGGER: Final = logging.getLogger("signalforge.safety") + +_REDACTED_VALUE: Final[str] = "" +"""Sentinel substituted for redacted cell values in :func:`redact_rows`.""" + +_SUSPICIOUS_SUBSTRINGS: Final[tuple[str, ...]] = ( + "email", + "phone", + "ssn", + "password", + "token", + "secret", + "api_key", +) +"""Lowercased substrings that, when present in an unmatched column name, hint +at a misconfigured redaction policy. DEC-020.""" + + +def hash_column_name(name: str) -> str: + """Return a stable ``col_<8-hex>`` placeholder for ``name`` (DEC-010). + + Schema-only and aggregate-only modes redact column NAMES too — a column + named ``customer_ssn`` leaks PII via the name itself. This function + yields a deterministic ``blake2b`` (digest_size=4) hash so the LLM can + still reference the column in its draft (e.g. ``col_a3f29c61``). The + real-name -> hashed-name mapping lives in the audit log only. + """ + digest = hashlib.blake2b(name.encode("utf-8"), digest_size=4).hexdigest() + return f"col_{digest}" + + +# --------------------------------------------------------------------------- +# _classify_column — precedence-resolved opt-out / pattern matcher +# --------------------------------------------------------------------------- + + +def _is_truthy_pii_flag(value: Any) -> bool: + """Return ``True`` when ``value`` should count as ``contains_pii=True``. + + Booleans pass through unchanged. ``None`` is always false. Strings are + truthy iff non-empty; numbers iff non-zero; other objects fall back to + Python truthiness. Coercions (anything other than a plain ``bool``) + emit a DEBUG log so the audit trail records the loose-typing decision. + """ + if isinstance(value, bool): + return value + if value is None: + return False + if isinstance(value, (int, float)): + coerced = bool(value) + if coerced: + _LOGGER.debug( + "meta.contains_pii coerced from %s to True", + type(value).__name__, + ) + return coerced + if isinstance(value, str): + if value: + _LOGGER.debug("meta.contains_pii coerced from non-empty str to True") + return True + return False + if bool(value): + _LOGGER.debug( + "meta.contains_pii coerced from %s to True", + type(value).__name__, + ) + return True + return False + + +def _has_pii_tag(tags: Iterable[Any] | None) -> bool: + """Case-insensitive check for the literal tag ``"pii"``.""" + if not tags: + return False + return any(isinstance(t, str) and t.lower() == "pii" for t in tags) + + +def _model_meta(model: Model) -> dict[str, Any]: + """Resolve the model's meta dict. + + ``Model`` has no top-level ``meta`` field; meta lives in ``config.meta`` + (DEC-011 in the manifest reader). This helper centralises that lookup + so the classifier reads ``model_meta`` rather than poking at + ``model.config.meta`` inline at every signal site. + """ + return getattr(model.config, "meta", {}) or {} + + +def _classify_column( + column: Column, + model: Model, + policy: SafetyPolicy, +) -> RedactionRecord | None: + """Return a redaction record for ``column`` or ``None`` if not redacted. + + Precedence (first match wins, top to bottom): + + 1. column ``meta.signalforge.sample == False`` -> ``column_meta_optout`` + 2. column tag ``pii`` (case-insensitive) -> ``tag_pii_column`` + 3. column ``meta.contains_pii`` truthy -> ``meta_contains_pii_column`` + 4. model ``meta.signalforge.sample == False`` -> ``model_meta_optout`` + 5. model tag ``pii`` (case-insensitive) -> ``tag_pii_model`` + 6. model ``meta.contains_pii`` truthy -> ``meta_contains_pii_model`` + 7. column name matches a redact pattern -> ``pattern_match`` + + When no signal fires, returns ``None``. As a side effect, columns whose + name contains a "suspicious" substring (DEC-020) but matched no + pattern emit a single WARNING with a JSON-encoded payload (ANSI-safe). + """ + name = column.name + name_lower = name.lower() + hashed = hash_column_name(name) + + # ----- column-level signals ----- + column_meta = column.meta or {} + sf_meta = column_meta.get("signalforge") + if isinstance(sf_meta, dict) and sf_meta.get("sample") is False: + return RedactionRecord( + column_name=name, + hashed_name=hashed, + redacted=True, + reason="column_meta_optout", + ) + if _has_pii_tag(getattr(column, "tags", ())): + return RedactionRecord( + column_name=name, + hashed_name=hashed, + redacted=True, + reason="tag_pii_column", + ) + if _is_truthy_pii_flag(column_meta.get("contains_pii")): + return RedactionRecord( + column_name=name, + hashed_name=hashed, + redacted=True, + reason="meta_contains_pii_column", + ) + + # ----- model-level signals ----- + model_meta = _model_meta(model) + sf_model_meta = model_meta.get("signalforge") + if isinstance(sf_model_meta, dict) and sf_model_meta.get("sample") is False: + return RedactionRecord( + column_name=name, + hashed_name=hashed, + redacted=True, + reason="model_meta_optout", + ) + if _has_pii_tag(getattr(model, "tags", ())): + return RedactionRecord( + column_name=name, + hashed_name=hashed, + redacted=True, + reason="tag_pii_model", + ) + if _is_truthy_pii_flag(model_meta.get("contains_pii")): + return RedactionRecord( + column_name=name, + hashed_name=hashed, + redacted=True, + reason="meta_contains_pii_model", + ) + + # ----- pattern match ----- + for pattern in policy.redact_patterns: + if fnmatch.fnmatchcase(name_lower, pattern.lower()): + return RedactionRecord( + column_name=name, + hashed_name=hashed, + redacted=True, + reason="pattern_match", + ) + + # ----- no signal: maybe warn, then pass through ----- + if any(s in name_lower for s in _SUSPICIOUS_SUBSTRINGS): + _LOGGER.warning( + "suspicious column not redacted: %s", + json.dumps( + { + "model_unique_id": getattr(model, "unique_id", ""), + "column": name, + "message": ( + "column name contains a suspicious substring but " + "no redaction pattern matched" + ), + } + ), + ) + return None + + +# --------------------------------------------------------------------------- +# redact_rows +# --------------------------------------------------------------------------- + + +def redact_rows( + rows: tuple[dict[str, Any], ...] | list[dict[str, Any]], + redacted_real_names: frozenset[str] | set[str] | tuple[str, ...] | list[str], +) -> tuple[dict[str, Any], ...]: + """Replace values for redacted columns with ``""``. + + Pure: does not mutate ``rows`` or any of its dicts. Missing-column-in-row + is silently ignored — a sampled row that lacks one of the redacted keys + just passes through with its keys unchanged. + + Operates on REAL column names. Callers that also want to rewrite keys to + hashed placeholders should use :func:`redact_column_names` for the + schema; rows themselves are kept keyed on real names so audit replay can + reconcile them with the records. + """ + redacted_set = frozenset(redacted_real_names) + new_rows: list[dict[str, Any]] = [] + for row in rows: + new_row = {k: (_REDACTED_VALUE if k in redacted_set else v) for k, v in row.items()} + new_rows.append(new_row) + return tuple(new_rows) + + +# --------------------------------------------------------------------------- +# redact_column_names +# --------------------------------------------------------------------------- + + +def redact_column_names( + columns: tuple[tuple[str, str], ...] | list[tuple[str, str]], + records: tuple[RedactionRecord, ...] | list[RedactionRecord], +) -> tuple[tuple[str, str], ...]: + """Substitute hashed placeholders for redacted columns in a schema tuple. + + ``columns`` is a sequence of ``(real_name, type_string)`` tuples for + every column in the model. ``records`` is the set of redaction records + produced by :func:`_classify_column` for the same model. + + Returns a tuple of ``(display_name, type_string)`` where + ``display_name`` is the hashed placeholder for redacted columns and the + real name for everyone else. Records with ``redacted=False`` are + ignored — the matching column passes through with its real name. + """ + redacted_lookup = {r.column_name: r.hashed_name for r in records if r.redacted} + result: list[tuple[str, str]] = [] + for real_name, type_str in columns: + display_name = redacted_lookup.get(real_name, real_name) + result.append((display_name, type_str)) + return tuple(result) + + +__all__ = [ + "hash_column_name", + "redact_rows", + "redact_column_names", +] diff --git a/tests/safety/test_classify.py b/tests/safety/test_classify.py new file mode 100644 index 00000000..807139fb --- /dev/null +++ b/tests/safety/test_classify.py @@ -0,0 +1,378 @@ +"""Classification matrix for ``_classify_column`` (US-008). + +Operationalises DEC-003 (the four opt-out signals + pattern), DEC-020 +(case-insensitivity + suspicious-column heuristic), and DEC-024 +(``RedactionReason`` ``Literal``). + +Hand-constructed ``Column`` / ``Model`` instances cover the matrix because +parametrisation is easier than building a manifest fixture for every cell. +The end-of-file ``test_classify_with_fixture_manifest`` then exercises the +real loader against ``tests/fixtures/safety/manifest_with_pii_meta.json``. +""" + +from __future__ import annotations + +import logging +from pathlib import Path + +import pytest + +from signalforge.manifest.loader import load +from signalforge.manifest.models import Column, Config, Model +from signalforge.safety.policy import SafetyPolicy +from signalforge.safety.redact import _classify_column, hash_column_name + +pytestmark = pytest.mark.safety + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _make_column( + name: str, + *, + tags: tuple[str, ...] = (), + meta: dict | None = None, +) -> Column: + return Column(name=name, tags=list(tags), meta=meta or {}) + + +def _make_model( + *, + unique_id: str = "model.test.x", + tags: tuple[str, ...] = (), + meta: dict | None = None, + columns: dict[str, Column] | None = None, +) -> Model: + # ``Model`` has no top-level ``meta`` field — meta lives in ``config.meta``. + # ``tags`` is top-level, so we set it both at the top and on the + # config (mirroring what dbt actually serialises). + return Model( + unique_id=unique_id, + name="x", + resource_type="model", + package_name="test", + original_file_path="models/x.sql", + path="x.sql", + tags=list(tags), + config=Config(materialized="table", tags=list(tags), meta=meta or {}), + columns=columns or {}, + raw_code="select 1", + ) + + +def _default_policy() -> SafetyPolicy: + return SafetyPolicy() + + +# --------------------------------------------------------------------------- +# No-signal baseline +# --------------------------------------------------------------------------- + + +def test_classify_no_signal_returns_none() -> None: + column = _make_column("created_at") + model = _make_model(columns={"created_at": column}) + assert _classify_column(column, model, _default_policy()) is None + + +# --------------------------------------------------------------------------- +# Column-level signals +# --------------------------------------------------------------------------- + + +def test_classify_column_meta_signalforge_sample_false_redacts() -> None: + column = _make_column("anything", meta={"signalforge": {"sample": False}}) + model = _make_model(columns={"anything": column}) + record = _classify_column(column, model, _default_policy()) + assert record is not None + assert record.redacted is True + assert record.reason == "column_meta_optout" + assert record.column_name == "anything" + assert record.hashed_name == hash_column_name("anything") + + +def test_classify_column_tag_pii_redacts() -> None: + column = _make_column("anything", tags=("pii",)) + model = _make_model(columns={"anything": column}) + record = _classify_column(column, model, _default_policy()) + assert record is not None + assert record.reason == "tag_pii_column" + assert record.redacted is True + + +def test_classify_column_tag_pii_case_insensitive() -> None: + column = _make_column("anything", tags=("PII",)) + model = _make_model(columns={"anything": column}) + record = _classify_column(column, model, _default_policy()) + assert record is not None + assert record.reason == "tag_pii_column" + + +def test_classify_column_meta_contains_pii_true_redacts() -> None: + column = _make_column("anything", meta={"contains_pii": True}) + model = _make_model(columns={"anything": column}) + record = _classify_column(column, model, _default_policy()) + assert record is not None + assert record.reason == "meta_contains_pii_column" + + +def test_classify_column_meta_contains_pii_truthy_string_redacts() -> None: + column = _make_column("anything", meta={"contains_pii": "yes"}) + model = _make_model(columns={"anything": column}) + record = _classify_column(column, model, _default_policy()) + assert record is not None + assert record.reason == "meta_contains_pii_column" + + +def test_classify_column_meta_contains_pii_truthy_int_redacts() -> None: + column = _make_column("anything", meta={"contains_pii": 1}) + model = _make_model(columns={"anything": column}) + record = _classify_column(column, model, _default_policy()) + assert record is not None + assert record.reason == "meta_contains_pii_column" + + +def test_classify_column_meta_contains_pii_falsy_zero_does_not_redact() -> None: + # 0 is falsy; should not trigger meta_contains_pii_column. Name is + # non-suspicious + does not match a default pattern, so the call + # returns None. + column = _make_column("created_at", meta={"contains_pii": 0}) + model = _make_model(columns={"created_at": column}) + assert _classify_column(column, model, _default_policy()) is None + + +def test_classify_column_meta_contains_pii_empty_string_does_not_redact() -> None: + column = _make_column("created_at", meta={"contains_pii": ""}) + model = _make_model(columns={"created_at": column}) + assert _classify_column(column, model, _default_policy()) is None + + +# --------------------------------------------------------------------------- +# Model-level signals +# --------------------------------------------------------------------------- + + +def test_classify_model_meta_optout_redacts_when_column_has_no_signal() -> None: + column = _make_column("created_at") + model = _make_model( + columns={"created_at": column}, + meta={"signalforge": {"sample": False}}, + ) + record = _classify_column(column, model, _default_policy()) + assert record is not None + assert record.reason == "model_meta_optout" + + +def test_classify_model_tag_pii_redacts() -> None: + column = _make_column("created_at") + model = _make_model(columns={"created_at": column}, tags=("pii",)) + record = _classify_column(column, model, _default_policy()) + assert record is not None + assert record.reason == "tag_pii_model" + + +def test_classify_model_tag_pii_case_insensitive() -> None: + column = _make_column("created_at") + model = _make_model(columns={"created_at": column}, tags=("Pii",)) + record = _classify_column(column, model, _default_policy()) + assert record is not None + assert record.reason == "tag_pii_model" + + +def test_classify_model_meta_contains_pii_true_redacts() -> None: + column = _make_column("created_at") + model = _make_model(columns={"created_at": column}, meta={"contains_pii": True}) + record = _classify_column(column, model, _default_policy()) + assert record is not None + assert record.reason == "meta_contains_pii_model" + + +# --------------------------------------------------------------------------- +# Pattern match +# --------------------------------------------------------------------------- + + +def test_classify_pattern_match_redacts() -> None: + column = _make_column("customer_email") + model = _make_model(columns={"customer_email": column}) + record = _classify_column(column, model, _default_policy()) + assert record is not None + assert record.reason == "pattern_match" + + +def test_classify_pattern_match_case_insensitive() -> None: + column = _make_column("CUSTOMER_EMAIL") + model = _make_model(columns={"CUSTOMER_EMAIL": column}) + record = _classify_column(column, model, _default_policy()) + assert record is not None + assert record.reason == "pattern_match" + + +def test_classify_pattern_match_uppercase_pattern_against_lowercase_name() -> None: + # Even when the policy carries an uppercase pattern, the matcher + # lowercases both sides before comparing. + policy = SafetyPolicy(redact_patterns=("*EMAIL",)) + column = _make_column("customer_email") + model = _make_model(columns={"customer_email": column}) + record = _classify_column(column, model, policy) + assert record is not None + assert record.reason == "pattern_match" + + +# --------------------------------------------------------------------------- +# Precedence +# --------------------------------------------------------------------------- + + +def test_classify_precedence_column_over_model_meta_optout() -> None: + # Both column and model carry meta sample=False; column wins. + column = _make_column("anything", meta={"signalforge": {"sample": False}}) + model = _make_model( + columns={"anything": column}, + meta={"signalforge": {"sample": False}}, + ) + record = _classify_column(column, model, _default_policy()) + assert record is not None + assert record.reason == "column_meta_optout" + + +def test_classify_precedence_column_tag_over_model_tag() -> None: + column = _make_column("anything", tags=("pii",)) + model = _make_model(columns={"anything": column}, tags=("pii",)) + record = _classify_column(column, model, _default_policy()) + assert record is not None + assert record.reason == "tag_pii_column" + + +def test_classify_precedence_signals_over_pattern_match() -> None: + # Column matches *email AND has a meta opt-out; meta wins. + column = _make_column("customer_email", meta={"signalforge": {"sample": False}}) + model = _make_model(columns={"customer_email": column}) + record = _classify_column(column, model, _default_policy()) + assert record is not None + assert record.reason == "column_meta_optout" + + +def test_classify_precedence_column_meta_over_column_tag() -> None: + # First-match-wins: column meta sample=False fires before tag pii. + column = _make_column( + "anything", + tags=("pii",), + meta={"signalforge": {"sample": False}}, + ) + model = _make_model(columns={"anything": column}) + record = _classify_column(column, model, _default_policy()) + assert record is not None + assert record.reason == "column_meta_optout" + + +# --------------------------------------------------------------------------- +# RedactionRecord shape +# --------------------------------------------------------------------------- + + +def test_classify_returns_redaction_record_with_correct_hashed_name() -> None: + column = _make_column("customer_email") + model = _make_model(columns={"customer_email": column}) + record = _classify_column(column, model, _default_policy()) + assert record is not None + assert record.hashed_name == hash_column_name(record.column_name) + + +# --------------------------------------------------------------------------- +# Suspicious-substring WARNING heuristic (DEC-020) +# --------------------------------------------------------------------------- + + +def test_classify_suspicious_unmatched_column_warns( + caplog: pytest.LogCaptureFixture, +) -> None: + # "customer_token" matches the suspicious substring "token" but is NOT + # matched by any default redact pattern — so we expect a WARNING and + # _classify_column returns None. + column = _make_column("customer_token") + model = _make_model(columns={"customer_token": column}) + with caplog.at_level(logging.WARNING, logger="signalforge.safety"): + result = _classify_column(column, model, _default_policy()) + assert result is None + warnings = [r for r in caplog.records if r.levelno == logging.WARNING] + assert len(warnings) == 1 + assert "customer_token" in warnings[0].getMessage() + + +def test_classify_redacted_suspicious_column_no_extra_warning( + caplog: pytest.LogCaptureFixture, +) -> None: + # "contact_email_addr" has "email" substring AND matches *email — but it + # is redacted by the pattern, so no warning should fire. + column = _make_column("user_email") + model = _make_model(columns={"user_email": column}) + with caplog.at_level(logging.WARNING, logger="signalforge.safety"): + record = _classify_column(column, model, _default_policy()) + assert record is not None + assert record.reason == "pattern_match" + warnings = [r for r in caplog.records if r.levelno == logging.WARNING] + assert warnings == [] + + +def test_classify_non_suspicious_unmatched_no_warning( + caplog: pytest.LogCaptureFixture, +) -> None: + column = _make_column("created_at") + model = _make_model(columns={"created_at": column}) + with caplog.at_level(logging.WARNING, logger="signalforge.safety"): + result = _classify_column(column, model, _default_policy()) + assert result is None + warnings = [r for r in caplog.records if r.levelno == logging.WARNING] + assert warnings == [] + + +# --------------------------------------------------------------------------- +# Real-fixture round trip +# --------------------------------------------------------------------------- + + +_REPO_ROOT = Path(__file__).resolve().parents[2] +_FIXTURE_PATH = _REPO_ROOT / "tests" / "fixtures" / "safety" / "manifest_with_pii_meta.json" + + +def test_classify_with_fixture_manifest() -> None: + """End-to-end: load the safety fixture, classify each documented column.""" + project_dir = _FIXTURE_PATH.parent + manifest = load(project_dir, manifest_path=_FIXTURE_PATH) + policy = _default_policy() + + customers = manifest.get_model("model.sf_demo.customers") + + # id: no signal, no pattern, not suspicious → None. + assert _classify_column(customers.columns["id"], customers, policy) is None + + # email: matches *email pattern. + rec = _classify_column(customers.columns["email"], customers, policy) + assert rec is not None + assert rec.reason == "pattern_match" + + # customer_ssn_optout: column meta signalforge.sample=False (also + # matches *ssn — meta should win). + rec = _classify_column(customers.columns["customer_ssn_optout"], customers, policy) + assert rec is not None + assert rec.reason == "column_meta_optout" + + # taxpayer_id: column tag pii. + rec = _classify_column(customers.columns["taxpayer_id"], customers, policy) + assert rec is not None + assert rec.reason == "tag_pii_column" + + # birth_date: column meta contains_pii=True. + rec = _classify_column(customers.columns["birth_date"], customers, policy) + assert rec is not None + assert rec.reason == "meta_contains_pii_column" + + # orders_pii_at_model.order_id: model carries tag pii. + orders = manifest.get_model("model.sf_demo.orders_pii_at_model") + rec = _classify_column(orders.columns["order_id"], orders, policy) + assert rec is not None + assert rec.reason == "tag_pii_model" diff --git a/tests/safety/test_redact.py b/tests/safety/test_redact.py new file mode 100644 index 00000000..26ba48b8 --- /dev/null +++ b/tests/safety/test_redact.py @@ -0,0 +1,144 @@ +"""Tests for the non-classify helpers in :mod:`signalforge.safety.redact`. + +Covers :func:`hash_column_name`, :func:`redact_rows`, and +:func:`redact_column_names`. The classify-matrix tests live in +``test_classify.py`` (separate file because the matrix is large). +""" + +from __future__ import annotations + +import re + +import pytest + +from signalforge.safety.models import RedactionRecord +from signalforge.safety.redact import ( + hash_column_name, + redact_column_names, + redact_rows, +) + +pytestmark = pytest.mark.safety + + +# --------------------------------------------------------------------------- +# hash_column_name +# --------------------------------------------------------------------------- + + +def test_hash_column_name_deterministic() -> None: + assert hash_column_name("foo") == hash_column_name("foo") + + +def test_hash_column_name_distinct_for_distinct_inputs() -> None: + assert hash_column_name("foo") != hash_column_name("bar") + + +def test_hash_column_name_format() -> None: + h = hash_column_name("customer_email") + assert re.fullmatch(r"col_[0-9a-f]{8}", h) is not None + + +def test_hash_column_name_handles_unicode() -> None: + # Should not raise on non-ASCII input. + h = hash_column_name("café_ñ_测试") + assert re.fullmatch(r"col_[0-9a-f]{8}", h) is not None + + +# --------------------------------------------------------------------------- +# redact_rows +# --------------------------------------------------------------------------- + + +def test_redact_rows_replaces_values() -> None: + rows = ({"a": 1, "b": 2},) + result = redact_rows(rows, {"a"}) + assert result == ({"a": "", "b": 2},) + + +def test_redact_rows_does_not_mutate_input() -> None: + rows = [{"a": 1, "b": 2}, {"a": 3, "b": 4}] + snapshot = [dict(r) for r in rows] + redact_rows(rows, {"a"}) + assert rows == snapshot + # And the inner dicts are still the original ones. + assert rows[0] == snapshot[0] + assert rows[1] == snapshot[1] + + +def test_redact_rows_returns_tuple() -> None: + result = redact_rows([{"a": 1}], {"a"}) + assert result.__class__ is tuple + + +def test_redact_rows_empty_redacted_list() -> None: + rows = ({"a": 1, "b": 2},) + result = redact_rows(rows, frozenset()) + assert result == ({"a": 1, "b": 2},) + + +def test_redact_rows_all_redacted() -> None: + rows = ({"a": 1, "b": 2},) + result = redact_rows(rows, {"a", "b"}) + assert result == ({"a": "", "b": ""},) + + +def test_redact_rows_missing_column_in_row_silent() -> None: + rows = ({"a": 1, "b": 2},) + # "c" is in the redacted set but not in the rows; should silently no-op. + result = redact_rows(rows, {"c"}) + assert result == ({"a": 1, "b": 2},) + + +def test_redact_rows_accepts_list_input() -> None: + result = redact_rows([{"a": 1}], {"a"}) + assert result == ({"a": ""},) + + +def test_redact_rows_empty_input() -> None: + assert redact_rows((), {"a"}) == () + + +# --------------------------------------------------------------------------- +# redact_column_names +# --------------------------------------------------------------------------- + + +def _record(name: str, *, redacted: bool, hashed: str | None = None) -> RedactionRecord: + return RedactionRecord( + column_name=name, + hashed_name=hashed if hashed is not None else hash_column_name(name), + redacted=redacted, + reason="pattern_match", + ) + + +def test_redact_column_names_substitutes_hashed_for_redacted() -> None: + columns = (("email", "STRING"), ("id", "INT64")) + records = (_record("email", redacted=True),) + result = redact_column_names(columns, records) + expected_hash = hash_column_name("email") + assert result == ((expected_hash, "STRING"), ("id", "INT64")) + + +def test_redact_column_names_preserves_non_redacted() -> None: + columns = (("email", "STRING"), ("id", "INT64")) + # A record with redacted=False should leave the name alone, even if + # listed. + records = (_record("email", redacted=False),) + result = redact_column_names(columns, records) + assert result == (("email", "STRING"), ("id", "INT64")) + + +def test_redact_column_names_returns_tuple_of_tuples() -> None: + columns = [("email", "STRING")] + records = (_record("email", redacted=True),) + result = redact_column_names(columns, records) + assert result.__class__ is tuple + assert all(item.__class__ is tuple for item in result) + + +def test_redact_column_names_handles_no_records() -> None: + columns = (("email", "STRING"),) + result = redact_column_names(columns, ()) + assert result == (("email", "STRING"),) From 874a619832f2a0d30479b26cc8afbf5d22e1929c Mon Sep 17 00:00:00 2001 From: wjduenow Date: Tue, 28 Apr 2026 22:56:48 -0700 Subject: [PATCH 13/19] bd_1-scaffolding-3z5: Add aggregate_columns + FakeAdapter MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit aggregate_columns wraps adapter.column_stats inside `with adapter:` (DEC-008 batching). Redacted columns return None keyed by hashed name; non-redacted keyed by real name. FakeAdapter mirrors warehouse FakeBigQueryClient's expect_* API; never MagicMock. ColumnNotInModelError on unknown column. Empty columns list short-circuits without ever opening the adapter context. When every requested column is redacted, the warehouse is never touched. The PII fixture's `database: "dev"` was bumped to `sf-demo-proj` so it satisfies BigQuery's project-ID grammar (6-30 chars, lowercase start) — the classify tests don't depend on the value, but TableRef.from_model now does. Co-Authored-By: Claude Opus 4.7 (1M context) --- src/signalforge/safety/aggregate.py | 114 ++++++++ .../safety/manifest_with_pii_meta.json | 4 +- tests/safety/_fake_adapter.py | 151 ++++++++++ tests/safety/test_aggregate.py | 260 ++++++++++++++++++ tests/safety/test_fake_adapter.py | 83 ++++++ 5 files changed, 610 insertions(+), 2 deletions(-) create mode 100644 src/signalforge/safety/aggregate.py create mode 100644 tests/safety/_fake_adapter.py create mode 100644 tests/safety/test_aggregate.py create mode 100644 tests/safety/test_fake_adapter.py diff --git a/src/signalforge/safety/aggregate.py b/src/signalforge/safety/aggregate.py new file mode 100644 index 00000000..7f25651f --- /dev/null +++ b/src/signalforge/safety/aggregate.py @@ -0,0 +1,114 @@ +"""Aggregate-stats wrapper for the safety layer (US-009). + +Public surface is one function — :func:`aggregate_columns` — that classifies +each requested column via :func:`signalforge.safety.redact._classify_column` +and, for non-redacted columns, fetches per-column stats from the warehouse +adapter. Redacted columns yield ``None`` (keyed by the hashed placeholder +name) and a :class:`signalforge.safety.models.RedactionRecord`. + +Design commitments operationalised here: + +* **DEC-008** — All ``adapter.column_stats`` calls execute inside a single + ``with adapter:`` block so the BigQuery adapter can batch them into one + query rather than one round-trip per column. +* **DEC-010** — Only redacted columns are renamed to their hashed + placeholder in the returned dict; non-redacted columns keep their real + names so callers can still address them by the names the dbt model + declares. +""" + +from __future__ import annotations + +from collections.abc import Iterable + +from signalforge.manifest.models import Model +from signalforge.safety.errors import ColumnNotInModelError +from signalforge.safety.models import RedactionRecord +from signalforge.safety.policy import SafetyPolicy +from signalforge.safety.redact import _classify_column +from signalforge.warehouse.base import WarehouseAdapter +from signalforge.warehouse.models import ColumnStats, TableRef + + +def aggregate_columns( + adapter: WarehouseAdapter, + model: Model, + columns: Iterable[str], + policy: SafetyPolicy, +) -> tuple[dict[str, ColumnStats | None], tuple[RedactionRecord, ...]]: + """Collect column-level stats for the requested columns. + + Each requested column is classified up front via + :func:`_classify_column`. Redacted columns yield ``None`` in the returned + dict (keyed by the **hashed** placeholder name, per DEC-010) and one + :class:`RedactionRecord` in the redactions tuple. Non-redacted columns + invoke ``adapter.column_stats`` inside a single ``with adapter:`` block + so the adapter can batch the calls into one query (DEC-008); the + resulting :class:`ColumnStats` is stored under the column's **real** + name. + + Empty ``columns`` yields ``({}, ())`` without ever opening the adapter + context. + + Raises: + ColumnNotInModelError: ``columns`` references a name not declared on + ``model.columns``. + ManifestProjectNotFoundError / ManifestSchemaNotFoundError: the + model is missing the ``database`` / ``schema`` fields needed to + build a :class:`TableRef`. + + Returns: + ``(stats_by_name, redactions)``. ``stats_by_name`` keys are hashed + placeholders for redacted columns and real names otherwise; + ``redactions`` lists every redacted column in request order. + """ + requested = list(columns) + if not requested: + return {}, () + + # Build a name -> Column lookup so the per-column classifier call is O(1). + column_lookup = {c.name: c for c in model.columns_list} + + # Classify all columns up front so we know which to fetch *before* + # opening the adapter context — DEC-008 batches stats calls, but there's + # no point opening the context at all if every requested column is + # redacted. + classifications: dict[str, RedactionRecord | None] = {} + for name in requested: + col_obj = column_lookup.get(name) + if col_obj is None: + raise ColumnNotInModelError( + model_unique_id=model.unique_id, + column_name=name, + ) + classifications[name] = _classify_column(col_obj, model, policy) + + stats: dict[str, ColumnStats | None] = {} + redactions: list[RedactionRecord] = [] + + needs_warehouse = any(rec is None or not rec.redacted for rec in classifications.values()) + + if needs_warehouse: + # Resolve the table reference once. ``TableRef.from_model`` raises + # typed errors when database/schema are missing — those propagate. + table = TableRef.from_model(model) + with adapter: + for name in requested: + rec = classifications[name] + if rec is not None and rec.redacted: + redactions.append(rec) + stats[rec.hashed_name] = None + else: + stats[name] = adapter.column_stats(table, name) + else: + # Every requested column is redacted — never touch the warehouse. + for name in requested: + rec = classifications[name] + assert rec is not None and rec.redacted # guarded by needs_warehouse + redactions.append(rec) + stats[rec.hashed_name] = None + + return stats, tuple(redactions) + + +__all__ = ["aggregate_columns"] diff --git a/tests/fixtures/safety/manifest_with_pii_meta.json b/tests/fixtures/safety/manifest_with_pii_meta.json index 53a52c2c..03c2c758 100644 --- a/tests/fixtures/safety/manifest_with_pii_meta.json +++ b/tests/fixtures/safety/manifest_with_pii_meta.json @@ -11,7 +11,7 @@ }, "nodes": { "model.sf_demo.customers": { - "database": "dev", + "database": "sf-demo-proj", "schema": "main", "name": "customers", "resource_type": "model", @@ -80,7 +80,7 @@ "primary_key": [] }, "model.sf_demo.orders_pii_at_model": { - "database": "dev", + "database": "sf-demo-proj", "schema": "main", "name": "orders_pii_at_model", "resource_type": "model", diff --git a/tests/safety/_fake_adapter.py b/tests/safety/_fake_adapter.py new file mode 100644 index 00000000..42455baa --- /dev/null +++ b/tests/safety/_fake_adapter.py @@ -0,0 +1,151 @@ +"""Hand-rolled fake for :class:`signalforge.warehouse.base.WarehouseAdapter`. + +Tests register expectations via ``expect_column_stats`` / ``expect_sample_rows``; +the adapter methods consume one matching expectation per call. Unexpected calls +raise ``AssertionError`` so silent mismatches surface loudly. + +Mirrors the ``expect_*`` style of ``tests/warehouse/_fake.py``'s +``FakeBigQueryClient`` — never ``MagicMock``. Lives under ``tests/safety/`` and +is never imported by production code. +""" + +from __future__ import annotations + +from collections import deque +from typing import Any + +from signalforge.warehouse.base import WarehouseAdapter +from signalforge.warehouse.models import ( + ColumnStats, + Dialect, + PartitionFilter, + TableRef, + TestResult, +) + +_FAKE_DIALECT = Dialect( + name="fake", + supports_tablesample=False, + supports_qualify=False, + quote_char="`", + identifier_case="preserve", +) + + +class FakeAdapter(WarehouseAdapter): + """Explicit fake adapter for the safety-layer aggregate / sample tests. + + Calls to ``column_stats`` / ``sample_rows`` consume the head of the + matching FIFO queue; the expected ``(table, column)`` (or ``(table, n)``) + pair must match the call exactly. ``column_stats`` additionally enforces + the DEC-025 contract that callers must be inside an active context + manager. + + The fake also records ``enter_count`` / ``exit_count`` so tests can + verify that aggregator code opens the context exactly once. + """ + + def __init__(self) -> None: + self._column_stats_queue: deque[tuple[TableRef, str, ColumnStats | BaseException]] = deque() + self._sample_rows_queue: deque[ + tuple[TableRef, int, list[dict[str, Any]] | BaseException] + ] = deque() + self._entered: bool = False + self.enter_count: int = 0 + self.exit_count: int = 0 + + # ---- context-manager surface ----------------------------------------- + + def __enter__(self) -> FakeAdapter: + self._entered = True + self.enter_count += 1 + return self + + def __exit__(self, exc_type: object, exc: object, tb: object) -> None: + self._entered = False + self.exit_count += 1 + return None + + # ---- expectation API -------------------------------------------------- + + def expect_column_stats( + self, + *, + table: TableRef, + column: str, + returns: ColumnStats | BaseException, + ) -> None: + self._column_stats_queue.append((table, column, returns)) + + def expect_sample_rows( + self, + *, + table: TableRef, + n: int, + returns: list[dict[str, Any]] | BaseException, + ) -> None: + self._sample_rows_queue.append((table, n, returns)) + + def assert_all_expectations_met(self) -> None: + unmet: list[str] = [] + if self._column_stats_queue: + unmet.append(f"column_stats: {list(self._column_stats_queue)!r}") + if self._sample_rows_queue: + unmet.append(f"sample_rows: {list(self._sample_rows_queue)!r}") + if unmet: + raise AssertionError("unmet expectations: " + "; ".join(unmet)) + + # ---- WarehouseAdapter surface ---------------------------------------- + + def dialect(self) -> Dialect: + return _FAKE_DIALECT + + def column_stats(self, table: TableRef, column: str) -> ColumnStats: + if not self._entered: + raise AssertionError( + f"column_stats called outside `with adapter:` context " + f"(table={table!r}, column={column!r})" + ) + if not self._column_stats_queue: + raise AssertionError( + f"unexpected column_stats call: table={table!r}, column={column!r}" + ) + expected_table, expected_column, returns = self._column_stats_queue.popleft() + if (expected_table, expected_column) != (table, column): + raise AssertionError( + f"column_stats mismatch: expected " + f"(table={expected_table!r}, column={expected_column!r}); " + f"got (table={table!r}, column={column!r})" + ) + if isinstance(returns, BaseException): + raise returns + return returns + + def sample_rows( + self, + table: TableRef, + n: int, + *, + partition_filter: PartitionFilter | None = None, + ) -> list[dict[str, Any]]: + if not self._sample_rows_queue: + raise AssertionError(f"unexpected sample_rows call: table={table!r}, n={n}") + expected_table, expected_n, returns = self._sample_rows_queue.popleft() + if (expected_table, expected_n) != (table, n): + raise AssertionError( + f"sample_rows mismatch: expected " + f"(table={expected_table!r}, n={expected_n}); " + f"got (table={table!r}, n={n})" + ) + if isinstance(returns, BaseException): + raise returns + return returns + + def run_test_sql(self, sql: str, *, capture_failures: int = 0) -> TestResult: + raise AssertionError( + f"FakeAdapter does not support run_test_sql; got sql={sql!r}, " + f"capture_failures={capture_failures}" + ) + + +__all__ = ["FakeAdapter"] diff --git a/tests/safety/test_aggregate.py b/tests/safety/test_aggregate.py new file mode 100644 index 00000000..2d848888 --- /dev/null +++ b/tests/safety/test_aggregate.py @@ -0,0 +1,260 @@ +"""Tests for :func:`signalforge.safety.aggregate.aggregate_columns` (US-009). + +Exercises the redacted/non-redacted split, the DEC-008 single-context-open +batching, the hashed-name keying for redacted columns (DEC-010), and the +``ColumnNotInModelError`` raise for unknown column names. The end-of-file +fixture round trip uses ``tests/fixtures/safety/manifest_with_pii_meta.json`` +loaded via the real loader. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from signalforge.manifest.loader import load +from signalforge.manifest.models import Column, Config, Model +from signalforge.safety.aggregate import aggregate_columns +from signalforge.safety.errors import ColumnNotInModelError +from signalforge.safety.policy import SafetyPolicy +from signalforge.safety.redact import hash_column_name +from signalforge.warehouse.models import ColumnStats, TableRef +from tests.safety._fake_adapter import FakeAdapter + +pytestmark = pytest.mark.safety + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _make_column( + name: str, + *, + tags: tuple[str, ...] = (), + meta: dict | None = None, +) -> Column: + return Column(name=name, tags=list(tags), meta=meta or {}) + + +def _make_model( + *, + unique_id: str = "model.test.x", + database: str | None = "my-project", + schema: str | None = "ds", + name: str = "tbl", + alias: str | None = None, + tags: tuple[str, ...] = (), + meta: dict | None = None, + columns: dict[str, Column] | None = None, +) -> Model: + return Model( + unique_id=unique_id, + name=name, + resource_type="model", + package_name="test", + original_file_path="models/x.sql", + path="x.sql", + database=database, + schema=schema, + alias=alias, + tags=list(tags), + config=Config(materialized="table", tags=list(tags), meta=meta or {}), + columns=columns or {}, + raw_code="select 1", + ) + + +def _stats(count: int = 100, distinct: int = 80, nulls: int = 5) -> ColumnStats: + return ColumnStats( + count=count, + distinct=distinct, + nulls=nulls, + min=0, + max=999, + data_type="INT64", + ) + + +_POLICY = SafetyPolicy() + + +# --------------------------------------------------------------------------- +# Tests +# --------------------------------------------------------------------------- + + +def test_aggregate_columns_redacted_returns_none_keyed_by_hashed_name() -> None: + column = _make_column("customer_email") + model = _make_model(name="t", columns={"customer_email": column}) + fake = FakeAdapter() + + stats, redactions = aggregate_columns(fake, model, ["customer_email"], _POLICY) + + hashed = hash_column_name("customer_email") + assert stats == {hashed: None} + assert len(redactions) == 1 + assert redactions[0].column_name == "customer_email" + assert redactions[0].hashed_name == hashed + assert redactions[0].reason == "pattern_match" + fake.assert_all_expectations_met() + + +def test_aggregate_columns_calls_adapter_for_non_redacted() -> None: + column = _make_column("id") + model = _make_model(name="t", columns={"id": column}) + table = TableRef.from_model(model) + + fake = FakeAdapter() + fake.expect_column_stats(table=table, column="id", returns=_stats()) + + stats, redactions = aggregate_columns(fake, model, ["id"], _POLICY) + + assert set(stats.keys()) == {"id"} + assert isinstance(stats["id"], ColumnStats) + assert stats["id"].count == 100 + assert redactions == () + fake.assert_all_expectations_met() + + +def test_aggregate_columns_does_not_call_adapter_for_redacted() -> None: + column = _make_column("user_email") + model = _make_model(name="t", columns={"user_email": column}) + + fake = FakeAdapter() # no expectations queued + stats, redactions = aggregate_columns(fake, model, ["user_email"], _POLICY) + + assert stats == {hash_column_name("user_email"): None} + assert len(redactions) == 1 + fake.assert_all_expectations_met() + + +def test_aggregate_columns_uses_with_adapter_context() -> None: + column = _make_column("id") + model = _make_model(name="t", columns={"id": column}) + table = TableRef.from_model(model) + + fake = FakeAdapter() + fake.expect_column_stats(table=table, column="id", returns=_stats()) + + assert fake.enter_count == 0 + assert fake.exit_count == 0 + + aggregate_columns(fake, model, ["id"], _POLICY) + + # Context opened exactly once around the (single) column_stats call, + # then closed. + assert fake.enter_count == 1 + assert fake.exit_count == 1 + + +def test_aggregate_columns_returns_redaction_records_for_redacted() -> None: + columns = { + "user_email": _make_column("user_email"), + "phone": _make_column("phone"), + } + model = _make_model(name="t", columns=columns) + fake = FakeAdapter() + + _, redactions = aggregate_columns(fake, model, ["user_email", "phone"], _POLICY) + + assert len(redactions) == 2 + reasons = {r.reason for r in redactions} + assert reasons == {"pattern_match"} + + +def test_aggregate_columns_handles_mixed_redacted_and_non_redacted() -> None: + columns = { + "id": _make_column("id"), + "email": _make_column("email"), + } + model = _make_model(name="t", columns=columns) + table = TableRef.from_model(model) + + fake = FakeAdapter() + # Only one call expected (for the non-redacted column). + fake.expect_column_stats(table=table, column="id", returns=_stats()) + + stats, redactions = aggregate_columns(fake, model, ["id", "email"], _POLICY) + + hashed_email = hash_column_name("email") + assert set(stats.keys()) == {"id", hashed_email} + assert isinstance(stats["id"], ColumnStats) + assert stats[hashed_email] is None + assert len(redactions) == 1 + assert redactions[0].column_name == "email" + fake.assert_all_expectations_met() + + +def test_aggregate_columns_unknown_column_raises_column_not_in_model() -> None: + model = _make_model( + unique_id="model.test.t", + name="t", + columns={"id": _make_column("id")}, + ) + fake = FakeAdapter() + + with pytest.raises(ColumnNotInModelError) as exc_info: + aggregate_columns(fake, model, ["does_not_exist"], _POLICY) + + assert exc_info.value.column_name == "does_not_exist" + assert exc_info.value.model_unique_id == "model.test.t" + + +def test_aggregate_columns_with_fixture_manifest() -> None: + """End-to-end: load the safety fixture and run the customer model. + + The fixture's ``model.sf_demo.customers`` has five columns; four are + redacted via DEC-003 signals (email/pattern, customer_ssn_optout/meta, + taxpayer_id/tag, birth_date/meta_contains_pii) and ``id`` is the + non-redacted control. + """ + repo_root = Path(__file__).resolve().parents[2] + fixture = repo_root / "tests" / "fixtures" / "safety" / "manifest_with_pii_meta.json" + + manifest = load(fixture.parent, manifest_path=fixture) + customers = manifest.get_model("model.sf_demo.customers") + table = TableRef.from_model(customers) + + fake = FakeAdapter() + fake.expect_column_stats(table=table, column="id", returns=_stats()) + + requested = list(customers.columns.keys()) + stats, redactions = aggregate_columns(fake, customers, requested, _POLICY) + + # Four redacted columns; one (id) keyed by real name with stats. + assert len(redactions) == 4 + assert "id" in stats + assert isinstance(stats["id"], ColumnStats) + + # The four redacted columns are keyed by hashed name and map to None. + redacted_names = {"email", "customer_ssn_optout", "taxpayer_id", "birth_date"} + for name in redacted_names: + assert stats[hash_column_name(name)] is None + + fake.assert_all_expectations_met() + + +def test_aggregate_columns_returns_tuple_for_redactions_field() -> None: + column = _make_column("user_email") + model = _make_model(name="t", columns={"user_email": column}) + fake = FakeAdapter() + + result = aggregate_columns(fake, model, ["user_email"], _POLICY) + + assert result[1].__class__ is tuple + + +def test_aggregate_columns_empty_columns_list_returns_empty_dict() -> None: + model = _make_model(name="t", columns={"id": _make_column("id")}) + fake = FakeAdapter() + + stats, redactions = aggregate_columns(fake, model, [], _POLICY) + + assert stats == {} + assert redactions == () + # Empty request must never open the adapter context. + assert fake.enter_count == 0 + fake.assert_all_expectations_met() diff --git a/tests/safety/test_fake_adapter.py b/tests/safety/test_fake_adapter.py new file mode 100644 index 00000000..ff59b0df --- /dev/null +++ b/tests/safety/test_fake_adapter.py @@ -0,0 +1,83 @@ +"""Self-tests for ``tests/safety/_fake_adapter.py``'s :class:`FakeAdapter`. + +Mirrors ``tests/warehouse/test_fake.py`` for the warehouse fake: regressions +in the test fake itself must not masquerade as bugs in the safety-layer +aggregator. Each test is capable of failing on a real regression +(``testing-signal.md`` — no ``assert True``-shaped placeholders). +""" + +from __future__ import annotations + +import pytest + +from signalforge.warehouse.base import WarehouseAdapter +from signalforge.warehouse.models import ColumnStats, TableRef +from tests.safety._fake_adapter import FakeAdapter + +pytestmark = pytest.mark.safety + + +def _make_table_ref(name: str = "tbl") -> TableRef: + return TableRef(project="my-project", dataset="ds", name=name) + + +def _make_stats() -> ColumnStats: + return ColumnStats(count=10, distinct=8, nulls=0, data_type="INT64") + + +def test_fake_adapter_satisfies_warehouse_adapter_abc() -> None: + """Construction succeeds + ``isinstance`` check passes — proves every + ``@abc.abstractmethod`` on ``WarehouseAdapter`` is implemented.""" + fake = FakeAdapter() + assert isinstance(fake, WarehouseAdapter) + + +def test_fake_adapter_unexpected_column_stats_raises() -> None: + fake = FakeAdapter() + with fake, pytest.raises(AssertionError, match="unexpected column_stats"): + fake.column_stats(_make_table_ref(), "col1") + + +def test_fake_adapter_column_stats_mismatch_raises() -> None: + fake = FakeAdapter() + table = _make_table_ref() + fake.expect_column_stats(table=table, column="col1", returns=_make_stats()) + with fake, pytest.raises(AssertionError, match="column_stats mismatch"): + fake.column_stats(table, "col2") + + +def test_fake_adapter_assert_all_expectations_met_succeeds_when_empty() -> None: + fake = FakeAdapter() + # Should not raise. + fake.assert_all_expectations_met() + + +def test_fake_adapter_assert_all_expectations_met_raises_on_unmet() -> None: + fake = FakeAdapter() + fake.expect_column_stats(table=_make_table_ref(), column="c", returns=_make_stats()) + with pytest.raises(AssertionError, match="unmet expectations"): + fake.assert_all_expectations_met() + + +def test_fake_adapter_outside_context_column_stats_raises_assertion() -> None: + """DEC-025: column_stats must be called inside an active ``with``.""" + fake = FakeAdapter() + fake.expect_column_stats(table=_make_table_ref(), column="c", returns=_make_stats()) + with pytest.raises(AssertionError, match="outside"): + fake.column_stats(_make_table_ref(), "c") + + +def test_fake_adapter_returns_exception_raises_it() -> None: + fake = FakeAdapter() + fake.expect_column_stats(table=_make_table_ref(), column="c", returns=ValueError("boom")) + with fake, pytest.raises(ValueError, match="boom"): + fake.column_stats(_make_table_ref(), "c") + + +def test_fake_adapter_sample_rows_basic() -> None: + fake = FakeAdapter() + table = _make_table_ref() + fake.expect_sample_rows(table=table, n=5, returns=[{"x": 1}]) + rows = fake.sample_rows(table, 5) + assert rows == [{"x": 1}] + fake.assert_all_expectations_met() From fd07fbd481d568c4723790852d8a0b85939bbd72 Mon Sep 17 00:00:00 2001 From: wjduenow Date: Tue, 28 Apr 2026 23:03:47 -0700 Subject: [PATCH 14/19] bd_1-scaffolding-969: Add build_llm_request + default-mode regression suite MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Single entry point per DEC-009: classifies columns, dispatches per-mode (schema-only zero warehouse calls — DEC-012(c)), writes AuditEvent before returning (DEC-011 fail-closed). AuditEvent carries signalforge_version, policy_hash, audit_schema_version=1 (DEC-014). policy_flags populated from policy state (sample_mode_enabled, redaction_disabled, audit_path_overridden). Three default-mode regression tests cluster DEC-012 at policy/config/request layers. Co-Authored-By: Claude Opus 4.7 (1M context) --- src/signalforge/safety/request.py | 223 ++++++++ tests/safety/test_default_mode_regression.py | 69 +++ tests/safety/test_request.py | 505 +++++++++++++++++++ 3 files changed, 797 insertions(+) create mode 100644 src/signalforge/safety/request.py create mode 100644 tests/safety/test_default_mode_regression.py create mode 100644 tests/safety/test_request.py diff --git a/src/signalforge/safety/request.py b/src/signalforge/safety/request.py new file mode 100644 index 00000000..5104b8b9 --- /dev/null +++ b/src/signalforge/safety/request.py @@ -0,0 +1,223 @@ +"""Single-entry request builder for the PII safety layer (US-010). + +:func:`build_llm_request` is the only sanctioned constructor of an +:class:`signalforge.safety.models.LLMRequest`. It classifies every column on +the input :class:`~signalforge.manifest.models.Model`, dispatches to the +warehouse adapter according to the configured :class:`SamplingMode`, writes +exactly one :class:`~signalforge.safety.models.AuditEvent` to the JSONL log, +and returns the request only if the audit write succeeded. + +Design commitments operationalised here: + +* **DEC-009 — Single entry.** Direct construction of :class:`LLMRequest` + bypasses the audit log; the AST scan in US-011 enforces this convention + at lint time. This module is the human-readable companion: every code + path that produces a request flows through one of three branches below. +* **DEC-010 — Column-name hashing.** Schema-only and aggregate-only modes + redact column NAMES (not just values) by replacing each redacted column's + identifier with its blake2b-derived ``col_`` placeholder. Sample + mode does the same and additionally rewrites the keys of every sampled + row so the LLM sees the same hashed identifiers it sees in + :attr:`LLMRequest.schema`. +* **DEC-011 — Fail-closed audit.** Any exception from + :func:`signalforge.safety.audit.write` propagates; the partial + :class:`LLMRequest` is dropped on the floor. Callers never receive a + request whose audit record didn't durably hit disk. +* **DEC-012(c) — Default mode is zero adapter calls.** Schema-only mode + must not invoke ``adapter.column_stats`` or ``adapter.sample_rows`` — + not even by opening the context manager. The companion regression test + in :mod:`tests.safety.test_default_mode_regression` uses a + :class:`FakeAdapter` with no expectations queued; any call would raise. +* **DEC-014 — Audit reproducibility.** Every emitted + :class:`AuditEvent` carries ``signalforge_version``, ``policy_hash``, + ``audit_schema_version``, and ``policy_flags``. +* **DEC-022 — Transitive immutability.** Sequences on the returned + :class:`LLMRequest` are :class:`tuple`, never :class:`list`, so the + payload cannot be mutated between audit-write time and LLM-call time. +""" + +from __future__ import annotations + +from datetime import datetime, timezone +from typing import Final + +import signalforge as _sf +from signalforge.manifest.models import Model +from signalforge.safety import audit +from signalforge.safety.aggregate import aggregate_columns +from signalforge.safety.errors import InvalidSamplingModeError +from signalforge.safety.models import ( + AuditEvent, + LLMRequest, + RedactionRecord, + SamplingMode, +) +from signalforge.safety.policy import ( + DEFAULT_AUDIT_PATH, + SafetyPolicy, + _compute_policy_hash, +) +from signalforge.safety.redact import ( + _classify_column, + redact_column_names, + redact_rows, +) +from signalforge.warehouse.base import WarehouseAdapter +from signalforge.warehouse.models import TableRef + +_AUDIT_SCHEMA_VERSION: Final[int] = 1 + + +def build_llm_request( + model: Model, + adapter: WarehouseAdapter, + policy: SafetyPolicy, +) -> LLMRequest: + """Produce a typed :class:`LLMRequest` and durably audit it (DEC-009). + + Per-mode behaviour: + + * :attr:`SamplingMode.SCHEMA_ONLY` — no adapter calls (DEC-012(c)). + ``sampled_rows=None``, ``aggregates=None``. ``schema`` carries + ``(hashed_name, type)`` for redacted columns and ``(real_name, type)`` + for everyone else. + * :attr:`SamplingMode.AGGREGATE_ONLY` — delegates to + :func:`signalforge.safety.aggregate.aggregate_columns`. + ``sampled_rows=None``; ``aggregates`` is a dict keyed by hashed name + (redacted columns, value=``None``) or real name (otherwise, value= + :class:`ColumnStats`). + * :attr:`SamplingMode.SAMPLE` — calls ``adapter.sample_rows`` inside the + adapter context, redacts values for redacted columns to + ``""``, and rewrites the keys of every row so the LLM sees + the same hashed identifiers it sees in :attr:`LLMRequest.schema`. + + Audit semantics (DEC-011 fail-closed): an exception from + :func:`signalforge.safety.audit.write` propagates; the partial request + is dropped on the floor. + + Args: + model: a manifest :class:`Model` whose columns will be classified. + adapter: a :class:`WarehouseAdapter`. Must be unused for + schema-only mode; opened-and-closed once for aggregate-only + and sample modes. + policy: a :class:`SafetyPolicy` selecting the sampling mode and + redaction patterns. + + Returns: + A :class:`LLMRequest` ready to hand to the LLM-drafting layer + (issue #5). + + Raises: + AuditWriteError: ``audit.write`` failed for any reason. The partial + request is not returned. + AuditRecordTooLargeError: The serialised audit line exceeded the + POSIX-atomic-append size cap. + InvalidSamplingModeError: defensive — should be unreachable given + :class:`SamplingMode`'s closed enum. + """ + # ---- 1. Classify every column up front ---------------------------------- + # Doing this once gives us the redaction set for downstream branches + # (schema rewrite, row-key rewrite, value redaction) without re-walking + # ``policy.redact_patterns`` per branch. + classifications: dict[str, RedactionRecord | None] = { + column.name: _classify_column(column, model, policy) for column in model.columns_list + } + redactions: tuple[RedactionRecord, ...] = tuple( + rec for rec in classifications.values() if rec is not None and rec.redacted + ) + + # ``schema`` is what the LLM sees: (display_name, type_str) for every + # column. Display name is the hashed placeholder for redacted columns. + raw_schema: tuple[tuple[str, str], ...] = tuple( + (column.name, column.data_type or "") for column in model.columns_list + ) + schema = redact_column_names(raw_schema, redactions) + + # ---- 2. Per-mode dispatch ---------------------------------------------- + sampled_rows: tuple[dict[str, object], ...] | None = None + aggregates = None + + if policy.mode is SamplingMode.SCHEMA_ONLY: + # DEC-012(c): zero adapter calls. Even opening the context manager + # is forbidden — schema-only mode must be a pure transform of the + # already-loaded manifest. + pass + elif policy.mode is SamplingMode.AGGREGATE_ONLY: + # ``aggregate_columns`` re-runs ``_classify_column`` internally; + # since ``_classify_column`` is a pure function of ``column``, + # ``model``, and ``policy.redact_patterns``, the redaction set it + # produces matches ours up-front. We pass ``redactions`` from our + # classification as the source of truth on the audit/request side. + aggregates_dict, _ = aggregate_columns( + adapter, + model, + [c.name for c in model.columns_list], + policy, + ) + aggregates = aggregates_dict + elif policy.mode is SamplingMode.SAMPLE: + table = TableRef.from_model(model) + with adapter: + raw_rows = adapter.sample_rows(table, policy.sample_size) + # Redact values for redacted columns (keys still on real names). + redacted_real_names = frozenset(rec.column_name for rec in redactions) + value_redacted = redact_rows(raw_rows, redacted_real_names) + # Rewrite each redacted column's key from real -> hashed so the row + # keys match the identifiers the LLM sees in ``schema`` / + # ``columns_sent``. Non-redacted columns keep their real names. + real_to_hashed = {rec.column_name: rec.hashed_name for rec in redactions} + sampled_rows = tuple( + {real_to_hashed.get(k, k): v for k, v in row.items()} for row in value_redacted + ) + else: + # Defensive: ``SamplingMode`` is a closed enum, so this branch is + # unreachable in practice. Surfacing it as a typed error rather + # than a bare ``RuntimeError`` keeps the safety layer's failure + # surface uniform. + raise InvalidSamplingModeError( + value=policy.mode, + allowed=tuple(m.value for m in SamplingMode), + ) + + # ---- 3. Assemble columns_sent + audit fields ---------------------------- + columns_sent: tuple[str, ...] = tuple(name for name, _ in schema) + row_count = len(sampled_rows) if sampled_rows is not None else None + + policy_flags: list[str] = [] + if policy.mode is SamplingMode.SAMPLE: + policy_flags.append("sample_mode_enabled") + if len(policy.redact_patterns) == 0: + policy_flags.append("redaction_disabled") + if policy.audit_path != DEFAULT_AUDIT_PATH: + policy_flags.append("audit_path_overridden") + + event = AuditEvent( + timestamp=datetime.now(timezone.utc), + model_unique_id=model.unique_id, + mode=policy.mode, + columns_sent=columns_sent, + redactions=redactions, + row_count=row_count, + signalforge_version=_sf.__version__, + policy_hash=_compute_policy_hash(policy), + audit_schema_version=_AUDIT_SCHEMA_VERSION, + policy_flags=tuple(policy_flags), + ) + + # ---- 4. Build the request, then audit, then return ---------------------- + # DEC-011 fail-closed: ``audit.write`` runs BEFORE we hand back the + # request. Any exception propagates and the request is dropped. + request = LLMRequest( + model_unique_id=model.unique_id, + mode=policy.mode, + columns_sent=columns_sent, + redactions=redactions, + sampled_rows=sampled_rows, + aggregates=aggregates, + schema=schema, + ) + audit.write(event, policy.audit_path) + return request + + +__all__ = ["build_llm_request"] diff --git a/tests/safety/test_default_mode_regression.py b/tests/safety/test_default_mode_regression.py new file mode 100644 index 00000000..ed76b8ae --- /dev/null +++ b/tests/safety/test_default_mode_regression.py @@ -0,0 +1,69 @@ +"""Default-mode regression suite — DEC-012. + +If a future bug flips the default :class:`SamplingMode` away from +``SCHEMA_ONLY``, every team that hasn't authored ``signalforge.yml`` silently +leaks PII. These three tests are the load-bearing safety net at three layers: + +* **DEC-012(a)** — the field default on :class:`SafetyPolicy` itself. +* **DEC-012(b)** — the loader fallback when no ``signalforge.yml`` is present. +* **DEC-012(c)** — :func:`build_llm_request` issues *zero* adapter calls under + the default policy. + +All three are also covered by their respective module test files +(``test_policy.py``, ``test_config.py``, ``test_request.py``); this file +clusters them so future-you can ``grep -r "default_mode"`` and find every +layer in one place. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from signalforge.manifest.loader import load +from signalforge.safety.config import load_safety_config +from signalforge.safety.models import SamplingMode +from signalforge.safety.policy import SafetyPolicy +from signalforge.safety.request import build_llm_request +from tests.safety._fake_adapter import FakeAdapter + +pytestmark = pytest.mark.safety + + +_FIXTURE = ( + Path(__file__).resolve().parent.parent / "fixtures" / "safety" / "manifest_with_pii_meta.json" +) + + +def test_safety_policy_no_args_is_schema_only() -> None: + """DEC-012(a) — policy field default.""" + assert SafetyPolicy().mode is SamplingMode.SCHEMA_ONLY + + +def test_load_safety_config_no_file_is_schema_only(tmp_path: Path) -> None: + """DEC-012(b) — loader fallback with no ``signalforge.yml``.""" + assert load_safety_config(tmp_path).mode is SamplingMode.SCHEMA_ONLY + + +def test_build_llm_request_default_policy_zero_warehouse_calls(tmp_path: Path) -> None: + """DEC-012(c) — default policy must trigger zero adapter calls. + + A regression here means a default-mode flip silently issues sample/aggregate + queries against the warehouse. ``FakeAdapter`` with no expectations queued + raises ``AssertionError`` on any call, so this passes only if no warehouse + method is invoked at all. + """ + manifest = load(_FIXTURE.parent, manifest_path=_FIXTURE) + customers = manifest.get_model("model.sf_demo.customers") + + fake = FakeAdapter() + policy = SafetyPolicy(audit_path=tmp_path / "audit.jsonl") + + request = build_llm_request(customers, fake, policy) + + fake.assert_all_expectations_met() + assert fake.enter_count == 0 + assert request.mode is SamplingMode.SCHEMA_ONLY + assert request.sampled_rows is None + assert request.aggregates is None diff --git a/tests/safety/test_request.py b/tests/safety/test_request.py new file mode 100644 index 00000000..434268d4 --- /dev/null +++ b/tests/safety/test_request.py @@ -0,0 +1,505 @@ +"""Tests for :func:`signalforge.safety.request.build_llm_request` (US-010). + +The request builder is the single entry point that ties together every piece +of the safety layer. These tests cover: + +* Per-mode dispatch (schema-only / aggregate-only / sample) and DEC-012(c)'s + zero-adapter-calls invariant for schema-only. +* DEC-010 column-name hashing — both in the ``schema`` field handed to the + LLM and in the keys of ``sampled_rows`` for sample mode. +* DEC-014 ``AuditEvent`` reproducibility — every audit row carries + ``signalforge_version``, ``policy_hash``, ``audit_schema_version``, and + ``policy_flags``. +* DEC-011 fail-closed semantics — any exception from ``audit.write`` + propagates and the partial :class:`LLMRequest` is dropped. +* DEC-022 transitive immutability — every sequence on the returned + :class:`LLMRequest` is a :class:`tuple`, not a :class:`list`. +""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + +import pytest + +import signalforge +from signalforge.manifest.loader import load +from signalforge.manifest.models import Model +from signalforge.safety import audit +from signalforge.safety.errors import AuditWriteError +from signalforge.safety.models import AuditEvent, LLMRequest, SamplingMode +from signalforge.safety.policy import SafetyPolicy, _compute_policy_hash +from signalforge.safety.redact import hash_column_name +from signalforge.safety.request import build_llm_request +from signalforge.warehouse.models import ColumnStats, TableRef +from tests.safety._fake_adapter import FakeAdapter + +pytestmark = pytest.mark.safety + + +_FIXTURE = ( + Path(__file__).resolve().parent.parent / "fixtures" / "safety" / "manifest_with_pii_meta.json" +) + + +# --------------------------------------------------------------------------- +# Fixtures +# --------------------------------------------------------------------------- + + +@pytest.fixture +def customers_model() -> Model: + """The ``model.sf_demo.customers`` model from the safety fixture manifest. + + Five columns: ``id`` (no signal), ``email`` (pattern), ``customer_ssn_optout`` + (column meta opt-out), ``taxpayer_id`` (PII tag), ``birth_date`` + (``meta.contains_pii``). Four redacted, one passes through. + """ + manifest = load(_FIXTURE.parent, manifest_path=_FIXTURE) + return manifest.get_model("model.sf_demo.customers") + + +def _stats(*, count: int = 100, distinct: int = 80, nulls: int = 5) -> ColumnStats: + return ColumnStats( + count=count, + distinct=distinct, + nulls=nulls, + min=0, + max=999, + data_type="INT64", + ) + + +def _policy(tmp_path: Path, **overrides: Any) -> SafetyPolicy: + """Construct a :class:`SafetyPolicy` with a tmp-path audit file. + + Direct construction skips ``load_safety_config``'s ``audit_path`` sanity + gate, which is fine for tests that need an absolute path. + """ + base: dict[str, Any] = {"audit_path": tmp_path / "audit.jsonl"} + base.update(overrides) + return SafetyPolicy(**base) + + +class _AuditRecorder: + """Hand-rolled ``audit.write`` wrapper that records calls. + + No ``MagicMock`` here per task instructions — a plain class with an + explicit ``__call__`` is what tests want, and it composes cleanly with + ``monkeypatch.setattr``. + """ + + def __init__(self, *, raise_on_call: BaseException | None = None) -> None: + self.calls: list[tuple[AuditEvent, Path]] = [] + self._raise = raise_on_call + + def __call__(self, event: AuditEvent, audit_path: Path) -> None: + self.calls.append((event, audit_path)) + if self._raise is not None: + raise self._raise + + +# --------------------------------------------------------------------------- +# Schema-only mode +# --------------------------------------------------------------------------- + + +def test_build_llm_request_schema_only_zero_warehouse_calls( + customers_model: Model, tmp_path: Path +) -> None: + fake = FakeAdapter() + policy = _policy(tmp_path, mode=SamplingMode.SCHEMA_ONLY) + + request = build_llm_request(customers_model, fake, policy) + + fake.assert_all_expectations_met() + assert fake.enter_count == 0 + assert request.mode is SamplingMode.SCHEMA_ONLY + assert request.sampled_rows is None + assert request.aggregates is None + + +def test_build_llm_request_schema_only_returns_columns_sent_with_hashed_for_redacted( + customers_model: Model, tmp_path: Path +) -> None: + fake = FakeAdapter() + policy = _policy(tmp_path, mode=SamplingMode.SCHEMA_ONLY) + + request = build_llm_request(customers_model, fake, policy) + + # The PII columns must be hashed, the safe column must remain. + assert "id" in request.columns_sent + assert "customer_ssn_optout" not in request.columns_sent + assert "email" not in request.columns_sent + assert "taxpayer_id" not in request.columns_sent + assert "birth_date" not in request.columns_sent + assert hash_column_name("customer_ssn_optout") in request.columns_sent + assert hash_column_name("email") in request.columns_sent + + +def test_build_llm_request_schema_only_schema_field_pairs_hashed_with_type( + customers_model: Model, tmp_path: Path +) -> None: + fake = FakeAdapter() + policy = _policy(tmp_path, mode=SamplingMode.SCHEMA_ONLY) + + request = build_llm_request(customers_model, fake, policy) + + schema_dict = dict(request.schema) + # Hashed names appear for redacted columns; types are pulled from the + # manifest column.data_type (None becomes "" — the columns in the fixture + # have no data_type set). + assert hash_column_name("email") in schema_dict + assert "id" in schema_dict + + +# --------------------------------------------------------------------------- +# Aggregate-only mode +# --------------------------------------------------------------------------- + + +def test_build_llm_request_aggregate_only_calls_column_stats_per_non_redacted_column( + customers_model: Model, tmp_path: Path +) -> None: + table = TableRef.from_model(customers_model) + fake = FakeAdapter() + fake.expect_column_stats(table=table, column="id", returns=_stats()) + policy = _policy(tmp_path, mode=SamplingMode.AGGREGATE_ONLY) + + request = build_llm_request(customers_model, fake, policy) + + fake.assert_all_expectations_met() + assert request.sampled_rows is None + assert request.aggregates is not None + # ``id`` keyed by real name with stats; redacted columns keyed by hashed + # name with None. + assert "id" in request.aggregates + assert isinstance(request.aggregates["id"], ColumnStats) + assert request.aggregates[hash_column_name("email")] is None + + +def test_build_llm_request_aggregate_only_no_sample_rows_calls( + customers_model: Model, tmp_path: Path +) -> None: + table = TableRef.from_model(customers_model) + fake = FakeAdapter() + fake.expect_column_stats(table=table, column="id", returns=_stats()) + policy = _policy(tmp_path, mode=SamplingMode.AGGREGATE_ONLY) + + # No sample_rows expectations queued — any call raises. + build_llm_request(customers_model, fake, policy) + fake.assert_all_expectations_met() + + +# --------------------------------------------------------------------------- +# Sample mode +# --------------------------------------------------------------------------- + + +def _sample_rows() -> list[dict[str, Any]]: + return [ + { + "id": 1, + "email": "alice@example.com", + "customer_ssn_optout": "111-11-1111", + "taxpayer_id": "12345", + "birth_date": "1990-01-01", + }, + { + "id": 2, + "email": "bob@example.com", + "customer_ssn_optout": "222-22-2222", + "taxpayer_id": "67890", + "birth_date": "1985-06-15", + }, + ] + + +def test_build_llm_request_sample_calls_sample_rows(customers_model: Model, tmp_path: Path) -> None: + table = TableRef.from_model(customers_model) + fake = FakeAdapter() + fake.expect_sample_rows(table=table, n=100, returns=_sample_rows()) + policy = _policy(tmp_path, mode=SamplingMode.SAMPLE) + + request = build_llm_request(customers_model, fake, policy) + + fake.assert_all_expectations_met() + assert request.sampled_rows is not None + assert len(request.sampled_rows) == 2 + + +def test_build_llm_request_sample_redacts_values_to_redacted_constant( + customers_model: Model, tmp_path: Path +) -> None: + table = TableRef.from_model(customers_model) + fake = FakeAdapter() + fake.expect_sample_rows(table=table, n=100, returns=_sample_rows()) + policy = _policy(tmp_path, mode=SamplingMode.SAMPLE) + + request = build_llm_request(customers_model, fake, policy) + + assert request.sampled_rows is not None + first = request.sampled_rows[0] + # ``id`` not redacted; everything else is. + assert first["id"] == 1 + assert first[hash_column_name("email")] == "" + assert first[hash_column_name("taxpayer_id")] == "" + + +def test_build_llm_request_sample_redacts_names_in_rows_to_hashed( + customers_model: Model, tmp_path: Path +) -> None: + table = TableRef.from_model(customers_model) + fake = FakeAdapter() + fake.expect_sample_rows(table=table, n=100, returns=_sample_rows()) + policy = _policy(tmp_path, mode=SamplingMode.SAMPLE) + + request = build_llm_request(customers_model, fake, policy) + + assert request.sampled_rows is not None + first = request.sampled_rows[0] + # Redacted real names absent; hashed names present. + assert "email" not in first + assert "customer_ssn_optout" not in first + assert "taxpayer_id" not in first + assert "birth_date" not in first + assert hash_column_name("email") in first + assert hash_column_name("customer_ssn_optout") in first + assert hash_column_name("taxpayer_id") in first + assert hash_column_name("birth_date") in first + + +# --------------------------------------------------------------------------- +# Audit-event content + write semantics +# --------------------------------------------------------------------------- + + +def test_build_llm_request_audit_emitted_exactly_once( + customers_model: Model, tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + rec = _AuditRecorder() + monkeypatch.setattr("signalforge.safety.request.audit.write", rec) + + fake = FakeAdapter() + policy = _policy(tmp_path, mode=SamplingMode.SCHEMA_ONLY) + build_llm_request(customers_model, fake, policy) + + assert len(rec.calls) == 1 + + +def test_build_llm_request_audit_carries_signalforge_version( + customers_model: Model, tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + rec = _AuditRecorder() + monkeypatch.setattr("signalforge.safety.request.audit.write", rec) + + fake = FakeAdapter() + policy = _policy(tmp_path, mode=SamplingMode.SCHEMA_ONLY) + build_llm_request(customers_model, fake, policy) + + event, _ = rec.calls[0] + assert event.signalforge_version == signalforge.__version__ + + +def test_build_llm_request_audit_carries_policy_hash( + customers_model: Model, tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + rec = _AuditRecorder() + monkeypatch.setattr("signalforge.safety.request.audit.write", rec) + + fake = FakeAdapter() + policy = _policy(tmp_path, mode=SamplingMode.SCHEMA_ONLY) + build_llm_request(customers_model, fake, policy) + + event, _ = rec.calls[0] + assert event.policy_hash == _compute_policy_hash(policy) + + +def test_build_llm_request_audit_carries_schema_version_1( + customers_model: Model, tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + rec = _AuditRecorder() + monkeypatch.setattr("signalforge.safety.request.audit.write", rec) + + fake = FakeAdapter() + policy = _policy(tmp_path, mode=SamplingMode.SCHEMA_ONLY) + build_llm_request(customers_model, fake, policy) + + event, _ = rec.calls[0] + assert event.audit_schema_version == 1 + + +def test_build_llm_request_audit_policy_flags_sample_mode_enabled( + customers_model: Model, tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + rec = _AuditRecorder() + monkeypatch.setattr("signalforge.safety.request.audit.write", rec) + + table = TableRef.from_model(customers_model) + fake = FakeAdapter() + fake.expect_sample_rows(table=table, n=100, returns=_sample_rows()) + policy = _policy(tmp_path, mode=SamplingMode.SAMPLE) + build_llm_request(customers_model, fake, policy) + + event, _ = rec.calls[0] + assert "sample_mode_enabled" in event.policy_flags + + +def test_build_llm_request_audit_policy_flags_redaction_disabled( + customers_model: Model, tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + rec = _AuditRecorder() + monkeypatch.setattr("signalforge.safety.request.audit.write", rec) + + fake = FakeAdapter() + # ``redact: {replace: []}`` resolves to an empty pattern tuple. + policy = SafetyPolicy( + mode=SamplingMode.SCHEMA_ONLY, + redact_patterns=(), + audit_path=tmp_path / "audit.jsonl", + ) + build_llm_request(customers_model, fake, policy) + + event, _ = rec.calls[0] + assert "redaction_disabled" in event.policy_flags + + +def test_build_llm_request_audit_policy_flags_audit_path_overridden( + customers_model: Model, tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + rec = _AuditRecorder() + monkeypatch.setattr("signalforge.safety.request.audit.write", rec) + + fake = FakeAdapter() + policy = _policy(tmp_path, mode=SamplingMode.SCHEMA_ONLY) + build_llm_request(customers_model, fake, policy) + + event, _ = rec.calls[0] + assert "audit_path_overridden" in event.policy_flags + + +def test_build_llm_request_audit_policy_flags_default_no_flags( + customers_model: Model, monkeypatch: pytest.MonkeyPatch +) -> None: + rec = _AuditRecorder() + monkeypatch.setattr("signalforge.safety.request.audit.write", rec) + + fake = FakeAdapter() + # Construct directly to keep the default ``audit_path``. + policy = SafetyPolicy() + build_llm_request(customers_model, fake, policy) + + event, _ = rec.calls[0] + assert event.policy_flags == () + + +# --------------------------------------------------------------------------- +# Fail-closed audit (DEC-011) +# --------------------------------------------------------------------------- + + +def test_build_llm_request_audit_write_failure_raises_audit_write_error_no_request_returned( + customers_model: Model, tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """DEC-011 — an ``audit.write`` failure aborts the call. + + The partial :class:`LLMRequest` must never escape; the function must + raise rather than return. + """ + boom = AuditWriteError(path=tmp_path / "audit.jsonl", cause=OSError("disk full")) + rec = _AuditRecorder(raise_on_call=boom) + monkeypatch.setattr("signalforge.safety.request.audit.write", rec) + + fake = FakeAdapter() + policy = _policy(tmp_path, mode=SamplingMode.SCHEMA_ONLY) + + with pytest.raises(AuditWriteError): + build_llm_request(customers_model, fake, policy) + + +# --------------------------------------------------------------------------- +# Immutability (DEC-022) and shape invariants +# --------------------------------------------------------------------------- + + +def test_build_llm_request_returned_request_is_transitively_immutable( + customers_model: Model, tmp_path: Path +) -> None: + table = TableRef.from_model(customers_model) + fake = FakeAdapter() + fake.expect_sample_rows(table=table, n=100, returns=_sample_rows()) + policy = _policy(tmp_path, mode=SamplingMode.SAMPLE) + + request = build_llm_request(customers_model, fake, policy) + + assert request.columns_sent.__class__ is tuple + assert request.redactions.__class__ is tuple + assert request.schema.__class__ is tuple + assert request.sampled_rows is not None + assert request.sampled_rows.__class__ is tuple + + +def test_build_llm_request_redactions_match_classifications( + customers_model: Model, tmp_path: Path +) -> None: + fake = FakeAdapter() + policy = _policy(tmp_path, mode=SamplingMode.SCHEMA_ONLY) + + request = build_llm_request(customers_model, fake, policy) + + redacted_real = {r.column_name for r in request.redactions} + assert redacted_real == { + "email", + "customer_ssn_optout", + "taxpayer_id", + "birth_date", + } + reasons = {r.column_name: r.reason for r in request.redactions} + assert reasons["customer_ssn_optout"] == "column_meta_optout" + assert reasons["taxpayer_id"] == "tag_pii_column" + assert reasons["birth_date"] == "meta_contains_pii_column" + assert reasons["email"] == "pattern_match" + + +def test_build_llm_request_columns_sent_count_matches_columns_in_model( + customers_model: Model, tmp_path: Path +) -> None: + fake = FakeAdapter() + policy = _policy(tmp_path, mode=SamplingMode.SCHEMA_ONLY) + + request = build_llm_request(customers_model, fake, policy) + + assert len(request.columns_sent) == len(customers_model.columns) + # Sanity: schema field has the same arity. + assert len(request.schema) == len(customers_model.columns) + + +def test_build_llm_request_returns_llmrequest_instance( + customers_model: Model, tmp_path: Path +) -> None: + fake = FakeAdapter() + policy = _policy(tmp_path, mode=SamplingMode.SCHEMA_ONLY) + + request = build_llm_request(customers_model, fake, policy) + + assert isinstance(request, LLMRequest) + assert request.model_unique_id == customers_model.unique_id + + +def test_build_llm_request_writes_audit_to_disk_under_default_path( + customers_model: Model, tmp_path: Path +) -> None: + """End-to-end: with no ``audit.write`` patch, the JSONL file is created.""" + fake = FakeAdapter() + policy = _policy(tmp_path, mode=SamplingMode.SCHEMA_ONLY) + + build_llm_request(customers_model, fake, policy) + + audit_file = tmp_path / "audit.jsonl" + assert audit_file.exists() + assert audit_file.read_text(encoding="utf-8").count("\n") == 1 + + +# Re-export so the import-only ``audit`` reference does not get culled by lint. +_ = audit From e1f15fa47bd2b48be891429382a87fee80621fa7 Mon Sep 17 00:00:00 2001 From: wjduenow Date: Tue, 28 Apr 2026 23:09:30 -0700 Subject: [PATCH 15/19] bd_1-scaffolding-2fb: Wire signalforge.safety public API + drift detector + AST scan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit __init__.py re-exports the documented surface (DEC-001). StrictAuditEvent drift detector validates the committed JSONL fixture and asserts field-set parity with production AuditEvent (DEC-026). AST scan rejects LLMRequest construction outside request.py — the audit-completeness convention from DEC-020(a). Includes negative test that the AST visitor catches a planted violation. Note: ``_path_safety`` is asserted absent from ``__all__`` rather than ``dir()``: Python attaches imported submodules to the parent package's namespace once any sibling (``config.py``) imports them, regardless of ``__init__.py``. Intent (private-helpers-stay-private) preserved via the ``__all__`` check, mirroring the warehouse package's pattern. Co-Authored-By: Claude Opus 4.7 (1M context) --- src/signalforge/safety/__init__.py | 73 ++++++++++++- tests/safety/test_drift_detector.py | 80 ++++++++++++++ tests/safety/test_public_api.py | 161 ++++++++++++++++++++++++++++ 3 files changed, 313 insertions(+), 1 deletion(-) create mode 100644 tests/safety/test_drift_detector.py create mode 100644 tests/safety/test_public_api.py diff --git a/src/signalforge/safety/__init__.py b/src/signalforge/safety/__init__.py index 0b37c15f..a4af33e0 100644 --- a/src/signalforge/safety/__init__.py +++ b/src/signalforge/safety/__init__.py @@ -1 +1,72 @@ -"""PII safety layer for SignalForge (placeholder; public re-exports land in US-011).""" +"""SignalForge PII safety layer. + +Library-only safety primitives sitting between the warehouse adapter and the +LLM-drafting layer. Three sampling modes (`schema-only` (default), +`aggregate-only`, `sample`) plus column redaction (per-column dbt meta/tags + +configurable name-pattern matching) plus a fail-closed audit log. + +Public API (v0.1): + - SamplingMode, RedactionRecord, AuditEvent, LLMRequest — typed shapes + - SafetyPolicy — user-facing config + - load_safety_config — config-file loader + - build_llm_request — single entry that produces an LLMRequest and writes + the audit record (DEC-009) + - aggregate_columns, redact_rows — composable helpers + - SafetyError + 9 subclasses — typed exception hierarchy + +Construct LLMRequest only via build_llm_request — direct construction bypasses +the audit log and is asserted against by tests/safety/test_public_api.py. +""" + +from __future__ import annotations + +from signalforge.safety.aggregate import aggregate_columns +from signalforge.safety.config import load_safety_config +from signalforge.safety.errors import ( + AuditRecordTooLargeError, + AuditWriteError, + ColumnNotInModelError, + ConfigNotFoundError, + InvalidConfigError, + InvalidPatternError, + InvalidSamplingModeError, + PolicyValidationError, + SafetyError, + UnknownConfigKeyError, +) +from signalforge.safety.models import ( + AuditEvent, + LLMRequest, + RedactionReason, + RedactionRecord, + SamplingMode, +) +from signalforge.safety.policy import SafetyPolicy +from signalforge.safety.redact import redact_rows +from signalforge.safety.request import build_llm_request + +__all__ = [ + # Models + "SamplingMode", + "RedactionReason", + "RedactionRecord", + "AuditEvent", + "LLMRequest", + "SafetyPolicy", + # Functions + "load_safety_config", + "build_llm_request", + "aggregate_columns", + "redact_rows", + # Errors + "SafetyError", + "ConfigNotFoundError", + "InvalidConfigError", + "InvalidSamplingModeError", + "InvalidPatternError", + "ColumnNotInModelError", + "AuditWriteError", + "AuditRecordTooLargeError", + "PolicyValidationError", + "UnknownConfigKeyError", +] diff --git a/tests/safety/test_drift_detector.py b/tests/safety/test_drift_detector.py new file mode 100644 index 00000000..1717a277 --- /dev/null +++ b/tests/safety/test_drift_detector.py @@ -0,0 +1,80 @@ +"""Drift detector for AuditEvent. + +Production AuditEvent uses extra='ignore' for forward-compat (DEC-015). Pair it +with a one-off StrictAuditEvent (extra='forbid') validated against a committed +JSONL fixture. Adding a field to production AuditEvent without updating the +fixture or this strict model breaks the test loudly. + +Reference: .claude/rules/testing-signal.md (drift detection pattern). +""" + +from __future__ import annotations + +import json +from datetime import datetime +from pathlib import Path + +import pytest +from pydantic import BaseModel, ConfigDict, ValidationError + +from signalforge.safety.models import RedactionRecord, SamplingMode + + +class StrictAuditEvent(BaseModel): + """Mirror of production AuditEvent with extra='forbid'. + + If you add a field to AuditEvent (signalforge.safety.models), you MUST: + 1. Add it here, and + 2. Update tests/fixtures/safety/audit_events_sample.jsonl via + tests/fixtures/safety/regenerate.sh. + """ + + model_config = ConfigDict(frozen=True, extra="forbid", populate_by_name=True) + timestamp: datetime + model_unique_id: str + mode: SamplingMode + columns_sent: tuple[str, ...] + redactions: tuple[RedactionRecord, ...] + row_count: int | None + signalforge_version: str + policy_hash: str + audit_schema_version: int + policy_flags: tuple[str, ...] + + +_FIXTURE = Path("tests/fixtures/safety/audit_events_sample.jsonl") + + +def test_audit_event_drift_detector_validates_committed_fixture(): + line = _FIXTURE.read_text(encoding="utf-8").strip() + assert line, f"expected one JSON line in {_FIXTURE}" + payload = json.loads(line) + # If this raises, an unknown field was introduced. Update production model + # AND update both this StrictAuditEvent class AND the fixture's regenerate.sh. + StrictAuditEvent.model_validate(payload) + + +def test_audit_event_drift_detector_rejects_unknown_field(): + line = _FIXTURE.read_text(encoding="utf-8").strip() + payload = json.loads(line) + payload["phantom_field"] = "x" + with pytest.raises(ValidationError): + StrictAuditEvent.model_validate(payload) + + +def test_audit_event_drift_detector_strict_model_field_set_matches_production(): + """Production AuditEvent and StrictAuditEvent must declare the same field set.""" + from signalforge.safety.models import AuditEvent + + prod_fields = set(AuditEvent.model_fields.keys()) + strict_fields = set(StrictAuditEvent.model_fields.keys()) + missing_in_strict = prod_fields - strict_fields + extra_in_strict = strict_fields - prod_fields + assert not missing_in_strict, ( + f"StrictAuditEvent is missing fields present in AuditEvent: {missing_in_strict}. " + "Update StrictAuditEvent to match." + ) + assert not extra_in_strict, ( + f"StrictAuditEvent has fields absent from AuditEvent: {extra_in_strict}. " + "Remove from StrictAuditEvent or add to AuditEvent." + ) diff --git a/tests/safety/test_public_api.py b/tests/safety/test_public_api.py new file mode 100644 index 00000000..ab8c8123 --- /dev/null +++ b/tests/safety/test_public_api.py @@ -0,0 +1,161 @@ +"""Public-API enforcement. + +DEC-001: signalforge.safety re-exports the documented surface; underscore- +prefixed helpers stay reachable via dotted import only. + +DEC-020(a): LLMRequest is constructed only via build_llm_request — direct +construction bypasses the audit log. AST-scan all safety modules except +request.py and assert no Call(func=Name(id='LLMRequest')) appears. +""" + +from __future__ import annotations + +import ast +from pathlib import Path + +import signalforge.safety as safety_pkg + +_DOCUMENTED_PUBLIC = ( + # Models + "SamplingMode", + "RedactionReason", + "RedactionRecord", + "AuditEvent", + "LLMRequest", + "SafetyPolicy", + # Functions + "load_safety_config", + "build_llm_request", + "aggregate_columns", + "redact_rows", + # Errors + "SafetyError", + "ConfigNotFoundError", + "InvalidConfigError", + "InvalidSamplingModeError", + "InvalidPatternError", + "ColumnNotInModelError", + "AuditWriteError", + "AuditRecordTooLargeError", + "PolicyValidationError", + "UnknownConfigKeyError", +) + + +def test_documented_surface_importable_from_package_root(): + for name in _DOCUMENTED_PUBLIC: + assert hasattr(safety_pkg, name), f"signalforge.safety is missing {name!r}" + + +def test_all_lists_documented_surface(): + assert sorted(safety_pkg.__all__) == sorted(_DOCUMENTED_PUBLIC), ( + "signalforge.safety.__all__ does not match the documented surface" + ) + + +def test_private_helpers_not_in_dir(): + """Underscore-prefixed helpers are reachable via dotted import but not in dir(). + + Note: ``_path_safety`` is checked separately — Python attaches imported + submodules to their parent package's namespace regardless of what + ``__init__.py`` does, so we assert it stays out of ``__all__`` (the + ``from package import *`` surface) instead. + """ + public = set(dir(safety_pkg)) + forbidden = { + "_classify_column", + "_compute_policy_hash", + "_resolve_redact_patterns", + } + leaked = forbidden & public + assert not leaked, f"private helpers leaked into public surface: {leaked}" + assert "_path_safety" not in safety_pkg.__all__, ( + "_path_safety leaked into signalforge.safety.__all__" + ) + + +def test_classify_column_reachable_via_dotted_import(): + from signalforge.safety.redact import _classify_column # noqa: F401 + + +def test_compute_policy_hash_reachable_via_dotted_import(): + from signalforge.safety.policy import _compute_policy_hash # noqa: F401 + + +# --- AST audit-completeness scan --- + + +_SAFETY_DIR = Path("src/signalforge/safety") + + +class _LLMRequestCallFinder(ast.NodeVisitor): + """Records every Call(func=Name(id='LLMRequest')) in the visited tree.""" + + def __init__(self): + self.calls: list[tuple[int, int]] = [] + + def visit_Call(self, node): # noqa: N802 + if isinstance(node.func, ast.Name) and node.func.id == "LLMRequest": + self.calls.append((node.lineno, node.col_offset)) + self.generic_visit(node) + + +def _collect_llm_request_calls_outside_request_module() -> list[tuple[Path, int]]: + hits: list[tuple[Path, int]] = [] + for path in _SAFETY_DIR.rglob("*.py"): + if path.name == "request.py": + continue + if path.name.startswith("_"): + # Private helpers may legitimately not call LLMRequest, but check anyway + pass + tree = ast.parse(path.read_text(encoding="utf-8")) + finder = _LLMRequestCallFinder() + finder.visit(tree) + for line, _col in finder.calls: + hits.append((path, line)) + return hits + + +def test_llm_request_construction_only_in_request_module(): + """DEC-020(a): direct LLMRequest(...) construction outside request.py bypasses + the audit log. AST-scan rejects any other location.""" + hits = _collect_llm_request_calls_outside_request_module() + formatted = "\n".join(f" {p}:{line}" for p, line in hits) + assert not hits, ( + "LLMRequest constructed outside signalforge.safety.request:\n" + f"{formatted}\n" + "Construct only via build_llm_request — direct construction bypasses the audit log." + ) + + +def test_llm_request_construction_in_request_module_is_present(): + """Sanity: confirm at least one LLMRequest(...) call exists in request.py. + If this fails, the AST-scan above is no longer meaningful.""" + request_path = _SAFETY_DIR / "request.py" + tree = ast.parse(request_path.read_text(encoding="utf-8")) + finder = _LLMRequestCallFinder() + finder.visit(tree) + assert finder.calls, ( + "Expected LLMRequest(...) call in signalforge.safety.request.py — " + "the AST-scan test in test_llm_request_construction_only_in_request_module " + "is no longer load-bearing if the legitimate constructor disappears." + ) + + +def test_llm_request_construction_negative_planted_violation_is_caught(): + """Negative test: a planted LLMRequest call in a string-fed AST tree is caught.""" + src = """ +from signalforge.safety.models import LLMRequest +def make(): + return LLMRequest( + model_unique_id="x", + mode="schema-only", + columns_sent=(), + redactions=(), + schema=(), + ) +""" + tree = ast.parse(src) + finder = _LLMRequestCallFinder() + finder.visit(tree) + assert len(finder.calls) == 1, "AST visitor failed to detect planted construction" From f7ab690b522d6afb65e1ca7298cd0cce2839f205 Mon Sep 17 00:00:00 2001 From: wjduenow Date: Tue, 28 Apr 2026 23:15:06 -0700 Subject: [PATCH 16/19] bd_1-scaffolding-gx3: Add docs/safety-ops.md + README 'Data safety' section + .gitignore Operational reference for the PII safety layer matching manifest- and warehouse-ops doc precedent: default posture, modes, signalforge.yml schema, redaction patterns, four opt-out signals + precedence, column-name redaction (DEC-010 blake2b), audit JSONL schema with audit_schema_version=1, sensitivity caveat (column names are plaintext), rotation as user responsibility, debugging (logger levels), typed-error reference, CLI integration note pointing at #9. --- .gitignore | 3 + README.md | 25 +++ docs/safety-ops.md | 380 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 408 insertions(+) create mode 100644 docs/safety-ops.md diff --git a/.gitignore b/.gitignore index e69b4a1d..ec28c360 100644 --- a/.gitignore +++ b/.gitignore @@ -25,6 +25,9 @@ dist/ coverage.xml htmlcov/ +# SignalForge runtime state (audit log lives here; sensitive — see docs/safety-ops.md) +.signalforge/ + # Local-only symlinked / synced skills — ignore all except maintainer-owned skills .claude/skills/* !.claude/skills/release-manager diff --git a/README.md b/README.md index a05f8008..45d79a23 100644 --- a/README.md +++ b/README.md @@ -59,6 +59,31 @@ via `WarehouseAdapter.from_profile(profile)`. See setup, cost defaults, sampling strategy (and the TABLESAMPLE cost-asterisk), `PartitionFilter` use, and the typed-error reference. +## Data safety + +Schema-only is the default. The LLM never sees row data unless you +explicitly opt in via `safety.mode: sample` in `signalforge.yml` (or +the post-#9 `--mode` CLI flag). Even column *names* that match the +built-in PII patterns (`*email`, `*phone`, `*ssn`) — or that you flag +via dbt `tags: ["pii"]` / `meta.contains_pii: true` / +`meta.signalforge.sample: false` — are replaced with stable hashed +placeholders (`col_<8 hex>`) before reaching the LLM. + +Every LLM call produces one structured record at +`.signalforge/audit.jsonl` (default; configurable via +`safety.audit_path`). The file contains plaintext column-name metadata +and should be treated as sensitive: this repo's `.gitignore` already +covers `.signalforge/`; the writer creates the directory at `0o700` +and the audit file at `0o600`. The audit writer is fail-closed — if +the write fails, the LLM call is aborted (no silent drafts without an +audit trail). See [docs/safety-ops.md](docs/safety-ops.md) for the +JSONL schema. + +Full reference — mode semantics, the four opt-out signals and their +precedence, the `signalforge.yml` schema, the audit schema, debugging, +and the typed-error reference — is in +[docs/safety-ops.md](docs/safety-ops.md). + ## Roadmap | Version | Scope | diff --git a/docs/safety-ops.md b/docs/safety-ops.md new file mode 100644 index 00000000..e902e9c7 --- /dev/null +++ b/docs/safety-ops.md @@ -0,0 +1,380 @@ +# PII safety layer — operations guide + +Operational reference for users of `signalforge.safety`. Companion to +[`docs/manifest-loader-ops.md`](manifest-loader-ops.md) and +[`docs/warehouse-adapter-ops.md`](warehouse-adapter-ops.md), and to the +design record in [`plans/super/4-pii-safety.md`](../plans/super/4-pii-safety.md). + +The safety layer sits between the warehouse adapter and the LLM-drafting +layer. Every LLM call goes through one entry point — +`signalforge.safety.build_llm_request` — which writes a structured audit +record before the request is handed back. There are no other sanctioned +constructors of `LLMRequest`. + +## Default posture + +Schema-only is the default sampling mode. Architectural Commitment #1 +(signal over volume) means the LLM should see the *least* data needed to +draft useful artifacts; raw row data is opt-in, not opt-out. See §4.5 of +[`docs/research/dbt-claude-technical-surface.md`](research/dbt-claude-technical-surface.md) +for the threat model the layer is sized against. + +The layer is fail-closed: any audit-write failure aborts the LLM call +(DEC-011) and propagates as `AuditWriteError`. The default mode is +asserted at three layers — config defaults, policy defaults, and the +schema-only branch in `build_llm_request` performs zero adapter calls +(DEC-012). Column NAMES are redacted in addition to values, because a +column name alone (`john_smith_ssn_1234`) can leak PII (DEC-010). + +User-facing tagline: **the LLM never sees data unless you've explicitly +told it it can.** + +## Modes + +Three modes, set via `safety.mode` in `signalforge.yml` or overridden +via the CLI's `--mode` flag (post-#9): + +### `schema-only` (default) + +Only column names + types reach the LLM. No warehouse queries are +issued — `build_llm_request` does not even open the adapter context. +Names matching a redaction signal (per-column meta/tags or a redact +pattern) are replaced with stable hashed placeholders of the form +`col_<8 hex>` (DEC-010). The resulting `LLMRequest.schema` is a tuple +of `(display_name, type_string)` pairs; `sampled_rows` and `aggregates` +are both `None`. + +```python +from signalforge.safety import SafetyPolicy, SamplingMode, build_llm_request + +policy = SafetyPolicy(mode=SamplingMode.SCHEMA_ONLY) +request = build_llm_request(model, adapter, policy) +# request.schema == (("customer_id", "INT64"), ("col_a3f29c61", "STRING"), ...) +# request.sampled_rows is None +# request.aggregates is None +``` + +### `aggregate-only` + +Column-level statistics — `count`, `distinct`, `nulls`, `min`, `max`, +`data_type` — reach the LLM via the `aggregates` field. Calls +`WarehouseAdapter.column_stats` once per non-redacted column inside a +single `with adapter:` block. Redacted columns are still keyed in the +dict, but the value is `None` keyed by their hashed name. + +```python +policy = SafetyPolicy(mode=SamplingMode.AGGREGATE_ONLY) +request = build_llm_request(model, adapter, policy) +# request.aggregates == { +# "customer_id": ColumnStats(count=42, distinct=42, nulls=0, ...), +# "col_a3f29c61": None, # redacted +# ... +# } +``` + +### `sample` + +Row-level data reaches the LLM. Calls `WarehouseAdapter.sample_rows` +inside a `with adapter:` block. Default sample size is 100 rows +(`safety.sample_size`). Values for redacted columns are replaced with +the literal string `""`. Row dict keys for redacted columns +are rewritten to their hashed placeholders so the keys match the +identifiers the LLM sees in `schema` and `columns_sent`. + +Constructing a `SafetyPolicy(mode=SamplingMode.SAMPLE)` emits a single +WARNING via `signalforge.safety` on policy load (DEC-021). + +```python +policy = SafetyPolicy(mode=SamplingMode.SAMPLE, sample_size=100) +request = build_llm_request(model, adapter, policy) +# request.sampled_rows == ( +# {"customer_id": 1, "col_a3f29c61": "", "country": "US"}, +# ... +# ) +``` + +## `signalforge.yml` reference + +Top-level namespace is locked to `safety:` per DEC-025; every other +top-level key (`llm:`, `prune:`, …) is reserved for future stages and +silently ignored by the safety loader. + +```yaml +safety: + mode: schema-only # one of schema-only / aggregate-only / sample (case-insensitive) + sample_size: 100 + audit_path: .signalforge/audit.jsonl # default; must stay inside project_dir + redact: + extend: ["*custom_*"] # appends to built-ins; mutually exclusive with replace + # replace: [...] # substitutes built-ins entirely; empty list disables (WARNING) +``` + +Field-by-field: + +- **`mode`** — `schema-only` | `aggregate-only` | `sample`. Case- and + separator-insensitive: `Schema-Only`, `schema_only`, `SCHEMA-ONLY` + all parse. Anything else raises `InvalidSamplingModeError` at load + time. +- **`sample_size`** — Integer row count for `sample` mode; ignored by + the other two. Default `100`. +- **`audit_path`** — JSONL audit-log path. Project-relative or + absolute; must canonicalise to a path inside `project_dir`. Default + is `.signalforge/audit.jsonl`. Containment is enforced via the same + symlink-hardened gate used by the manifest loader (`..` segments + rejected outright; symlink loops raise `InvalidConfigError`). +- **`redact.extend`** — List of fnmatch globs appended to the built-in + patterns. Mutually exclusive with `redact.replace`. +- **`redact.replace`** — List of fnmatch globs that substitutes the + built-ins entirely. An empty list disables redaction and emits a + WARNING. + +Unknown keys under `safety:` or `safety.redact:` raise +`UnknownConfigKeyError` (the policy uses Pydantic's `extra="forbid"`, +DEC-015). Typos like `redacts:` or `mode_:` fail loud at load time +rather than silently doing nothing. + +## Redaction patterns + +Patterns are case-insensitive `fnmatch` globs matched against the +*lowercased* column name. The six built-ins are: + +``` +*email +email +*phone +phone +*ssn +ssn +``` + +Each PII class has a prefixed form (`*email`) and a bare form (`email`) +so both `user_email` and `email` match. + +**Override semantics.** + +- `redact.extend` appends to the six built-ins. +- `redact.replace` substitutes the built-ins entirely. +- `redact.replace: []` disables redaction (with a WARNING). +- Specifying both `extend` and `replace` raises `InvalidConfigError`. + +**Footgun-rejected patterns.** Three values are rejected at policy-load +time with `InvalidPatternError`: + +- `""` — never matches anything. +- `"*"` — matches every column; use `redact.replace: []` to disable + explicitly. +- `"?"` — matches every single-character column. + +**Suspicious-unmatched-column heuristic.** When a column's lowercased +name contains one of `email` / `phone` / `ssn` / `password` / `token` / +`secret` / `api_key` AND no signal fired (no opt-out, no pattern +match), the redactor logs a single WARNING with the model unique_id +and column name. The column is **not** auto-redacted — the heuristic +flags potential misconfiguration for the operator to decide. + +## Per-column opt-out + +Four signals override the pattern matcher (DEC-003). Listed by +precedence — first match wins, top to bottom: + +1. **Column-level `meta.signalforge.sample: false`** — strongest; + beats every other signal. Reason code `column_meta_optout`. +2. **Column-level `tags: ["pii"]`** — case-insensitive; `["PII"]` and + `["Pii"]` both fire. Lowercase recommended. Reason code + `tag_pii_column`. +3. **Column-level `meta.contains_pii: true`** — truthy values + accepted; non-bool truthy values (`"yes"`, `1`) emit a DEBUG log + noting the coercion. Reason code `meta_contains_pii_column`. +4. **Model-level fallbacks** — the same three signals at the model + level (`meta.signalforge.sample`, `tags: [pii]`, + `meta.contains_pii`) cascade to every column. Reason codes + `model_meta_optout` / `tag_pii_model` / `meta_contains_pii_model`. +5. **Pattern match** — last resort. Reason code `pattern_match`. + +Concrete dbt YAML: + +```yaml +# models/marts/customers.yml +version: 2 +models: + - name: customers + columns: + - name: customer_email + # Signal 1: strongest opt-out, beats everything else. + meta: + signalforge: + sample: false + - name: phone_number + # Signal 2: case-insensitive tag match. + tags: ["pii"] + - name: home_address + # Signal 3: truthy values accepted. + meta: + contains_pii: true +``` + +```yaml +# Model-level fallback — every column on the model is redacted. +models: + - name: hr_employees + config: + meta: + signalforge: + sample: false +``` + +## Column-name redaction + +Every redacted column's name is hashed via `blake2b` with +`digest_size=4` (DEC-010), yielding a placeholder of the form +`col_<8 hex>`: + +``` +customer_ssn -> col_a3f29c61 +``` + +The mapping (`real_name -> hashed_name`) is recorded in the audit log's +`redactions` array; the LLM never sees the real name. This closes the +"column name itself leaks PII" gap — names like `john_smith_ssn_1234` +or `card_number_last4` would otherwise reach the LLM even when the +*values* were redacted. + +The hash is deterministic across runs, so re-running a draft against +the same model produces the same placeholder; reviewers can correlate +audit records to manifest columns by re-hashing the real name. + +## Audit JSONL schema + +Every LLM call produces exactly one JSONL record at `safety.audit_path` +(default `.signalforge/audit.jsonl`). One record per line; atomic +append via `O_APPEND` + a single `os.write` (DEC-005). + +`AuditEvent` fields: + +| Field | Type | Meaning | Example | +| ------------------------ | ----------------------------- | ----------------------------------------------------------------------- | -------------------------------------- | +| `timestamp` | ISO 8601 datetime | UTC timestamp of the LLM call. | `"2026-04-28T14:33:01.122Z"` | +| `model_unique_id` | string | dbt unique_id of the drafted model. | `"model.shop.dim_customers"` | +| `mode` | string | Sampling mode in effect. | `"schema-only"` | +| `columns_sent` | array of string | LLM-visible column names (hashed for redacted columns). | `["customer_id", "col_a3f29c61"]` | +| `redactions` | array of `RedactionRecord` | Full redaction decisions — both real and hashed names. **Sensitive.** | `[{"column_name": "customer_ssn", ...}]` | +| `row_count` | integer or `null` | Row count for sample mode; `null` for schema-only / aggregate-only. | `100` or `null` | +| `signalforge_version` | PEP-440 version string | The package version that produced the record. | `"0.1.0"` | +| `policy_hash` | 16 hex chars | First 16 hex chars of `SHA-256(policy)`. DEC-014. | `"6f1c0e3d2c44c012"` | +| `audit_schema_version` | integer | Audit shape version. Currently `1`. | `1` | +| `policy_flags` | array of string | Closed set of flag literals — see below. | `["sample_mode_enabled"]` | + +**`policy_flags` closed set:** + +- `sample_mode_enabled` — `policy.mode is SamplingMode.SAMPLE`. +- `redaction_disabled` — `policy.redact_patterns` is empty. +- `audit_path_overridden` — `policy.audit_path != DEFAULT_AUDIT_PATH`. + +`RedactionRecord` fields: `column_name` (real), `hashed_name` +(`col_<8 hex>`), `redacted` (bool), `reason` (one of seven literal +strings — see [Per-column opt-out](#per-column-opt-out)). + +## Audit log sensitivity + +The audit JSONL contains plaintext column names in +`RedactionRecord.column_name`. For PII-laden schemas this metadata can +itself be sensitive; treat the file at-rest as such. + +Recommendations: + +- **Gitignore `.signalforge/`** (already configured in this repo's + `.gitignore`). +- **Restrict at-rest permissions.** The writer creates + `.signalforge/` at `0o700` and `audit.jsonl` at `0o600` on first + call. Don't relax these. +- **Don't ship as a build artifact.** Strip from container images and + CI uploads. +- **Don't check in.** The "explainable diffs" commitment applies to + the YAML SignalForge writes — not to the audit log. + +## Audit log rotation + +User responsibility. v0.1 has no built-in rotation. + +Suggestions: + +- **`logrotate` on the JSONL.** Standard Linux log rotation; works + because each line is self-contained. +- **External log shipping.** Tail to a SIEM or centralised log store; + rotate the on-disk file on the shipping side. +- **Per-run prefixed paths.** Set `safety.audit_path` to + `.signalforge/audit-.jsonl` per CI run if rotation is too + coarse. + +Why no in-process rotation: it would put failure modes (rename races, +disk-full mid-rotate, fsync ordering) inside the fail-closed +audit-write hot path, which conflicts with DEC-011. v0.2 may revisit. + +## Debugging + +Logger name: `signalforge.safety`. + +```python +import logging +logging.getLogger("signalforge.safety").setLevel(logging.DEBUG) +``` + +Levels: + +- **INFO** — One line per `audit.write` (the JSON-encoded summary: + `unique_id`, `mode`, `columns_sent` count, `redactions` count, + `audit_schema_version`). +- **WARNING** — Sample-mode-enabled (one per policy construction); the + empty-redaction `redact: replace: []` warning; the + suspicious-unmatched-column heuristic (one per offending column). + Oversize records raise `AuditRecordTooLargeError` rather than log. +- **DEBUG** — `meta.contains_pii` coercion notes (e.g. value `"yes"` + coerced to `True`); empty-config-file fallback to defaults. + +The safety layer never logs full row data, full column lists, or the +real names of redacted columns. INFO output is a hint about *that* a +call happened, not *what* was in it. + +**Reading a fail-closed `AuditWriteError`.** The cause is exposed as +`.cause`; the path is exposed as `.path`. Common causes: + +- Parent directory not writable (no `+w` for the user, or + `.signalforge/` is a symlink to a read-only mount). +- Disk full (`ENOSPC`). +- Oversize record (raises `AuditRecordTooLargeError` instead — reduce + `columns_sent` or `redactions` count; the cap is 4000 bytes for + POSIX-atomic concurrent appends). + +## Typed-error reference + +Public API: `from signalforge.safety import errors`. Every exception +subclasses `SafetyError` and carries a class-level `default_remediation` +rendered on a `↳ Remediation:` line by `__str__`. + +| Class | When raised | Where it surfaces | How to fix | +| ------------------------------ | ---------------------------------------------------------------------------------------- | ---------------------------------------------- | --------------------------------------------------------------------------------------- | +| `SafetyError` | Base class; never raised directly. | `signalforge.safety.errors` | Catch it to handle every safety-layer failure uniformly. | +| `ConfigNotFoundError` | Explicit `path=` argument to `load_safety_config` pointed at a missing file. | `load_safety_config` | Verify the path, or pass `path=None` to fall back to defaults. | +| `InvalidConfigError` | Parent for parse / schema failures in `signalforge.yml`. Free-form message. | `load_safety_config` | Check `signalforge.yml` against this doc's schema. | +| `InvalidSamplingModeError` | `safety.mode` is not one of `schema-only` / `aggregate-only` / `sample`. | `SafetyPolicy._normalise_mode` | Set `safety.mode` to one of the documented values. | +| `InvalidPatternError` | A redact pattern is empty or one of the bare wildcards `"*"` / `"?"`. | `SafetyPolicy._validate_patterns` | Use a non-empty fnmatch glob; use `redact.replace: []` to disable redaction explicitly. | +| `ColumnNotInModelError` | A safety helper looked up a column not declared on the manifest model. | `aggregate_columns` / sibling helpers | Verify the column exists in `manifest.nodes[model].columns`. | +| `AuditWriteError` | Appending to the JSONL audit log failed (any I/O or encoding error). DEC-011 fail-closed.| `audit.write` | Check `/.signalforge/` exists and is writable; resolve disk / permission. | +| `AuditRecordTooLargeError` | Serialised audit line exceeded the POSIX-atomic-append cap (4000 bytes). | `audit.write` | Reduce `columns_sent` or `redactions` count. | +| `PolicyValidationError` | Generic Pydantic validation failure not covered by a more specific subclass. | `load_safety_config` (last-resort wrap) | Inspect `.field`, `.value`, `.reason`; reconcile against the documented field types. | +| `UnknownConfigKeyError` | A typo'd / unsupported key under a known scope (`safety.redacts:`, etc.). | `SafetyPolicy.model_validate` / redact resolver | Remove or rename the unknown key; see this doc's schema. | + +## CLI integration note + +Tracked in [issue #9](https://github.com/wjduenow/SignalForge/issues/9). +The `signalforge generate` CLI's `--mode` flag will load the policy via +`load_safety_config(...)` and override via +`policy.with_mode(SamplingMode.SAMPLE)` (DEC-018) — that is the +canonical override seam. `SafetyPolicy` is frozen, so `with_mode` is +the only sanctioned mutation path. + +**There is no env-var override for `mode`.** The mode must be set in +`signalforge.yml` or via `--mode` (post-#9). This is intentional: +an env-var override would let a CI misconfiguration silently flip +schema-only into sample mode, which conflicts with the layer's +fail-closed posture. From c560a8719745718e8ed504973114c1aefe715156 Mon Sep 17 00:00:00 2001 From: wjduenow Date: Tue, 28 Apr 2026 23:24:52 -0700 Subject: [PATCH 17/19] =?UTF-8?q?bd=5F1-scaffolding-8av:=20Quality=20gate?= =?UTF-8?q?=20=E2=80=94=20fix=20bugs=20from=20code=20review?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three blockers from quality-gate review: 1. LLMRequest.aggregates was dict[str, ColumnStats|None] — frozen=True only blocks attribute reassignment, not dict mutation. Downstream consumers (#5) could rewrite values after the audit log was written, desyncing the audit from what the LLM actually saw. Switched to tuple[tuple[str, ColumnStats|None], ...] for transitive immutability (DEC-022). Conversion happens at the request-builder boundary; aggregate_columns' public dict return is unchanged. 2. SafetyPolicy.with_mode() used model_copy(update=...) which silently skipped @model_validator(mode="after"). Calling policy.with_mode(SamplingMode.SAMPLE) — the documented CLI override seam (#9) — silently enabled sample mode without emitting the DEC-021 WARNING. Now goes through model_validate so the warning fires every time, regardless of construction path. 3. SafetyPolicy._normalise_mode returned non-string non-enum values unchanged, letting Pydantic raise a generic ValidationError instead of the typed InvalidSamplingModeError. Now every invalid type raises the typed error so the safety-layer hierarchy stays homogeneous. Plus: documented the .signalforge/ pre-existing-permissions caveat in docs/safety-ops.md (mkdir(exist_ok=True, mode=...) does not tighten pre-existing directories — user must verify pre-deploy). Twelve new regression tests: - test_safety_policy_with_mode_sample_emits_warning (DEC-021) - test_safety_policy_with_mode_schema_only_emits_no_warning - test_safety_policy_mode_non_string_non_enum_raises_invalid_sampling_mode_error (parametrised across 7 bad types: int, float, None, list, dict, tuple, object) - test_llm_request_aggregates_is_tuple_of_tuples_when_present (DEC-022) - test_llm_request_aggregates_immutable_when_none - test_llm_request_aggregates_field_reassignment_blocked_by_frozen Validation: 423 passed (up from 411), ruff/pyright/format all clean. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/safety-ops.md | 8 ++++- src/signalforge/safety/models.py | 7 +++- src/signalforge/safety/policy.py | 19 +++++++++-- src/signalforge/safety/request.py | 8 +++-- tests/safety/test_models.py | 29 +++++++++++++++++ tests/safety/test_policy.py | 54 +++++++++++++++++++++++++++++++ tests/safety/test_request.py | 12 ++++--- 7 files changed, 126 insertions(+), 11 deletions(-) diff --git a/docs/safety-ops.md b/docs/safety-ops.md index e902e9c7..9c0d9838 100644 --- a/docs/safety-ops.md +++ b/docs/safety-ops.md @@ -286,7 +286,13 @@ Recommendations: `.gitignore`). - **Restrict at-rest permissions.** The writer creates `.signalforge/` at `0o700` and `audit.jsonl` at `0o600` on first - call. Don't relax these. + call. Don't relax these. Note: if `.signalforge/` already exists + with looser permissions (e.g. created by a different process or + user), the writer's `mkdir(exist_ok=True, mode=0o700)` will NOT + tighten the existing directory — Python's `mkdir` only applies + `mode` when creating. Verify pre-existing directory permissions + before deploying; tighten manually if needed + (`chmod 700 .signalforge/`). - **Don't ship as a build artifact.** Strip from container images and CI uploads. - **Don't check in.** The "explainable diffs" commitment applies to diff --git a/src/signalforge/safety/models.py b/src/signalforge/safety/models.py index 2b025f56..8c3358d3 100644 --- a/src/signalforge/safety/models.py +++ b/src/signalforge/safety/models.py @@ -126,7 +126,12 @@ class LLMRequest(BaseModel): columns_sent: tuple[str, ...] redactions: tuple[RedactionRecord, ...] sampled_rows: tuple[dict[str, Any], ...] | None = None - aggregates: dict[str, ColumnStats | None] | None = None + # Tuple-of-tuples (not dict) so frozen=True actually prevents mutation: + # downstream consumers cannot do `request.aggregates["x"] = ...` post-audit + # (DEC-022 transitive immutability). Convention: list order matches + # ``columns_sent``; redacted columns appear with their hashed name as key + # and ``None`` as value. + aggregates: tuple[tuple[str, ColumnStats | None], ...] | None = None # ``schema`` overrides Pydantic v1's deprecated :meth:`BaseModel.schema` # method on this subclass. The override is intentional — the field name is # part of the documented LLMRequest contract — so pyright's structural diff --git a/src/signalforge/safety/policy.py b/src/signalforge/safety/policy.py index 4082db6f..64d43554 100644 --- a/src/signalforge/safety/policy.py +++ b/src/signalforge/safety/policy.py @@ -187,7 +187,13 @@ def _normalise_mode(cls, value: Any) -> Any: value=value, allowed=tuple(m.value for m in SamplingMode), ) - return value + # Non-string, non-enum input (e.g. mode=42, mode=None, mode=[]) must + # raise the typed error rather than fall through to Pydantic's generic + # ValidationError — keeps the safety-layer error hierarchy homogeneous. + raise InvalidSamplingModeError( + value=value, + allowed=tuple(m.value for m in SamplingMode), + ) @field_validator("redact_patterns") @classmethod @@ -228,8 +234,17 @@ def with_mode(self, mode: SamplingMode) -> SafetyPolicy: Used by issue #9's CLI to apply ``--mode`` after loading from ``signalforge.yml``. Frozen Pydantic models cannot be mutated; this is the canonical override path (DEC-018). + + Re-runs all validators (including the sample-mode WARNING per + DEC-021) by going through ``model_validate`` rather than + ``model_copy``. ``model_copy(update=...)`` is a shallow shortcut + that skips ``@model_validator(mode="after")``, which would + silently enable sample mode without emitting the WARNING — a + regression caught by Quality-Gate review. """ - return self.model_copy(update={"mode": mode}) + data = self.model_dump() + data["mode"] = mode + return SafetyPolicy.model_validate(data) def _compute_policy_hash(policy: SafetyPolicy) -> str: diff --git a/src/signalforge/safety/request.py b/src/signalforge/safety/request.py index 5104b8b9..408bb922 100644 --- a/src/signalforge/safety/request.py +++ b/src/signalforge/safety/request.py @@ -83,7 +83,7 @@ def build_llm_request( for everyone else. * :attr:`SamplingMode.AGGREGATE_ONLY` — delegates to :func:`signalforge.safety.aggregate.aggregate_columns`. - ``sampled_rows=None``; ``aggregates`` is a dict keyed by hashed name + ``sampled_rows=None``; ``aggregates`` is a tuple of (hashed-name, (redacted columns, value=``None``) or real name (otherwise, value= :class:`ColumnStats`). * :attr:`SamplingMode.SAMPLE` — calls ``adapter.sample_rows`` inside the @@ -154,7 +154,11 @@ def build_llm_request( [c.name for c in model.columns_list], policy, ) - aggregates = aggregates_dict + # Convert to tuple-of-tuples so the LLMRequest.aggregates field is + # transitively immutable (DEC-022). A bare dict on a frozen Pydantic + # model is still mutable in its contents, which would let a downstream + # consumer rewrite values after the audit log has been written. + aggregates = tuple(aggregates_dict.items()) elif policy.mode is SamplingMode.SAMPLE: table = TableRef.from_model(model) with adapter: diff --git a/tests/safety/test_models.py b/tests/safety/test_models.py index df892635..53b1569f 100644 --- a/tests/safety/test_models.py +++ b/tests/safety/test_models.py @@ -199,6 +199,35 @@ def test_llm_request_sampled_rows_immutable_when_present() -> None: request.sampled_rows = None # type: ignore[misc] +def test_llm_request_aggregates_is_tuple_of_tuples_when_present() -> None: + """Regression: ``aggregates`` was ``dict`` (mutable post-frozen) — caught by + Quality-Gate review. Now ``tuple[tuple[str, ColumnStats|None], ...]`` so + downstream consumers (#5) cannot ``request.aggregates["x"] = ...`` after + the audit log has been written (DEC-022 transitive immutability).""" + from signalforge.warehouse.models import ColumnStats + + stats = ColumnStats(count=10, distinct=5, nulls=0, min=0, max=9, data_type="INT64") + request = _valid_llm_request(aggregates=(("id", stats), ("col_a3f29c61", None))) + assert request.aggregates is not None + assert request.aggregates.__class__ is tuple + for entry in request.aggregates: + assert entry.__class__ is tuple + assert len(entry) == 2 + assert isinstance(entry[0], str) + assert entry[1] is None or isinstance(entry[1], ColumnStats) + + +def test_llm_request_aggregates_immutable_when_none() -> None: + request = _valid_llm_request(aggregates=None) + assert request.aggregates is None + + +def test_llm_request_aggregates_field_reassignment_blocked_by_frozen() -> None: + request = _valid_llm_request(aggregates=(("id", None),)) + with pytest.raises(ValidationError): + request.aggregates = None # type: ignore[misc] + + def test_llm_request_schema_field_is_tuple_of_tuples() -> None: request = _valid_llm_request() assert request.schema.__class__ is tuple diff --git a/tests/safety/test_policy.py b/tests/safety/test_policy.py index 449ee993..3717fb95 100644 --- a/tests/safety/test_policy.py +++ b/tests/safety/test_policy.py @@ -13,6 +13,7 @@ import logging import re from pathlib import Path +from typing import Any import pytest from pydantic import ValidationError @@ -138,6 +139,19 @@ def test_safety_policy_mode_unknown_raises_invalid_sampling_mode_error() -> None SafetyPolicy.model_validate({"mode": "phantom"}) +@pytest.mark.parametrize("bad_value", [42, 3.14, None, [], {}, ("schema-only",), object()]) +def test_safety_policy_mode_non_string_non_enum_raises_invalid_sampling_mode_error( + bad_value: Any, +) -> None: + """Regression: the ``@field_validator(mode="before")`` previously fell + through to Pydantic's generic ``ValidationError`` when given non-string, + non-enum input (e.g. ``mode=42``). Now every invalid type raises the + typed ``InvalidSamplingModeError`` so the safety-layer error hierarchy + stays homogeneous. Caught by Quality-Gate review.""" + with pytest.raises(InvalidSamplingModeError): + SafetyPolicy.model_validate({"mode": bad_value}) + + # --------------------------------------------------------------------------- # Pattern-injection rejection (DEC-023) # --------------------------------------------------------------------------- @@ -218,6 +232,46 @@ def test_safety_policy_schema_only_mode_no_warning( assert warning_records == [] +def test_safety_policy_with_mode_sample_emits_warning( + caplog: pytest.LogCaptureFixture, +) -> None: + """Regression: ``with_mode(SAMPLE)`` MUST re-trigger the validator chain. + + Pydantic v2's ``model_copy(update=...)`` skips ``@model_validator(mode="after")`` + by default — that path was the original implementation and silently + enabled sample mode without WARNING when the CLI's ``--mode sample`` flag + flowed through. Caught by Quality-Gate review; ``with_mode`` now goes + through ``model_validate`` so the WARNING fires every time sample mode is + enabled, regardless of whether construction was direct or via override. + """ + policy = SafetyPolicy() # SCHEMA_ONLY, no warning + caplog.clear() + with caplog.at_level(logging.WARNING, logger="signalforge.safety"): + overridden = policy.with_mode(SamplingMode.SAMPLE) + assert overridden.mode is SamplingMode.SAMPLE + warning_records = [ + r for r in caplog.records if r.name == "signalforge.safety" and r.levelno == logging.WARNING + ] + assert len(warning_records) == 1 + assert "Sample mode enabled" in warning_records[0].getMessage() + + +def test_safety_policy_with_mode_schema_only_emits_no_warning( + caplog: pytest.LogCaptureFixture, +) -> None: + """Symmetric check: switching FROM sample mode (or staying off it) must + not spuriously fire the warning.""" + policy = SafetyPolicy(mode=SamplingMode.SAMPLE) + caplog.clear() + with caplog.at_level(logging.WARNING, logger="signalforge.safety"): + overridden = policy.with_mode(SamplingMode.SCHEMA_ONLY) + assert overridden.mode is SamplingMode.SCHEMA_ONLY + warning_records = [ + r for r in caplog.records if r.name == "signalforge.safety" and r.levelno == logging.WARNING + ] + assert warning_records == [] + + # --------------------------------------------------------------------------- # Frozen # --------------------------------------------------------------------------- diff --git a/tests/safety/test_request.py b/tests/safety/test_request.py index 434268d4..06aa6fdc 100644 --- a/tests/safety/test_request.py +++ b/tests/safety/test_request.py @@ -172,11 +172,13 @@ def test_build_llm_request_aggregate_only_calls_column_stats_per_non_redacted_co fake.assert_all_expectations_met() assert request.sampled_rows is None assert request.aggregates is not None - # ``id`` keyed by real name with stats; redacted columns keyed by hashed - # name with None. - assert "id" in request.aggregates - assert isinstance(request.aggregates["id"], ColumnStats) - assert request.aggregates[hash_column_name("email")] is None + # aggregates is tuple[tuple[name, stats], ...] — convert to dict for the + # membership checks. The tuple shape (vs. dict) is the DEC-022 immutability + # guarantee: downstream consumers can't mutate values post-audit. + aggregates_by_name = dict(request.aggregates) + assert "id" in aggregates_by_name + assert isinstance(aggregates_by_name["id"], ColumnStats) + assert aggregates_by_name[hash_column_name("email")] is None def test_build_llm_request_aggregate_only_no_sample_rows_calls( From 0b8647a5eb47c2dafd367db03f801d6b6ee7bcb9 Mon Sep 17 00:00:00 2001 From: wjduenow Date: Wed, 29 Apr 2026 08:01:35 -0700 Subject: [PATCH 18/19] =?UTF-8?q?bd=5F1-scaffolding-51d:=20US-014=20?= =?UTF-8?q?=E2=80=94=20Patterns=20&=20Memory=20(rules/safety-layer.md=20+?= =?UTF-8?q?=20CLAUDE.md=20+=20bd=20remember)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New .claude/rules/safety-layer.md distils the load-bearing patterns from issue #4: fail-closed audit semantics, column-name redaction via stable blake2b-4 hash, AuditEvent reproducibility fields (signalforge_version + policy_hash + audit_schema_version), extra=forbid on config-shaped models vs extra=ignore on read-back, the four opt-out signals + precedence, ANSI-safe lazy-format logger, AST audit-completeness scan, model_copy re-validation gotcha, signalforge.yml top-level namespace. CLAUDE.md updated: 'Repository status' bullet for issue #4, 'Public API surface (v0.1)' entries for signalforge.safety (SamplingMode, SafetyPolicy, LLMRequest, load_safety_config, build_llm_request, SafetyError hierarchy). Four bd remember entries: - schema-only mode redacts column names too - audit writes are fail-closed - Pydantic v2 extra=forbid (config) vs extra=ignore (read-back) split - model_copy doesn't re-run @model_validator(mode='after') Co-Authored-By: Claude Opus 4.7 (1M context) --- .claude/rules/safety-layer.md | 89 +++++++++++++++++++++++++++++++++++ CLAUDE.md | 8 ++-- 2 files changed, 94 insertions(+), 3 deletions(-) create mode 100644 .claude/rules/safety-layer.md diff --git a/.claude/rules/safety-layer.md b/.claude/rules/safety-layer.md new file mode 100644 index 00000000..59c18912 --- /dev/null +++ b/.claude/rules/safety-layer.md @@ -0,0 +1,89 @@ +# Safety layer (PII redaction + audit log) + +Established by issue #4 (PII safety layer). Apply to every module under `signalforge.safety` and to any new code that constructs an `LLMRequest`, redacts data heading to an LLM, or writes an audit record. + +The safety layer sits between the warehouse adapter (#3) and the LLM-drafting pipeline (#5). It enforces SignalForge's deployment-blocker safety posture: schema-only by default, sample is explicit opt-in, every LLM call leaves a durable audit receipt. + +## Fail-closed audit semantics (DEC-011) + +Any exception inside `audit.write` propagates as `AuditWriteError` from `build_llm_request`. The function never returns an `LLMRequest` whose audit record didn't durably hit disk. Concretely: + +- `audit.write` opens with `O_APPEND | O_CREAT | 0o600`, writes one JSONL line, calls `os.fsync`, closes. +- Catches **no** exceptions internally — `OSError`, `PermissionError`, `IOError`, encoding failures all propagate. +- Size cap (`_AUDIT_RECORD_LIMIT_BYTES = 4000`) checked **before** any file open, so an oversize record leaves no artifact. +- `build_llm_request` calls `audit.write` AFTER constructing the request but BEFORE returning it. If audit fails, the partial request is dropped. + +An unaudited LLM call is, by definition, PII leaving the warehouse without a receipt — exactly the failure mode this layer exists to prevent. Don't add try/except around audit writes "to be defensive"; the propagation IS the defence. + +## Column NAMES leak PII too — redact them with stable hashes (DEC-010) + +A column named `customer_ssn` or `john_smith_email` leaks PII via the name itself, even when no values are sampled. The layer redacts NAMES in `schema-only` and `aggregate-only` modes, not just values in `sample` mode. + +The hash is `f"col_{blake2b(name.encode(), digest_size=4).hexdigest()}"` — 8 hex chars, deterministic per name, computed once and recorded in `RedactionRecord.hashed_name`. The (real → hashed) mapping is persisted to the audit JSONL so a reviewer can map back; the LLM only ever sees the hash. + +When adding a new mode or surface, make sure the LLM-bound payload uses hashed names for every column where `RedactionRecord.redacted is True`. + +## AuditEvent reproducibility fields (DEC-014) + +Every `AuditEvent` carries three fields that look minor but are load-bearing: + +- `signalforge_version: str` — read from `signalforge.__version__` at write time. Lets a reviewer know which code produced the record. +- `policy_hash: str` — 16-hex-char SHA-256 of the resolved `SafetyPolicy.model_dump_json` (sorted keys, canonical form via the `_compute_policy_hash` helper). Lets a reviewer verify all records in a run came from the same policy. +- `audit_schema_version: int = 1` — frozen at the constant in production code. Bump when the JSONL schema evolves; v0.2 readers gate on this. + +The drift-detector test (`tests/safety/test_drift_detector.py`) pairs production `AuditEvent` (`extra="ignore"`) with a one-off `StrictAuditEvent` (`extra="forbid"`) validated against the committed JSONL fixture. Adding a field to production without updating the strict model OR the fixture breaks the test loudly. Don't bypass. + +## Config-shaped models use `extra="forbid"`; read-back models use `extra="ignore"` (DEC-015) + +The default in `manifest-readers.md` is `extra="ignore"` for forward-compat. The safety layer **deliberately overrides** for config files: + +- `SafetyPolicy`, `_SafetyPolicyContent` (the inner `signalforge.yml` block), `_SafetyConfigFile` → `extra="forbid"`. A typo like `redacts:` (vs `redact:`) in a user-authored YAML file MUST fail loud — silent no-op is exactly the failure mode this ticket exists to prevent. +- `AuditEvent`, `RedactionRecord`, `LLMRequest` → `extra="ignore"`. These are read back from JSONL files / consumed by downstream stages where forward-compat matters. + +Pair every `extra="ignore"` reader-shaped model with a one-off `extra="forbid"` drift detector. + +## The four opt-out signals + precedence (DEC-003) + +`_classify_column` returns `RedactionRecord | None` based on (in precedence order, first match wins, column-level always beats model-level when both fire): + +1. Column-level `meta.signalforge.sample == False` → `column_meta_optout` +2. Column-level `tags: ["pii"]` (case-insensitive) → `tag_pii_column` +3. Column-level `meta.contains_pii` truthy → `meta_contains_pii_column` +4. Model-level `meta.signalforge.sample == False` → `model_meta_optout` +5. Model-level `tags: ["pii"]` → `tag_pii_model` +6. Model-level `meta.contains_pii` truthy → `meta_contains_pii_model` +7. Pattern match against `policy.redact_patterns` (case-insensitive `fnmatch`) → `pattern_match` + +Every coercion path emits a DEBUG log when it normalises (e.g., `tags: [PII]` → lowercase, `meta.contains_pii: "yes"` → True). The seven reasons are a `Literal[...]` so audit-log consumers can pattern-match exhaustively. Don't add an eighth without updating both production `RedactionReason` and the drift detector. + +## ANSI-safe lazy-format logger (DEC-022) + +Every `_LOGGER.{info,warning,debug,error}` call in `signalforge.safety.*` uses lazy-format with `json.dumps()` for any user-controlled string: + +```python +_LOGGER.info("audit event: %s", json.dumps({"unique_id": event.model_unique_id, ...})) +``` + +**Never** `_LOGGER.info(f"... {model_unique_id} ...")` — a column name or model id containing ANSI escapes (`\x1b[31m...`) would inject into log viewers. JSON encoding handles this; f-string interpolation does not. Quality-gate validation greps for `_LOGGER.\w+\(f"` and rejects any hits in `src/signalforge/safety/`. + +## AST audit-completeness scan (DEC-020(a)) + +`tests/safety/test_public_api.py::test_llm_request_construction_only_in_request_module` scans every `.py` under `src/signalforge/safety/` (excluding `request.py`) for `Call(func=Name(id="LLMRequest"))` and rejects any hits. The convention is "construct `LLMRequest` only via `build_llm_request`" — the docstring on `LLMRequest` says so, and the AST scan enforces it. + +If you add a new module that genuinely needs to construct an `LLMRequest` (e.g., a deserialiser for resumption), update the AST-scan exclusion list AND document the audit-write seam. Don't suppress the test. + +## Pydantic v2 `with_mode` re-runs validators (DEC-018) + +`SafetyPolicy.with_mode(mode)` does NOT use `model_copy(update=...)` — that path silently skips `@model_validator(mode="after")`, which means going through the documented CLI override seam would silently enable sample mode without emitting the DEC-021 WARNING. + +Use `model_validate({**self.model_dump(), "mode": mode})` so every validator re-runs. This was caught by Quality-Gate review and is a regression worth defending against in any future "factory that produces a mutated copy" helpers — `model_copy` is the wrong tool any time a validator side-effect is part of the contract. + +## `signalforge.yml` top-level namespace: `safety:` (DEC-025) + +The config file's top level is `{ safety: { ... } }`. Other top-level keys (`llm:`, `prune:`, `grade:`, ...) are reserved for future stages and silently ignored by the safety loader. Each stage validates its own subtree independently. + +When introducing a new pipeline stage with config, claim its own top-level key. Don't pile config under `safety:` — that violates the namespacing reservation and forces a v2-config migration when you eventually split. + +## Reference + +`plans/super/4-pii-safety.md` — DEC-001 … DEC-026. `src/signalforge/safety/` — current implementation. `tests/safety/_fake_adapter.py` — `FakeAdapter` + `expect_*` API. `docs/safety-ops.md` — operational reference. `tests/fixtures/safety/manifest_with_pii_meta.json` — fixture exercising all four opt-out signals. diff --git a/CLAUDE.md b/CLAUDE.md index 75e746a9..48b71e03 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,20 +4,22 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co ## Repository status -Pre-alpha. Three issues shipped: +Pre-alpha. Four issues shipped: - **#1 (project scaffolding)** — `pyproject.toml` (Hatchling + src layout), `src/signalforge/__init__.py` with `__version__`, smoke test, ruff + pyright + pytest configs, GitHub Actions CI on PRs into `dev` and pushes to `main`, and `CONTRIBUTING.md`. - **#2 (manifest loader)** — `signalforge.manifest` subpackage: typed `Manifest` / `Model` (Pydantic v2), `load(project_dir, manifest_path=None) -> Manifest`, single-model resolver by `unique_id` or file path, schema-version tolerance v9–v12, symlink-hardened path canonicalisation, soft 200 MB warning. See `docs/manifest-loader-ops.md` for the operational reference. - **#3 (BigQuery warehouse adapter)** — `signalforge.warehouse` subpackage: `WarehouseAdapter` ABC + `from_profile` factory, `BigQueryAdapter` (the only v0.1 concrete adapter), `load_profile` for dbt `profiles.yml`, deterministic hash-mod sampling with fail-loud sizing checks, identifier-validation at construction time, `QueryJobConfig` defaults that pin `use_query_cache=False`. See `docs/warehouse-adapter-ops.md` for the operational reference. +- **#4 (PII safety layer)** — `signalforge.safety` subpackage: schema-only-default sampling-mode policy, fail-closed audit JSONL writer (`O_APPEND` + `fsync` + size cap), column-name redaction via stable blake2b-4 hashes (DEC-010), four opt-out signals (column meta, model meta, `tags:[pii]`, `meta.contains_pii`) with documented precedence, AST audit-completeness scan rejecting direct `LLMRequest` construction, drift-detector test for `AuditEvent`. `signalforge.yml` config namespace `{ safety: { ... } }` reserved for the safety layer. See `docs/safety-ops.md` for the operational reference and `.claude/rules/safety-layer.md` for the rules distilled from this ticket. -Design is happening in the open on the `dev` branch; remaining feature work (LLM client, prune logic) lands next. +Design is happening in the open on the `dev` branch; remaining feature work (LLM client #5, prune logic #6, grader #7, diff renderer #8, CLI #9) lands next. ## Public API surface (v0.1) - `signalforge.manifest.load`, `Manifest`, `Model`, and the `ManifestError` hierarchy. Documented in `docs/manifest-loader-ops.md`. - `signalforge.warehouse.load_profile`, `DbtProfileTarget`, the `WarehouseAdapter` ABC + `from_profile` factory, the `BigQueryAdapter` concrete, the typed value objects (`Dialect`, `BIGQUERY_DIALECT`, `TableRef`, `PartitionFilter`, `ColumnStats`, `TestResult`), and the `WarehouseError` hierarchy. Documented in `docs/warehouse-adapter-ops.md`. +- `signalforge.safety.load_safety_config`, `SafetyPolicy`, `build_llm_request`, `aggregate_columns`, `redact_rows`, the typed shapes (`SamplingMode`, `RedactionReason`, `RedactionRecord`, `AuditEvent`, `LLMRequest`), and the `SafetyError` hierarchy (10 classes). Documented in `docs/safety-ops.md`. -Internals (`_loader_helpers`, `_sql_safety`, `_path_safety`, `_test_result_repr`, `adapters/_client`, etc.) are `_`-prefixed and not part of the public contract. +Internals (`_loader_helpers`, `_sql_safety`, `_path_safety`, `_test_result_repr`, `adapters/_client`, `_classify_column`, `_compute_policy_hash`, `_resolve_redact_patterns`, etc.) are `_`-prefixed and not part of the public contract. ## Validation From 476b1960b3aac95c0d43f63b392c6014413d08ee Mon Sep 17 00:00:00 2001 From: wjduenow Date: Wed, 29 Apr 2026 08:18:15 -0700 Subject: [PATCH 19/19] Address Copilot PR #18 review (3 real bugs + 2 doc fixes) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five Copilot review comments. Three real bugs and two doc inaccuracies: 1. config.py:101 (real bug) — load_safety_config's default-fallback branches returned SafetyPolicy() with the relative Path('.signalforge/audit.jsonl') field default, making the audit log resolve relative to CWD not project_dir. Now every default-fallback branch (no file / empty file / comments-only / missing safety: key / safety: present but no audit_path) canonicalises DEFAULT_AUDIT_PATH against project_dir before passing to SafetyPolicy. Symmetric with the user-override path. 2. config.py:136 (real bug) — audit_path: 123 (or [a,b], or true) crashed inside Path(audit_path_raw) with TypeError, leaking a non-SafetyError. Now type-checked at the gate: non-(str|os.PathLike) raises InvalidConfigError with a clear remediation pointing at YAML quoting. 3. config.py validator path (real bug) — Pydantic's extra="forbid" raises ValidationError with type='extra_forbidden', which the loader was wrapping as the generic PolicyValidationError. DEC-026 specified UnknownConfigKeyError for typos like `redacts:` or `mode_:`. Now the loader walks the Pydantic error list a second time and translates extra_forbidden into the typed UnknownConfigKeyError so the contract in safety-ops.md actually holds. 4. models.py:70 (doc bug) — RedactionRecord docstring claimed records are emitted for every column considered. They aren't — only redacted columns produce records (None pass-through for non-redacted). Updated docstring to match. 5. safety-ops.md:72 (doc bug) — aggregate-only example showed request.aggregates as a dict. After the Quality-Gate fix that made LLMRequest.aggregates a tuple-of-tuples (DEC-022 transitive immutability), the example was stale. Updated. The sixth Copilot comment (request.py:5) flagged stale PR metadata ('Phase: detailing (awaiting approval)' / 'Review the plan in this PR') that I had already fixed via REST API after the PR was opened. The title is now '4: PII safety layer' and the description reflects the shipped implementation. Will mark that comment as outdated. Eight new regression tests: - test_load_safety_config_no_file_default_audit_path_is_inside_project - test_load_safety_config_empty_file_default_audit_path_is_inside_project - test_load_safety_config_missing_safety_key_default_audit_path_is_inside_project - test_load_safety_config_no_audit_path_in_yaml_canonicalises_default - test_load_safety_config_typo_fixture_raises_unknown_config_key_error (renamed + tightened from generic SafetyError check) - test_load_safety_config_top_level_typo_raises_unknown_config_key_error - test_load_safety_config_audit_path_int_raises_invalid_config_error - test_load_safety_config_audit_path_list_raises_invalid_config_error - test_load_safety_config_audit_path_bool_raises_invalid_config_error Validation: 431 passed (up from 423), ruff/pyright/format all clean. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/safety-ops.md | 17 +++-- src/signalforge/safety/config.py | 58 ++++++++++++-- src/signalforge/safety/models.py | 13 ++-- tests/safety/test_config.py | 126 ++++++++++++++++++++++++++++--- 4 files changed, 188 insertions(+), 26 deletions(-) diff --git a/docs/safety-ops.md b/docs/safety-ops.md index 9c0d9838..71bc75e6 100644 --- a/docs/safety-ops.md +++ b/docs/safety-ops.md @@ -59,17 +59,22 @@ request = build_llm_request(model, adapter, policy) Column-level statistics — `count`, `distinct`, `nulls`, `min`, `max`, `data_type` — reach the LLM via the `aggregates` field. Calls `WarehouseAdapter.column_stats` once per non-redacted column inside a -single `with adapter:` block. Redacted columns are still keyed in the -dict, but the value is `None` keyed by their hashed name. +single `with adapter:` block. Redacted columns still appear as entries +in the returned tuple, but their statistic value is `None` and their +column name is the hashed placeholder. + +`LLMRequest.aggregates` is a `tuple[tuple[str, ColumnStats | None], ...]` +(not a dict) so `frozen=True` actually prevents mutation downstream +(DEC-022 transitive immutability). ```python policy = SafetyPolicy(mode=SamplingMode.AGGREGATE_ONLY) request = build_llm_request(model, adapter, policy) -# request.aggregates == { -# "customer_id": ColumnStats(count=42, distinct=42, nulls=0, ...), -# "col_a3f29c61": None, # redacted +# request.aggregates == ( +# ("customer_id", ColumnStats(count=42, distinct=42, nulls=0, ...)), +# ("col_a3f29c61", None), # redacted # ... -# } +# ) ``` ### `sample` diff --git a/src/signalforge/safety/config.py b/src/signalforge/safety/config.py index f114b5dd..ef9215aa 100644 --- a/src/signalforge/safety/config.py +++ b/src/signalforge/safety/config.py @@ -33,6 +33,7 @@ from __future__ import annotations import logging +import os from pathlib import Path from typing import Final @@ -48,7 +49,7 @@ PolicyValidationError, UnknownConfigKeyError, ) -from signalforge.safety.policy import SafetyPolicy +from signalforge.safety.policy import DEFAULT_AUDIT_PATH, SafetyPolicy _LOGGER: Final = logging.getLogger("signalforge.safety") _DEFAULT_CONFIG_FILENAME: Final = "signalforge.yml" @@ -86,6 +87,15 @@ def load_safety_config(project_dir: Path, path: Path | None = None) -> SafetyPol PolicyValidationError: Generic Pydantic validation failure not covered by the more specific exceptions above. """ + # Default audit_path is project-relative (`.signalforge/audit.jsonl`). + # Canonicalise against project_dir so every default-fallback branch ends + # up with the same absolute, symlink-hardened path the user-override + # branch produces. Without this, a user who omits `signalforge.yml` + # gets an audit log at CWD-relative `.signalforge/audit.jsonl` rather + # than `/.signalforge/audit.jsonl`. Reported by Copilot + # PR review. + default_audit_path = canonicalise_path(DEFAULT_AUDIT_PATH, project_dir) + if path is not None: config_file = path if not config_file.exists(): @@ -93,12 +103,12 @@ def load_safety_config(project_dir: Path, path: Path | None = None) -> SafetyPol else: config_file = project_dir / _DEFAULT_CONFIG_FILENAME if not config_file.exists(): - return SafetyPolicy() + return SafetyPolicy(audit_path=default_audit_path) raw_text = config_file.read_text(encoding="utf-8").strip() if not raw_text: _LOGGER.debug("safety config file %r is empty; using defaults", str(config_file)) - return SafetyPolicy() + return SafetyPolicy(audit_path=default_audit_path) try: loaded = yaml.safe_load(raw_text) @@ -109,7 +119,7 @@ def load_safety_config(project_dir: Path, path: Path | None = None) -> SafetyPol if loaded is None: # File parses to None (e.g. only comments) — same as empty. - return SafetyPolicy() + return SafetyPolicy(audit_path=default_audit_path) if not isinstance(loaded, dict): raise InvalidConfigError( @@ -119,7 +129,7 @@ def load_safety_config(project_dir: Path, path: Path | None = None) -> SafetyPol safety_block = loaded.get("safety") if safety_block is None: # Missing safety: key — other top-level keys reserved per DEC-025. - return SafetyPolicy() + return SafetyPolicy(audit_path=default_audit_path) if not isinstance(safety_block, dict): raise InvalidConfigError( @@ -132,6 +142,21 @@ def load_safety_config(project_dir: Path, path: Path | None = None) -> SafetyPol # then canonicalise + containment-check via the symlink-hardened helper. audit_path_raw = safety_block.get("audit_path") if audit_path_raw is not None: + # Validate the type up front. YAML can parse `audit_path: 123` to + # an int (or `audit_path:` to None — already filtered above) which + # would crash inside `Path(...)` with TypeError, leaking a + # non-SafetyError exception. Reported by Copilot PR review. + if not isinstance(audit_path_raw, (str, os.PathLike)): + raise InvalidConfigError( + message=( + f"audit_path must be a string or path-like; " + f"got {type(audit_path_raw).__name__} ({audit_path_raw!r})" + ), + remediation=( + "Quote the value in signalforge.yml so YAML parses it as a string " + "(default: .signalforge/audit.jsonl)." + ), + ) candidate = Path(audit_path_raw) if any(part == ".." for part in candidate.parts): raise InvalidConfigError( @@ -140,10 +165,17 @@ def load_safety_config(project_dir: Path, path: Path | None = None) -> SafetyPol "Use a path inside the project directory (default: .signalforge/audit.jsonl)." ), ) - resolved = canonicalise_path(audit_path_raw, project_dir) + # `canonicalise_path` accepts `Path | str`; narrow `os.PathLike` + # input through `Path(...)` (already done as `candidate` above). + resolved = canonicalise_path(candidate, project_dir) # Replace the raw value with the resolved Path so SafetyPolicy # stores the canonical form. safety_block = {**safety_block, "audit_path": resolved} + else: + # User did not override audit_path — apply the canonicalised default + # so the policy's audit_path is always absolute (symmetric with the + # branches above). + safety_block = {**safety_block, "audit_path": default_audit_path} try: return SafetyPolicy.model_validate(safety_block) @@ -172,6 +204,20 @@ def load_safety_config(project_dir: Path, path: Path | None = None) -> SafetyPol ), ): raise inner from exc + # Translate Pydantic's `extra_forbidden` into our typed + # UnknownConfigKeyError so DEC-026's contract holds: typos like + # `redacts:` (instead of `redact:`) at the top of the safety block + # surface as UnknownConfigKeyError, not the generic policy-validation + # error. Reported by Copilot PR review. + for err in exc.errors(): + if err.get("type") == "extra_forbidden": + loc = err.get("loc", ()) + if loc: + bad_key = str(loc[-1]) + scope = ( + "safety." + ".".join(str(p) for p in loc[:-1]) if len(loc) > 1 else "safety" + ) + raise UnknownConfigKeyError(key=bad_key, scope=scope) from exc # Last-resort wrap: surface the Pydantic failure as a typed # safety-layer error so callers can pattern-match. raise PolicyValidationError( diff --git a/src/signalforge/safety/models.py b/src/signalforge/safety/models.py index 8c3358d3..754adf25 100644 --- a/src/signalforge/safety/models.py +++ b/src/signalforge/safety/models.py @@ -63,11 +63,14 @@ class SamplingMode(str, Enum): class RedactionRecord(BaseModel): - """One column's redaction outcome. - - Emitted by the redactor for every column considered (whether kept or - dropped) so the audit log records the *full* decision surface, not just - the redactions actually applied. + """One applied column redaction. + + Emitted only for columns the redactor actually drops or masks. Columns + that pass through unchanged do not produce a :class:`RedactionRecord`, + so audit/request payloads capture the redactions that were applied + rather than the full set of columns considered. The ``reason`` field is + a closed :data:`RedactionReason` literal so audit-log consumers can + pattern-match exhaustively on the seven possible signals. """ model_config = _BASE_MODEL_CONFIG diff --git a/tests/safety/test_config.py b/tests/safety/test_config.py index fcbe17e1..a93a6c48 100644 --- a/tests/safety/test_config.py +++ b/tests/safety/test_config.py @@ -51,6 +51,60 @@ def test_load_safety_config_no_file_default_redact_patterns(tmp_path: Path) -> N assert load_safety_config(tmp_path).redact_patterns == DEFAULT_REDACT_PATTERNS +def test_load_safety_config_no_file_default_audit_path_is_inside_project( + tmp_path: Path, +) -> None: + """Regression (Copilot PR #18 review): the default ``audit_path`` must be + canonicalised against ``project_dir`` even when no config file exists. + Without the fix, ``SafetyPolicy()``'s relative default + (``.signalforge/audit.jsonl``) makes the audit log resolve relative to + the orchestrator's CWD, not the project — silently writing the audit + record to the wrong place and bypassing the DEC-013 containment gate. + """ + policy = load_safety_config(tmp_path) + # Resolve the project root the same way the loader does (canonicalise + # follows symlinks, so on macOS `/private/var/...` vs `/var/...`). + project_resolved = tmp_path.resolve() + assert policy.audit_path.is_absolute() + assert policy.audit_path.is_relative_to(project_resolved) + assert policy.audit_path.name == "audit.jsonl" + assert policy.audit_path.parent.name == ".signalforge" + + +def test_load_safety_config_empty_file_default_audit_path_is_inside_project( + tmp_path: Path, +) -> None: + (tmp_path / "signalforge.yml").write_bytes(b"") + policy = load_safety_config(tmp_path) + assert policy.audit_path.is_absolute() + assert policy.audit_path.is_relative_to(tmp_path.resolve()) + + +def test_load_safety_config_missing_safety_key_default_audit_path_is_inside_project( + tmp_path: Path, +) -> None: + """Same regression as the no-file branch: when only an unrelated + top-level key is present (DEC-025 namespace reservation), the + ``audit_path`` default must still be canonicalised.""" + (tmp_path / "signalforge.yml").write_text("llm:\n model: x\n", encoding="utf-8") + policy = load_safety_config(tmp_path) + assert policy.audit_path.is_absolute() + assert policy.audit_path.is_relative_to(tmp_path.resolve()) + + +def test_load_safety_config_no_audit_path_in_yaml_canonicalises_default( + tmp_path: Path, +) -> None: + """User authored ``signalforge.yml`` but didn't override ``audit_path`` — + the policy's audit_path must still be the canonicalised default, not the + relative ``Path('.signalforge/audit.jsonl')`` from ``SafetyPolicy``'s + field default.""" + (tmp_path / "signalforge.yml").write_text("safety:\n mode: schema-only\n", encoding="utf-8") + policy = load_safety_config(tmp_path) + assert policy.audit_path.is_absolute() + assert policy.audit_path.is_relative_to(tmp_path.resolve()) + + def test_load_safety_config_empty_file_returns_defaults(tmp_path: Path) -> None: (tmp_path / "signalforge.yml").write_bytes(b"") policy = load_safety_config(tmp_path) @@ -171,16 +225,70 @@ def test_load_safety_config_unknown_mode_fixture_raises(tmp_path: Path) -> None: load_safety_config(tmp_path) -def test_load_safety_config_typo_fixture_raises(tmp_path: Path) -> None: - """The ``redacts:`` typo (rather than ``redact:``) under ``safety:`` — - the loader must surface a typed safety-layer error. The exact subclass - depends on whether the validator can map it to - :class:`UnknownConfigKeyError` or whether it falls through to the - Pydantic generic path; either way it must inherit from - :class:`SafetyError`. - """ +def test_load_safety_config_typo_fixture_raises_unknown_config_key_error( + tmp_path: Path, +) -> None: + """Regression (Copilot PR #18 review): a ``redacts:`` typo (vs ``redact:``) + under ``safety:`` must surface as the typed + :class:`UnknownConfigKeyError`, not the generic + :class:`PolicyValidationError`. Pydantic's ``extra="forbid"`` raises + ``ValidationError`` with ``type=='extra_forbidden'``; the loader + translates that into our typed error so DEC-026 holds.""" + from signalforge.safety.errors import UnknownConfigKeyError + _copy_fixture("signalforge_typo.yml", tmp_path) - with pytest.raises(SafetyError): + with pytest.raises(UnknownConfigKeyError) as excinfo: + load_safety_config(tmp_path) + # The bad key surfaces in the error so users can find it. + assert "redacts" in str(excinfo.value) + + +def test_load_safety_config_top_level_typo_raises_unknown_config_key_error( + tmp_path: Path, +) -> None: + """Same translation applies to typos at any level inside ``safety:``.""" + from signalforge.safety.errors import UnknownConfigKeyError + + (tmp_path / "signalforge.yml").write_text("safety:\n mode_: schema-only\n", encoding="utf-8") + with pytest.raises(UnknownConfigKeyError) as excinfo: + load_safety_config(tmp_path) + assert "mode_" in str(excinfo.value) + + +# --------------------------------------------------------------------------- +# audit_path type validation (Copilot PR #18 review) +# --------------------------------------------------------------------------- + + +def test_load_safety_config_audit_path_int_raises_invalid_config_error( + tmp_path: Path, +) -> None: + """Regression: ``audit_path: 123`` previously crashed inside ``Path(123)`` + with ``TypeError``, leaking a non-:class:`SafetyError` exception. Now + rejected at the type-validation gate with :class:`InvalidConfigError`.""" + (tmp_path / "signalforge.yml").write_text("safety:\n audit_path: 123\n", encoding="utf-8") + with pytest.raises(InvalidConfigError) as excinfo: + load_safety_config(tmp_path) + assert "audit_path" in str(excinfo.value) + assert "string or path-like" in str(excinfo.value) + + +def test_load_safety_config_audit_path_list_raises_invalid_config_error( + tmp_path: Path, +) -> None: + """Same gate — a YAML list like ``audit_path: [a, b]`` must fail loud.""" + (tmp_path / "signalforge.yml").write_text( + "safety:\n audit_path:\n - a\n - b\n", encoding="utf-8" + ) + with pytest.raises(InvalidConfigError): + load_safety_config(tmp_path) + + +def test_load_safety_config_audit_path_bool_raises_invalid_config_error( + tmp_path: Path, +) -> None: + (tmp_path / "signalforge.yml").write_text("safety:\n audit_path: true\n", encoding="utf-8") + with pytest.raises(InvalidConfigError): load_safety_config(tmp_path)