diff --git a/README.md b/README.md index 375e549b..0c7cebf1 100644 --- a/README.md +++ b/README.md @@ -130,7 +130,9 @@ signalforge lint --project-dir /tmp/sf-austin On success, stdout is silent (git-style) and the exit code is `0`. Failures are listed on stderr with the offending block(s) named — single-failure runs use the `ERROR: ` shape; multi-failure -runs emit a header + one bullet per block. See +runs emit a header + one bullet per block. Pass `--model ` to +also confirm a specific model resolves in the manifest (accepts a +bare name, a `unique_id`, or a file path). See [`docs/cli-ops.md`](docs/cli-ops.md) § `signalforge lint` for the full contract. @@ -157,11 +159,16 @@ contract. ### 7. Expected output The diff lists drafted column descriptions and signal-bearing tests -alongside dropped tests with a one-line "why". The kept/dropped/flagged -table looks like this (truncated): +alongside dropped tests with a one-line "why". Every artifact lands +in one of four tiers — `kept` (survived prune with positive +evidence), `kept-uncertain` (kept, but the warehouse couldn't be +reached to evaluate it — e.g. a budget or connectivity issue), +`dropped` (prune found it adds no signal), and `flagged` (kept, but +graded below the quality threshold). The table looks like this +(truncated): ```text -diff: model.austin.stg_bikeshare_trips kept=8 dropped=2 flagged=1 +diff: model.austin.stg_bikeshare_trips kept=8 kept-uncertain=0 dropped=2 flagged=1 TIER ARTIFACT TEST REASON SCORE WHY kept column.trip_id.description 0.97 Description added; passed all grading criteria. @@ -195,7 +202,7 @@ full rendered diff). The committed `.gitignore` covers `.signalforge/`. | Symptom | Likely cause | Fix | | --- | --- | --- | | `User does not have bigquery.jobs.create permission in project bigquery-public-data` | `GOOGLE_CLOUD_PROJECT` not set; SDK fell back to the source project | Export `GOOGLE_CLOUD_PROJECT=` where you have the `BigQuery Job User` role | -| `Query exceeded max_bytes_billed (limit=100000000, ...)` | Editing the profile dropped or lowered `maximum_bytes_billed` | Keep `maximum_bytes_billed: 1000000000` (1 GB) — the smoke test ships this cap intentionally | +| `Query exceeded max_bytes_billed (limit=100000000, ...)` | Editing the profile dropped or lowered `maximum_bytes_billed` | Keep `maximum_bytes_billed: 1000000000` (1 GB) — the bundled demo `profiles.yml` ships this cap intentionally so the materialised-sample scan clears the adapter's 100 MB default | | `Manifest not found` / `dbt_project.yml not found at ...` | CLI walked up from the wrong cwd, or `--project-dir` doesn't directly contain `dbt_project.yml` | Either `cd` into the project root, or pass `--project-dir ` pointing at the directory holding `dbt_project.yml` | | `aggregate_complete=False` in `grade.json` | Network blip during a grade call exhausted retries | Re-run; if it persists, raise `grade.total_budget_seconds` in `signalforge.yml` | | `LLM response did not match the CandidateSchema shape` | Anthropic response shape drifted vs. the parser | Set `ANTHROPIC_LOG=info` and inspect `~/.anthropic-debug/`; file an issue | @@ -213,15 +220,25 @@ of the same flow as a gated test (`pytest -m e2e --no-cov`): Four subcommands ship in v0.1: ```bash -signalforge generate # full pipeline; --mode, --min-score, --write/--dry-run, --format -signalforge init-demo [] # copy the bundled Austin demo project into ; --force -signalforge lint # validate signalforge.yml config blocks +signalforge generate # full draft -> prune -> grade -> diff pipeline for one model +signalforge init-demo [] # copy the bundled Austin demo project into +signalforge lint # validate signalforge.yml config blocks (no LLM/warehouse calls) signalforge version # print the SignalForge version ``` +Key `generate` flags: `--project-dir`, `--manifest`, `--profiles-dir` +(point at the project / manifest / profile); `--mode +{schema-only,aggregate-only,sample}` and `--min-score` (pipeline +behaviour); `--write` / `--dry-run` and `--format {ansi,markdown,json}` +(output); `--estimate` (cost preview, no billable calls); `--select +` (run across many models); `--scope`, `--sample-strategy`; and +the `--quiet` / `--verbose` / `--no-color` observability triad. +`init-demo` takes `--force`; `lint` takes `--config`, `--manifest`, +`--model`, `--project-dir`. + `signalforge --help` prints the top-level help; each subcommand has its own `--help` page. See [docs/cli-ops.md](docs/cli-ops.md) for the full -reference. +reference, exit-code taxonomy, and environment variables. ## Configuration diff --git a/docs/audits.md b/docs/audits.md index 7d111df2..799e8f2c 100644 --- a/docs/audits.md +++ b/docs/audits.md @@ -14,9 +14,9 @@ For the per-stage *production* contracts (defaults, error remediation, cost guid | File | One-line content | Per-record version field | Drift-detector test | |------|------------------|--------------------------|---------------------| -| `.signalforge/audit.jsonl` | One safety `AuditEvent` per `build_llm_request` call (LLM request leaving the warehouse boundary, plus column-name redaction map) | `audit_schema_version: 1` | `tests/safety/test_drift_detector.py` | +| `.signalforge/audit.jsonl` | One safety `AuditEvent` per `build_llm_request` call (LLM request leaving the warehouse boundary, plus column-name redaction map) | `audit_schema_version: 3` | `tests/safety/test_drift_detector.py` | | `.signalforge/llm_responses.jsonl` | One `LLMResponseEvent` per successful LLM round-trip in the draft layer (prompt + response hashes, cache-token economics) | `audit_schema_version: 1` | `tests/draft/test_drift_detector.py` | -| `.signalforge/prune.jsonl` | One `PruneEvent` per candidate test routed through the prune orchestrator (decision, reason, compiled SQL hash, optional sample failures) | `audit_schema_version: 1` | `tests/prune/test_drift_detector.py` | +| `.signalforge/prune.jsonl` | One `PruneEvent` per candidate test routed through the prune orchestrator (decision, reason, compiled SQL hash, optional sample failures) | `audit_schema_version: 2` | `tests/prune/test_drift_detector.py` | | `.signalforge/grade.jsonl` | One `GradeEvent` per `(artifact × criterion)` LLM-as-judge call (score, evidence, reasoning, response-token economics) | `audit_schema_version: 1` | `tests/grade/test_drift_detector.py` | | `.signalforge/grade.json` | End-of-run `GradingReport` sidecar — aggregate `pass_rate` / `mean_score`, `aggregate_complete`, every per-result row | `grade_schema_version: 1` (top-level) | `tests/grade/test_drift_detector.py` | | `.signalforge/diff.json` | End-of-run `DiffReport` sidecar — kept / kept-uncertain / dropped / flagged entries, proposed YAML, unified diff, reproducibility hashes | `schema_version: 1` + `audit_schema_version: 2` (top-level) | `tests/diff/test_drift_detector.py` | @@ -206,9 +206,9 @@ Every event and report model across the five stages uses Pydantic v2 `extra="ign `audit_schema_version` is per-shape, not project-wide. The pinned values today: -- `AuditEvent.audit_schema_version: int = 1` (safety) +- `AuditEvent.audit_schema_version: int = 3` (safety — bumped 1 → 2 in issue #54 for the `draft_skip_*` redaction reasons, 2 → 3 in issue #55 for the `policy_hash` recipe change) - `LLMResponseEvent.audit_schema_version: int = 1` (draft) -- `PruneEvent.audit_schema_version: Literal[1] = 1` (prune) +- `PruneEvent.audit_schema_version: int = 2` (prune — bumped 1 → 2 in issue #55 when `config_hash` migrated to the `blake2b-8` recipe) - `GradeEvent.audit_schema_version: Literal[1] = 1` (grade per-call) - `GradingReport.grade_schema_version: Literal[1] = 1` (grade sidecar — separate field from the per-call event) - `DiffReport.schema_version: Literal[1] = 1` (diff sidecar overall) plus `DiffReport.audit_schema_version: Literal[2] = 2` (diff entries — bumped from 1 in issue #50 alongside the `kept-uncertain` tier literal) diff --git a/docs/e2e-smoke-test.md b/docs/e2e-smoke-test.md index 6564405a..8164ed7e 100644 --- a/docs/e2e-smoke-test.md +++ b/docs/e2e-smoke-test.md @@ -133,8 +133,11 @@ per-run `tmp_path`, substitutes `GOOGLE_CLOUD_PROJECT` into `profiles.yml`, and runs `signalforge generate` against the temp project. If you want to run the same flow as a one-off CLI invocation (rather than through pytest), follow the README's -[Quick start](../README.md#quick-start) walkthrough — it -shows the manual copy + profile-rewrite incantation. +[Quick start](../README.md#quick-start) walkthrough — it uses +`signalforge init-demo` to copy out the bundled demo project, whose +`profiles.yml` already reads `GOOGLE_CLOUD_PROJECT` from the +environment and ships the raised `maximum_bytes_billed` cap, so no +profile rewrite is needed. ### Why the profile rewrite matters diff --git a/src/signalforge/_demo/profiles.yml b/src/signalforge/_demo/profiles.yml index 5cdac61f..684fd876 100644 --- a/src/signalforge/_demo/profiles.yml +++ b/src/signalforge/_demo/profiles.yml @@ -12,6 +12,12 @@ # The Austin bikeshare data is served from the public # `bigquery-public-data.austin_bikeshare` dataset; your `GOOGLE_CLOUD_PROJECT` # is the BILLING project the BigQuery SDK uses to issue the read. +# +# `maximum_bytes_billed: 1000000000` (1 GB) raises the adapter's default +# 100 MB cap — the `prune.sample_strategy: materialised` run in the demo's +# `signalforge.yml` does a CTAS over the full ~2.27M-row `bikeshare_trips` +# source, which exceeds 100 MB. 1 GB is still a hard ceiling well inside the +# BigQuery free tier. austin: target: dev outputs: @@ -21,3 +27,4 @@ austin: project: "{{ env_var('GOOGLE_CLOUD_PROJECT') }}" dataset: austin_bikeshare location: US + maximum_bytes_billed: 1000000000 diff --git a/tests/test_demo_fixture_parity.py b/tests/test_demo_fixture_parity.py index 02393639..0d07b08f 100644 --- a/tests/test_demo_fixture_parity.py +++ b/tests/test_demo_fixture_parity.py @@ -5,8 +5,11 @@ documented rewrites: 1. ``profiles.yml`` — the shipped copy uses dbt's ``env_var('GOOGLE_CLOUD_PROJECT')`` - macro for the BigQuery project field and drops the maintainer-only - "DO NOT signalforge against this" header (DEC-009). + macro for the BigQuery project field, drops the maintainer-only + "DO NOT signalforge against this" header (DEC-009), and adds + ``maximum_bytes_billed: 1000000000`` so the demo's materialised-sample + CTAS clears the adapter's default 100 MB cap (the e2e test fixture + does the equivalent bump in ``tmp_path`` at run time instead). 2. ``.gitignore`` — the shipped copy is slimmed to a single ``.signalforge/`` exclusion; the test-fixture copy keeps the issue-#10 / DEC-021 maintainer commentary.