Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 26 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,9 @@ signalforge lint --project-dir /tmp/sf-austin
On success, stdout is silent (git-style) and the exit code is `0`.
Failures are listed on stderr with the offending block(s) named —
single-failure runs use the `ERROR: <message>` shape; multi-failure
runs emit a header + one bullet per block. See
runs emit a header + one bullet per block. Pass `--model <name>` to
also confirm a specific model resolves in the manifest (accepts a
bare name, a `unique_id`, or a file path). See
[`docs/cli-ops.md`](docs/cli-ops.md) § `signalforge lint` for the full
contract.

Expand All @@ -157,11 +159,16 @@ contract.
### 7. Expected output

The diff lists drafted column descriptions and signal-bearing tests
alongside dropped tests with a one-line "why". The kept/dropped/flagged
table looks like this (truncated):
alongside dropped tests with a one-line "why". Every artifact lands
in one of four tiers — `kept` (survived prune with positive
evidence), `kept-uncertain` (kept, but the warehouse couldn't be
reached to evaluate it — e.g. a budget or connectivity issue),
`dropped` (prune found it adds no signal), and `flagged` (kept, but
graded below the quality threshold). The table looks like this
(truncated):

```text
diff: model.austin.stg_bikeshare_trips kept=8 dropped=2 flagged=1
diff: model.austin.stg_bikeshare_trips kept=8 kept-uncertain=0 dropped=2 flagged=1

TIER ARTIFACT TEST REASON SCORE WHY
kept column.trip_id.description 0.97 Description added; passed all grading criteria.
Expand Down Expand Up @@ -195,7 +202,7 @@ full rendered diff). The committed `.gitignore` covers `.signalforge/`.
| Symptom | Likely cause | Fix |
| --- | --- | --- |
| `User does not have bigquery.jobs.create permission in project bigquery-public-data` | `GOOGLE_CLOUD_PROJECT` not set; SDK fell back to the source project | Export `GOOGLE_CLOUD_PROJECT=<billing-project>` where you have the `BigQuery Job User` role |
| `Query exceeded max_bytes_billed (limit=100000000, ...)` | Editing the profile dropped or lowered `maximum_bytes_billed` | Keep `maximum_bytes_billed: 1000000000` (1 GB) — the smoke test ships this cap intentionally |
| `Query exceeded max_bytes_billed (limit=100000000, ...)` | Editing the profile dropped or lowered `maximum_bytes_billed` | Keep `maximum_bytes_billed: 1000000000` (1 GB) — the bundled demo `profiles.yml` ships this cap intentionally so the materialised-sample scan clears the adapter's 100 MB default |
| `Manifest not found` / `dbt_project.yml not found at ...` | CLI walked up from the wrong cwd, or `--project-dir` doesn't directly contain `dbt_project.yml` | Either `cd` into the project root, or pass `--project-dir <abs-path>` pointing at the directory holding `dbt_project.yml` |
| `aggregate_complete=False` in `grade.json` | Network blip during a grade call exhausted retries | Re-run; if it persists, raise `grade.total_budget_seconds` in `signalforge.yml` |
| `LLM response did not match the CandidateSchema shape` | Anthropic response shape drifted vs. the parser | Set `ANTHROPIC_LOG=info` and inspect `~/.anthropic-debug/`; file an issue |
Expand All @@ -213,15 +220,25 @@ of the same flow as a gated test (`pytest -m e2e --no-cov`):
Four subcommands ship in v0.1:

```bash
signalforge generate <model> # full pipeline; --mode, --min-score, --write/--dry-run, --format
signalforge init-demo [<dest>] # copy the bundled Austin demo project into <dest>; --force
signalforge lint # validate signalforge.yml config blocks
signalforge generate <model> # full draft -> prune -> grade -> diff pipeline for one model
signalforge init-demo [<dest>] # copy the bundled Austin demo project into <dest>
signalforge lint # validate signalforge.yml config blocks (no LLM/warehouse calls)
signalforge version # print the SignalForge version
```

Key `generate` flags: `--project-dir`, `--manifest`, `--profiles-dir`
(point at the project / manifest / profile); `--mode
{schema-only,aggregate-only,sample}` and `--min-score` (pipeline
behaviour); `--write` / `--dry-run` and `--format {ansi,markdown,json}`
(output); `--estimate` (cost preview, no billable calls); `--select
<expr>` (run across many models); `--scope`, `--sample-strategy`; and
the `--quiet` / `--verbose` / `--no-color` observability triad.
`init-demo` takes `--force`; `lint` takes `--config`, `--manifest`,
`--model`, `--project-dir`.

`signalforge --help` prints the top-level help; each subcommand has its
own `--help` page. See [docs/cli-ops.md](docs/cli-ops.md) for the full
reference.
reference, exit-code taxonomy, and environment variables.

## Configuration

Expand Down
8 changes: 4 additions & 4 deletions docs/audits.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,9 @@ For the per-stage *production* contracts (defaults, error remediation, cost guid

| File | One-line content | Per-record version field | Drift-detector test |
|------|------------------|--------------------------|---------------------|
| `.signalforge/audit.jsonl` | One safety `AuditEvent` per `build_llm_request` call (LLM request leaving the warehouse boundary, plus column-name redaction map) | `audit_schema_version: 1` | `tests/safety/test_drift_detector.py` |
| `.signalforge/audit.jsonl` | One safety `AuditEvent` per `build_llm_request` call (LLM request leaving the warehouse boundary, plus column-name redaction map) | `audit_schema_version: 3` | `tests/safety/test_drift_detector.py` |
| `.signalforge/llm_responses.jsonl` | One `LLMResponseEvent` per successful LLM round-trip in the draft layer (prompt + response hashes, cache-token economics) | `audit_schema_version: 1` | `tests/draft/test_drift_detector.py` |
| `.signalforge/prune.jsonl` | One `PruneEvent` per candidate test routed through the prune orchestrator (decision, reason, compiled SQL hash, optional sample failures) | `audit_schema_version: 1` | `tests/prune/test_drift_detector.py` |
| `.signalforge/prune.jsonl` | One `PruneEvent` per candidate test routed through the prune orchestrator (decision, reason, compiled SQL hash, optional sample failures) | `audit_schema_version: 2` | `tests/prune/test_drift_detector.py` |
| `.signalforge/grade.jsonl` | One `GradeEvent` per `(artifact × criterion)` LLM-as-judge call (score, evidence, reasoning, response-token economics) | `audit_schema_version: 1` | `tests/grade/test_drift_detector.py` |
| `.signalforge/grade.json` | End-of-run `GradingReport` sidecar — aggregate `pass_rate` / `mean_score`, `aggregate_complete`, every per-result row | `grade_schema_version: 1` (top-level) | `tests/grade/test_drift_detector.py` |
| `.signalforge/diff.json` | End-of-run `DiffReport` sidecar — kept / kept-uncertain / dropped / flagged entries, proposed YAML, unified diff, reproducibility hashes | `schema_version: 1` + `audit_schema_version: 2` (top-level) | `tests/diff/test_drift_detector.py` |
Expand Down Expand Up @@ -206,9 +206,9 @@ Every event and report model across the five stages uses Pydantic v2 `extra="ign

`audit_schema_version` is per-shape, not project-wide. The pinned values today:

- `AuditEvent.audit_schema_version: int = 1` (safety)
- `AuditEvent.audit_schema_version: int = 3` (safety — bumped 1 → 2 in issue #54 for the `draft_skip_*` redaction reasons, 2 → 3 in issue #55 for the `policy_hash` recipe change)
- `LLMResponseEvent.audit_schema_version: int = 1` (draft)
- `PruneEvent.audit_schema_version: Literal[1] = 1` (prune)
- `PruneEvent.audit_schema_version: int = 2` (prune — bumped 1 → 2 in issue #55 when `config_hash` migrated to the `blake2b-8` recipe)
- `GradeEvent.audit_schema_version: Literal[1] = 1` (grade per-call)
- `GradingReport.grade_schema_version: Literal[1] = 1` (grade sidecar — separate field from the per-call event)
- `DiffReport.schema_version: Literal[1] = 1` (diff sidecar overall) plus `DiffReport.audit_schema_version: Literal[2] = 2` (diff entries — bumped from 1 in issue #50 alongside the `kept-uncertain` tier literal)
Expand Down
7 changes: 5 additions & 2 deletions docs/e2e-smoke-test.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,8 +133,11 @@ per-run `tmp_path`, substitutes `GOOGLE_CLOUD_PROJECT` into
`profiles.yml`, and runs `signalforge generate` against the temp
project. If you want to run the same flow as a one-off CLI
invocation (rather than through pytest), follow the README's
[Quick start](../README.md#quick-start) walkthrough — it
shows the manual copy + profile-rewrite incantation.
[Quick start](../README.md#quick-start) walkthrough — it uses
`signalforge init-demo` to copy out the bundled demo project, whose
`profiles.yml` already reads `GOOGLE_CLOUD_PROJECT` from the
environment and ships the raised `maximum_bytes_billed` cap, so no
profile rewrite is needed.

### Why the profile rewrite matters

Expand Down
7 changes: 7 additions & 0 deletions src/signalforge/_demo/profiles.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,12 @@
# The Austin bikeshare data is served from the public
# `bigquery-public-data.austin_bikeshare` dataset; your `GOOGLE_CLOUD_PROJECT`
# is the BILLING project the BigQuery SDK uses to issue the read.
#
# `maximum_bytes_billed: 1000000000` (1 GB) raises the adapter's default
# 100 MB cap — the `prune.sample_strategy: materialised` run in the demo's
# `signalforge.yml` does a CTAS over the full ~2.27M-row `bikeshare_trips`
# source, which exceeds 100 MB. 1 GB is still a hard ceiling well inside the
# BigQuery free tier.
austin:
target: dev
outputs:
Expand All @@ -21,3 +27,4 @@ austin:
project: "{{ env_var('GOOGLE_CLOUD_PROJECT') }}"
dataset: austin_bikeshare
location: US
maximum_bytes_billed: 1000000000
7 changes: 5 additions & 2 deletions tests/test_demo_fixture_parity.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,11 @@
documented rewrites:

1. ``profiles.yml`` — the shipped copy uses dbt's ``env_var('GOOGLE_CLOUD_PROJECT')``
macro for the BigQuery project field and drops the maintainer-only
"DO NOT signalforge against this" header (DEC-009).
macro for the BigQuery project field, drops the maintainer-only
"DO NOT signalforge against this" header (DEC-009), and adds
``maximum_bytes_billed: 1000000000`` so the demo's materialised-sample
CTAS clears the adapter's default 100 MB cap (the e2e test fixture
does the equivalent bump in ``tmp_path`` at run time instead).
2. ``.gitignore`` — the shipped copy is slimmed to a single ``.signalforge/``
exclusion; the test-fixture copy keeps the issue-#10 / DEC-021 maintainer
commentary.
Expand Down
Loading