2: dbt manifest loader (plan)

.claude/rules/manifest-readers.md

@@ -0,0 +1,55 @@
+# External-format readers (Pydantic v2)
+
+Established by issue #2 (manifest loader). Apply to any module in this repo that parses an externally-defined JSON/YAML format (dbt's `manifest.json`, `catalog.json`, `run_results.json`, BigQuery `INFORMATION_SCHEMA` snapshots, etc.).
+
+## Pydantic v2: frozen + extra=ignore in production
+
+```python
+from pydantic import BaseModel, ConfigDict
+
+_BASE = ConfigDict(frozen=True, extra="ignore", populate_by_name=True)
+```
+
+`frozen=True` makes models immutable post-construction. `extra="ignore"` survives upstream schema additions (e.g. dbt Fusion v20 fields) without code changes. `populate_by_name=True` lets callers use either the field name or the alias when constructing.
+
+Do **not** use `extra="forbid"` in production. Reserve it for *test-only* `StrictModel(BaseModel)` subclasses constructed inline in drift-detector tests — those catch silent schema expansion before a live manifest does.
+
+## Symlink-hardened path resolution
+
+Any function that takes a user-supplied path and reads from disk MUST canonicalise via:
+
+```python
+def _canonicalise_path(input_path: Path | str, project_dir: Path) -> Path:
+    try:
+        project_resolved = project_dir.resolve(strict=True)
+        if not Path(input_path).is_absolute():
+            p = project_resolved / input_path
+        else:
+            p = Path(input_path)
+        resolved = p.resolve(strict=False)
+    except RuntimeError as exc:        # symlink loop
+        raise ModelPathOutsideProjectError(...) from exc
+    if not resolved.is_relative_to(project_resolved):
+        raise ModelPathOutsideProjectError(...)
+    return resolved
+```
+
+Three traps to remember:
+
+1. `Path.relative_to()` does **not** follow symlinks. Use `.resolve()` first.
+2. `Path.resolve()` raises `RuntimeError` on cycles regardless of `strict=`. Wrap.
+3. The "default" path (e.g. `target/manifest.json`) must go through the same gate as a user-supplied override. Don't trust convention.
+
+Issue #2's pass-2 review caught all three by accident; they're now baked into the loader and its regression tests.
+
+## Errors carry remediation
+
+Every typed exception in an external-format reader subclasses a module base (e.g. `ManifestError`) and accepts a `remediation: str` kwarg. `__str__` renders both the message and a `↳ Remediation:` line. This makes "explainable diffs" (CLAUDE.md commitment #5) load-bearing from the very first stage of the pipeline.
+
+## No logging / metrics in stage-0 modules
+
+Reader modules are deterministic JSON-to-typed-objects. They do not emit logs or metrics in v0.1. Observability lives in the stage that *consumes* the data (LLM drafting, prune, grade) — that's where signal-vs-volume tradeoffs surface. Adding logs here just generates noise.
+
+## Reference
+
+`plans/super/2-manifest-loader.md` — DEC-001, DEC-007, DEC-008, DEC-013, DEC-014, DEC-017. `src/signalforge/manifest/loader.py` — current implementation of all three traps.

.claude/rules/testing-signal.md

@@ -26,6 +26,20 @@ To verify locally: temporarily decorate a test with `@pytest.mark.does_not_exist
 
 Do NOT create `tests/__init__.py`. Pytest's rootdir handles discovery for src layouts; an empty `__init__.py` masks import errors and makes failures harder to read.
 
+## Fixture regeneration via ephemeral `uvx`
+
+When a fixture's correctness depends on an external tool's output (dbt's `manifest.json`, etc.), commit the *generated* artefact and document a regeneration script that runs the tool via `uvx` (or `pipx run`) at a pinned version:
+
+```bash
+uvx --python 3.11 --from "dbt-duckdb==X.Y.*" --with "dbt-core==X.Y.*" dbt parse
+```
+
+Pin the tool version in dev-deps for the *latest* schema only — older versions are summoned ephemerally. Strip non-deterministic fields (`generated_at`, `invocation_id`, `user_id`, etc.) with `jq` before committing so the JSON is reproducible. Reference: `tests/fixtures/regenerate.sh` (issue #2).
+
+## Drift detection via one-off `extra="forbid"` model
+
+If a parser uses `extra="ignore"` in production (forward-compat), pair it with a test that constructs a one-off `StrictModel(BaseModel)` with `extra="forbid"` and validates a known-current fixture against it. Adding a key to the fixture without updating the model breaks the test loudly. Reference: `tests/manifest/test_models.py::test_drift_detector_extra_forbid`.
+
 ## Reference
 
-`plans/super/1-project-scaffolding.md` — DEC-010. `tests/test_smoke.py` — current implementation. The `strict_markers = true` ini setting was discovered during US-003 of issue #1.
+`plans/super/1-project-scaffolding.md` — DEC-010. `plans/super/2-manifest-loader.md` — DEC-005, DEC-009, DEC-012, DEC-017. `tests/test_smoke.py`, `tests/manifest/`, `tests/fixtures/regenerate.sh` — current implementations.

CLAUDE.md

@@ -4,7 +4,18 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 
 ## Repository status
 
-Pre-alpha. Issue #1 (project scaffolding) shipped: `pyproject.toml` (Hatchling + src layout), `src/signalforge/__init__.py` with `__version__`, smoke test, ruff + pyright + pytest configs, GitHub Actions CI on PRs into `dev` and pushes to `main`, and `CONTRIBUTING.md`. Design is happening in the open on the `dev` branch; feature work (BigQuery adapter, LLM client, prune logic) lands next.
+Pre-alpha. Two issues shipped:
+
+- **#1 (project scaffolding)** — `pyproject.toml` (Hatchling + src layout), `src/signalforge/__init__.py` with `__version__`, smoke test, ruff + pyright + pytest configs, GitHub Actions CI on PRs into `dev` and pushes to `main`, and `CONTRIBUTING.md`.
+- **#2 (manifest loader)** — `signalforge.manifest` subpackage: typed `Manifest` / `Model` (Pydantic v2), `load(project_dir, manifest_path=None) -> Manifest`, single-model resolver by `unique_id` or file path, schema-version tolerance v9–v12, symlink-hardened path canonicalisation, soft 200 MB warning. First stable v0.1 public API surface. See `docs/manifest-loader-ops.md` for the operational reference.
+
+Design is happening in the open on the `dev` branch; remaining feature work (BigQuery adapter, LLM client, prune logic) lands next.
+
+## Public API surface (v0.1)
+
+- `signalforge.manifest.load`, `Manifest`, `Model`, and the `ManifestError` hierarchy. Documented in `docs/manifest-loader-ops.md`.
+
+Internals (`_loader_helpers`, etc.) are `_`-prefixed and not part of the public contract.
 
 ## Validation
 

CONTRIBUTING.md

@@ -20,6 +20,18 @@ Validate before pushing:
 ruff check . && ruff format --check . && pyright && pytest
 ```
 
+## Test markers
+
+Tests are tagged with `@pytest.mark.{unit, integration, error}` (declared in
+`pyproject.toml`). Run a single category with `pytest -m unit`. New tests
+SHOULD use a marker; bare tests are fine for true smoke checks.
+
+## Regenerating fixtures
+
+Fixture regen lives in [`tests/fixtures/README.md`](tests/fixtures/README.md).
+v12 is a one-liner against the in-`[dev]` `dbt-core` install; older schemas
+(v9 / v10 / v11) use ephemeral `uvx` invocations.
+
 ## License
 
 Contributions are Apache-2.0. The repo-level [LICENSE](LICENSE) covers it — do not add per-file license preambles.

README.md

@@ -36,8 +36,9 @@ SignalForge generates the same artifacts, then asks a different question: **does
 The grading layer reuses [clauditor](https://github.com/wjduenow/clauditor)'s LLM-as-judge methodology, applied to a new artifact class.
 
 > **Status (v0.1, in progress):** Not yet on PyPI. The CLI shape below is the
-> intended target — the CLI itself ships in a follow-up v0.1 ticket. Today
-> the package installs from a clone with `pip install -e ".[dev]"`.
+> intended target — library API lands first (`signalforge.manifest`); the CLI
+> ships in a later v0.1 ticket. Today the package installs from a clone with
+> `pip install -e ".[dev]"`.
 
 ## Quick start
 

docs/manifest-loader-ops.md

@@ -0,0 +1,75 @@
+# Manifest loader — operations guide
+
+Operational reference for users of `signalforge.manifest`. Companion to
+[`tests/fixtures/README.md`](../tests/fixtures/README.md) and the design
+record in [`plans/super/2-manifest-loader.md`](../plans/super/2-manifest-loader.md).
+
+## Memory profile
+
+The loader's soft-size warning uses a 3× expansion ratio between on-disk
+manifest bytes and resident Python memory; size your CI runner accordingly.
+
+| Manifest size on disk | Approx. resident memory |
+| --------------------- | ----------------------- |
+| small (~50 KB)        | ~5 MB                   |
+| medium (~5 MB)        | ~50–150 MB              |
+| large (~30+ MB)       | ~300–500+ MB            |
+
+## Soft size warning (DEC-008)
+
+The loader exposes `MAX_MANIFEST_BYTES = 200 * 1024 * 1024` at module scope.
+If `os.path.getsize(manifest_path)` exceeds it, `load()` emits a single
+`UserWarning` (not an exception) and proceeds — v0.1 has no hard ceiling.
+Tests that need to exercise the threshold can monkeypatch:
+
+```python
+import signalforge.manifest.loader as loader_mod
+loader_mod.MAX_MANIFEST_BYTES = 1024  # force the warning on tiny fixtures
+```
+
+The warning text includes the 3× rule of thumb so users can plan capacity.
+
+## Multi-version fixture regeneration (DEC-009 / DEC-012)
+
+Cross-link: [`tests/fixtures/README.md`](../tests/fixtures/README.md) holds
+the canonical recipe, including the per-schema-version `uvx dbt-core==X.Y.x`
+incantation for v9 / v10 / v11.
+
+- v12 can be regenerated with the in-`[dev]` `dbt-core>=1.8` install — no
+  `uvx` needed; `pip install -e ".[dev]"` is sufficient.
+- v9 / v10 / v11 use ephemeral `uvx` installs of dbt-core 1.5.x / 1.6.x /
+  1.7.x; the older lines need `--python 3.11` because they import the
+  removed `distutils` module.
+- `bash tests/fixtures/regenerate.sh` drives the full matrix and strips
+  non-deterministic metadata fields via `jq` so PR diffs don't churn.
+
+## Supported schema versions
+
+| Manifest schema | dbt-core lines       | Notes                          |
+| --------------- | -------------------- | ------------------------------ |
+| v9              | 1.5.x                | regen via `uvx`                |
+| v10             | 1.6.x                | regen via `uvx`                |
+| v11             | 1.7.x                | regen via `uvx`                |
+| v12             | 1.8 / 1.9 / 1.10 / 1.11 | regen via in-`[dev]` install |
+
+Schema **v20** (Fusion engine) is tracked as future work and currently
+raises `UnsupportedManifestVersionError`.
+
+## Error class quick reference
+
+Public API: `from signalforge.manifest import errors`.
+
+- **`ManifestNotFoundError`** — `load()` was given a path that does not
+  exist or is not a regular file.
+- **`UnsupportedManifestVersionError`** — `metadata.dbt_schema_version`
+  resolves to a schema outside v9–v12 (e.g. v8 or v20/Fusion).
+- **`ModelNotFoundError`** — `Manifest.get_model(unique_id)` was called
+  with a unique_id absent from `nodes` and `disabled`.
+- **`ModelDisabledError`** — `get_model()` matched a node, but it lives
+  in the `disabled` dict; callers must opt in to disabled nodes
+  explicitly.
+- **`ModelPathOutsideProjectError`** — the resolver detected a model
+  whose `original_file_path` (after symlink resolution) escapes the
+  project root.
+- **`ModelMissingSqlError`** — a model node has `raw_code: ""` or no
+  resolvable SQL on disk.

docs/research/README.md

@@ -0,0 +1,48 @@
+# Research snapshot
+
+This folder is a pinned snapshot of seven dbt research files vendored from
+[clauditor](https://github.com/wjduenow/clauditor)'s `docs/temp/` directory
+(per DEC-006 of `plans/super/2-manifest-loader.md`). The originals live in
+clauditor's gitignored `docs/temp/` tree, so without this copy, anyone outside
+the maintainer's machine could not read the references that informed
+SignalForge's manifest-loader design. Treat this folder as a snapshot — not a
+sync source — and update it deliberately.
+
+## Files
+
+- **`dbt-claude-technical-surface.md`** — implementer-facing reference for
+  Claude-powered dbt tooling: artifact schemas, MCP surface, SDK options,
+  warehouse integration, CI shapes, token economics. **Section 1.1 is the
+  canonical schema reference for the manifest module** — it informed
+  DEC-001 (manifest-only scope), DEC-011 (schema-version detection via the
+  `metadata.dbt_schema_version` URL), DEC-012 (fixture matrix shape), and
+  DEC-017 (the resolver / `iter_models` / `schema_version` API surface).
+- **`dbt-research-index.md`** — master index across the five research
+  artifacts; the entry point if you're skim-reading the corpus.
+- **`dbt-research.pdf`** — binary PDF compilation of the underlying research;
+  supplementary, not load-bearing for code.
+- **`dbt-ai-tools-deep-dive.md`** — survey of every meaningful AI/LLM tool in
+  or adjacent to dbt as of April 2026 (Copilot, codegen, DinoAI, datapilot,
+  Cortex Code), with user complaints and gap analysis.
+- **`dbt-pain-deep-dive.md`** — direct quotes and failure stories from HN,
+  Reddit, Substack, and dbt Slack about what hurts when running dbt in
+  production; the source for SignalForge's "signal over volume" framing.
+- **`dbt-tool-design-sketches.md`** — three concrete tool designs evaluated
+  for viability; SignalForge is the design that survived.
+- **`dbt-tooling-opportunity-report.md`** — product-framing synthesis,
+  including the top-10 ranked pain points; the companion to
+  `dbt-claude-technical-surface.md`.
+
+## Refreshing this snapshot
+
+If the originals update in clauditor, the maintainer can refresh this folder
+in place:
+
+```bash
+cp clauditor/docs/temp/dbt-*.md docs/research/
+cp clauditor/docs/temp/dbt-research.pdf docs/research/
+```
+
+Contributors outside the maintainer's machine can read this committed
+snapshot but cannot reach the live clauditor copy — that's the point of
+vendoring per DEC-006.