Skip to content
Merged
Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
6deb0c9
#155: super-plan — Gemini truncation + per-provider e2e gap
wjduenow May 28, 2026
96ead48
#155: devolved — epic + 10 tasks live in bd
wjduenow May 28, 2026
ede0259
bd_1-scaffolding-eu0.4: #155 US-003 — bump Gemini live fixture to 204…
wjduenow May 28, 2026
4060d9b
Merge bead bd_1-scaffolding-eu0.4: #155 US-003 — bump Gemini live fix…
wjduenow May 28, 2026
ad75a89
bd_1-scaffolding-eu0.5: #155 US-004 — apply_provider_override helper …
wjduenow May 28, 2026
8cd7107
Merge bead bd_1-scaffolding-eu0.5: #155 US-004 — apply_provider_overr…
wjduenow May 28, 2026
c3253fd
bd_1-scaffolding-eu0.2: #155 US-001 — LLMProvider.is_clean_completion…
wjduenow May 28, 2026
0d6f9a3
Merge bead bd_1-scaffolding-eu0.2: #155 US-001 — LLMProvider.is_clean…
wjduenow May 28, 2026
0b24dd8
bd_1-scaffolding-eu0.6: #155 US-005 — new tests/cli/test_e2e_openai_s…
wjduenow May 28, 2026
2c25eff
Merge bead bd_1-scaffolding-eu0.6: #155 US-005 — tests/cli/test_e2e_o…
wjduenow May 28, 2026
619233a
bd_1-scaffolding-eu0.7: #155 US-006 — new tests/cli/test_e2e_gemini_s…
wjduenow May 28, 2026
ead38ff
Merge bead bd_1-scaffolding-eu0.7: #155 US-006 — tests/cli/test_e2e_g…
wjduenow May 28, 2026
9a23e3f
bd_1-scaffolding-eu0.3: #155 US-002 — per-provider unclean-path tests…
wjduenow May 28, 2026
ef83e22
Merge bead bd_1-scaffolding-eu0.3 (worker portion): #155 US-002 — per…
wjduenow May 28, 2026
cd052e9
bd_1-scaffolding-eu0.8: #155 US-007 — parametrize tests/cli/test_e2e_…
wjduenow May 28, 2026
71cc293
bd_1-scaffolding-eu0.3: #155 US-002 (orchestrator portion) — clarify …
wjduenow May 28, 2026
b354c8b
Merge bead bd_1-scaffolding-eu0.8: #155 US-007 — parametrize tests/cl…
wjduenow May 28, 2026
ec5b1a8
bd_1-scaffolding-eu0.9: #155 US-008 — CONTRIBUTING.md live-suite pre-…
wjduenow May 28, 2026
9709f8e
Merge bead bd_1-scaffolding-eu0.9: #155 US-008 — CONTRIBUTING.md live…
wjduenow May 28, 2026
194d187
bd_1-scaffolding-eu0.10: #155 US-009 — Quality Gate fixes (4-pass cod…
wjduenow May 28, 2026
ddb5120
bd_1-scaffolding-eu0.11: #155 US-010 — Patterns & Memory
wjduenow May 28, 2026
ce7e050
#155 US-009 follow-up: close codecov patch-coverage gap (7 uncovered …
wjduenow May 28, 2026
2124097
#155: Address PR review feedback (CodeRabbit + Copilot, 0 false posit…
wjduenow May 28, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .claude/rules/llm-drafter.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Every `# pyright: ignore[...]` and `# type: ignore[...]` comment for the Anthrop

`call_llm(*, system, cached_block, dynamic_block, model, max_tokens, cache_ttl="5m", prompt_version, max_retries_*, provider="anthropic", client=None) -> LLMResult` (renamed from `call_anthropic` by #135, which dropped the old name) is the single shared seam for **both** the drafter and grader. It owns the generic machinery — retry loop + backoff (`2**attempt*_rand_uniform(0.75,1.25)`), per-class budgets, WARNING/INFO logs, the min/cap token validation, and `LLMResult` assembly — and dispatches the vendor-specific bits to a provider strategy resolved from a registry. When `client is None`, `call_llm` builds it via `strategy.make_client()` (DEC-006 — client construction lives at the seam, not the CLI).

- **`LLMProvider` ABC + registry** in `signalforge.llm.providers`: `register_provider(provider)` / `provider_for(name) -> LLMProvider`; unknown name → `UnknownProviderError(LLMError)` listing available keys (CLI tier 2). A provider supplies `make_client`, `build_create_kwargs`, `build_count_tokens_kwargs`, `extract_text_blocks`, `extract_usage` (→ `UsageMetrics`), `classify_exception` (→ `ExceptionCategory`), `estimate_input_tokens(model, text, *, system="", client=None) -> int` (#136 US-005; powers `--estimate`), plus capability flags `supports_prompt_caching` / `supports_token_count`. Wiring a new provider = that class + `register_provider` + a config enum value. Registered in v0.3: `AnthropicProvider` (`name="anthropic"`, both flags `True`); `OpenAIProvider` (`name="openai"`, both flags `False`, #136).
- **`LLMProvider` ABC + registry** in `signalforge.llm.providers`: `register_provider(provider)` / `provider_for(name) -> LLMProvider`; unknown name → `UnknownProviderError(LLMError)` listing available keys (CLI tier 2). A provider supplies `make_client`, `build_create_kwargs`, `build_count_tokens_kwargs`, `extract_text_blocks`, `extract_usage` (→ `UsageMetrics`), `classify_exception` (→ `ExceptionCategory`), `estimate_input_tokens(model, text, *, system="", client=None) -> int` (#136 US-005; powers `--estimate`), `is_clean_completion(response) -> bool` + `unclean_finish_reason_message(response) -> str` (#155 US-001 / DEC-005 — provider-neutral "non-clean finish_reason raises `LLMResponseFormatError`" gate called at the `call_llm` boundary; see Gemini-shape § "Non-clean finish_reason → typed `LLMResponseFormatError`" below for the full contract + the `_CLEAN_STOP_REASONS` convention each concrete declares), plus capability flags `supports_prompt_caching` / `supports_token_count`. Wiring a new provider = that class + `register_provider` + a config enum value. Registered in v0.3: `AnthropicProvider` (`name="anthropic"`, both flags `True`); `OpenAIProvider` (`name="openai"`, both flags `False`, #136).
- **Neutral value objects:** `UsageMetrics` + the `ExceptionCategory` enum (`AUTH`, `RATE_LIMIT`, `SERVER_ERROR`, `CONNECTION`, `NO_RETRY`) keep the orchestrator off vendor-shaped dicts.
- **Capability-gated behaviour (DEC-008):** `supports_prompt_caching=False` ⇒ no `cache_control` marker, no `extended-cache-ttl` beta header, 0 cache tokens, no dual-zero anomaly WARNING. `supports_token_count=False` ⇒ skip the pre-send count gate (no pre-send `LLMCacheTooLargeError`). Anthropic sets both `True`, so its emitted bytes/control flow are unchanged — the byte-identity gate (fixtures + prompt-cache snapshot + drift detectors) is the regression guard.
- **`provider` config field (DEC-007):** `DraftConfig.provider` (`llm:` block) and `GradeConfig.provider` (`grade:` block), both registry-validated `str` defaulting to `"anthropic"` — **deliberately NOT a `Literal`** (a registry is a plugin point that grows; #136/#137 register a provider instead of editing a Literal in two configs). The validator raises `UnknownProviderError` (an `LLMError`, so Pydantic v2 does NOT wrap it into `ValidationError` — it propagates raw with the available-keys remediation).
Expand Down Expand Up @@ -47,7 +47,7 @@ A new provider added under #137+ MUST implement `estimate_input_tokens` (it's `@

1. **`.messages.create` façade adapter — applies even when the SDK doesn't ship a `.messages` namespace at all.** The orchestrator hard-calls `llm_client.messages.create(**kwargs)`, but `google-genai`'s native surface is `client.models.generate_content(...)`. `_GeminiClientAdapter.messages` is a `_GeminiMessagesAdapter` instance whose `.create(**kwargs)` forwards to `models.generate_content(**kwargs)` and whose `.count_tokens(**kwargs)` forwards to `models.count_tokens(**kwargs)`. The pattern generalises #136's adapter shape to any vendor whose SDK exposes a differently-named namespace; the orchestrator stays vendor-neutral by construction.
2. **Server-side JSON via `response_mime_type="application/json"` (DEC-018 of #137).** `GeminiProvider.build_create_kwargs` attaches it inside the `config=...` dict that `models.generate_content` accepts. Mirrors OpenAI's `response_format={"type":"json_object"}` exactly per the precedent above — server-side enforcement eliminates the prose-preamble drift class; the tolerant `extract_json_payload` parser remains the fallback.
3. **Safety-filter / no-content → typed `LLMResponseFormatError` (DEC-005 of #137).** Gemini blocks responses via `finish_reason ∈ {SAFETY, RECITATION, OTHER, MAX_TOKENS, ...}` and surfaces a candidate with no text parts. `GeminiProvider.extract_text_blocks` walks `response.candidates[*].content.parts[*].text`; when no candidate yields a non-empty text part it raises `LLMResponseFormatError(f"Gemini response produced no text (finish_reason={fr!r}).")` with the first candidate's reason name. `call_llm` propagates as an `LLMError`; `grade_artifacts` wraps to `GradeLLMError` → conservative degrade per `grade-layer.md`. **The typed degrade is the contract — a future vendor with an analogous safety surface (e.g. content-filter responses) MUST route through the same `LLMResponseFormatError` → typed-degrade path rather than returning empty text or a sentinel.** Don't add a Gemini-specific error class — the existing `LLMError` hierarchy already covers it (Open notes for implementation in `plans/super/137-gemini-grading.md`).
3. **Non-clean finish_reason → typed `LLMResponseFormatError` (DEC-005 of #137, generalised provider-neutral by DEC-001/DEC-005 of #155).** Every provider exposes a "completion is unusable" signal via its native finish-reason field: Anthropic `stop_reason ∈ {max_tokens, refusal, tool_use, ...}`, OpenAI `choices[0].finish_reason ∈ {length, content_filter, tool_calls, ...}`, Gemini `candidates[0].finish_reason.name ∈ {MAX_TOKENS, SAFETY, RECITATION, OTHER, ...}`. The rule is enforced uniformly at the orchestrator boundary via the `LLMProvider.is_clean_completion(response) -> bool` ABC method (DEC-005 of #155) called by `call_llm` AFTER `messages.create` returns and BEFORE `extract_text_blocks` is invoked; when it returns `False`, `call_llm` raises `LLMResponseFormatError(strategy.unclean_finish_reason_message(response))`. Each provider declares its `_CLEAN_STOP_REASONS: frozenset[str]` (Anthropic `{end_turn, stop_sequence}`, OpenAI `{stop}`, Gemini `{STOP}`) — **anything else is unclean, regardless of whether text is also present.** This load-bearing post-#155 generalisation closes a Finding-1-shaped gap: a `MAX_TOKENS` truncation that emits *partial* text would otherwise slip through any "no text at all" gate and the downstream parser would degrade with `GradeOutputError(json_parse)` masking the actionable typed cause. `call_llm` propagates the raise as an `LLMError`; `grade_artifacts` wraps to `GradeLLMError` → conservative degrade per `grade-layer.md` (the `reasoning="call failed: GradeLLMError"` contract pin lives in `tests/grade/test_gemini_neutrality.py`, and the post-#155 per-provider unclean-path pins live in `tests/llm/test_{anthropic,openai,gemini}_provider_via_fake.py` + `tests/llm/test_client.py`). **The typed degrade is the contract — a future vendor with an analogous safety / truncation surface MUST route through the same `is_clean_completion` → `LLMResponseFormatError` → typed-degrade path rather than returning empty text or a sentinel.** `tool_use` is deliberately UNCLEAN for Anthropic (DEC-006 of #155) since the codebase doesn't use tools today; if/when tool-use intentionally lands the clean set expands deliberately. Don't add a per-provider error class — the existing `LLMError` hierarchy already covers it. Don't push the check into individual `extract_text_blocks` implementations — that splits responsibility across providers (one might forget; the next vendor lands with no enforcement); the ABC method centralises the rule. Per-provider `unclean_finish_reason_message(response) -> str` (DEC-007 of #155) keeps operator-facing diagnostics vendor-accurate (Anthropic says "stop_reason"; OpenAI/Gemini say "finish_reason").
4. **Native `models.count_tokens` for `estimate_input_tokens` (DEC-016 of #137).** Distinct from OpenAI's local `tiktoken` path: Gemini has a first-party server-side count API, so `GeminiProvider.estimate_input_tokens(model, text, *, system="", client=None)` calls `client.models.count_tokens(model=model, contents=[system + text])` and reads `response.total_tokens`. One extra API round-trip per estimate (comparable in shape to the Anthropic path). `system + text` is concatenated into a single `contents` entry — Gemini's count endpoint doesn't distinguish a system envelope from regular tokens (unlike Anthropic), so every token contributes to the same total and the figure matches what `generate_content` will bill at runtime. **Don't reach for `tiktoken` for a vendor that ships its own count API.** Verify the response field name against the installed SDK version — `total_tokens` is correct for `google-genai>=0.5,<1`; a future SDK rev may rename it.

### Namespace-package SDKs (#137 generalisation)
Expand Down
8 changes: 8 additions & 0 deletions .claude/rules/testing-signal.md
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,14 @@ A user-facing CLI flag's value can drift across surfaces (README, test argv, pla

Issue #10's gotcha: `signalforge generate stg_bikeshare_trips` (bare name) failed with `ModelNotFoundError`; only the file path or unique_id forms work. Caught only by Pass 4 of Quality Gate review — no unit test exercises the CLI's full model-arg path against the real `Manifest.get_model`. Mitigation: pre-merge review explicitly verifies CLI examples by running them locally.

### Per-test provider overlay via `apply_provider_override` (#155 US-004 / DEC-012)

When an e2e test needs to swap the LLM provider config (e.g. `tests/cli/test_e2e_openai_smoke.py` runs `signalforge generate` against the same Austin bikeshare fixture as the baseline BQ smoke, but with `grade.provider: openai` instead of the Anthropic default), the canonical helper is `tests.cli._e2e_helpers.apply_provider_override(project_dir, *, grade_provider=None, grade_model=None, grade_max_output_tokens=None) -> None`. Reads `<project_dir>/signalforge.yml`, overlays the `grade:` block deltas, writes back. Non-destructive: unset knobs left alone. Raises `FileNotFoundError` if the fixture's `signalforge.yml` is missing. This is the seam #155 US-005/US-006/US-007 (the three e2e provider variants) all flow through.

Two load-bearing rules: (1) **per-test overlay, not a `GradeConfig` default bump.** Lowering `GradeConfig.max_output_tokens` default would over-budget Anthropic/OpenAI calls; the per-test overlay scopes the Gemini-specific 2048 floor to the test that needs it (DEC-009 of #155). (2) **Drafter stays Anthropic Sonnet across all three e2e providers per DEC-011** — fixture stability requires only the grader varies. Tests that overlay `grade_provider` MUST still gate on `ANTHROPIC_API_KEY` (drafter) AND the provider-specific key (grader); a 3-env-var skip gate is insufficient when the drafter remains Anthropic — five is the contract (drafter API key + grader API key + their respective `SF_RUN_*` opt-ins + `GOOGLE_CLOUD_PROJECT` for the BigQuery leg). #155 QG Pass 4 caught the openai-smoke 3-vs-5 drift exactly because the docstring contract said "three" while the BQ smoke + Gemini sibling already used the five-var pattern.

The same belt-and-suspenders gating still applies: marker + runtime `_skip_reason()` + `tmp_path` isolation. The overlay is applied AFTER `copy_fixture_to_tmp` so the committed fixture is never modified.

## Reference

`plans/super/1-project-scaffolding.md` — DEC-010. `plans/super/2-manifest-loader.md` — DEC-005, DEC-009, DEC-012, DEC-017. `plans/super/27-codecov-coverage.md` — DEC-001, DEC-004, DEC-009. `plans/super/10-e2e-bigquery-smoke.md` — DEC-001, DEC-002, DEC-004, DEC-008, DEC-010, DEC-022. `tests/test_smoke.py`, `tests/manifest/`, `tests/fixtures/regenerate.sh`, `tests/cli/_e2e_helpers.py`, `tests/cli/test_e2e_bigquery_smoke.py`.
111 changes: 111 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,117 @@ is likely a redistribution (a code path moved behind a gated marker) rather than
a true regression. A drop in the *combined* total is a real regression worth
chasing before the release goes out.

## Live e2e suite (pre-release only)

The live e2e and live-API tests below hit real warehouses and real LLM
providers, so each invocation costs real money. They run on a
**pre-release cadence only** — NOT per-PR, NOT CI-gated. The
`addopts -m 'not …'` exclusion in `pyproject.toml` keeps every gated
marker out of default runs automatically; this section just documents how
a maintainer invokes the full suite when cutting a release (per DEC-010
of [`plans/super/155-gemini-truncation-e2e-gap.md`](plans/super/155-gemini-truncation-e2e-gap.md)).

**Cost ceiling:** ≈ **$0.30 per full-suite run** across the five paid e2e
tests below (plus a handful of cents for the grade-only live-API
companions). At ~2–3 pre-release audits per month for a one-maintainer
project, that lands at roughly $0.60–$1.00/month — small enough that a
shell wrapper around the invocation would add surface area without
changing the contract.

### Tests in the live e2e suite

Five paid e2e tests cover the full `signalforge generate` pipeline
(manifest → safety → draft → prune → grade → diff) against real
warehouses and real graders:

1. **`tests/cli/test_e2e_bigquery_smoke.py`** — `@pytest.mark.e2e`.
Parametrized over `grade.provider ∈ {"anthropic", "openai", "gemini"}`
(issue #155 US-007). The baseline gate is `SF_RUN_BQ=1` +
`ANTHROPIC_API_KEY` + `GOOGLE_CLOUD_PROJECT` (every variant uses the
Anthropic drafter); the `openai` variant additionally requires
`SF_RUN_OPENAI=1` + `OPENAI_API_KEY`; the `gemini` variant
additionally requires `SF_RUN_GEMINI=1` + `GOOGLE_API_KEY`. Variants
missing their extra env vars skip cleanly; the baseline variant
always runs when the BQ + Anthropic gate is satisfied.
2. **`tests/cli/test_e2e_openai_smoke.py`** — `@pytest.mark.e2e` +
`@pytest.mark.openai`. Anthropic drafter, OpenAI `gpt-4o` grader.
Five-env-var gate (mirrors the Gemini sibling and the parametrized
BQ smoke — drafter stays Anthropic Sonnet per DEC-011, so the
Anthropic auth + BigQuery opt-in are part of the contract): `SF_RUN_OPENAI=1`
+ `OPENAI_API_KEY` + `SF_RUN_BQ=1` + `ANTHROPIC_API_KEY` +
`GOOGLE_CLOUD_PROJECT`.
3. **`tests/cli/test_e2e_gemini_smoke.py`** — `@pytest.mark.e2e` +
`@pytest.mark.gemini`. Anthropic drafter, Gemini
`gemini-2.5-flash` grader with `grade.max_output_tokens=2048` (the
floor from DEC-008 of #155 — Gemini's verbose `reasoning` needs at
least 1024 tokens to land a clean `STOP` finish_reason; 2048 is the
verified-safe floor). Gate: `SF_RUN_GEMINI=1` + `GOOGLE_API_KEY` +
`SF_RUN_BQ=1` + `ANTHROPIC_API_KEY` + `GOOGLE_CLOUD_PROJECT`.
4. **`tests/cli/test_e2e_snowflake_smoke.py`** — `@pytest.mark.e2e` +
`@pytest.mark.snowflake`. The other-warehouse path: Anthropic drafter,
Anthropic grader, Snowflake adapter against read-only
`SNOWFLAKE_SAMPLE_DATA.TPCH_SF1` with `prune.sample_strategy: oneshot`.
Gate: `SF_RUN_SNOWFLAKE=1` + `ANTHROPIC_API_KEY` + `SNOWFLAKE_ACCOUNT`
+ `SNOWFLAKE_USER` + `SNOWFLAKE_PASSWORD` + `SNOWFLAKE_WAREHOUSE`.
See [`docs/snowflake-e2e-setup.md`](docs/snowflake-e2e-setup.md) for
warehouse-side setup (resource monitor, XS warehouse, aggressive
auto-suspend — guardrails against runaway cost).

Six **grade-only / draft-only live-API smokes** complement the e2e
suite. They exercise a single layer (no warehouse) against a real
provider and are gated by `@pytest.mark.anthropic` /
`@pytest.mark.openai` / `@pytest.mark.gemini` (NOT `e2e`):

- `tests/grade/test_smoke_real_api.py` (Anthropic),
`tests/grade/test_smoke_real_api_openai.py` (OpenAI),
`tests/grade/test_gemini_grade_live.py` (Gemini).
- `tests/draft/test_smoke_real_api_openai.py` (OpenAI),
`tests/draft/test_gemini_draft_live.py` (Gemini).
- `tests/llm/test_gemini_live.py` (Gemini raw `call_llm` round-trip).

Per-marker drilldowns (env vars, cost-per-call, single-marker
invocations) live in the `## Gemini live smoke`,
`## OpenAI live-API smoke tests`, and `## BigQuery integration tests`
sections below. The block here documents how to run the **whole** suite
in one go.

### Full pre-release invocation

```bash
SF_RUN_BQ=1 \
SF_RUN_OPENAI=1 \
SF_RUN_GEMINI=1 \
SF_RUN_SNOWFLAKE=1 \
ANTHROPIC_API_KEY=sk-ant-... \
OPENAI_API_KEY=sk-proj-... \
GOOGLE_API_KEY=... \
GOOGLE_CLOUD_PROJECT=<billing-project> \
SNOWFLAKE_ACCOUNT=... SNOWFLAKE_USER=... SNOWFLAKE_PASSWORD=... \
SNOWFLAKE_WAREHOUSE=... SNOWFLAKE_DATABASE=... SNOWFLAKE_SCHEMA=... \
uv run pytest -m 'e2e or anthropic or openai or gemini' --no-cov
```

`--no-cov` is required per `.claude/rules/python-build.md` § "Python
version: advertised floor matches the tested floor" — the
`--cov-fail-under=80` gate inherited from `addopts` would fail any
marker-specific run that exercises only a fraction of the codebase
(mirrors `uv run pytest -m bigquery --no-cov` and
`uv run pytest -m cli_subprocess --no-cov`).

The `snowflake` marker is deliberately **omitted** from the `-m`
expression above: that marker also covers the offline `fakesnow` +
`sqlglot` validation suite (no env vars required), which the default
post-PR validation already runs ad-hoc. The live Snowflake e2e is
reached via the `e2e` marker on `test_e2e_snowflake_smoke.py` instead;
if a maintainer wants to add the gated live `snowflake` tests
(`test_snowflake_estimate_live.py`, `test_snowflake_prune_live.py`),
append ` or snowflake` to the `-m` expression.

If you need to skip a specific provider for a given release (e.g. an
OpenAI quota hold), simply omit its `SF_RUN_<X>=1` env var — that
provider's tests self-skip with a named reason while the rest of the
suite proceeds.

## Test markers

Tests are tagged with `@pytest.mark.{unit, integration, error}` (declared in
Expand Down
Loading
Loading