Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions docs/draft-ops.md
Original file line number Diff line number Diff line change
Expand Up @@ -588,11 +588,11 @@ but must validate quality afterward (truncated draft responses surface as
`LLMOutputJSONError` or `LLMOutputAnchorContractError` and fail the run rather
than ship a partial `schema.yml`).

| Provider | Recommended floor | Rationale |
|--------------------------|-------------------|--------------------------------------------------------------------------------------------|
| Anthropic (Sonnet 4.6+) | 1024 | Sufficient for full reasoning; tested in BQ smoke. |
| OpenAI (gpt-4o) | 1024 | Same headroom; no observed truncation. |
| Gemini (2.5-flash+) | **2048** | Verbose reasoning style; 512 / 1024 observed truncating mid-string (issue #155 DEC-008). |
| Provider | Recommended floor | Rationale |
|--------------------------|-------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Anthropic (Sonnet 4.6+) | 1024 | Sufficient for full reasoning; tested in BQ smoke. |
| OpenAI (gpt-4o) | 1024 | Same headroom; no observed truncation. |
| Gemini (2.5-flash+) | **4096** | Verbose reasoning style; 512 / 1024 observed truncating mid-string (#155 DEC-008). Drafter floor mirrors the grader floor raised by #158 — Gemini's per-call output is high-variance enough that the grader's lesson applies. |
Comment thread
wjduenow marked this conversation as resolved.
Resolved

## OpenAI provider

Expand Down
30 changes: 22 additions & 8 deletions docs/grade-ops.md
Original file line number Diff line number Diff line change
Expand Up @@ -506,13 +506,27 @@ but must validate quality afterward. Truncated judge responses surface as
`LLMResponseFormatError` (the provider-neutral `is_clean_completion` gate raises
on any non-clean finish_reason — Anthropic `stop_reason="max_tokens"`, OpenAI
`finish_reason="length"`, Gemini `finish_reason="MAX_TOKENS"`) and degrade
the pair with `reasoning="call failed: GradeLLMError"` per #155 DEC-005.

| Provider | Recommended floor | Rationale |
|--------------------------|-------------------|--------------------------------------------------------------------------------------------|
| Anthropic (Sonnet 4.6+) | 1024 | Sufficient for full reasoning; tested in BQ smoke. |
| OpenAI (gpt-4o) | 1024 | Same headroom; no observed truncation. |
| Gemini (2.5-flash+) | **2048** | Verbose reasoning style; 512 / 1024 observed truncating mid-string (issue #155 DEC-008). |
the pair with `reasoning="call failed: GradeLLMError: <inner finish_reason message>"`
per #155 DEC-005 + #158 (the inner provider message — naming the actual
`finish_reason` value — is surfaced into the audit JSONL so a residual
degrade is self-diagnosing without re-reading stderr).

| Provider | Recommended floor | Rationale |
|--------------------------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Anthropic (Sonnet 4.6+) | 1024 | Sufficient for full reasoning; tested in BQ smoke. |
| OpenAI (gpt-4o) | 1024 | Same headroom; no observed truncation. |
| Gemini (2.5-flash+) | **4096** | Verbose reasoning style; 512 / 1024 observed truncating mid-string (#155 DEC-008). 2048 verified safe at the 5-pair in-isolation smoke scale but **#158 found 5–6/108 pairs still degrade at 2048 on the full Austin fixture** — 4096 is the fixture-scale floor. |

> **Fixture-scale caveat (issue #158):** these floors are *necessary
> but not sufficient* — Gemini's per-pair `reasoning` length is
> high-variance, so a fixture with substantially more artifacts than
> the Austin bikeshare e2e (~27 artifacts × 4 criteria = ~108 pairs)
> may still see residual `MAX_TOKENS` degrades at 4096. The honest
> guidance is to treat the floor as fixture-scale-dependent: validate
> with a full-fixture run, watch `GradingReport.aggregate_complete`,
> and bump if any pair degrades with `reasoning` mentioning
> `finish_reason='MAX_TOKENS'`. The diagnostic in the degrade reasoning
> tells you exactly which `finish_reason` fired.

## OpenAI provider

Expand Down Expand Up @@ -582,7 +596,7 @@ grade:
- **Install extra:** `pip install signalforge-dbt[gemini]` (or `uv sync --dev` in a contributor checkout). Pulls `google-genai>=0.5,<1`.
- **Env var:** `GOOGLE_API_KEY` (read by the SDK; SignalForge never logs it).
- **Server-side JSON enforcement:** `GeminiProvider.build_create_kwargs` sets `response_mime_type="application/json"` on the `GenerateContentConfig` (DEC-018 of #137). Belt-and-braces with the tolerant `extract_json_payload` parser.
- **Safety-filter handling:** When Gemini blocks a response (`finish_reason` ∈ `{SAFETY, RECITATION, OTHER, ...}` with no text parts), `GeminiProvider.extract_text_blocks` raises `LLMResponseFormatError` naming the reason (DEC-005 of #137); the grade engine wraps it as `GradeLLMError` and degrades the affected pair via the conservative `score=None` / `reasoning="call failed: GradeLLMError"` taxonomy.
- **Safety-filter handling:** When Gemini blocks a response (`finish_reason` ∈ `{SAFETY, RECITATION, OTHER, ...}` with no text parts), `GeminiProvider.extract_text_blocks` raises `LLMResponseFormatError` naming the reason (DEC-005 of #137); the grade engine wraps it as `GradeLLMError` and degrades the affected pair via the conservative `score=None` / `reasoning="call failed: GradeLLMError: <inner finish_reason message>"` taxonomy (#158 surfaces the inner provider message into the audit field so the actual `finish_reason` value — `SAFETY` vs `RECITATION` vs `MAX_TOKENS` — is recoverable from `.signalforge/grade.jsonl` alone).
Comment thread
wjduenow marked this conversation as resolved.
Resolved

**No prompt caching (cost note — DEC-013 of #137).** v0.3 Gemini ships
**without** prompt caching. `GeminiProvider` reports
Expand Down
2 changes: 1 addition & 1 deletion plans/super/155-gemini-truncation-e2e-gap.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ Three findings surfaced by live validation of the #134 epic, all rooted in the s
| **DEC-005** | Provider seam = new abstract method `LLMProvider.is_clean_completion(response) -> bool` called by `call_llm` before `extract_text_blocks` (Option B). Each provider declares `_CLEAN_STOP_REASONS: frozenset[str]`. | Centralizes the rule in the orchestrator (where cross-provider invariants live), forces every future provider to declare its clean-set (can't silently forget), zero AST/confinement impact. |
| **DEC-006** | Anthropic's `stop_reason="tool_use"` is **unclean** in v0.3 (clean set = `{end_turn, stop_sequence}`). | Codebase doesn't use tools today; `tool_use` would signal system-prompt drift or unexpected LLM behaviour. When tool-use intentionally lands, the clean set expands deliberately. |
| **DEC-007** | Error-message text = provider-specific via override `LLMProvider.unclean_finish_reason_message(response) -> str`. Default in ABC; each concrete overrides to surface its vendor-native field name. | Operator-facing diagnostic stays vendor-accurate (`stop_reason` for Anthropic, `finish_reason` for OpenAI/Gemini). |
| **DEC-008** | Per-provider `max_output_tokens` floor table in `docs/grade-ops.md` + `docs/draft-ops.md`: Anthropic **1024**, OpenAI **1024**, Gemini **2048**. Documented as recommended floor for grading workloads, not enforced cap. | Honest floors from observed data. Gemini's verbose `reasoning` provably needs ≥1024; 2048 verified safe in #155 probe. |
| **DEC-008** | Per-provider `max_output_tokens` floor table in `docs/grade-ops.md` + `docs/draft-ops.md`: Anthropic **1024**, OpenAI **1024**, Gemini **2048**. Documented as recommended floor for grading workloads, not enforced cap. | Honest floors from observed data. Gemini's verbose `reasoning` provably needs ≥1024; 2048 verified safe in #155 probe. **Reframed by #158 (2026-05-28):** the 2048 "verified safe" claim was scoped to the 5-pair in-isolation probe (`tests/grade/test_gemini_grade_live.py`). The first full-pipeline e2e run against Gemini surfaced 5–6/108 pairs still degrading at 2048 on the Austin bikeshare fixture — Gemini's per-pair `reasoning` is high-variance enough that the in-isolation floor is not the full-fixture floor. The docs table now reads **4096** for Gemini, framed as "fixture-scale-dependent" rather than a single safe number; this DEC stays as the historical record of the 2048 figure's provenance. See `plans/super/155-...` was the in-isolation verification; #158 was the full-pipeline correction. The same lesson lives in memory `in-isolation-smoke-misses-pipeline-drift`. |
Comment thread
wjduenow marked this conversation as resolved.
Resolved
| **DEC-009** | Gemini e2e sibling's `max_output_tokens=2048` lives in the test's `signalforge.yml` overlay, NOT a bumped `GradeConfig`/`DraftConfig` production default. | Tested-by-construction; no production-config change. Avoids over-budgeting Anthropic/OpenAI default calls. |
| **DEC-010** | Live-suite cadence = pre-release only, documented in `CONTRIBUTING.md`. NO `make e2e-live-all` wrapper, NO per-PR CI integration. | $0.30/full-suite × ~2-3 pre-release audits/month = ~$0.60-1.00/month for a one-maintainer project. Env-var gating is already explicit; a shell wrapper adds surface area without changing the contract. |
| **DEC-011** | Keep 3 separate e2e test files (`test_e2e_bigquery_smoke.py`, `test_e2e_openai_smoke.py`, `test_e2e_gemini_smoke.py`), do NOT collapse into one parametrized test. Parametrize is internal to BQ over `grade.provider`. | Per-file failure messages name the broken provider; per-file cost is transparent in CI logs; per-file marker gating aligns with the existing `@pytest.mark.openai` / `@pytest.mark.gemini` convention. |
Expand Down
28 changes: 26 additions & 2 deletions src/signalforge/grade/engine.py
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@
)
from signalforge.llm import AnthropicClientProtocol
from signalforge.llm.client import call_llm
from signalforge.llm.errors import LLMError
from signalforge.llm.errors import LLMError, LLMResponseFormatError
from signalforge.manifest.models import Model
from signalforge.prune.models import PruneResult

Expand Down Expand Up @@ -359,6 +359,30 @@ def _grade_one(
return grading_result, event


def _format_degrade_reasoning(exc: BaseException) -> str:
"""Render the ``GradingResult.reasoning`` string for a degraded pair.

Resolves issue #158: the bare ``f"call failed: {type(exc).__name__}"``
shape loses the vendor ``finish_reason`` value when a provider's
response-shape gate fires (Gemini ``MAX_TOKENS`` vs ``SAFETY`` vs
``RECITATION`` all collapse to ``"call failed: GradeLLMError"``).
When the wrapped cause is :class:`LLMResponseFormatError`, surface its
bare ``message`` field (which names the vendor field + value per
:meth:`LLMProvider.unclean_finish_reason_message`) so operators can
diagnose from the audit JSONL / sidecar alone without re-reading
stderr.

For every other cause (auth, rate-limit, parser failure, budget
exhausted) the existing ``"call failed: <ClassName>"`` shape is
preserved verbatim — the audit corpus stays diff-clean for the 90%
case, and only the response-shape branch grows the diagnostic.
"""
base = f"call failed: {type(exc).__name__}"
if isinstance(exc, GradeLLMError) and isinstance(exc.cause, LLMResponseFormatError):
return f"{base}: {exc.cause.message}"
return base


def _build_degraded(
*,
artifact_id: str,
Expand Down Expand Up @@ -704,7 +728,7 @@ def grade_artifacts(
grading_result, event = _build_degraded(
artifact_id=artifact_id,
criterion=criterion,
reasoning=f"call failed: {type(exc).__name__}",
reasoning=_format_degrade_reasoning(exc),
config=resolved_config,
rubric_hash=rubric_hash,
template_hash=template_hash,
Expand Down
21 changes: 11 additions & 10 deletions tests/cli/test_e2e_bigquery_smoke.py
Original file line number Diff line number Diff line change
Expand Up @@ -219,21 +219,22 @@ def test_e2e_signalforge_generate_against_austin_bikeshare(
grade_model="gpt-4o",
)
elif grade_provider == "gemini":
# ``grade_max_output_tokens=2048`` is **load-bearing**, not
# cosmetic. Issue #155 Finding 2: Gemini 2.5-flash's verbose
# ``reasoning`` field routinely exceeds
# ``max_output_tokens=512``/``1024`` on this fixture's grading
# workload, hitting MAX_TOKENS and truncating mid-string. That
# truncation surfaces downstream as a degraded grade result
# (``aggregate_complete=False``) and would flake assertion #6
# below. The #155 live probe verified ``2048`` passes cleanly.
# Per DEC-009 the floor lives here in the test overlay rather
# ``grade_max_output_tokens=4096`` is **load-bearing**, not
# cosmetic. Issue #155 Finding 2 + issue #158: Gemini 2.5-flash's
# verbose ``reasoning`` field truncates at low caps (512/1024)
# on every fixture; the #155 probe found 2048 sufficient for
# the 5-pair in-isolation smoke, but #158 caught that this
# full-pipeline fixture runs 108–116 (artifact × criterion)
# pairs and 5–6 of them still exceed 2048 (typed-degrading to
# ``GradeLLMError`` and flipping ``aggregate_complete=False``).
# 4096 is the #158 floor for the full-fixture workload. Per
# DEC-009 the floor still lives here in the test overlay rather
# than as a bumped ``GradeConfig`` production default.
apply_provider_override(
project_dir,
grade_provider="gemini",
grade_model="gemini-2.5-flash",
grade_max_output_tokens=2048,
grade_max_output_tokens=4096,
)
else: # pragma: no cover — parametrize guards the value space.
raise AssertionError(f"unhandled grade_provider: {grade_provider!r}")
Expand Down
Loading
Loading