diff --git a/docs/draft-ops.md b/docs/draft-ops.md index 37bfa027..2e36f4ce 100644 --- a/docs/draft-ops.md +++ b/docs/draft-ops.md @@ -588,11 +588,11 @@ but must validate quality afterward (truncated draft responses surface as `LLMOutputJSONError` or `LLMOutputAnchorContractError` and fail the run rather than ship a partial `schema.yml`). -| Provider | Recommended floor | Rationale | -|--------------------------|-------------------|--------------------------------------------------------------------------------------------| -| Anthropic (Sonnet 4.6+) | 1024 | Sufficient for full reasoning; tested in BQ smoke. | -| OpenAI (gpt-4o) | 1024 | Same headroom; no observed truncation. | -| Gemini (2.5-flash+) | **2048** | Verbose reasoning style; 512 / 1024 observed truncating mid-string (issue #155 DEC-008). | +| Provider | Recommended floor | Rationale | +|--------------------------|-------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Anthropic (Sonnet 4.6+) | 1024 | Sufficient for full reasoning; tested in BQ smoke. | +| OpenAI (gpt-4o) | 1024 | Same headroom; no observed truncation. | +| Gemini (2.5-flash+) | **4096** | Verbose reasoning style; 512 / 1024 observed truncating mid-string (#155 DEC-008). The 4096 figure is a **conservative mirror of the #158 Gemini-grader floor** (Gemini's per-pair output is high-variance enough that the grader's 5–6/108 degrades-at-2048 finding applies to any verbose response). The drafter currently runs Anthropic in every shipped e2e, so this row is **pending Gemini-drafter live validation** — when that lands, update with measured evidence. | ## OpenAI provider diff --git a/docs/grade-ops.md b/docs/grade-ops.md index fd5665c8..0089d13b 100644 --- a/docs/grade-ops.md +++ b/docs/grade-ops.md @@ -506,13 +506,27 @@ but must validate quality afterward. Truncated judge responses surface as `LLMResponseFormatError` (the provider-neutral `is_clean_completion` gate raises on any non-clean finish_reason — Anthropic `stop_reason="max_tokens"`, OpenAI `finish_reason="length"`, Gemini `finish_reason="MAX_TOKENS"`) and degrade -the pair with `reasoning="call failed: GradeLLMError"` per #155 DEC-005. - -| Provider | Recommended floor | Rationale | -|--------------------------|-------------------|--------------------------------------------------------------------------------------------| -| Anthropic (Sonnet 4.6+) | 1024 | Sufficient for full reasoning; tested in BQ smoke. | -| OpenAI (gpt-4o) | 1024 | Same headroom; no observed truncation. | -| Gemini (2.5-flash+) | **2048** | Verbose reasoning style; 512 / 1024 observed truncating mid-string (issue #155 DEC-008). | +the pair with `reasoning="call failed: GradeLLMError: "` +per #155 DEC-005 + #158 (the inner provider message — naming the actual +`finish_reason` value — is surfaced into the audit JSONL so a residual +degrade is self-diagnosing without re-reading stderr). + +| Provider | Recommended floor | Rationale | +|--------------------------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Anthropic (Sonnet 4.6+) | 1024 | Sufficient for full reasoning; tested in BQ smoke. | +| OpenAI (gpt-4o) | 1024 | Same headroom; no observed truncation. | +| Gemini (2.5-flash+) | **4096** | Verbose reasoning style; 512 / 1024 observed truncating mid-string (#155 DEC-008). 2048 verified safe at the 5-pair in-isolation smoke scale but **#158 found 5–6/108 pairs still degrade at 2048 on the full Austin fixture** — 4096 is the fixture-scale floor. | + +> **Fixture-scale caveat (issue #158):** these floors are *necessary +> but not sufficient* — Gemini's per-pair `reasoning` length is +> high-variance, so a fixture with substantially more artifacts than +> the Austin bikeshare e2e (~27 artifacts × 4 criteria = ~108 pairs) +> may still see residual `MAX_TOKENS` degrades at 4096. The honest +> guidance is to treat the floor as fixture-scale-dependent: validate +> with a full-fixture run, watch `GradingReport.aggregate_complete`, +> and bump if any pair degrades with `reasoning` mentioning +> `finish_reason='MAX_TOKENS'`. The diagnostic in the degrade reasoning +> tells you exactly which `finish_reason` fired. ## OpenAI provider @@ -582,7 +596,7 @@ grade: - **Install extra:** `pip install signalforge-dbt[gemini]` (or `uv sync --dev` in a contributor checkout). Pulls `google-genai>=0.5,<1`. - **Env var:** `GOOGLE_API_KEY` (read by the SDK; SignalForge never logs it). - **Server-side JSON enforcement:** `GeminiProvider.build_create_kwargs` sets `response_mime_type="application/json"` on the `GenerateContentConfig` (DEC-018 of #137). Belt-and-braces with the tolerant `extract_json_payload` parser. -- **Safety-filter handling:** When Gemini blocks a response (`finish_reason` ∈ `{SAFETY, RECITATION, OTHER, ...}` with no text parts), `GeminiProvider.extract_text_blocks` raises `LLMResponseFormatError` naming the reason (DEC-005 of #137); the grade engine wraps it as `GradeLLMError` and degrades the affected pair via the conservative `score=None` / `reasoning="call failed: GradeLLMError"` taxonomy. +- **Safety-filter / non-clean finish_reason handling:** Two related paths both route through the same degrade. (1) The provider-neutral `LLMProvider.is_clean_completion(response)` gate inside `call_llm` (DEC-005 of #155) raises `LLMResponseFormatError` when `finish_reason` is anything but `STOP` — including `SAFETY`, `RECITATION`, `OTHER`, `MAX_TOKENS` (even when partial text is present). (2) The legacy `GeminiProvider.extract_text_blocks` raise (DEC-005 of #137) still fires when zero text parts are returned. Either way, the grade engine wraps the result as `GradeLLMError` and degrades the affected pair via the conservative `score=None` / `reasoning="call failed: GradeLLMError: "` taxonomy (#158 surfaces the inner provider message into the audit field so the actual `finish_reason` value — `SAFETY` vs `RECITATION` vs `MAX_TOKENS` — is recoverable from `.signalforge/grade.jsonl` alone). **No prompt caching (cost note — DEC-013 of #137).** v0.3 Gemini ships **without** prompt caching. `GeminiProvider` reports diff --git a/plans/super/155-gemini-truncation-e2e-gap.md b/plans/super/155-gemini-truncation-e2e-gap.md index 38666489..7e821c57 100644 --- a/plans/super/155-gemini-truncation-e2e-gap.md +++ b/plans/super/155-gemini-truncation-e2e-gap.md @@ -53,7 +53,7 @@ Three findings surfaced by live validation of the #134 epic, all rooted in the s | **DEC-005** | Provider seam = new abstract method `LLMProvider.is_clean_completion(response) -> bool` called by `call_llm` before `extract_text_blocks` (Option B). Each provider declares `_CLEAN_STOP_REASONS: frozenset[str]`. | Centralizes the rule in the orchestrator (where cross-provider invariants live), forces every future provider to declare its clean-set (can't silently forget), zero AST/confinement impact. | | **DEC-006** | Anthropic's `stop_reason="tool_use"` is **unclean** in v0.3 (clean set = `{end_turn, stop_sequence}`). | Codebase doesn't use tools today; `tool_use` would signal system-prompt drift or unexpected LLM behaviour. When tool-use intentionally lands, the clean set expands deliberately. | | **DEC-007** | Error-message text = provider-specific via override `LLMProvider.unclean_finish_reason_message(response) -> str`. Default in ABC; each concrete overrides to surface its vendor-native field name. | Operator-facing diagnostic stays vendor-accurate (`stop_reason` for Anthropic, `finish_reason` for OpenAI/Gemini). | -| **DEC-008** | Per-provider `max_output_tokens` floor table in `docs/grade-ops.md` + `docs/draft-ops.md`: Anthropic **1024**, OpenAI **1024**, Gemini **2048**. Documented as recommended floor for grading workloads, not enforced cap. | Honest floors from observed data. Gemini's verbose `reasoning` provably needs ≥1024; 2048 verified safe in #155 probe. | +| **DEC-008** | Per-provider `max_output_tokens` floor table in `docs/grade-ops.md` + `docs/draft-ops.md`: Anthropic **1024**, OpenAI **1024**, Gemini **2048**. Documented as recommended floor for grading workloads, not enforced cap. | Honest floors from observed data. Gemini's verbose `reasoning` provably needs ≥1024; 2048 verified safe in #155 probe. **Reframed by #158 (2026-05-28):** the 2048 "verified safe" claim was scoped to the 5-pair in-isolation probe (`tests/grade/test_gemini_grade_live.py`). The first full-pipeline e2e run against Gemini surfaced 5–6/108 pairs still degrading at 2048 on the Austin bikeshare fixture — Gemini's per-pair `reasoning` is high-variance enough that the in-isolation floor is not the full-fixture floor. The docs table now reads **4096** for Gemini, framed as "fixture-scale-dependent" rather than a single safe number; this DEC stays as the historical record of the 2048 figure's provenance. The full-pipeline correction lives in `plans/super/` issue [#158](https://github.com/wjduenow/SignalForge/issues/158); the generalised lesson lives in memory `in-isolation-smoke-misses-pipeline-drift`. | | **DEC-009** | Gemini e2e sibling's `max_output_tokens=2048` lives in the test's `signalforge.yml` overlay, NOT a bumped `GradeConfig`/`DraftConfig` production default. | Tested-by-construction; no production-config change. Avoids over-budgeting Anthropic/OpenAI default calls. | | **DEC-010** | Live-suite cadence = pre-release only, documented in `CONTRIBUTING.md`. NO `make e2e-live-all` wrapper, NO per-PR CI integration. | $0.30/full-suite × ~2-3 pre-release audits/month = ~$0.60-1.00/month for a one-maintainer project. Env-var gating is already explicit; a shell wrapper adds surface area without changing the contract. | | **DEC-011** | Keep 3 separate e2e test files (`test_e2e_bigquery_smoke.py`, `test_e2e_openai_smoke.py`, `test_e2e_gemini_smoke.py`), do NOT collapse into one parametrized test. Parametrize is internal to BQ over `grade.provider`. | Per-file failure messages name the broken provider; per-file cost is transparent in CI logs; per-file marker gating aligns with the existing `@pytest.mark.openai` / `@pytest.mark.gemini` convention. | diff --git a/src/signalforge/grade/engine.py b/src/signalforge/grade/engine.py index 65be0630..a9f6ee4f 100644 --- a/src/signalforge/grade/engine.py +++ b/src/signalforge/grade/engine.py @@ -123,7 +123,7 @@ ) from signalforge.llm import AnthropicClientProtocol from signalforge.llm.client import call_llm -from signalforge.llm.errors import LLMError +from signalforge.llm.errors import LLMError, LLMResponseFormatError from signalforge.manifest.models import Model from signalforge.prune.models import PruneResult @@ -359,6 +359,30 @@ def _grade_one( return grading_result, event +def _format_degrade_reasoning(exc: BaseException) -> str: + """Render the ``GradingResult.reasoning`` string for a degraded pair. + + Resolves issue #158: the bare ``f"call failed: {type(exc).__name__}"`` + shape loses the vendor ``finish_reason`` value when a provider's + response-shape gate fires (Gemini ``MAX_TOKENS`` vs ``SAFETY`` vs + ``RECITATION`` all collapse to ``"call failed: GradeLLMError"``). + When the wrapped cause is :class:`LLMResponseFormatError`, surface its + bare ``message`` field (which names the vendor field + value per + :meth:`LLMProvider.unclean_finish_reason_message`) so operators can + diagnose from the audit JSONL / sidecar alone without re-reading + stderr. + + For every other cause (auth, rate-limit, parser failure, budget + exhausted) the existing ``"call failed: "`` shape is + preserved verbatim — the audit corpus stays diff-clean for the 90% + case, and only the response-shape branch grows the diagnostic. + """ + base = f"call failed: {type(exc).__name__}" + if isinstance(exc, GradeLLMError) and isinstance(exc.cause, LLMResponseFormatError): + return f"{base}: {exc.cause.message}" + return base + + def _build_degraded( *, artifact_id: str, @@ -704,7 +728,7 @@ def grade_artifacts( grading_result, event = _build_degraded( artifact_id=artifact_id, criterion=criterion, - reasoning=f"call failed: {type(exc).__name__}", + reasoning=_format_degrade_reasoning(exc), config=resolved_config, rubric_hash=rubric_hash, template_hash=template_hash, diff --git a/tests/cli/test_e2e_bigquery_smoke.py b/tests/cli/test_e2e_bigquery_smoke.py index 6306cb06..f5ec3c4f 100644 --- a/tests/cli/test_e2e_bigquery_smoke.py +++ b/tests/cli/test_e2e_bigquery_smoke.py @@ -219,21 +219,22 @@ def test_e2e_signalforge_generate_against_austin_bikeshare( grade_model="gpt-4o", ) elif grade_provider == "gemini": - # ``grade_max_output_tokens=2048`` is **load-bearing**, not - # cosmetic. Issue #155 Finding 2: Gemini 2.5-flash's verbose - # ``reasoning`` field routinely exceeds - # ``max_output_tokens=512``/``1024`` on this fixture's grading - # workload, hitting MAX_TOKENS and truncating mid-string. That - # truncation surfaces downstream as a degraded grade result - # (``aggregate_complete=False``) and would flake assertion #6 - # below. The #155 live probe verified ``2048`` passes cleanly. - # Per DEC-009 the floor lives here in the test overlay rather + # ``grade_max_output_tokens=4096`` is **load-bearing**, not + # cosmetic. Issue #155 Finding 2 + issue #158: Gemini 2.5-flash's + # verbose ``reasoning`` field truncates at low caps (512/1024) + # on every fixture; the #155 probe found 2048 sufficient for + # the 5-pair in-isolation smoke, but #158 caught that this + # full-pipeline fixture runs 108–116 (artifact × criterion) + # pairs and 5–6 of them still exceed 2048 (typed-degrading to + # ``GradeLLMError`` and flipping ``aggregate_complete=False``). + # 4096 is the #158 floor for the full-fixture workload. Per + # DEC-009 the floor still lives here in the test overlay rather # than as a bumped ``GradeConfig`` production default. apply_provider_override( project_dir, grade_provider="gemini", grade_model="gemini-2.5-flash", - grade_max_output_tokens=2048, + grade_max_output_tokens=4096, ) else: # pragma: no cover — parametrize guards the value space. raise AssertionError(f"unhandled grade_provider: {grade_provider!r}") diff --git a/tests/cli/test_e2e_gemini_smoke.py b/tests/cli/test_e2e_gemini_smoke.py index 8b0a2b38..b4701868 100644 --- a/tests/cli/test_e2e_gemini_smoke.py +++ b/tests/cli/test_e2e_gemini_smoke.py @@ -7,13 +7,18 @@ Anthropic Sonnet (per DEC-011 — fixture stability; only the grader is parametrised across providers). -The ``grade_max_output_tokens=2048`` overlay is **load-bearing**, not +The ``grade_max_output_tokens=4096`` overlay is **load-bearing**, not cosmetic. Issue #155 Finding 2: Gemini 2.5-flash's verbose ``reasoning`` field routinely exceeds ``max_output_tokens=512``/``1024`` -on this fixture's 5-pair grading workload, hitting ``MAX_TOKENS`` and -truncating mid-string. The live probe in #155 verified ``2048`` passes -cleanly. Without the cap, this test would flake the same way the -original validation broke. Per DEC-009 the floor lives in the test +on small fixtures, hitting ``MAX_TOKENS`` and truncating mid-string. +The #155 live probe verified ``2048`` passes cleanly on the in-isolation +5-pair smoke (:file:`tests/grade/test_gemini_grade_live.py`) — but +**issue #158** caught the structural gap: this full-pipeline fixture +runs 108–116 (artifact × criterion) pairs, and Gemini's per-pair +``reasoning`` length is high-variance enough that 5–6 of them still +exceed 2048 (typed-degrading as ``"call failed: GradeLLMError"`` and +flipping ``aggregate_complete=False``). 4096 is the #158 floor for the +full-fixture workload. Per DEC-009 the floor still lives in the test overlay rather than as a bumped production default — avoids over-budgeting Anthropic/OpenAI calls. @@ -61,7 +66,8 @@ Gemini's grading distribution should still produce at least one flag against thresholds this strict). 6. ``GradingReport.aggregate_complete is True`` — **the load-bearing - assertion that proves the 2048 cap fixes the truncation bug.** + assertion that proves the 4096 cap (#158) fixes the full-fixture + truncation bug.** If ``max_output_tokens`` were too low, Gemini's verbose ``reasoning`` would truncate mid-string. Post-#155 US-001 the provider-neutral ``is_clean_completion`` gate raises @@ -161,20 +167,22 @@ def test_e2e_signalforge_generate_against_austin_bikeshare_with_gemini_grader( # drafter stays Anthropic Sonnet per DEC-011 (no ``llm.provider`` # override); only ``grade:`` block knobs change here. # - # ``grade_max_output_tokens=2048`` is **load-bearing**, not cosmetic. - # Issue #155 Finding 2: Gemini 2.5-flash's verbose ``reasoning`` - # field routinely exceeds ``max_output_tokens=512``/``1024`` on this - # fixture's grading workload, hitting MAX_TOKENS and truncating - # mid-string. That truncation surfaces downstream as a degraded - # grade result (``aggregate_complete=False``) and would flake - # assertion #6 below. The #155 live probe verified ``2048`` passes - # cleanly. Per DEC-009 the floor lives here in the test overlay - # rather than as a bumped ``GradeConfig`` production default. + # ``grade_max_output_tokens=4096`` is **load-bearing**, not cosmetic. + # Issue #155 Finding 2 + issue #158: Gemini 2.5-flash's verbose + # ``reasoning`` field truncates at low caps (512/1024) on every + # fixture; the #155 probe found 2048 sufficient for the 5-pair + # in-isolation smoke, but #158 caught that this full-pipeline + # fixture runs 108–116 (artifact × criterion) pairs and 5–6 of them + # still exceed 2048 (typed-degrading to ``GradeLLMError`` and + # flipping ``aggregate_complete=False``). 4096 is the #158 floor for + # this fixture's grading workload. Per DEC-009 the floor still + # lives here in the test overlay rather than as a bumped + # ``GradeConfig`` production default. apply_provider_override( project_dir, grade_provider="gemini", grade_model="gemini-2.5-flash", - grade_max_output_tokens=2048, + grade_max_output_tokens=4096, ) # The committed `profiles.yml` pins ``project: bigquery-public-data`` @@ -256,25 +264,29 @@ def test_e2e_signalforge_generate_against_austin_bikeshare_with_gemini_grader( # 6. Grade aggregate_complete — no degraded calls. # This is the **load-bearing assertion** that proves the - # ``max_output_tokens=2048`` overlay fixes the truncation bug. - # Post-#155 US-001 the provider-neutral ``is_clean_completion`` - # gate raises ``LLMResponseFormatError`` on a MAX_TOKENS finish - # even when partial text is present, which ``call_llm`` - # propagates as ``LLMError`` and ``grade_artifacts`` wraps as + # ``max_output_tokens=4096`` overlay (#158) fixes the truncation + # bug at this fixture's scale. Post-#155 US-001 the + # provider-neutral ``is_clean_completion`` gate raises + # ``LLMResponseFormatError`` on a MAX_TOKENS finish even when + # partial text is present, which ``call_llm`` propagates as + # ``LLMError`` and ``grade_artifacts`` wraps as # ``GradeLLMError`` → - # ``GradingResult(score=None, passed=False, reasoning="call failed: GradeLLMError")`` - # → ``aggregate_complete=False``. The cap MUST keep every - # (artifact, criterion) pair scoring cleanly. + # ``GradingResult(score=None, passed=False, + # reasoning="call failed: GradeLLMError: ")`` + # (#158 broadened the reasoning to surface the inner finish_reason + # so a residual degrade is self-diagnosing) → ``aggregate_complete=False``. + # The cap MUST keep every (artifact, criterion) pair scoring cleanly. grade_sidecar = project_dir / ".signalforge" / "grade.json" assert grade_sidecar.is_file(), f"grade sidecar missing at {grade_sidecar}" grading_report = GradingReport.model_validate_json(grade_sidecar.read_text()) assert grading_report.aggregate_complete is True, ( "expected GradingReport.aggregate_complete=True (every (artifact, criterion) " - "pair scored cleanly with Gemini at max_output_tokens=2048); got False — " - "either the 2048 cap is no longer sufficient (see #155 Finding 2; raise the " - "overlay), Gemini hit a safety-filter / RECITATION block, or " + "pair scored cleanly with Gemini at max_output_tokens=4096); got False — " + "either the 4096 cap is no longer sufficient (see #158; raise the overlay " + "and re-measure), Gemini hit a safety-filter / RECITATION block, or " "grade.total_budget_seconds tripped. Inspect " - ".signalforge/grade.jsonl for the per-pair degrade reasons." + ".signalforge/grade.jsonl for the per-pair degrade reasons — #158 now " + "carries the inner finish_reason into the reasoning string." ) # 7. No traceback in stderr (DEC-016 of cli-layer.md — the CLI's diff --git a/tests/grade/test_engine.py b/tests/grade/test_engine.py index 4690a34a..a35c3dd8 100644 --- a/tests/grade/test_engine.py +++ b/tests/grade/test_engine.py @@ -598,6 +598,71 @@ def test_grade_artifacts_one_criterion_retry_exhausted_does_not_fail_whole_repor assert report.aggregate_complete is False +# --------------------------------------------------------------------------- +# Issue #158 — degrade-reasoning carries LLMResponseFormatError message +# --------------------------------------------------------------------------- + + +def test_format_degrade_reasoning_includes_response_format_error_message() -> None: + """``_format_degrade_reasoning`` surfaces the inner + :class:`LLMResponseFormatError` message — which names the vendor + ``finish_reason`` field + value — when the wrapped cause is a + response-shape error. + + This is the issue #158 diagnostic upgrade: before this change every + Gemini ``MAX_TOKENS`` / ``SAFETY`` / ``RECITATION`` degrade collapsed + to the bare ``"call failed: GradeLLMError"`` string, forcing + operators to re-read stderr to distinguish them. The new shape + includes the provider-emitted message so the audit JSONL / sidecar + is self-diagnosing. + """ + from signalforge.grade.engine import _format_degrade_reasoning + from signalforge.grade.errors import GradeLLMError + from signalforge.llm.errors import LLMResponseFormatError + + inner = LLMResponseFormatError( + "Gemini response unclean (finish_reason='MAX_TOKENS').", + ) + wrapped = GradeLLMError("LLM call failed", cause=inner) + + reasoning = _format_degrade_reasoning(wrapped) + + # Preserves the existing "call failed: " prefix so the audit + # corpus stays diff-clean for the 90% case, AND grows the inner + # message after a colon so the finish_reason value survives. + assert reasoning == ( + "call failed: GradeLLMError: Gemini response unclean (finish_reason='MAX_TOKENS')." + ) + + +def test_format_degrade_reasoning_preserves_bare_shape_for_non_response_format_causes() -> None: + """Every cause other than :class:`LLMResponseFormatError` keeps the + pre-#158 ``"call failed: "`` shape verbatim. + + Acceptance criterion of bd issue: rate-limit / auth / parser failure + / budget-exceeded degrades stay diff-clean in the audit corpus — + only the response-shape branch grows the diagnostic. + """ + from signalforge.grade.engine import _format_degrade_reasoning + from signalforge.grade.errors import GradeLLMError, GradeOutputError + from signalforge.llm.errors import LLMAuthError, LLMRateLimitError + + rate_limit = GradeLLMError( + "rate limit exhausted", + cause=LLMRateLimitError("429", attempts=3, cause=RuntimeError("upstream")), + ) + assert _format_degrade_reasoning(rate_limit) == "call failed: GradeLLMError" + + auth = GradeLLMError( + "auth failed", + cause=LLMAuthError("401", cause=RuntimeError("upstream")), + ) + assert _format_degrade_reasoning(auth) == "call failed: GradeLLMError" + + parser = GradeOutputError("bad json", violation_type="json_parse") + assert _format_degrade_reasoning(parser) == "call failed: GradeOutputError" + + # --------------------------------------------------------------------------- # Whole-run pre-flight envelope-breach (DEC-013) # --------------------------------------------------------------------------- diff --git a/tests/grade/test_gemini_neutrality.py b/tests/grade/test_gemini_neutrality.py index d4b5f63f..ad9a97c2 100644 --- a/tests/grade/test_gemini_neutrality.py +++ b/tests/grade/test_gemini_neutrality.py @@ -379,11 +379,20 @@ def test_grade_artifacts_safety_blocked_response_degrades_pair( # exercised here AND the MAX_TOKENS-with-partial-text path through # the same orchestrator gate, so this assertion is the shared # contract pin for both regressions). + # + # Issue #158 broadened the reasoning string so the inner + # ``LLMResponseFormatError`` message (which names the vendor's + # ``finish_reason`` value) survives into the audit JSONL / sidecar + # — operators can now distinguish ``SAFETY`` vs ``MAX_TOKENS`` vs + # ``RECITATION`` degrades without re-reading stderr. The bare + # ``"call failed: GradeLLMError"`` shape is preserved verbatim for + # every other cause (auth / rate-limit / parser failure). bad = degraded[0] assert bad.score is None assert bad.passed is False assert bad.evidence == "" - assert bad.reasoning == "call failed: GradeLLMError" + assert bad.reasoning.startswith("call failed: GradeLLMError: ") + assert "finish_reason='SAFETY'" in bad.reasoning # The other pairs were scored normally. for good in scored: