fix(noema): reject a completion the provider declares truncated

[seonghobae]

Lane: jan | session:session_01BV96rXhqoR3tYZ9AeAVur4

The hole

scripts/ci/noema_review_gate.py never reads finish_reason — the field where an OpenAI-compatible provider declares it stopped at the output-token budget rather than because the model finished. On main@f25063882, grep -c finish_reason returns 0.

Truncation is only sometimes caught downstream. Driving main's own parser directly:

'{"decision":"approve","summary":"looks fine","findings":[{"severity":"high"'
   → NoemaModelOutputError                              ✅ unbalanced, fails closed

'{"decision":"approve","summary":"reviewed","findings":[]}'
   → parsed as a valid verdict                          ⚠️ cut lands on a closed object

The second case is not a corner case. findings is emitted last, so the likeliest parseable truncation is an approval carrying an empty or short findings list — a review cut off mid-thought, arriving as a genuine-looking APPROVE on a required gate, with nothing anywhere saying it was incomplete.

main's local repair is deliberately lossless (_strip_trailing_commas_outside_strings only), so it does not manufacture this. The provider does, and the gate simply never looked at the field where the provider says so.

This is the same failure family as #1921: a reviewer that could not see everything returns APPROVE, silently. There the cause is allowed-locations starvation; here it is output-budget truncation.

The fix

reject_truncated_completion(raw) reads choices[0].finish_reason on the decoded body and raises NoemaModelOutputError for the unambiguous "length". It runs in call_llm immediately after decode_llm_response_body, ahead of content extraction.

Deliberately narrow, in three ways:

• Only "length" is rejected. Missing, empty, "stop", "tool_calls", "end_turn", "content_filter" — all pass untouched. A provider reporting a vocabulary this gate does not model must not be failed spuriously, and this gate fronts every repository in the organization.
• It never raises a shape error. Malformed envelopes keep being classified by extract_llm_message_content, which reports their actual cause with precise messages. Duplicating that here would attribute the wrong reason to a body this function cannot classify.
• No retry, no new exception type, no change to repair ownership. The gateway keeps that, per the caller attempts=1 (gateway owns repair/failover) contract already on main.

Relationship to #1606

This is the portable half of #1606. That PR cannot merge as written: alongside this check it adds caller-side retry (is_retry, truncated_after_retry, a stale-head check before retry) built on extract_llm_completion, which main replaced with extract_llm_message_content when it moved repair ownership to the gateway. Resolving its conflict would mean silently reversing that architectural decision. Detail in #1606.

So the defect is landed here in main's own idiom and its own error type, and #1606's remaining parts — the bounded-field validators and NOEMA_MAX_* constants — stay available as their own change if wanted.

Verification

• pytest tests/test_noema_review_gate.py -k "truncated_completion or call_llm_consults" → 15 passed
• coverage run -m pytest tests -q → 2905 passed, 1 skipped, 21 subtests passed
• coverage report → scripts/ci/noema_review_gate.py 100% (864 statements / 388 branches, 0 missing); TOTAL 100%
• interrogate → PASSED (100.0%)
• git diff --check clean
• Branched from origin/main@f25063882 in a fresh worktree

The first test asserts the truncated payload really would have parsed before the guard — otherwise the test would pass for the wrong reason and prove nothing about the hole being closed. A separate test pins that the guard runs ahead of content extraction, so a future reorder cannot silently move it after the parse.

Developer experience

A truncated review now fails with a message naming the cause (finish_reason=length) instead of either succeeding wrongly or failing later with a misleading JSON-shape error. The message embeds no part of the untrusted response body, so it stays safe in the public pull_request_target job log.

User experience

An APPROVE from noema-review can no longer be produced by a completion the provider itself declared incomplete. This closes a path by which a required gate could pass a change nobody finished reviewing.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4

---

Generated by Claude Code

[coderabbitai]

Warning

Review limit reached

Next included review available in 34 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: ebe42fcd-0a92-4a21-90d8-cd89adb11f28

📥 Commits

Reviewing files that changed from the base of the PR and between 4302463 and e13bd33.

📒 Files selected for processing (2)

• scripts/ci/noema_review_gate.py
• tests/test_noema_review_gate.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[seonghobae]

CI status note — standing down on the two red checks on head 88588b31b.

CodeQL compatibility analysis (python) and (actions) ended with VERDICT_STATE=pending: the compat job's designed fail-fast after dispatching a CodeQL PR scan whose terminal verdict has not been published yet. The dispatched scan (run 33977184168, re-dispatched 16:13Z by the ready-for-review event) is queued behind the organization job ceiling — of 70 CodeQL PR runs in .github since 13:00Z, 44 are queued, 20 cancelled, 2 failed, 0 succeeded. Nothing in this PR's diff is involved, and the dispatch workflow reruns this exact job once the verdict lands.

Not re-running it manually: a re-run dispatches again and enlarges the queue that is causing the failure (docs/doctoring/ci-failure-signature-triage.md §9). Capacity relief is in #1938.

---

Generated by Claude Code

[seonghobae]

CI note and push: head 88588b31b → 773e70139 (a base merge only; no change to the PR's own commits).

Two new red checks on the previous head, both the single-upstream gateway stall class and neither in this diff: strix (job 101353895177) ended "provider/backend was unavailable", and noema-review (job 101354120806) failed with the gateway 502 after 1685 s. #1939 removed that stall on main at 17:25Z, but a re-run cannot pick it up because workflow_sha is bound at run creation (catalog §2), so this push merges main@f2f91b806 into the branch to create the new event. The two CodeQL compatibility failures remain the designed pending state noted above.

Local gate on 773e70139: 2911 passed / 1 skipped, coverage 100% (13144 statements, 5312 branches; scripts/ci/noema_review_gate.py unchanged at 100%), interrogate 100%, git diff --check clean. Cadence: 9 h 19 m since the previous push.

---

Generated by Claude Code

[seonghobae]

CI note on head 773e70139: noema-review (run 33992904601, job 101388218941) failed in "Prepare Noema model verdict" with HTTP Error 429: Too Many Requests; caller attempts=1, duration=1444.7s, phase=response_error, served_model=deepseek-ai/deepseek-v4-pro-0813. Gateway-side; nothing in this diff (noema_review_gate.py's truncated-completion rejection and its tests) is on that path — the request never returned a completion to reject.

Same class as #1938's failure 20 minutes earlier, read per host 1's source trace on that thread: the gateway did fail over; served_model names the last route tried; 1444.7 / 548 ≈ 2.6 says two or three of the ready routes each waited out their per-recv timeouts before answering 429. The run was created at 21:23Z, so its sidecar is post-#1939 (diversified catalog) but its workflow predates #1944, so there is no noema-sidecar-evidence artifact to show the per-route walk.

Corrected tally, since the one I posted on #1938 was wrong: of the post-#1939 noema-review runs in this repository that actually reached the verdict step, 1 succeeded (#1902) and 3 failed (#1872 502 after 1989.9 s, #1938 429 after 551 s, this one 429 after 1444.7 s). The three run-level "successes" at 21:59–22:15Z were the closure-event runs of #1943/#1944/#1945 after merge, whose job was skipped entirely.

Action: not re-running yet. #1938's attempt-2 re-run (queued 23:49Z, same pre-#1944 workflow) is the measurement of whether the rate has moved; at 1 in 4 a re-run here would spend another ~25 minutes of a runner slot in a 234-deep queue at the same odds. I will issue this PR's single re-run once that result is in, or fold it into the next base-merge push, which also binds the artifact-producing workflow. The two CodeQL compatibility failures remain the designed pending state noted earlier.

---

Generated by Claude Code

[seonghobae]

CI note on head 773e70139, second failing check: strix (run 33992904674, job 101388406980) ended "provider/backend was unavailable" after 23.5 minutes in "Run Strix (quick)". Gateway-side, not this diff — and this one carries the evidence the Noema failure above could not, because the Strix workflow already ships the sidecar files in strix-reports (artifact 9979385164):

• contextual-orchestrator-preflight.json: ready 1 / rejected 11 of the 12 selected routes. All four openrouter routes 429; nvidia_nim deepseek-v4-pro-0813 and deepseek-v4-flash-0731 429; nvidia_nim_sub deepseek-v4-flash-0731 TimeoutError; the four gemma-3 routes 404 on both NVIDIA accounts. The only ready route was nvidia_nim_sub deepseek-v4-pro-0813.
• strix.log: that sole route answered 429 rate_limit_exceeded on the first request and on all five of Strix's replays (backoff 2 → 32 s, 00:06–00:12Z); run.json shows llm_usage.requests: 0, so the scan never got a single completion.
• contextual-orchestrator-sidecar.stderr.log: a wall of request_failed status=429 code=rate_limit_exceeded (the pre-fix(sidecar): let the stream sanitizer pass orchestrator route and circuit events #1945 sanitizer, so no per-route lines).

So at 23:48–00:12Z the free pool was rate-limited on every account at once — OpenRouter at preflight, the primary NVIDIA account at preflight, the sub account on request. #1939's interleave is working (4+4+4 selected); there was simply no free capacity to interleave. The same window produced #1938's and this PR's noema-review 429s, and it burned roughly 25 minutes of a runner per job in a 234-deep queue while doing so.

Action: no re-run now — with one ready route that 429s on contact, a re-run is 25 more minutes of slot at near-zero odds. This PR's single sanctioned re-run stays gated on #1938's queued attempt 2 (or the next base-merge push), same as the Noema check. The lever for the condition itself (a pool with ≤1 ready route that rate-limits on first contact should not cost 25 minutes) sits in the orchestrator's retry policy and the Strix replay policy, neither in this diff; I have handed the artifact to the observability lane.

---

Generated by Claude Code

[seonghobae]

CI note on head 773e70139, third failing check: opencode-review (run 33992904744, job 101394230478) failed in seconds at "Fail closed without a current-head OpenCode verdict" — catalog signature 1, not a review verdict; no opencode-agent review exists on this head. The dispatch step succeeded and created handler run 34002410471 (OpenCode Review Dispatch, queued 00:52Z), but every handler run completed since 2026-09-05T17:00Z (83 of 83) was rejected in validate-pr-metadata because the dispatcher identity opencode-agent[bot] is not in OPENCODE_REPOSITORY_DISPATCH_ACTOR (still github-actions[bot]). #1932's multi-identity parser is on main; the variable is an owner action tracked in #1929.

Nothing in this diff is involved, and no push or re-run from this side changes the outcome — a re-run re-dispatches into the same rejected gate, and the dispatch workflow re-runs this exact job by itself once a verdict is published. Standing down on this check until #1929's variable is reconciled; the Strix, Noema, and CodeQL notes above stand.

---

Generated by Claude Code

[seonghobae]

CI note and push: head 773e70139 → 9161e50b2 (a base merge only; no change to the PR's own commits; 5 h 53 m since the previous push).

Merges main@efb892692, the owner's bypass-merged sidecar pin bump to contextual-orchestrator@414f2297 (contextual-orchestrator#1081: _invoke's retry-then-failover no longer stacks on the client's max_retries — the loop behind this head's noema-review 429 after 1444.7 s and its 2 h 18 min Strix), plus #1943 / #1944 / #1945. A re-run could not pick any of that up (workflow_sha and the sidecar pin are bound at run creation), so this push is the remedy for this head's failed noema-review and strix.

Local gate on 9161e50b2 with GITHUB_ACTIONS=true: 2919 passed / 1 skipped, coverage 100% (scripts/ci/noema_review_gate.py unchanged at 100%), interrogate 100%, git diff --check clean. The pool condition (#1948) is unchanged by the bump, so a Noema or Strix failure on this head is still possible; if it happens, the run now ships its per-candidate trace as an artifact. opencode-review stays fail-closed until #1929's variable is set; the CodeQL shards will read designed-pending again.

---

Generated by Claude Code

[seonghobae]

CI note on head 9161e50b2: noema-review (run 34008575125, job 101427287280) failed at "Provision contextual-orchestrator review sidecar" after 5 min 9 s (05:05:55–05:11:04Z) on the new pin 414f2297; "Prepare Noema model verdict" skipped. Artifact noema-sidecar-evidence (9983138060): ready 0 / rejected 12 — both NVIDIA keys' deepseek-v4-flash 429, both keys' deepseek-v4-pro TimeoutError at the 90 s probe bound, the four gemma-3 404, all four OpenRouter free routes 429 — probed 05:08Z. With zero ready routes the launcher exits before /healthz and the sidecar script fails the job: signature 3's fail-fast shape, the sixth consecutive zero-ready artifact across my heads since 04:21Z. Not this PR's; no re-run while the latest artifact shows ≤1 ready route; nothing to push. strix and opencode-review on this head are still queued; their outcomes get one note each only if they differ from what is already recorded here.

---

Generated by Claude Code

[seonghobae]

CI note on head 9161e50b2: strix (run 34008575120, job 101428234002) failed after 1 h 34 m (05:12:24–06:47:09Z; "Run Strix (quick)" 86 min) with STRIX_PROVIDER_UNAVAILABLE — Strix's stream idle timeout (strix/config/models.py:371) after openai.InternalServerError: 500 internal_error — on the new pin 414f2297. It is the first post-co#1081 Strix sample on my heads, and the artifact strix-reports (9984330203) separates what the pin bump fixed from what it did not:

• Route preflight 05:17Z: ready 4 / rejected 8 — both keys' deepseek-v4-flash and deepseek-v4-pro ready; OpenRouter 429 ×4, gemma-3 404 ×4. The pool had come back.
• Serving 05:24–06:47Z: Strix completed 8 requests with usage (417,875 input / 1,235 output tokens in run.json) while the gateway logged 48 TimeoutErrors and 48 × status=500 code=internal_error. 44 of the timeouts were on nvidia_nim deepseek-v4-flash, the first-ranked route, at 90 s each: 66 of the 86 minutes. None of the 48 timeouts was followed by a circuit_failure record; all 6 fast HTTP failures were, and those did fail over to nvidia_nim_sub. So the stalled route was ranked first again on every retry — the tool-bearing passthrough class co#1082 fixes (a bare TimeoutError re-raised as 500 internal_error, _record_failure unreachable). The retry stacking co#1081 removed is gone: attempt=1/1 throughout, no 2/3.

Not this PR's, and nothing to push. This PR's sanctioned re-run is unspent; I am holding it, because the failure is deterministic while that first-ranked route stalls and a re-run costs ~90 runner-minutes for a coin flip. The two sibling scans (#1916, #1946) are still running on the same pool; their outcomes go to #1948, not here, unless they differ in kind.

---

Generated by Claude Code

[seonghobae]

CI note and push: head 9161e50b2 → e13bd337e (a base merge only; no change to the PR's own commits; 7 h 12 m since the previous push).

Merges main@43024633e — #1947 (deferred transient routes), #1949 (lazy fill + per-account 429 skip), #1950 and #1953 — on top of the efb892692 pin bump this head already carried. The merged launcher is measurably better at both ends of this head's failure mode: a fully rate-limited hour now stops after 6 probes and fails provisioning in 126 s instead of 300–844 s (#1913 run 34021019274), and a live hour reaches routes the old fill never did — contextual-orchestrator#1044's 08:26Z run preflighted ready 6 / probed 16 / skipped 4 / deferred 2 including both keys' llama-3.2-11b and carried a review to a served completion. Since this PR's subject is the Noema truncated-completion gate, a head that can actually reach a verdict is worth more than waiting.

The five red checks on the previous head were all attributed and none was this diff's: strix (run 34008575120, noted 06:53Z) was the tool-bearing passthrough timeout class — 48 timeouts, 48 × 500 internal_error, 0 circuit records, co#1082's scope; noema-review the zero-ready provisioning shape; opencode-review signature 1's fail-closed wait behind #1929; both CodeQL shards the designed dispatch-and-exit.

Gate on the pushed tree: 2943 passed, 1 skipped, 21 subtests; coverage 100 % (13,197 statements / 5,334 branches, 0 missed); interrogate 100 %; git diff --check clean.

---

Generated by Claude Code

[cwl-noema-review]

Noema LLM review

The PR adds a fail-closed guard reject_truncated_completion() that raises NoemaModelOutputError when the provider's first choice reports finish_reason=="length", preventing a budget-truncated completion from being parsed as a genuine APPROVE (the most likely parseable truncation, since findings is emitted last). The guard is invoked in call_llm() before extract_llm_message_content(), runs only on the decoded body, and only rejects the unambiguous "length" value, deferring all shape/malformed-envelope errors to the existing content parser. Concrete regression hypotheses were falsified at the guard's shape guards, JSON error handling, and call-site ordering. Tests cover the parseable-but-truncated APPROVE case, allow-list of other finish reasons, deferral of shape errors, and the guard's position ahead of content extraction.

Reviewed changed lines

• scripts/ci/noema_review_gate.py:1212 (RIGHT): Guard returns without raising for non-list or empty choices, deferring shape errors to extract_llm_message_content; no spurious truncation rejection.
• scripts/ci/noema_review_gate.py:1213 (RIGHT): Non-dict first choice returns before any finish_reason access, avoiding AttributeError and preserving the content parser's precise error reporting.
• scripts/ci/noema_review_gate.py:1237 (RIGHT): json.JSONDecodeError is caught and returns, so malformed bodies are not misclassified as truncated.
• scripts/ci/noema_review_gate.py:1641 (RIGHT): Guard is called immediately after _extract_served_model and before extract_llm_message_content, so the truncation signal is honored before any content parsing.

Adversarial validation

• scripts/ci/noema_review_gate.py:1212 (RIGHT) falsified: A response whose choices is not a list or is empty is rejected as truncated, causing a false REQUEST_CHANGES for a well-formed provider response. — reject_truncated_completion checks if not isinstance(choices, list) or not choices: return and returns without raising, allowing extract_llm_message_content to handle the shape error.
• scripts/ci/noema_review_gate.py:1213 (RIGHT) falsified: A first choice that is not a dict (e.g., a string) crashes with an AttributeError instead of deferring to the content parser. — The guard checks if not isinstance(first_choice, dict): return immediately after indexing, so a non-dict value returns None and extract_llm_message_content raises its precise shape error.
• scripts/ci/noema_review_gate.py:1237 (RIGHT) falsified: The guard rejects a response with finish_reason: "length" even when JSON parsing fails, causing a spurious NoemaModelOutputError for malformed bodies. — reject_truncated_completion catches json.JSONDecodeError and returns immediately, so unparseable bodies are not raised here but by the later content extraction path.
• scripts/ci/noema_review_gate.py:1641 (RIGHT) falsified: The guard runs after _extract_served_model, so a truncation signal in the model field could interfere with guard ordering. — reject_truncated_completion(raw) is called before content = extract_llm_message_content(raw), and _extract_served_model does not depend on content, so the ordering is correct for the truncation check.
• Residual risk: The guard only examines the first choice's finish_reason and only the literal "length" value; a provider reporting a non-standard truncation signal (e.g., a custom value in finish_reason or truncation indicated in another field) would not be caught. This is an acceptable residual limitation given the explicit fail-open choice for unmapped vocabularies.

Findings

• No blocking findings.

• Result: APPROVE
• Head SHA: e13bd337e0fc2bb6163c606d6b03bc95804f3ba6
• Reviewer credential: noema-review-github-app-refresh
• Actor: cwl-noema-review[bot]

[seonghobae]

First Noema APPROVE on any of my heads this session, and it landed on the head the base merge produced. cwl-noema-review[bot] approved e13bd337e at 12:29Z, with an adversarial section that falsified four concrete regression hypotheses against the guard's shape checks (non-list/empty choices, non-dict first choice, JSONDecodeError deferral, call-site ordering ahead of extract_llm_message_content) and recorded the residual limitation the design intends — only the first choice's literal "length" is treated as truncation, other vocabularies fail open.

That closes the loop this PR's own subject opened: a truncated completion is now rejected before it can be parsed as an APPROVE, and the reviewer that says so reached a verdict through the same gateway the change protects.

Remaining on this head: both CodeQL compatibility analysis shards (the designed dispatch-and-exit blocked at #1929's actor gate, noted here since 16:15Z yesterday), strix still running, and opencode-review / coverage jobs queued. The PR reads behind again since main has moved twice this hour; I am holding the next base merge rather than resetting a head that just earned a verdict — strix on it is still live, and a push now would discard that run.

---

Generated by Claude Code

[seonghobae]

Failing check on e13bd337e0fc2bb6163c606d6b03bc95804f3ba6: strix (job 101477621732, run 34027404208). Standing down — it is not this PR's — and recording three things this run establishes, because it is an unusually informative specimen.

Not this PR's, and it predates the relevant fix. The preflight report carries no postponed_probed_count key, which marks it as pre-#1957; the run was created at 10:26:01Z and #1957 merged at 11:37:30Z. Classifying by created_at rather than execution time is the only way to read this correctly — the job executed from 11:58Z, well after the merge, so by wall clock it looks like the fix failing.

Classified by the verdict line, not by a grep — and the two disagree here. The artifact does contain loginAsGuest (in strix-pr-scope-jibt1j_66a8/strix.log), which is the Strix sandbox signature. But the terminal verdict is:

STRIX_PROVIDER_UNAVAILABLE: contextual-orchestrator/orchestrator/free exhausted;
the gateway owns provider discovery and failover.

Not STRIX_SANDBOX_UNAVAILABLE. is_caido_bootstrap_timing_error() inspects only the last attempt's log, so an earlier sandbox failure can sit in the artifact while the run dies of something else. A grep would have filed this as a sandbox outage; it is a gateway exhaustion. Preflight was healthy — ready_count 6, probed_count 16, escalations_used 2, deferred_count 2 against target_ready 8 — so the pool was reachable and the failure came later, in serving.

The runner cost, which is the part worth escalating. The strix job held a runner from 11:58:00Z to 17:49:47Z — 351.8 minutes, very nearly six hours — for no verdict. That is not queue time; the surrounding jobs (Detect changed scope, Admit current pull request head, cancel-superseded-pr-runs) all completed in tenths of a minute at 11:20Z, so this job had a runner and spent the whole span executing. It is the longest single occupancy I have measured today by more than a factor of two, against a 10–30 minute normal scan and a shared 60-job ceiling. That is the concrete cost behind #1531.

No fix to port; not re-running. Nothing in this PR's diff can affect gateway capacity, and a re-run of a six-hour job that ends in provider exhaustion would take another slot from the pool that caused it.

I am folding the classification rule and this occupancy measurement into docs/doctoring/ci-failure-signature-triage.md on #1913.

Head unchanged at e13bd337e. Keeping the PR watched.

---

Generated by Claude Code