Skip to content

fix(orchestrator): stop verify-failed completions from shadowing the successful retry as origin best result - #11848

Merged
lalalune merged 2 commits into
developfrom
fix/rh-8
Jul 3, 2026
Merged

lalalune merged 2 commits into
developfrom
fix/rh-8

Conversation

@NubsCarson

Copy link
Copy Markdown
Member

Defect

35a9e81 (#11514) added captureOriginResultForCompletion on the router's task_complete early returns so a finished deliverable survives to the per-origin spawn cap. But the verify-retry handoff records a completion the router itself just judged a failed build (deadUrls.length > 0), and recordOriginResult is pure longest-wins — no verification-status gate. The #11514 justification ("recordOriginResult is monotonic so recording here is always safe") is wrong for known-failed results: longest-wins is not correctness-wins.

Failure path (refs on pre-fix develop):

  1. Build attempt 1 emits task_complete referencing a URL that fails liveness verification. annotateUnverifiedUrls stamps the narration with the dead-URL failure annotation (sub-agent-router.ts ~2988) — which systematically makes the failed text longer than a clean success text.
  2. retryIncompleteBuild spawns attempt 2, and the verify-retry handoff records the failed completion for the origin (sub-agent-router.ts:1135; same-shaped captures at 1150/1196). sanitizeSuccessorMetadata preserves the origin keys byte-for-byte, so both attempts record under the SAME origin-result key.
  3. Attempt 2 succeeds with a shorter deliverable. recordOriginResult (if (prev && candidate.length <= prevLen) return;) keeps the longer verify-FAILED text — the successful retry can never displace it.
  4. When the per-origin spawn cap fires, tasks.ts (~1199-1211) relays best.deliverable ?? best.text verbatim: the user receives the dead-URL completion the router judged a failed build — including the planner-only [verification: … do NOT tell the user the app is live …] directive — as the final answer.

Fix

Gate verify-FAILED completions (deadUrls.length > 0) out of origin-result capture at every site. A known-failed build is not a relayable deliverable; with nothing clean captured, the spawn cap's honest "attempted N times" fallback covers the case where no clean completion ever lands. The main-path record now routes through the same helper, so the #8875 key contract and the gate live in one place. recordOriginResult stays longest-wins for the results that are actually relayable.

Tests (real path, no mocks of the thing under test)

New origin-result-verify-failed-shadow.test.ts drives the real handleEvent and the real TASKS spawn-cap branch through the exact two-attempt lineage against a real local HTTP app host whose deploy state transitions 404 → 200 (attempt 1's dead URL, then the retry's verified-live URL; real safeFetch probes both ways). The retry successor session uses the byte-identical metadata shape retryIncompleteBuild produces (sanitizeSuccessorMetadata + buildVerifyRetryCount/retryOfSessionId).

RED on pre-fix develop — reproduced at the exact user-facing symptom:

FAIL … > does not record the verify-failed completion as the origin best result
  expected undefined, received the verify-FAILED annotated narration as bestResultFor

FAIL … > never relays the verification-failure text as the final answer
AssertionError: expected '[sub-agent: build game (codex) — task…' not to contain 'NOT reachable'
+ [verification: the following URL(s) the sub-agent referenced are NOT reachable — do NOT tell
+ the user the app is live; report the real status and that the build likely did not complete]
+   -  → HTTP 404

GREEN after the fix. The #11514 test that pinned the old behavior (verify-retry handoff records the failed completion) is updated to pin the corrected semantics, with the rationale in-line; its other pins (lineage-dedupe longest-wins for clean completions, honest cap fallback, deliverable relay at the cap) are unchanged and still pass.

Test Files  151 passed | 4 skipped (155)
     Tests  1576 passed | 8 skipped (1584)

bun run typecheck exit 0; bun run lint:check clean (plugin scope).

Evidence

  • Real-LLM trajectory: N/A — deterministic router bookkeeping (which completion is retained/relayed), no model/prompt/action-selection behavior change; the regression test drives the real completion pipeline end to end including real HTTP liveness probes.
  • Screenshots/video: N/A — no UI surface.
  • Backend logs: the tests exercise the real [verify] probe … → HTTP 404 / re-dispatched sub-agent after failed verification router log path.
  • Domain artifact: bestResultFor() state inspected before/after each attempt in the tests — failed capture absent, clean retry result retained, spawn-cap reply free of the failure annotation.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@coderabbitai

coderabbitai Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: b47e3ce3-b9d5-4643-970c-cedc0cd77e80

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/rh-8

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

NubsCarson and others added 2 commits July 3, 2026 03:22
…successful retry as origin best result

35a9e81 (#11514) records completions on the task_complete early
returns so the spawn cap can relay them instead of re-spawning. but the
verify-retry handoff records a completion the router itself just judged
a failed build: the dead-url verification annotation makes that text
systematically longer than a clean success, so longest-wins
recordOriginResult keeps the failure and the successful retry's shorter
deliverable can never displace it. at the per-origin spawn cap tasks.ts
then relays the dead-url completion — planner-only verification
directive included — verbatim to the user as the final answer.

gate verify-failed completions (deadUrls > 0) out of origin-result
capture at every site: a known-failed build is not a relayable
deliverable, and the cap's honest "attempted N times" fallback covers
the nothing-clean case. the main-path record now routes through the
same helper so the key contract (#8875) and the gate live in one place.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@lalalune
lalalune merged commit ffea64c into develop Jul 3, 2026
37 of 39 checks passed
@lalalune
lalalune deleted the fix/rh-8 branch July 3, 2026 10:24
@claude

claude Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Claude encountered an error —— View job


I'll analyze this and get back to you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants