Skip to content

fix(core): stop transcript guard from rewriting replies that quote a transcript - #11837

Merged
lalalune merged 1 commit into
developfrom
fix/rh-5
Jul 3, 2026
Merged

fix(core): stop transcript guard from rewriting replies that quote a transcript#11837
lalalune merged 1 commit into
developfrom
fix/rh-5

Conversation

@NubsCarson

Copy link
Copy Markdown
Member

Defect

Follow-up to 35e6a66 (#11712). The fail-closed raw-transcript guard and the text-mode transcript parser silently rewrote legitimate replies that QUOTE a shouldRespond:/replyText: transcript, dropping the actual answer:

  • Send boundary (packages/core/src/services/message.ts final_reply guard): looksLikeRawFieldTranscript(reply) matched /(^|\n)\s*replyText\s*:/ anywhere in the reply — including quoted lines and lines inside code fences. A user pastes a leaked transcript into Discord and asks the bot to diagnose it (this repo's own daily workflow); Stage 1 answers correctly in the canonical JSON envelope with replyText = diagnostic prose that quotes the transcript. The guard fired on the quoted replyText: line and replaced the whole reply with extractReplyTextFromTranscript() output — only the text after the QUOTED marker — destroying the entire diagnosis with nothing but a logger.warn.
  • Text-mode path (packages/core/src/runtime/message-handler.ts parseMessageHandlerFieldTranscript): claimed any non-JSON prose containing a replyText: line and discarded every preamble line ("lines before the first field marker are preamble; ignore them") — same silent content loss for plain-prose answers that quote a transcript.
  • The comment above the claim check said "routing field plus reply-bearing field" but the code was OR (!hasShouldRespond && !hasReplyText).

Fix (structural, not contains-check)

A genuine text-mode envelope echo IS the message — its first substantive line (outside code fences) is a known field line, with a shouldRespond:/replyText: hallmark at top level. A reply that opens with prose, or whose field lines sit inside a code fence, QUOTES a transcript and is content:

  • looksLikeRawFieldTranscript is now a fence-aware line scan implementing that rule. Quoting replies ship intact at the send boundary and through the plain-text synthesizer; the genuine raw RESPONSE_HANDLER structured output (shouldRespond:/replyText:/contexts:/topics:/emotion:) sent verbatim to discord when field parse falls through #11712 leak shape (leading skeleton) is still detected, blocked, and recovered.
  • parseFieldTranscript treats field lines inside ``` / ~~~ fences as value content, so a real leak whose replyText value quotes an envelope in a fence is no longer split at the quoted lines.
  • parseMessageHandlerFieldTranscript is gated on the same detector, so prose-with-preamble falls through to the tolerant plain-text handler with the full answer intact.
  • Comment/code OR-vs-AND mismatch resolved in favor of documented-correct OR: a lone shouldRespond: IGNORE echo must remain claimable; the comment now says so explicitly.

Trade-off made explicit: a hypothetical leak prefixed by scaffold prose would now ship as visible (ugly but complete) text instead of being rewritten — visible skeleton beats silent destruction of a correct answer, and the observed #11712 leak shape (leading skeleton) remains fully fail-closed.

Reproduction / evidence

Re-confirmed on develop tip 16b69a6edf before the fix — 6 new tests failed exactly as the defect describes:

FAIL  looksLikeRawFieldTranscript(QUOTING_DIAGNOSIS)  → expected false, got true
FAIL  looksLikeRawFieldTranscript(FENCED_QUOTING_DIAGNOSIS) → expected false, got true
FAIL  parseMessageHandlerOutput(QUOTING_DIAGNOSIS)    → expected null, got claimed
      transcript whose plan.reply = "it works\n\nthe blank line…" (diagnosis preamble dropped)
FAIL  fenced field lines split the replyText value of a real leak

After the fix:

  • packages/core/src/runtime/__tests__/response-field-transcript.test.ts — 22/22 (all 16 pre-existing raw RESPONSE_HANDLER structured output (shouldRespond:/replyText:/contexts:/topics:/emotion:) sent verbatim to discord when field parse falls through #11712 regression tests untouched and green, + new quoting/fence coverage)
  • packages/core/src/__tests__/message-routing-live-regression.test.ts — new end-to-end scenario: canonical JSON envelope quoting a transcript → routeMessageHandlerOutputfinal_reply → guard predicate false → diagnosis ships verbatim; plus fail-closed regression for the genuine leak
  • Affected suites (12 files: message-handler, json-output, response-handler field registry/evaluators, message service abort/stage1-retry/shortcut/voice gates): 222/222 green
  • Full packages/core suite: 2670 passed / 11 skipped; the single failure (link-extraction.test.ts, live example.com fetch) reproduces on clean HEAD with this change stashed — pre-existing and unrelated
  • bun run --cwd packages/core typecheck clean

N/A — UI/screenshots/video: runtime parser/guard change, no user-facing surface beyond message text integrity, fully covered by the routing-level tests above.
N/A — live-LLM trajectory: the defect is deterministic post-model-output plumbing; the failing-then-passing tests drive the exact routed path (parseMessageHandlerOutputrouteMessageHandlerOutput → send-boundary predicate) with the confirmed real-world payload shape.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@coderabbitai

coderabbitai Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: e10d2507-40df-4126-8610-4932e042dd80

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/rh-5

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…transcript

follow-up to 35e6a66 (#11712): looksLikeRawFieldTranscript fired on any
replyText: line anywhere in a reply, so a legitimate diagnosis that QUOTES a
leaked shouldRespond:/replyText: transcript (this repo's own daily debugging
workflow) was silently replaced at the send boundary by the QUOTED replyText
tail — the whole answer dropped with only a logger.warn. same hijack on the
text-mode path: parseMessageHandlerFieldTranscript claimed any prose with a
replyText: line and discarded every preamble line.

structural rule instead of contains-check: a raw envelope echo IS the message —
its first substantive line outside code fences is a known field line, with a
shouldRespond:/replyText: hallmark at top level. prose preamble or fenced field
lines mean the reply QUOTES a transcript and ships intact. parseFieldTranscript
now treats fenced field lines as value content, so a real leak whose replyText
quotes an envelope in a fence is no longer split at the quoted lines. the
text-mode claim is gated on the same detector, and the comment/code mismatch
(comment said routing AND reply field, code was OR) is resolved in favor of the
documented-correct OR (a lone shouldRespond: IGNORE echo must stay claimable).

genuine leak shape from #11712 (leading skeleton) is still detected, blocked,
and recovered — existing regression tests unchanged and green.
@lalalune
lalalune merged commit 9a9be08 into develop Jul 3, 2026
@lalalune
lalalune deleted the fix/rh-5 branch July 3, 2026 10:19

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@github-actions github-actions Bot added the Tests label Jul 3, 2026
@claude

claude Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Claude encountered an error —— View job


I'll analyze this and get back to you.

@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

LifeOps Benchmark — eliza

Run ID: lifeops-eliza-28654216533

LifeOps Benchmark

Model: gemma-4-31b
Judge: claude-opus-4-7
Scenarios: 25
pass@1: 0.000
pass@k: 0.000
Total cost: $0.0000

Full artifacts: see the lifeops-run-eliza-28654216533 upload on this run.

@github-actions

github-actions Bot commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

LifeOps Benchmark — hermes

Run ID: lifeops-hermes-28654216533

LifeOps Benchmark

Model: gemma-4-31b
Judge: claude-opus-4-7
Scenarios: 25
pass@1: 0.360
pass@k: 0.360
Total cost: $0.0000

Full artifacts: see the lifeops-run-hermes-28654216533 upload on this run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants