Conversation
…ctured replies (#11504) The Stage-1 junk-reply heuristic used an unanchored repeated-character check (/(.)\1{4,}/) that flagged any reply containing a 5+ run of the same character anywhere — which matched the 8-space indentation of any nested code block, pretty-printed JSON, and "=====" separator lines. Valid gemma-4-31b bare-code replies were silently replaced with "I'm not sure how to answer that.", dropping the eliza-harness HumanEval score to 0.40 vs 1.00 on raw harnesses. Every check is now anchored to the whole reply: - The degenerate-run check only fires when every whitespace-separated token is a single character repeated 5+ times ("aaaaa", "!!!!! !!!!!"), checked per token to avoid the catastrophic backtracking an anchored alternation regex exhibits. - A bare refusal stub ("I am not sure", "I don't know", optionally apology-prefixed) is now classified unusable, per the issue's ask to tighten the signal to actual refusals/empties. Refusal-plus-content and social apologies are kept. - The leading-fence exemption is removed as dead code: with every rule whole-reply anchored, no fenced reply can match a junk rule, and the old exemption never covered prose-then-fence replies anyway. Regression tests cover the gemma-style bare Python body, prose+fenced code, pretty-printed JSON, separator/emphasis prose (all previously deferred), and keep the true positives (empty, whitespace, scaffold fragments, refusal stubs, degenerate runs) deferring. Closes #11504 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
Contributor
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Member
Author
lalalune
changed the title
This was referenced Jul 2, 2026
lalalune
added a commit
that referenced
this pull request
Jul 2, 2026
…te (#11579) Port the six finer-grained regression tests from the closed PR #11553 atop the merged #11555 fix: bare unfenced code-body verbatim, fenced code with a prose lead-in, pretty-printed JSON, prose containing separator/emphasis runs, still-defers empty/whitespace/refusal-stub/degenerate-run, and refusal-that-continues-into-content + bare social apology. Running the ported suite against the merged implementation exposed two real coverage gaps in isUnusableStage1Reply: - Bare refusal stubs ("I am not sure.", "I don't know.", "I'm sorry, I can't help with that.") were delivered verbatim instead of deferring — #11504's ask was to tighten the unusable signal to actual refusals/empties. Port #11553's whole-reply-anchored isBareRefusalStage1Reply (apology-prefix stripped; refusal-plus-content and bare social apologies are kept). - Multi-token degenerate spam ("aaaaa bbbbb") slipped the merged whole-reply single-code-point check. Add #11553's per-token check (every whitespace-separated token is one character repeated 5+ times) alongside the existing set-based check, so nothing the merged heuristic already caught is weakened. Refs #11504 Co-authored-by: lalalune <shaw.nicola.walters@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #11504
Summary
The runtime fix for #11504 landed in #11555 (
c532e3b565e):isUnusableStage1Replyno longer defers valid bare-code or structured replies just because they contain repeated-character runs. This PR converts the remaining duplicate branch into the missing live evidence/closure artifact.Evidence Added
.github/issue-evidence/11504-stage1-code-replies/README.md.github/issue-evidence/11504-stage1-code-replies/live-cerebras-current/93432706-b3b2-08ea-ab6a-ba55340a8848/tj-74b5c8fef7d6a9.jsonManual review of the trajectory confirms:
status: finishedmessageHandlerstagegemma-4-31bcontexts:["simple"]withrequiresTool:falsedirect_replyinstead ofI'm not sure how to answer that.Verification
bun run --cwd packages/core test src/__tests__/message-runtime-stage1.test.ts->1 passed,75 passedbunx @biomejs/biome check packages/core/src/services/message.ts packages/core/src/__tests__/message-runtime-stage1.test.ts .github/issue-evidence/11504-stage1-code-replies/README.md->Checked 2 files ... No fixes applied.bun run --cwd packages/core typecheck-> passgemma-4-31bCerebras runtime capture -> pass, artifact abovebun run install:light-> pass, artifact sync skipped withELIZA_SKIP_ARTIFACT_SYNC=1bun installattempted, but postinstall artifact sync was still at7.0 MiB / 971 MiBwith141m 59sremaining, so it was stopped before completionbun run verifycurrently fails before typecheck/lint inaudit:type-safety-ratcheton current repo baseline (as unknown as: 77 / 76,?? 0core/agent/app-core:376 / 375). This PR changes only evidence artifacts, with no production-source diff.UI / Media
N/A - core runtime evidence only; no user-facing UI changed.