fix(core): stop isUnusableStage1Reply deferring valid bare-code replies (#11504) - #11555
Conversation
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…es (#11504) The Stage-1 junk heuristic flagged any character repeated 5+ times anywhere in the reply (/(.)\1{4,}/u), so the 8+ consecutive spaces of two-level code indentation — a gemma-4-31b HumanEval-style bare python body — plus markdown "-----" dividers and pretty-printed JSON all got replaced with "I'm not sure how to answer that." The starts-with-fence escape only rescued replies that BEGIN with a code fence, so bare code and prose-then-fenced replies both died. eliza-harness HumanEval scored 0.40 vs 1.00 for the same model on raw harnesses. Tighten the degenerate-run rule to fire only when the WHOLE reply is a single repeated code point (non-whitespace content, length >= 5, one unique char). Empty replies, JSON-scaffold fragments, digit-only replies, enum echoes ("RPPY"), and single-char spam ("aaaaa", "!!!!!!!!", "aaaaa aaaaa") all stay unusable. Regression tests: bare nested-indent code body, fenced code, prose + fenced code, pretty-printed JSON, and markdown-divider replies ship verbatim; extended the known-junk test with whitespace-only and degenerate-spam fragments. Refs #11504 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
731b70f to
66f3aa1
Compare
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
…te (#11579) Port the six finer-grained regression tests from the closed PR #11553 atop the merged #11555 fix: bare unfenced code-body verbatim, fenced code with a prose lead-in, pretty-printed JSON, prose containing separator/emphasis runs, still-defers empty/whitespace/refusal-stub/degenerate-run, and refusal-that-continues-into-content + bare social apology. Running the ported suite against the merged implementation exposed two real coverage gaps in isUnusableStage1Reply: - Bare refusal stubs ("I am not sure.", "I don't know.", "I'm sorry, I can't help with that.") were delivered verbatim instead of deferring — #11504's ask was to tighten the unusable signal to actual refusals/empties. Port #11553's whole-reply-anchored isBareRefusalStage1Reply (apology-prefix stripped; refusal-plus-content and bare social apologies are kept). - Multi-token degenerate spam ("aaaaa bbbbb") slipped the merged whole-reply single-code-point check. Add #11553's per-token check (every whitespace-separated token is one character repeated 5+ times) alongside the existing set-based check, so nothing the merged heuristic already caught is weakened. Refs #11504 Co-authored-by: lalalune <shaw.nicola.walters@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
Root cause
isUnusableStage1Reply(packages/core/src/services/message.ts) classified a Stage-1 simple-path reply as junk when any character repeats 5+ times anywhere in the text:Two-level code indentation is 8+ consecutive spaces, so a gemma-4-31b HumanEval-style bare python body
matched the rule and got replaced with
"I'm not sure how to answer that.". The starts-with-fence escape (/^```[a-z0-9_-]*\s+/iu) only rescued replies that begin with a code fence — bare code bodies,prose + fenced blockreplies, pretty-printed JSON, and markdown-----dividers all died. This is the runtime-pipeline gap called out in.github/issue-evidence/10199-gemma-4-31b-cutover/review-package-multiharness/scorecard.md: eliza-harness HumanEval 0.40 vs 1.00 for the same model (gemma-4-31b on Cerebras) on hermes/openclaw raw harnesses (~60% of code turns deferred), and in production it discards legitimate code/structured replies.Fix
Tighten the degenerate-run rule so it fires only when the whole reply is a single repeated code point (non-whitespace content, length ≥ 5, exactly one unique character):
No benchmark-mode special case — the unusable signal is now scoped to actual degenerate output, which fixes production code-reply UX and the benchmark cell at once.
Deliberately preserved protections (all still return unusable):
{},::::,,,,)isTerseReplyWorthKeepingwhen the answer is genuinely numeric, e.g."4"for "What is 2+2?")RPPY,RESPOND) with theSay PONGintentional-token rescue intactaaaaa,!!!!!!!!,aaaaa aaaaa)Behavior deliberately loosened: a 5+ repeated-char run inside a longer reply (code indentation,
-----dividers,.....ellipses, "soooo") no longer marks the whole reply unusable — that was the bug.Tests
New regression test
delivers bare-code and structured Stage 1 replies verbatimdrives the fullrunV5MessageRuntimeStage1path with: (a) a gemma-style bare nested-indent python body, (b) a fenced code block, (c) prose + fenced code, (d) pretty-printed JSON, (e) a markdown divider reply — each ships verbatim with exactly one model call. Extended the known-junk test with" "(whitespace-only),"!!!!!!!!", and"aaaaa aaaaa".Refs #11504
🤖 Generated with Claude Code