Skip to content

fix(core): stop isUnusableStage1Reply deferring valid bare-code replies (#11504) - #11555

Merged
lalalune merged 1 commit into
developfrom
fix/11504-stage1-bare-code-reply
Jul 2, 2026
Merged

lalalune merged 1 commit into
developfrom
fix/11504-stage1-bare-code-reply

Conversation

@lalalune

@lalalune lalalune commented Jul 2, 2026

Copy link
Copy Markdown
Member

Root cause

isUnusableStage1Reply (packages/core/src/services/message.ts) classified a Stage-1 simple-path reply as junk when any character repeats 5+ times anywhere in the text:

if (/(.)\1{4,}/u.test(trimmed)) return true;

Two-level code indentation is 8+ consecutive spaces, so a gemma-4-31b HumanEval-style bare python body

def has_close_elements(numbers: List[float], threshold: float) -> bool:
    for idx, elem in enumerate(numbers):
        for idx2, elem2 in enumerate(numbers):
            ...

matched the rule and got replaced with "I'm not sure how to answer that.". The starts-with-fence escape (/^```[a-z0-9_-]*\s+/iu) only rescued replies that begin with a code fence — bare code bodies, prose + fenced block replies, pretty-printed JSON, and markdown ----- dividers all died. This is the runtime-pipeline gap called out in .github/issue-evidence/10199-gemma-4-31b-cutover/review-package-multiharness/scorecard.md: eliza-harness HumanEval 0.40 vs 1.00 for the same model (gemma-4-31b on Cerebras) on hermes/openclaw raw harnesses (~60% of code turns deferred), and in production it discards legitimate code/structured replies.

Fix

Tighten the degenerate-run rule so it fires only when the whole reply is a single repeated code point (non-whitespace content, length ≥ 5, exactly one unique character):

const nonWhitespace = [...trimmed.replace(/\s+/gu, "")];
if (nonWhitespace.length >= 5 && new Set(nonWhitespace).size === 1) {
	return true;
}

No benchmark-mode special case — the unusable signal is now scoped to actual degenerate output, which fixes production code-reply UX and the benchmark cell at once.

Deliberately preserved protections (all still return unusable):

  • empty / whitespace-only replies
  • JSON-scaffold fragments ({}, ::::, ,,,)
  • digit-only replies (still rescued downstream by isTerseReplyWorthKeeping when the answer is genuinely numeric, e.g. "4" for "What is 2+2?")
  • non-allowlisted 2–8 char all-caps enum echoes (RPPY, RESPOND) with the Say PONG intentional-token rescue intact
  • single-character spam, including across whitespace (aaaaa, !!!!!!!!, aaaaa aaaaa)

Behavior deliberately loosened: a 5+ repeated-char run inside a longer reply (code indentation, ----- dividers, ..... ellipses, "soooo") no longer marks the whole reply unusable — that was the bug.

Tests

New regression test delivers bare-code and structured Stage 1 replies verbatim drives the full runV5MessageRuntimeStage1 path with: (a) a gemma-style bare nested-indent python body, (b) a fenced code block, (c) prose + fenced code, (d) pretty-printed JSON, (e) a markdown divider reply — each ships verbatim with exactly one model call. Extended the known-junk test with " " (whitespace-only), "!!!!!!!!", and "aaaaa aaaaa".

bunx vitest run src/__tests__/message-runtime-stage1.test.ts
 Test Files  1 passed (1)
      Tests  74 passed (74)

bunx vitest run src/__tests__/message-action-dedupe.test.ts \
  src/__tests__/message-benchmark-integration.contract.test.ts \
  src/__tests__/message-routing-live-regression.test.ts \
  src/__tests__/messages-format.test.ts \
  src/__tests__/message-stable-prefix.test.ts \
  src/__tests__/message-stage1-context-catalog.test.ts
 Test Files  6 passed (6)
      Tests  54 passed (54)

bunx turbo typecheck lint "--filter=@elizaos/core"
 Tasks:    61 successful, 61 total

Refs #11504

🤖 Generated with Claude Code

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@coderabbitai

coderabbitai Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: c7fbad06-f041-4923-adf8-0df44d103a15

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/11504-stage1-bare-code-reply

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…es (#11504)

The Stage-1 junk heuristic flagged any character repeated 5+ times
anywhere in the reply (/(.)\1{4,}/u), so the 8+ consecutive spaces of
two-level code indentation — a gemma-4-31b HumanEval-style bare python
body — plus markdown "-----" dividers and pretty-printed JSON all got
replaced with "I'm not sure how to answer that." The starts-with-fence
escape only rescued replies that BEGIN with a code fence, so bare code
and prose-then-fenced replies both died. eliza-harness HumanEval scored
0.40 vs 1.00 for the same model on raw harnesses.

Tighten the degenerate-run rule to fire only when the WHOLE reply is a
single repeated code point (non-whitespace content, length >= 5, one
unique char). Empty replies, JSON-scaffold fragments, digit-only
replies, enum echoes ("RPPY"), and single-char spam ("aaaaa",
"!!!!!!!!", "aaaaa aaaaa") all stay unusable.

Regression tests: bare nested-indent code body, fenced code, prose +
fenced code, pretty-printed JSON, and markdown-divider replies ship
verbatim; extended the known-junk test with whitespace-only and
degenerate-spam fragments.

Refs #11504

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@lalalune
lalalune force-pushed the fix/11504-stage1-bare-code-reply branch from 731b70f to 66f3aa1 Compare July 2, 2026 20:18

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@lalalune
lalalune merged commit c532e3b into develop Jul 2, 2026
32 of 54 checks passed
@lalalune
lalalune deleted the fix/11504-stage1-bare-code-reply branch July 2, 2026 20:18
lalalune added a commit that referenced this pull request Jul 2, 2026
…te (#11579)

Port the six finer-grained regression tests from the closed PR #11553
atop the merged #11555 fix: bare unfenced code-body verbatim, fenced
code with a prose lead-in, pretty-printed JSON, prose containing
separator/emphasis runs, still-defers
empty/whitespace/refusal-stub/degenerate-run, and
refusal-that-continues-into-content + bare social apology.

Running the ported suite against the merged implementation exposed two
real coverage gaps in isUnusableStage1Reply:

- Bare refusal stubs ("I am not sure.", "I don't know.", "I'm sorry,
  I can't help with that.") were delivered verbatim instead of
  deferring — #11504's ask was to tighten the unusable signal to
  actual refusals/empties. Port #11553's whole-reply-anchored
  isBareRefusalStage1Reply (apology-prefix stripped;
  refusal-plus-content and bare social apologies are kept).
- Multi-token degenerate spam ("aaaaa bbbbb") slipped the merged
  whole-reply single-code-point check. Add #11553's per-token check
  (every whitespace-separated token is one character repeated 5+
  times) alongside the existing set-based check, so nothing the
  merged heuristic already caught is weakened.

Refs #11504

Co-authored-by: lalalune <shaw.nicola.walters@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
@github-actions github-actions Bot added the Tests label Jul 3, 2026
@claude

claude Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Claude encountered an error —— View job


I'll analyze this and get back to you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant