Skip to content

feat(agent): false-stop detection — nudge past premature stop after tool calls - #78113

Open
yingliang-zhang wants to merge 1 commit into
NousResearch:mainfrom
yingliang-zhang:feat/false-stop-detection
Open

feat(agent): false-stop detection — nudge past premature stop after tool calls#78113
yingliang-zhang wants to merge 1 commit into
NousResearch:mainfrom
yingliang-zhang:feat/false-stop-detection

Conversation

@yingliang-zhang

Copy link
Copy Markdown
Contributor

Problem

When an LLM produces finish_reason=stop with text indicating intent to continue (e.g., a colon-preamble that lost its tool_calls, or a narrated continuation), the conversation silently ends without executing the intended tool call (#42503).

This is observed across model families — K3, GLM, DeepSeek, Qwen, and others. The model writes a preamble like 接下来读取文件: or 继续把剩余页面读完。 with finish_reason=stop and no tool_calls, right after a tool round completed.

Fix

Add a bounded false-stop detection mechanism that nudges the model to issue the actual tool call:

  • Signature A — lost tool-call preamble: short text (<120 chars) ending with a colon (: or ), produced after a tool round. Likely the start of a tool-call narration that lost its tool_calls.
  • Signature B — narrated continuation: text (≤200 chars) ending with CJK sentence punctuation, containing at least one intent marker, and NOT containing done/wait/question markers, produced after a tool round.

Gated by agent.false_stop_detection: "auto" (on for interactive surfaces — CLI, TUI, desktop, codex, local — off for messaging platforms). 2-attempt budget, resets on tool round and genuine completion. Synthetic nudge is stripped from durable transcript and trajectory via _false_stop_synthetic ephemeral flag.

Complementary to existing features

  • intent_ack_continuation (already upstream): fires at turn START (no prior tool calls) when model "announces an action but calls no tool." False-stop detection is the POST-tool-round version — it fires after a tool round completed (the _was_in_tool_round gate checks last 8 messages for assistant with tool_calls).
  • _dropped_toolcall_nudge (already upstream): handles finish_reason=tool_calls with empty tool_calls array. Signature A handles finish_reason=stop with a text preamble — adjacent but distinct.
  • fix(agent): nudge past progress-placeholder responses after tool call… #57610 (OPEN, "nudge past progress-placeholder responses after tool calls"): addresses a similar problem space. This PR's Signature A spec + CJK corpus was offered as a complementary contribution in a comment on fix(agent): nudge past progress-placeholder responses after tool call… #57610.

Implementation

Extracted to a dedicated agent/false_stop.py module following the agent/verification_stop.py (273 lines) and agent/kanban_stop.py (108 lines) pattern:

  • build_false_stop_nudge(content, messages, attempts) — pure function returning nudge text or None
  • false_stop_detection_enabled(config) — config + env gate, mirrors verify_on_stop_enabled()
  • Module-level constants for marker tuples (previously rebuilt per iteration)

Marker lists are production-hardened from real deployments: 24 CJK + 13 English intent markers, with done/wait/question exclusion guards. Signature B's ending set is CJK-only (。!?;…) to avoid over-triggering on English completions ending in ..

Test plan

  • scripts/run_tests.sh tests/agent/test_false_stop.py — all pass (35 tests across 5 categories: gating, Signature A, Signature B, budget/reset, transcript hygiene)
  • No regression on tests/agent/test_verification_stop.py, tests/agent/test_verification_stop_caching.py, tests/run_agent/test_verification_continuation_budget.py
  • py_compile on all changed files
  • No circular imports

@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard labels Aug 4, 2026
@yingliang-zhang
yingliang-zhang force-pushed the feat/false-stop-detection branch from 5fe882f to 2ad5f50 Compare August 12, 2026 08:53
…ool calls

When an LLM produces finish_reason=stop with text indicating intent to
continue (a colon-preamble that lost its tool_calls, or a narrated
continuation) right after a tool round, the conversation silently ends
without executing the intended tool call (NousResearch#42503).

Extract the inlined detection logic into a dedicated agent/false_stop.py
module following the verification_stop.py / kanban_stop.py pattern:

- build_false_stop_nudge(): pure function returning nudge text or None
- false_stop_detection_enabled(): config + env gate, mirrors
  verify_on_stop_enabled() with surface-aware "auto" default
- Module-level marker constants (24 CJK + 13 English intent markers,
  done/wait/question exclusion guards) — previously rebuilt per iteration

Two detection signatures:
- Signature A: short colon-preamble (<120 chars) after a tool round
- Signature B: narrated continuation (≤200 chars, CJK ending, intent
  markers, no done/wait/question markers) after a tool round

Both gated by _was_in_tool_round (last 8 messages). Bounded to 2
nudges per turn, resets on tool round and genuine completion. Synthetic
nudge flagged _false_stop_synthetic in _EPHEMERAL_SCAFFOLDING_FLAGS
for transcript hygiene.

Complementary to intent_ack_continuation (turn-start, not post-tool)
and NousResearch#57610 (progress-placeholder text).

Config: agent.false_stop_detection: "auto" (on for CLI/TUI/desktop/
codex/local, off for messaging platforms). Env: HERMES_FALSE_STOP_DETECTION.

Tests: 39 tests in 5 categories (gating, Signature A, Signature B,
budget/reset, transcript hygiene). No regressions on verification_stop,
verification_stop_caching, or verification_continuation_budget.
@yingliang-zhang
yingliang-zhang force-pushed the feat/false-stop-detection branch from 2ad5f50 to bee0183 Compare August 16, 2026 01:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard P3 Low — cosmetic, nice to have type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants