Skip to content

fix: report spent limits as ACP stop reasons - #246

Merged
hallerite merged 2 commits into
mainfrom
fix/acp-stop-reasons
Nov 1, 2026
Merged

hallerite merged 2 commits into
mainfrom
fix/acp-stop-reasons

Conversation

@hallerite

@hallerite hallerite commented Nov 1, 2026 •

Copy link
Copy Markdown
Member

Why

A prompt that a limit cut short ended with ACP stop reason end_turn unless the limit was the agent's own max_tokens. A client that reads only the ACP stop reason took a run stopped by max_total_turns, max_total_tokens or a failed compaction for one the agent finished. ACP has stop reasons for exactly this: max_turn_requests and max_tokens.

What

last_stop_reason ACP stop reason before after
done end_turn end_turn
max_total_turns end_turn max_turn_requests
max_total_tokens end_turn max_tokens
token_budget (the policy's max_tokens) max_tokens max_tokens
compaction_failed end_turn max_tokens

The snapshot's last_stop_reason still names the exact cause. depth_limit stays end_turn: it only stops a child started past max_depth, never an ACP session's root.

What verifiers sees

Verifiers reads both the ACP stop reason and the snapshot, but for rlm the snapshot wins, so its stop conditions do not change. Checked against verifiers main at eaab317a8. Each rlm turn goes through three steps, in order (verifiers/v1/acp/__init__.py):

  1. It stores the ACP stop reason unchanged in trace.info["acp_stop_reason"].

  2. The rlm harness reads the snapshot's last_stop_reason and stops the trace (verifiers/v1/harnesses/rlm/harness.py, _TRUNCATION_STOPS):

    last_stop_reason verifiers stop condition
    max_total_turns max_turns
    max_total_tokens max_total_tokens
    token_budget max_output_tokens
    compaction_failed compaction_failed
  3. The generic ACP mapping (_STOP_CONDITIONS) turns max_tokens into max_output_tokens and max_turn_requests into max_turns.

Trace.stop keeps the first condition it is given, so step 2 always beats step 3. With this change the two agree anyway; only the snapshot can tell max_total_tokens and compaction_failed apart.

The only change verifiers will see: for an rlm rollout cut short by a limit, trace.info["acp_stop_reason"] changes from end_turn to max_turn_requests or max_tokens. Nothing in verifiers or prime-rl reads that field; it is only stored.

The change matters for ACP clients that do not read the rlm snapshot: editors such as Zed, other generic ACP hosts, and the end-to-end suite in #244. They used to be told end_turn for a run that a limit cut short.

Tests and docs

Test. test_acp_stop_reason_names_the_spent_limit runs the real engine behind RLMACPAgent with scripted replies, one case per limit plus a normal answer, and checks both the ACP stop reason and the snapshot. The turn and tree-token cases fail without the fix. A failed compaction is not triggered here; it shares the table. The unit suite passes (400).

Docs. The README states the mapping. #245 moves the README into docs/, and #244's end-to-end budget test checks only the snapshot; whichever lands after this updates docs/runtime-contract.md and can assert the ACP stop reason too.

🤖 Generated with Claude Code


Note

Low Risk
Behavior change is limited to ACP PromptResponse.stop_reason for limit cases; snapshot metadata and non-limit paths are unchanged.

Overview
ACP prompt responses now surface limit-driven stops instead of reporting end_turn whenever the engine stopped for tree turn/token caps or compaction failure.

RLMACPAgent maps engine stop_reason values through _ACP_STOP_REASONS: max_total_turns → max_turn_requests, and max_total_tokens, token_budget, and compaction_failed → max_tokens. Normal completions still return end_turn; the session snapshot’s last_stop_reason continues to carry the precise engine cause.

The README documents this mapping. test_acp_stop_reason_names_the_spent_limit exercises the real engine via ACP for normal completion and each limit policy, asserting both the ACP stop_reason and snapshot last_stop_reason.

Reviewed by Cursor Bugbot for commit 3b6f844. Bugbot is set up for automated code reviews on this repo. Configure here.

hallerite and others added 2 commits November 1, 2026 19:17
A prompt stopped by max_total_turns, max_total_tokens or a failed
compaction ended with end_turn, so a client that reads only the ACP stop
reason took a truncated run for a finished one. The turn cap now ends
with max_turn_requests and the token limits with max_tokens; the
snapshot's last_stop_reason still names the exact cause.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hallerite
hallerite marked this pull request as ready for review November 1, 2026 19:20
@hallerite
hallerite merged commit b7fcc06 into main Nov 1, 2026
9 checks passed
@hallerite
hallerite deleted the fix/acp-stop-reasons branch November 1, 2026 22:55
hallerite added a commit that referenced this pull request Nov 1, 2026
## Why

The README had grown to 600 lines in the order features were added:
shell jobs, crash recovery and subscriptions sat under "Testing", agent
cleanup notes after the test command, and `rlm.services`, `rlm.hints`
and the policy fields were not documented at all. Someone new could not
find how the pieces fit, and someone looking for one fact had to read
everything.

## What

The README becomes a landing page, and the content moves into a guide
under `docs/`, where each page opens with what its subject is for.

<picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://raw.githubusercontent.com/PrimeIntellect-ai/nano-rlm/477080dca24ad094c5543bf2dbc1766fd9f00a19/docs/images/overview-dark.svg">
<img alt="An ACP client sends the runtime contract and prompts to rlm.
The engine runs the agent loop against the model API and appends to the
session ledger; its tool calls run in a persistent IPython kernel, whose
rlm.* calls reach the supervisor; the supervisor owns child agents, each
with its own engine, kernel and session directory."
src="https://raw.githubusercontent.com/PrimeIntellect-ai/nano-rlm/477080dca24ad094c5543bf2dbc1766fd9f00a19/docs/images/overview.svg">
</picture>

- **README:** what nano-rlm is, the overview figure above, install, a
quickstart ACP client, and an index of the pages. I ran the quickstart
against a hosted model with only the endpoint swapped.
- **The pages:**

  | Page | Covers |
  |---|---|
|
[Architecture](https://github.com/PrimeIntellect-ai/nano-rlm/blob/477080dca24ad094c5543bf2dbc1766fd9f00a19/docs/architecture.md)
| the components, one turn through them, and trust boundaries |
| [Runtime
contract](https://github.com/PrimeIntellect-ai/nano-rlm/blob/477080dca24ad094c5543bf2dbc1766fd9f00a19/docs/runtime-contract.md)
| every runtime field and policy, prompts, steering, snapshots and stop
reasons, tracing headers |
| [Python
API](https://github.com/PrimeIntellect-ai/nano-rlm/blob/477080dca24ad094c5543bf2dbc1766fd9f00a19/docs/python-api.md)
| agents, inbox and `wait`, shell jobs, services, watches, history,
hints |
| [Tools and
skills](https://github.com/PrimeIntellect-ai/nano-rlm/blob/477080dca24ad094c5543bf2dbc1766fd9f00a19/docs/tools.md)
| native and built-in Python tools, tool packages, `SKILL.md`, MCP, the
kernel's environment |
| [Context and
history](https://github.com/PrimeIntellect-ai/nano-rlm/blob/477080dca24ad094c5543bf2dbc1766fd9f00a19/docs/context.md)
| compaction, tool-output truncation, the ledger |
| [Kernel and
processes](https://github.com/PrimeIntellect-ai/nano-rlm/blob/477080dca24ad094c5543bf2dbc1766fd9f00a19/docs/processes.md)
| timeouts, crash recovery, and which processes outlive a call, a
restart or the session |
| [Split
execution](https://github.com/PrimeIntellect-ai/nano-rlm/blob/477080dca24ad094c5543bf2dbc1766fd9f00a19/docs/split-execution.md)
| the workspace worker over stdio or TCP, its cost and responsibilities
|

- **Checked against the code.** Every fact was checked against the code,
which corrected a few the README had wrong:
- a compaction gives up after `max_compaction_attempts` (5) attempts,
not three;
  - `max_tokens` is a per-agent completion budget;
  - the native `wait` tool is offered whenever `ipython` is;
  - the built-in Python tools include `bash`.
- **AGENTS.md** asks a change to user-facing behavior to update its page
in the same PR.

Links, anchors and images were checked across all pages, and the pages
render with GitHub's markdown renderer, figures in light and dark.

#246 added a README paragraph on ACP stop reasons; it is now a table in
the runtime contract, keyed by the snapshot's `last_stop_reason`.

#244 also edits the README's testing section. Whichever merges second
moves the end-to-end notes to the new layout.
rasdani added a commit to PrimeIntellect-ai/verifiers that referenced this pull request Nov 2, 2026
Update the RLM harness's pinned nano-rlm commit from `15513ae8` to
`48da1d9c43bdcf2ef20b7cff9e4c6df800810b30`.

`15513ae8` was the head of
PrimeIntellect-ai/nano-rlm#240 when
#3011 pinned it. #240
has since been squash-merged as `48da1d9c`, so `15513ae8` is on no
branch and will be garbage collected. `48da1d9c` has the same tree as
#240's final head.

[Upstream
changes](PrimeIntellect-ai/nano-rlm@15513ae...48da1d9)
also bring in nano-rlm main since #240's original base:
- PrimeIntellect-ai/nano-rlm#218: separates
persistent tool execution from the agent loop. Local mode is affected
too: bash background jobs outlive the kernel, and output keeps the first
and last 512 KiB.
- PrimeIntellect-ai/nano-rlm#246: spent limits
are reported as ACP stop reasons.
- PrimeIntellect-ai/nano-rlm#244 and
PrimeIntellect-ai/nano-rlm#245: tests and docs.

`tests/unit/harnesses/test_rlm.py` passes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Default dependency bump changes runtime agent/tool behavior for
harnesses using the pinned ref, though unit tests for the harness still
pass.
> 
> **Overview**
> Updates the **RLM harness** default `version` git ref for **nano-rlm**
from `15513ae8…` to **`48da1d9c…`**, so installs track the squash-merged
commit from upstream PR #240 instead of a ref that is no longer on any
branch.
> 
> Rollouts that rely on the default pin will pull the newer nano-rlm
tree, which (per upstream) includes **persistent tool execution outside
the agent loop** (e.g. bash background jobs surviving the kernel with
head/tail output truncation), **ACP stop reasons when spend limits
hit**, and related test/doc updates. Custom `version` overrides are
unchanged.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
cea2226. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

<!-- Macroscope's pull request summary starts here -->
<!-- Macroscope will only edit the content between these invisible
markers, and the markers themselves will not be visible in the GitHub
rendered markdown. -->
<!-- If you delete either of the start / end markers from your PR's
description, Macroscope will append its summary at the bottom of the
description. -->
> [!NOTE]
> ### Bump default nano-rlm ref in `RLMHarnessConfig` to `48da1d9c`
> Updates the default `version` for nano-rlm from `15513ae8` to
`48da1d9c` in
[harness.py](https://github.com/PrimeIntellect-ai/verifiers/pull/3044/files#diff-3955da24b5fd88f62482297bbda7ba0936f633927f15576db27fed6e58f668ec).
Applies only when no version is explicitly configured. Behavioral
Change: runs without an explicit version now use nano-rlm at ref
`48da1d9c43bdcf2ef20b7cff9e4c6df800810b30`.
>
> <!-- Macroscope's review summary starts here -->
>
> <sup><a href="https://app.macroscope.com">Macroscope</a> summarized
cea2226.</sup>
> <!-- Macroscope's review summary ends here -->
>
<!-- Macroscope's pull request summary ends here -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant