Repository navigation
fix: report spent limits as ACP stop reasons - #246
Merged
Merged
Conversation
A prompt stopped by max_total_turns, max_total_tokens or a failed compaction ended with end_turn, so a client that reads only the ACP stop reason took a truncated run for a finished one. The turn cap now ends with max_turn_requests and the token limits with max_tokens; the snapshot's last_stop_reason still names the exact cause. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hallerite
marked this pull request as ready for review
November 1, 2026 19:20
hallerite
added a commit
that referenced
this pull request
Nov 1, 2026
## Why The README had grown to 600 lines in the order features were added: shell jobs, crash recovery and subscriptions sat under "Testing", agent cleanup notes after the test command, and `rlm.services`, `rlm.hints` and the policy fields were not documented at all. Someone new could not find how the pieces fit, and someone looking for one fact had to read everything. ## What The README becomes a landing page, and the content moves into a guide under `docs/`, where each page opens with what its subject is for. <picture> <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/PrimeIntellect-ai/nano-rlm/477080dca24ad094c5543bf2dbc1766fd9f00a19/docs/images/overview-dark.svg"> <img alt="An ACP client sends the runtime contract and prompts to rlm. The engine runs the agent loop against the model API and appends to the session ledger; its tool calls run in a persistent IPython kernel, whose rlm.* calls reach the supervisor; the supervisor owns child agents, each with its own engine, kernel and session directory." src="https://raw.githubusercontent.com/PrimeIntellect-ai/nano-rlm/477080dca24ad094c5543bf2dbc1766fd9f00a19/docs/images/overview.svg"> </picture> - **README:** what nano-rlm is, the overview figure above, install, a quickstart ACP client, and an index of the pages. I ran the quickstart against a hosted model with only the endpoint swapped. - **The pages:** | Page | Covers | |---|---| | [Architecture](https://github.com/PrimeIntellect-ai/nano-rlm/blob/477080dca24ad094c5543bf2dbc1766fd9f00a19/docs/architecture.md) | the components, one turn through them, and trust boundaries | | [Runtime contract](https://github.com/PrimeIntellect-ai/nano-rlm/blob/477080dca24ad094c5543bf2dbc1766fd9f00a19/docs/runtime-contract.md) | every runtime field and policy, prompts, steering, snapshots and stop reasons, tracing headers | | [Python API](https://github.com/PrimeIntellect-ai/nano-rlm/blob/477080dca24ad094c5543bf2dbc1766fd9f00a19/docs/python-api.md) | agents, inbox and `wait`, shell jobs, services, watches, history, hints | | [Tools and skills](https://github.com/PrimeIntellect-ai/nano-rlm/blob/477080dca24ad094c5543bf2dbc1766fd9f00a19/docs/tools.md) | native and built-in Python tools, tool packages, `SKILL.md`, MCP, the kernel's environment | | [Context and history](https://github.com/PrimeIntellect-ai/nano-rlm/blob/477080dca24ad094c5543bf2dbc1766fd9f00a19/docs/context.md) | compaction, tool-output truncation, the ledger | | [Kernel and processes](https://github.com/PrimeIntellect-ai/nano-rlm/blob/477080dca24ad094c5543bf2dbc1766fd9f00a19/docs/processes.md) | timeouts, crash recovery, and which processes outlive a call, a restart or the session | | [Split execution](https://github.com/PrimeIntellect-ai/nano-rlm/blob/477080dca24ad094c5543bf2dbc1766fd9f00a19/docs/split-execution.md) | the workspace worker over stdio or TCP, its cost and responsibilities | - **Checked against the code.** Every fact was checked against the code, which corrected a few the README had wrong: - a compaction gives up after `max_compaction_attempts` (5) attempts, not three; - `max_tokens` is a per-agent completion budget; - the native `wait` tool is offered whenever `ipython` is; - the built-in Python tools include `bash`. - **AGENTS.md** asks a change to user-facing behavior to update its page in the same PR. Links, anchors and images were checked across all pages, and the pages render with GitHub's markdown renderer, figures in light and dark. #246 added a README paragraph on ACP stop reasons; it is now a table in the runtime contract, keyed by the snapshot's `last_stop_reason`. #244 also edits the README's testing section. Whichever merges second moves the end-to-end notes to the new layout.
rasdani
added a commit
to PrimeIntellect-ai/verifiers
that referenced
this pull request
Nov 2, 2026
Update the RLM harness's pinned nano-rlm commit from `15513ae8` to `48da1d9c43bdcf2ef20b7cff9e4c6df800810b30`. `15513ae8` was the head of PrimeIntellect-ai/nano-rlm#240 when #3011 pinned it. #240 has since been squash-merged as `48da1d9c`, so `15513ae8` is on no branch and will be garbage collected. `48da1d9c` has the same tree as #240's final head. [Upstream changes](PrimeIntellect-ai/nano-rlm@15513ae...48da1d9) also bring in nano-rlm main since #240's original base: - PrimeIntellect-ai/nano-rlm#218: separates persistent tool execution from the agent loop. Local mode is affected too: bash background jobs outlive the kernel, and output keeps the first and last 512 KiB. - PrimeIntellect-ai/nano-rlm#246: spent limits are reported as ACP stop reasons. - PrimeIntellect-ai/nano-rlm#244 and PrimeIntellect-ai/nano-rlm#245: tests and docs. `tests/unit/harnesses/test_rlm.py` passes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Default dependency bump changes runtime agent/tool behavior for harnesses using the pinned ref, though unit tests for the harness still pass. > > **Overview** > Updates the **RLM harness** default `version` git ref for **nano-rlm** from `15513ae8…` to **`48da1d9c…`**, so installs track the squash-merged commit from upstream PR #240 instead of a ref that is no longer on any branch. > > Rollouts that rely on the default pin will pull the newer nano-rlm tree, which (per upstream) includes **persistent tool execution outside the agent loop** (e.g. bash background jobs surviving the kernel with head/tail output truncation), **ACP stop reasons when spend limits hit**, and related test/doc updates. Custom `version` overrides are unchanged. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit cea2226. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> <!-- Macroscope's pull request summary starts here --> <!-- Macroscope will only edit the content between these invisible markers, and the markers themselves will not be visible in the GitHub rendered markdown. --> <!-- If you delete either of the start / end markers from your PR's description, Macroscope will append its summary at the bottom of the description. --> > [!NOTE] > ### Bump default nano-rlm ref in `RLMHarnessConfig` to `48da1d9c` > Updates the default `version` for nano-rlm from `15513ae8` to `48da1d9c` in [harness.py](https://github.com/PrimeIntellect-ai/verifiers/pull/3044/files#diff-3955da24b5fd88f62482297bbda7ba0936f633927f15576db27fed6e58f668ec). Applies only when no version is explicitly configured. Behavioral Change: runs without an explicit version now use nano-rlm at ref `48da1d9c43bdcf2ef20b7cff9e4c6df800810b30`. > > <!-- Macroscope's review summary starts here --> > > <sup><a href="https://app.macroscope.com">Macroscope</a> summarized cea2226.</sup> > <!-- Macroscope's review summary ends here --> > <!-- Macroscope's pull request summary ends here --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A prompt that a limit cut short ended with ACP stop reason
end_turnunless the limit was the agent's ownmax_tokens. A client that reads only the ACP stop reason took a run stopped bymax_total_turns,max_total_tokensor a failed compaction for one the agent finished. ACP has stop reasons for exactly this:max_turn_requestsandmax_tokens.What
last_stop_reasondoneend_turnend_turnmax_total_turnsend_turnmax_turn_requestsmax_total_tokensend_turnmax_tokenstoken_budget(the policy'smax_tokens)max_tokensmax_tokenscompaction_failedend_turnmax_tokensThe snapshot's
last_stop_reasonstill names the exact cause.depth_limitstaysend_turn: it only stops a child started pastmax_depth, never an ACP session's root.What verifiers sees
Verifiers reads both the ACP stop reason and the snapshot, but for rlm the snapshot wins, so its stop conditions do not change. Checked against verifiers main at eaab317a8. Each rlm turn goes through three steps, in order (
verifiers/v1/acp/__init__.py):It stores the ACP stop reason unchanged in
trace.info["acp_stop_reason"].The rlm harness reads the snapshot's
last_stop_reasonand stops the trace (verifiers/v1/harnesses/rlm/harness.py,_TRUNCATION_STOPS):last_stop_reasonmax_total_turnsmax_turnsmax_total_tokensmax_total_tokenstoken_budgetmax_output_tokenscompaction_failedcompaction_failedThe generic ACP mapping (
_STOP_CONDITIONS) turnsmax_tokensintomax_output_tokensandmax_turn_requestsintomax_turns.Trace.stopkeeps the first condition it is given, so step 2 always beats step 3. With this change the two agree anyway; only the snapshot can tellmax_total_tokensandcompaction_failedapart.The only change verifiers will see: for an rlm rollout cut short by a limit,
trace.info["acp_stop_reason"]changes fromend_turntomax_turn_requestsormax_tokens. Nothing in verifiers or prime-rl reads that field; it is only stored.The change matters for ACP clients that do not read the rlm snapshot: editors such as Zed, other generic ACP hosts, and the end-to-end suite in #244. They used to be told
end_turnfor a run that a limit cut short.Tests and docs
Test.
test_acp_stop_reason_names_the_spent_limitruns the real engine behindRLMACPAgentwith scripted replies, one case per limit plus a normal answer, and checks both the ACP stop reason and the snapshot. The turn and tree-token cases fail without the fix. A failed compaction is not triggered here; it shares the table. The unit suite passes (400).Docs. The README states the mapping. #245 moves the README into
docs/, and #244's end-to-end budget test checks only the snapshot; whichever lands after this updatesdocs/runtime-contract.mdand can assert the ACP stop reason too.🤖 Generated with Claude Code
Note
Low Risk
Behavior change is limited to ACP
PromptResponse.stop_reasonfor limit cases; snapshot metadata and non-limit paths are unchanged.Overview
ACP prompt responses now surface limit-driven stops instead of reporting
end_turnwhenever the engine stopped for tree turn/token caps or compaction failure.RLMACPAgentmaps enginestop_reasonvalues through_ACP_STOP_REASONS:max_total_turns→max_turn_requests, andmax_total_tokens,token_budget, andcompaction_failed→max_tokens. Normal completions still returnend_turn; the session snapshot’slast_stop_reasoncontinues to carry the precise engine cause.The README documents this mapping.
test_acp_stop_reason_names_the_spent_limitexercises the real engine via ACP for normal completion and each limit policy, asserting both the ACPstop_reasonand snapshotlast_stop_reason.Reviewed by Cursor Bugbot for commit 3b6f844. Bugbot is set up for automated code reviews on this repo. Configure here.