From 7e5c41efb78ee9f5e4f114ddd5eaea8f7a5902e0 Mon Sep 17 00:00:00 2001 From: Konstantin Dunas Date: Mon, 2 Nov 2026 00:20:58 +0100 Subject: [PATCH 1/3] docs: a short README and a docs/ guide (#245) ## Why The README had grown to 600 lines in the order features were added: shell jobs, crash recovery and subscriptions sat under "Testing", agent cleanup notes after the test command, and `rlm.services`, `rlm.hints` and the policy fields were not documented at all. Someone new could not find how the pieces fit, and someone looking for one fact had to read everything. ## What The README becomes a landing page, and the content moves into a guide under `docs/`, where each page opens with what its subject is for. An ACP client sends the runtime contract and prompts to rlm. The engine runs the agent loop against the model API and appends to the session ledger; its tool calls run in a persistent IPython kernel, whose rlm.* calls reach the supervisor; the supervisor owns child agents, each with its own engine, kernel and session directory. - **README:** what nano-rlm is, the overview figure above, install, a quickstart ACP client, and an index of the pages. I ran the quickstart against a hosted model with only the endpoint swapped. - **The pages:** | Page | Covers | |---|---| | [Architecture](https://github.com/PrimeIntellect-ai/nano-rlm/blob/477080dca24ad094c5543bf2dbc1766fd9f00a19/docs/architecture.md) | the components, one turn through them, and trust boundaries | | [Runtime contract](https://github.com/PrimeIntellect-ai/nano-rlm/blob/477080dca24ad094c5543bf2dbc1766fd9f00a19/docs/runtime-contract.md) | every runtime field and policy, prompts, steering, snapshots and stop reasons, tracing headers | | [Python API](https://github.com/PrimeIntellect-ai/nano-rlm/blob/477080dca24ad094c5543bf2dbc1766fd9f00a19/docs/python-api.md) | agents, inbox and `wait`, shell jobs, services, watches, history, hints | | [Tools and skills](https://github.com/PrimeIntellect-ai/nano-rlm/blob/477080dca24ad094c5543bf2dbc1766fd9f00a19/docs/tools.md) | native and built-in Python tools, tool packages, `SKILL.md`, MCP, the kernel's environment | | [Context and history](https://github.com/PrimeIntellect-ai/nano-rlm/blob/477080dca24ad094c5543bf2dbc1766fd9f00a19/docs/context.md) | compaction, tool-output truncation, the ledger | | [Kernel and processes](https://github.com/PrimeIntellect-ai/nano-rlm/blob/477080dca24ad094c5543bf2dbc1766fd9f00a19/docs/processes.md) | timeouts, crash recovery, and which processes outlive a call, a restart or the session | | [Split execution](https://github.com/PrimeIntellect-ai/nano-rlm/blob/477080dca24ad094c5543bf2dbc1766fd9f00a19/docs/split-execution.md) | the workspace worker over stdio or TCP, its cost and responsibilities | - **Checked against the code.** Every fact was checked against the code, which corrected a few the README had wrong: - a compaction gives up after `max_compaction_attempts` (5) attempts, not three; - `max_tokens` is a per-agent completion budget; - the native `wait` tool is offered whenever `ipython` is; - the built-in Python tools include `bash`. - **AGENTS.md** asks a change to user-facing behavior to update its page in the same PR. Links, anchors and images were checked across all pages, and the pages render with GitHub's markdown renderer, figures in light and dark. #246 added a README paragraph on ACP stop reasons; it is now a table in the runtime contract, keyed by the snapshot's `last_stop_reason`. #244 also edits the README's testing section. Whichever merges second moves the end-to-end notes to the new layout. --- AGENTS.md | 4 + README.md | 652 ++++------------------------------ docs/architecture.md | 38 ++ docs/context.md | 64 ++++ docs/images/overview-dark.svg | 1 + docs/images/overview.svg | 1 + docs/processes.md | 40 +++ docs/python-api.md | 168 +++++++++ docs/runtime-contract.md | 103 ++++++ docs/split-execution.md | 66 ++++ docs/tools.md | 116 ++++++ 11 files changed, 666 insertions(+), 587 deletions(-) create mode 100644 docs/architecture.md create mode 100644 docs/context.md create mode 100644 docs/images/overview-dark.svg create mode 100644 docs/images/overview.svg create mode 100644 docs/processes.md create mode 100644 docs/python-api.md create mode 100644 docs/runtime-contract.md create mode 100644 docs/split-execution.md create mode 100644 docs/tools.md diff --git a/AGENTS.md b/AGENTS.md index 8b51b5eb..822e867a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -6,6 +6,10 @@ - **Targeted comments**: don't explain your work process or reference old code. Use comments sparingly to clarify ambiguous logic or non-obvious constraints. - **No "this replaced that" comments**: never write comments or docstrings that reference code that used to exist. That context belongs in the PR, not in the file. Describe what the code does now, not what it isn't anymore. +## Docs + +- **Keep docs current**: user-facing behavior is documented in `docs/` (indexed from the README). A change to that behavior updates the page that describes it in the same PR. + ## Running code - **Always use uv**: run code with `uv run`, never raw `python`. diff --git a/README.md b/README.md index 400bd722..a9c4fc1d 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,20 @@ # nano-rlm -A minimal CLI coding agent with a persistent IPython execution environment and optional recursive sub-agents. For a full-fledged coding agent built on the same RLM principles, see [prime-agent](https://github.com/PrimeIntellect-ai/prime-agent). +A minimal recursive language-model agent. The model works through one persistent IPython kernel: Python, shell commands, file edits and orchestration all happen there, and the kernel's state survives across turns and context compaction. From that kernel the agent can spawn sub-agents, each with its own kernel, and coordinate them like any other Python object. -By default the model gets a single built-in tool, `ipython`: a persistent IPython kernel for Python, and Bash commands and background jobs via `rlm.shell.run`. File edits, shell work, and orchestration all go through it. The runtime contract's `builtin_tools` list can select a different tool set (`bash`, `edit`, `fetch`, `ipython`) for native tool-calling runs. +nano-rlm runs as an [Agent Client Protocol](https://agentclientprotocol.com/) agent and is built as a training harness: every session is configured by one explicit runtime contract, and every model call, message and tool result is recorded. For a full-fledged coding agent built on the same principles, see [prime-agent](https://github.com/PrimeIntellect-ai/prime-agent). -For convenience, rlm ships built-in Python tools that can be enabled per session via the runtime contract's `skills` list (off by default): `edit` (single-occurrence string replacement), `search` (web search via Serper, needs `SERPER_API_KEY`), and `fetch` (retrieve a URL as cleaned text). Enabled tools are pre-imported into the IPython kernel (see [Tools](#tools)), so the agent calls `await edit(path=..., old_str=..., new_str=...)`, `await search(query=...)`, or `await fetch(url=...)`. `fetch` also exists as a native builtin tool with the same semantics, for tool-calling runs (opt-in via the contract's `builtin_tools`). [Skills](#skills) are `SKILL.md` instructions that teach the agent workflows using these tools. - -Context compaction is on by default: the engine compacts when 16k tokens remain below an advertised model context window. Termination comes from the default tree-wide budget of 1M new tokens (`max_total_tokens`). The policy can set an explicit `summarize_at_tokens` threshold. The IPython kernel keeps running across compaction, so REPL state survives (see [Compaction](#compaction)). + + + An ACP client sends the runtime contract and prompts to rlm. The engine runs the agent loop against the model API and appends to the session ledger; its tool calls run in a persistent IPython kernel, whose rlm.* calls reach the supervisor; the supervisor owns child agents, each with its own engine, kernel and session directory. + -Inside IPython, the `rlm` package is available in the namespace. When recursion is allowed, `await rlm.agent.spawn(task=...)` returns a handle to a supervisor-owned agent that continues across cells. Python tool packages supplied by the host environment (see [Tools](#tools)) are importable directly by name, e.g. `import websearch`. +- **One persistent kernel.** By default the model has a single tool, `ipython`. Shell work goes through `rlm.shell.run` or `!cmd`, edits through the pre-imported `edit`, and variables, imports and loaded data persist from cell to cell. +- **Recursion from Python.** `await rlm.agent.spawn(task=...)` returns a handle to a sub-agent that keeps running across cells. Agents report to their parent's inbox, and the parent waits, steers, reads their history or cancels them. +- **Supervised background work.** Shell jobs, long-lived services and watches on agents, jobs and paths are owned by a supervisor, so they survive a lost variable or a kernel crash. +- **Unbounded sessions.** The engine compacts the context into a handoff summary when it runs out of room; the kernel keeps its state and the full history stays readable from Python. +- **Tools where they belong.** Native tool calls (`bash`, `edit`, `fetch`), Python tool packages, `SKILL.md` workflows, and MCP servers whose credentials never reach the kernel. +- **Split execution.** The agent loop and credentials can stay in the harness while the tools run in a separate workspace, over stdio or TCP. ## Install @@ -16,603 +22,75 @@ Inside IPython, the `rlm` package is available in the namespace. When recursion git clone https://github.com/PrimeIntellect-ai/nano-rlm.git cd nano-rlm uv sync -source .venv/bin/activate -``` - -## CLI - -rlm runs exclusively as an [Agent Client Protocol](https://agentclientprotocol.com/) agent: - -```bash -rlm --acp -``` - -There is no standalone prompt mode: every session is created by an ACP client that -supplies the full runtime configuration (see below). Tool CLIs provided by the host -environment are on `$PATH` inside the kernel (e.g. `websearch --queries "..."` when -the `websearch` tool package is installed). - -## Agent Client Protocol - -Each ACP session owns one persistent RLM engine. Repeated `session/prompt` -requests retain both the model conversation and the live IPython kernel. The -agent accepts text prompts and stdio or streamable HTTP MCP servers, including -HTTP request headers. It supports cancellation and `session/close`; cancelling -a turn leaves its session available for the next prompt. It intentionally does -not advertise `session/load`: an arbitrary live Python kernel cannot be -reconstructed after the ACP process exits, so clients must keep the process -alive for the lifetime of a session. - -RLM advertises `_meta.steering.supported` during ACP initialization. Clients can -send `_session/steering` with `sessionId` and text `prompt` blocks while a turn is -running. Messages enter the conversation as user messages at the next model step, -after any tool results; tools are not interrupted. Steering wakes the native -`wait` tool so the next model step can consume the message. The request returns `injected` -when the message has entered the conversation, or `promptRequired` with reason -`noRunningTurn` when no active turn can consume it. Idle steering never starts a -new turn. Clients can set `_meta.steering.idleBehavior` to `promptRequired`. - -An optional `messageId` deduplicates retries within the session; reusing an ID with -different content fails. Injected messages survive prompt rollback and are kept in -the user-request history used by compaction. Queued messages that cannot be applied -remain unconsumed. Worlds uses this interface to deliver its attention inbox. - -RLM's ACP surface is a versioned training contract. `initialize` advertises the exact -`ai.prime.rlm/contract-v1` marker in its response `_meta`; clients must require -it, then provide one complete `ai.prime.rlm/runtime-v1` object in -`session/new._meta`. The runtime object contains the ACP session ID, model, -provider, execution policy, prompt configuration, enabled built-in Python tools, -optional builtin tool selection, -explicit kernel environment, and optional search credential. Nullable and -disabled values are sent explicitly as `null` or empty collections. Missing, -partial, unknown, or unsupported contracts are rejected; ACP sessions never -fall back to process environment configuration. - -Credentials travel over the private ACP stdio channel and are never echoed. -The `session/close` response carries one authoritative, credential-free -snapshot of cumulative usage, metrics, tool-call stats, supervisor counters, -and limits under `ai.prime.rlm/session-v1`. - -When a limit cuts a prompt short, its ACP stop reason says so: -`max_turn_requests` when the snapshot's `last_stop_reason` is -`max_total_turns`, and `max_tokens` when it is `max_total_tokens`, -`token_budget` (the policy's `max_tokens`) or `compaction_failed`. Every other -prompt ends with `end_turn`. - -Every actual model call carries a standard HTTP `Idempotency-Key` header that -stays stable across SDK and outer retries (retry attempts are distinguished by -`x-stainless-retry-count`), so an inference proxy can deduplicate replayed -requests. Both header names are reserved and rejected in provider -configuration. - -Model calls also carry a private `X-ACP-Model-Request-ID` correlation header. -RLM publishes sparse, labeled relationships between those request IDs under -`ai.prime.acp/semantic-edges-v1`: `continuation`, `subagent_call`, -`subagent_return`, and `compaction`. `continuation` preserves same-agent causal -order even when a consumer's physical token-prefix graph splits. ACP consumers -can resolve the request IDs onto their own message nodes while harnesses that do -not understand the extension ignore it. - -## Python API (inside a session) - -`rlm.agent` exposes `spawn`, `list`, and `get` inside a running session. These async calls contact the session supervisor; spawning waits only for registration, not task completion. `history()` also supports explicit session-directory loading outside a running session. - -## Configuration - -All runtime configuration enters through the `ai.prime.rlm/runtime-v1` contract object -(model, provider credentials, execution policy, prompt configuration, Python tools in `skills`, builtin tools, kernel -environment, search credential). Prompt configuration is role-aware: optional -`subagent_append_to_system_prompt` (nodes that can still recurse) and -`leaf_append_to_system_prompt` (depth == max_depth) override `append_to_system_prompt` -for sub-agents, each falling back to the next-more-general tier. Recursive children inherit the parent's configuration -in-memory (`model_copy`); nothing is re-read from the process environment. - -`policy.max_concurrent_subagents` caps concurrent model calls across the entire -tree, including the root and compaction requests. Queued calls closer to the root -have priority; calls at the same depth are admitted FIFO. In-flight calls are not -preempted, so continuous shallow work can delay deeper requests. Shell/tool -execution, child waits, and outer transport retry backoff do not hold inference -slots; SDK-internal retries remain within a slot. `policy.max_depth` independently -limits recursion. This limit is per tree; a serving-wide cap across multiple trees -belongs in the eval runner or inference gateway. - -`policy.delegation_prompt` (default off) appends the delegation guidance — when to spawn children, how to brief, watch, collect and reconcile them — for every agent that can still delegate. - -`system_prompt_path` supplies task instructions in place of the default task role. -The runtime guide is always appended, including when a custom prompt file is used. -The role-appropriate append instructions are included between the task instructions -and runtime guide. This keeps tool/API documentation and lifecycle rules available -to root agents, persistent children, and leaves. Credentials are not included in -agent identity metadata. - -The generated guide distinguishes Python state from supervisor-owned resources, -shows inbox dictionaries versus handle/metadata objects, and explains waiting, -completion, history recovery, jobs, and subscriptions. Delegation instructions are -shown only when IPython and recursion are available. Tool descriptions follow the -same shell execution guidance. - -Checkpoint prompts preserve task requirements, evidence, outstanding assignments, -jobs, subscriptions, event actions, output cursors, and history references. Commands -and edits are included only when relevant. Compaction thresholds, summary validation, -and context retention policy are unchanged. - -The process environment configures only process infrastructure: - -| Variable | Default | Description | -| ---------- | --------- | ------------- | -| `RLM_HOME` | `~/.rlm` | Root directory for sessions, data, and skills (`$RLM_HOME/skills`) | - -## Recursion - -The supervisor owns each agent's identity, task, runtime, and lifetime. Python variables hold handles, so losing a variable or ending a cell does not stop its agent. - -```python -researcher = await rlm.agent.spawn( - task="Check authentication behavior", name="researcher" -) -other = await rlm.agent.spawn(task="Check login behavior", name="login") - -agents = await rlm.agent.list() # Direct children, including completed agents -info = await researcher.info() # Fresh metadata snapshot -h = await researcher.history() # Fresh conversation snapshot -researcher = await rlm.agent.get("researcher") # Recover by sibling name or ID ``` -Names are unique among siblings and remain reserved for the session, including after completion. Immutable IDs are shown alongside names. Metadata includes the parent ID, initial task, status, persistence flag, creation time, elapsed lifetime in seconds, session directory, and any failure. `list(recursive=True)` includes descendants, but only direct children can be retrieved as handles or controlled through the supervisor. +## Quickstart -```python -result = await researcher.result(yield_after=60) -if result.running: - print("still working, answer so far:", result.answer) -else: - print(result.answer) -``` - -`result()` waits up to `yield_after` seconds (default 300) for the agent to finish and returns an `AgentResult` (`status`, `answer`, `usage`, `turns`, `session_dir`, `running`). `answer` is `None` while the agent is still working on its first answer; a persistent agent's latest answer stays available while it runs again. Terminal failure or cancellation raises. Use `info()` or `wait()` for current activity; a retained answer does not mean a follow-up has finished. `wait()` returns current metadata after an outcome or its timeout (default 30 seconds, range 0–300). It waits inside the Python cell and uses the cell's normal execution timeout. Cancelling or timing out a wait does not cancel the agent. - -`await researcher.cancel()` terminates the agent and its descendants and waits for cleanup. Ordinary agents release their kernels after answering. An agent spawned with `persistent=True` becomes idle after answering and retains its conversation and kernel; a parent instruction or new inbox event wakes it. Parent termination tears down all descendants, including persistent agents. Closing the ACP session tears down the tree. Cancelling an individual prompt or cell leaves its accepted children registered and recoverable. - -Status is `starting` while waiting for capacity, `running` during execution, `waiting` during a native event wait, `idle` for a persistent agent that has answered, or `completed`, `failed`, or `cancelled` after termination. Completed metadata, results, and history remain accessible for the supervisor's lifetime. Shared filesystem access is trusted; handle permissions are orchestration controls, not filesystem isolation. - -Depth, concurrency, total-spawn, and shared token/turn limits remain supervisor-enforced. Rejected spawns raise before registration. Persistent idle kernels release inference capacity but still count toward the session's total-spawn limit. A child return enters the ACP semantic trace when the parent retrieves its result, so background completion is not mistaken for result consumption. - -### Messages and event waiting +`rlm --acp` serves one ACP agent over stdio. A client starts it, sends the runtime contract with `session/new`, and prompts it. This client works against any OpenAI-compatible endpoint: ```python -await researcher.send("Also check logout") # Deliver after its answer or native wait -await researcher.steer("Focus on login first") # Deliver at the next model/tool boundary - -# Inside the child: -await rlm.agent.send_to_parent("Found a missing permission check") - -# Inside its parent: -for event in await rlm.inbox.list(): - report = await rlm.inbox.read(event["id"]) - if report["type"] == "agent.message": - print(report["content"]["name"], report["content"]["text"]) -``` - -Parent instructions are pushed into the child's conversation. Steering does not interrupt a running model request or tool. Queued messages wait until the child answers or calls the native `wait` tool. Both operations wake an idle persistent child; sending to a terminated child raises. - -Reports and `agent.completed` events enter the parent's inbox at every depth. The model sees the unread count whenever it changes or new events arrive, and chooses when to retrieve payloads. `list()` returns metadata without marking events read; `read(id)` returns the payload and marks it read. `list(unread_only=False)` includes previously read events. Completion payloads identify the agent and status; retrieve the answer with its handle's `result()`. - -ACP dependency edges use `agent_message` for delivered instructions and explicitly read inbox events. Spawning uses `subagent_call`; retrieving an answer uses `subagent_return`. Delivery policy and event type remain in the event records. An unread-count notification alone creates no `agent_message` edge. - -The native `wait` tool suspends inference without occupying an IPython cell, until a new inbox arrival, parent instruction, or timeout (default 300 seconds, range 0–300). It also yields to queued instructions. Already-announced unread events do not repeatedly wake it. A final root answer returns control to the ACP caller; use `wait` to keep the current prompt available for events. - -The supervisor owns inboxes and instruction queues. Each agent's `inbox.jsonl` records arrivals, instruction delivery, and explicit reads; delivered messages and notifications also enter `messages.jsonl`. Message payloads are limited to 65,536 JSON-encoded UTF-8 bytes. Reports are rejected once an inbox contains 10,000 events, and pending instruction queues have the same limit; lifecycle events remain recordable. This state lasts while the supervisor lives. Restarting a crashed IPython kernel and restoring a crashed supervisor are separate features. - -## Compaction - -There is no model-driven compaction tool. Compaction is on by default and unlimited (the policy's `compaction` field turns it off, `max_compactions` caps it); the default 1M `max_total_tokens` tree budget keeps sessions bounded, since each compaction cycle itself spends new tokens. The engine reads the model context window from the provider's `/models` response and compacts when 16k tokens remain below it; small windows keep at least half. Set `summarize_at_tokens` to pin the threshold explicitly. Without a known window or explicit threshold, proactive compaction stays off, but a provider overflow still triggers it reactively. A tool result larger than 20KB is truncated to its head and tail before it enters the conversation, with a warning naming the original size. - -The engine asks the model for a plain-text handoff summary and resumes the task on a fresh branch seeded with that summary; reasoning is never part of it. A provider overflow (a 400 or 413 naming a context limit) triggers the same compaction reactively from the current state. A rejected checkpoint request falls back to the last state that passed a threshold check - by definition a state with a full reserve of room - and an empty or tool-calling reply is resampled; after three failed attempts the run ends cleanly with what the conversation holds. An overflow with no history beyond the task propagates: the task alone approaches the window and there is nothing to reclaim. - -The IPython kernel keeps running across the compaction, so all variables, imports, and in-memory data are preserved. The model is told to mention important variable names in its summary so the resumed branch knows what is available. The same policy applies to the main agent and all recursive agents. - -A session creates a fresh ledger and refuses to reopen an existing `messages.jsonl` for writing. Existing histories remain readable. The session owns active messages and their indices; the engine uses context snapshots. A ledger write or flush failure makes the writer unusable and prevents further prompts. Rollback restores in-memory context and counters even if its ledger write fails. Supervisor crash recovery is not supported. - -The working conversation remains in the session's append-only `messages.jsonl`, which the model can search and process from Python. Message records have stable event `id` fields, zero-based `message_index` values, and a `message` object containing the role, content, and tool-call fields. Full tool outputs and shortened context versions have separate indices; the shortened tool record links to its `source_message_index`. System messages and installed summaries are also logged. Checkpoint prompts and raw checkpoint responses are omitted: this ledger serves agent context recovery. Verifiers, the ACP consumer, records model requests/responses through its interception endpoint and receives semantic links and metrics over ACP, independently of this file. A `user` record begins a prompt attempt; `prompt_rollback.prompt_id` identifies a failed or cancelled attempt, whose records remain available as history. - -Each `context_window` record declares a zero-based `window` index, a `reason` (`start`, `compaction`, or `rollback`), and the ordered `message_indices` that seed that window. Subsequent message records with a `window` field append to it. Closed windows never change. Rollback opens a new window containing the restored context, preserving addresses in the failed window. The `turn` field is an execution-loop counter, not a message or window index. - -Each agent writes its own `messages.jsonl` in its session directory. Message and context-window indices are local to that agent. Agent discovery belongs to the supervisor; the history reader accepts an explicit session directory. - -The kernel can inspect its own history or a child's, including while the child is running: - -```python -from rlm import history - -h = await history() # Defaults to $RLM_SESSION_DIR -requests = h.user_messages() # Original inputs, including rolled-back attempts -message = h.messages[3] # Session-wide message index -earlier = h.windows[0].messages # Initial working context -child_history = history(session_dir="/path/to/child-session") -message = child_history.windows[4].messages[2] # If that child has reached window 4 -``` - -Snapshots contain complete records as of the read; call `history(...)` again to observe new activity. `h.events` exposes lifecycle records, including each child's spawn prompt and rollback markers. The compacted context points to this API so the model can retrieve omitted details without putting the whole transcript back in context. - -## Session Directory - -Every invocation writes to `$RLM_HOME/sessions//`. Nested session directories mirror the call tree. - -```text -.rlm/sessions/abc123/ -├── meta.json -├── messages.jsonl -├── sub-d4e5/ -│ ├── meta.json -│ ├── messages.jsonl -│ └── sub-f6g7/ -└── sub-h8i9/ -``` - -These artifacts are consumable for debugging, visualization, or training-data extraction. - -## Tools - -Tools are executable capabilities. They can be native model tool calls, Python callables in IPython, or shell commands. Skills are `SKILL.md` workflows that guide the agent's use of tools. - -`rlm` ships a small set of built-in Python tools enabled per session via the runtime contract's `skills` list (`edit`, `search`, `fetch`; see [MCP tools](#mcp-tools) for the related MCP path). `edit` runs in the kernel. Credentialed `search` runs in the supervisor through the capability broker, so `SERPER_API_KEY` is unavailable to IPython and its subprocesses; it returns title/URL/snippet for a single query (`await search(query="...")`). The contract's `skills` field selects Python tools; it does not select `SKILL.md` workflows. - -Additional Python tool packages are supplied by the host environment: before `install.sh` runs, the environment places packages under `$RLM_HOME/skills//`, and `install.sh` installs them alongside `rlm` so they're both importable and on `$PATH`. The prompt advertises pre-imported callables as "Python tools (pre-imported)" and packages with console commands as "Shell-enabled Python tools". - -From IPython, call a pre-imported tool's async `run(...)` entrypoint through its callable module: - -```python -import websearch - -help(websearch) # signature + docstring -results = await websearch(queries=["latest jupyter_client release"]) -``` - -Installed `rlm-skill-*` tool packages also expose their declared console command. Built-in Python tools and MCP proxy callables are IPython-only: - -```bash -websearch --queries "latest jupyter_client release" -``` - -### Python tool package contract - -A Python tool package is laid out like this: - -```text -/ -├── SKILL.md # Optional workflow instructions -├── pyproject.toml -└── src/ - └── / - ├── __init__.py - └── .py -``` - -Required public surface: - -- async `run(...)`: the single entrypoint. Type-annotated parameters and a Google-style docstring (`Args:`, `Returns:`) drive both the Python API and the CLI. - -The `pyproject.toml` points its console script at the shared CLI entry: - -```toml -[project.scripts] - = "rlm.skill:cli" -``` - -`rlm.skill:cli` reads `sys.argv[0]`, imports the matching module, and uses [tyro](https://brentyi.github.io/tyro/) to build the argparse CLI from `run`'s signature. The return value of `run` is printed; a raise surfaces as a normal Python traceback. - -Naming expectations (all match): - -- package directory name: `` -- distribution name in `pyproject.toml`: `rlm-skill-` -- import name: `` -- console script name: `` - -Keyword arguments on `run(...)` and CLI flags line up automatically — `queries: list[str]` becomes `--queries` on the CLI. - -The distribution prefix `rlm-skill-` and CLI entrypoint `rlm.skill:cli` identify Python tool packages. Neither requires a `SKILL.md` file. Dependencies go in the package's own `pyproject.toml`; declare `rlm` there so the shared CLI entry resolves. Version conflicts between packages installed side-by-side are the user's responsibility. - -### Local development - -Place Python tool packages under `$RLM_HOME/skills/` (`~/.rlm/skills/` by default) before running `install.sh`. [The `say` fixture](tests/fixtures/skills/say) is a minimal package example. - -## Skills - -A skill is a folder holding a `SKILL.md` with `name` and `description` frontmatter. It contains instructions for a workflow and may include scripts, references, or a Python tool package. A skill's Markdown is read by the agent; its scripts and tools are executed through the available runtime tools. - -Every folder under `$RLM_HOME/skills` holding a `SKILL.md` is listed in a `## Skills` prompt section, one `- name: description (file: path)` line each (the codex convention). Only metadata and paths are included; the agent reads the full `SKILL.md` when its task matches. This discovery is independent of installed Python tool packages. A folder with both a `SKILL.md` and a tool package contributes workflow instructions to the skills catalog and a callable to the Python tools listing. - -## MCP tools - -A host harness can wire task-specific [MCP](https://modelcontextprotocol.io) tool servers to `rlm` through the ACP session (a standard `mcpServers` config shape). Streamable HTTP and stdio transports are supported: - -```json -{ - "mcpServers": { - "remote": {"url": "http://127.0.0.1:8000/mcp"}, - "local": { - "command": "/path/to/tool-server", - "args": ["--stdio"], - "env": {"API_KEY": "..."} +import asyncio +import os + +from acp import PROTOCOL_VERSION, spawn_agent_process, text_block + + +class Client: + async def session_update(self, session_id, update, **kwargs): + if getattr(update, "session_update", None) == "agent_message_chunk": + print(update.content.text) + + +async def main(): + runtime = { + "session_id": "quickstart", + "model": "your-model", + "provider": { + "base_url": "https://your-endpoint/v1", + "api_key": os.environ["API_KEY"], + "headers": {}, + }, + "policy": {}, + "system_prompt_path": None, + "append_to_system_prompt": None, + "skills": ["edit"], + "kernel_env": {}, + "search_api_key": None, } - } -} -``` - -Programmatically, pass `mcp_servers={"tools": "http://127.0.0.1:8000/mcp"}` to `RLMEngine`. An HTTP server may instead be `{"url": "...", "headers": {"Authorization": "..."}}` when it needs request headers; stdio servers use the same `command` / `args` / `env` shape shown above. + async with spawn_agent_process(Client(), "rlm", "--acp") as (agent, _process): + await agent.initialize(PROTOCOL_VERSION) + session = await agent.new_session( + cwd=os.getcwd(), mcp_servers=[], **{"ai.prime.rlm/runtime-v1": runtime} + ) + await agent.prompt( + session_id=session.session_id, + prompt=[text_block("How many Python files are in this directory?")], + ) + await agent.close_session(session.session_id) -At startup `rlm` connects to each server and indexes all pages of its raw MCP catalog in the supervisor. MCP listing includes input schemas; these stay in the supervisor's catalog. Startup and search do not create Python callables, public descriptors, broker handlers, or per-tool modules. The first `get(id)` creates and registers that tool's descriptor; subsequent loads reuse it. Only requested tools are converted into async callables in IPython, regardless of catalog size. -The discovery guide appears in the system prompt only when IPython is active and at least one MCP tool was discovered. Server summaries include available descriptions from MCP initialization. From IPython, search by keywords, load a selected callable, and inspect its full JSON input schema before calling it: - -```python -await rlm.tools.servers() # server names, tool counts and available descriptions -page = await rlm.tools.search("calendar events", limit=5, server="tools") -tool = await rlm.tools.get(page["matches"][0]["id"]) -print(tool.schema) # includes nested fields, enums and constraints -help(tool) # Python signature and description -result = await tool(day="monday", title="standup") # returns str +asyncio.run(main()) ``` -Search returns `matches` (at most 20 summaries per page: id, server and description, without schemas) and `next_cursor`. Retrieve another page using the same query and server filter; `next_cursor=None` means there are no more matches: +Run it with `uv run python quickstart.py`. The session's ledger lands in `~/.rlm/sessions/` (`$RLM_HOME` moves it). [The runtime contract](docs/runtime-contract.md) lists every field and policy. -```python -if page["next_cursor"] is not None: - page = await rlm.tools.search( - "calendar events", limit=5, server="tools", cursor=page["next_cursor"] - ) -``` - -Save loaded callables in variables to reuse them. Search uses keyword matching; try other keywords when no result fits. Cursors belong to the current session's catalog and query/server filter. Raw catalog metadata, including listing cursors, is limited to 64 MiB, with at most 1 MiB per tool; bounds are checked while pages are indexed. Discovery allows at most 4,096 pages per server. Individual tool names are omitted from the system prompt. - -Loaded tools are async functions called with `await tool(...)`. Calls return text strings; when a tool returns JSON text, decode it before accessing fields. Prose and CSV remain text: +## Documentation -```python -raw = await tool(...) -data = json.loads(raw) # only for tools that return JSON text -``` - -Each call connects using the configured transport, invokes the tool, and returns its text content (a tool-reported error is raised as `RuntimeError`). Discovery and invocation run in the session supervisor through the session-local capability broker. The kernel receives only a selected tool's public schema, description, name, and opaque capability; server URLs, headers, commands, and environment variables stay in the supervisor. MCP tools are IPython-only. - -## Kernel - -The IPython kernel always runs in rlm's own Python (`sys.executable`). `install.sh` puts `rlm` and all discovered Python tool packages into the same `uv tool install` environment, so their imports work natively from inside an IPython cell. Enabled built-in tools are also pre-imported by name, e.g. `edit`. - -The kernel starts from a small platform environment (`PATH`, home/user/shell, locale, temporary-directory, certificate, and virtual-environment variables) plus the contract's explicit `kernel_env` mapping. It receives private Jupyter/IPython config directories and does not inherit the rest of the supervisor process environment. This de-ambients credentials; it is not hostile-code containment because the kernel still shares the sandbox user, filesystem, process namespace, and network with the supervisor. - -To exercise packages from the target project's `.venv` (e.g. running its test suite), shell out from an IPython cell: `!./.venv/bin/python3 -m pytest`. The kernel itself stays isolated from whatever project venv the agent is working on — no cross-cell state involving sandbox packages. +| Page | What it covers | +|---|---| +| [Architecture](docs/architecture.md) | The engine, kernel, supervisor and session directory, and one turn through them | +| [Runtime contract](docs/runtime-contract.md) | The ACP surface: the runtime object, policy, prompts, steering, snapshots and tracing headers | +| [Python API](docs/python-api.md) | The `rlm` namespace in the kernel: agents, inbox, shell jobs, services, watches, history, hints | +| [Tools and skills](docs/tools.md) | Native tools, built-in Python tools, tool packages, `SKILL.md` skills, MCP servers, the kernel's environment | +| [Context and history](docs/context.md) | Compaction, tool-output truncation, the `messages.jsonl` ledger and reading it back | +| [Kernel and processes](docs/processes.md) | Timeouts, crash recovery, and which processes outlive a call, a restart or the session | +| [Split execution](docs/split-execution.md) | Running the tools in a separate workspace, over stdio or TCP | ## Developing -After `uv sync`, enable the pre-commit hooks: - -```bash -uv run pre-commit install -``` - -Lint and format manually: - -```bash -uv run ruff check --fix . -uv run ruff format . -``` - -## Testing - -Install dev dependencies and run the suite: - ```bash uv sync --group dev +uv run pre-commit install +uv run ruff check --fix . && uv run ruff format . uv run pytest tests/ ``` - -Agent execution status and cleanup are separate: `info.cleanup_error` reports failed resource cleanup without hiding a completed result. Calling `cancel()` again retries unfinished cleanup. - -`send()` and `steer()` return acceptance IDs, not processing acknowledgements. They reject instructions when the tree budget is already exhausted. If an accepted instruction cannot be delivered because the budget runs out or the child terminates, the supervisor records `instruction_failed` in the child's inbox journal and sends the parent an `agent.delivery_failed` event containing `agent_id`, `message_id`, and `reason`. Failed instructions are removed from the pending queue. - -### Bash commands and background jobs - -For quick commands, wait for the result in the current cell: - -```python -result = await rlm.shell.run("git status --short") -print(result.text, result.exit_code) -``` - -`run(command, cwd=..., env=..., yield_after=10, timeout=None)` returns a `ShellJob`: a snapshot with `running`, -`ok`, combined stdout/stderr `text` (up to 16 KiB: the first and last 8 KiB around an -omitted-range marker when longer), `exit_code`, `id`, `truncated`, `error` and `timed_out`, plus the -methods `result()`, `read()`, `info()` and `cancel()`. `yield_after` is how long `run()` -waits before yielding a handle instead of a result: 10 s by default, 0 returns at once, capped -at 300. A command still going then comes back with `running=True` and the output so far while -the job continues; nothing is killed by yielding. `timeout`, if given, kills the process group after that many seconds and -sets `timed_out`. `await job.result(yield_after=...)` waits (up to 300 s per call) and returns -the finished snapshot, again with `running=True` if the job is still not done; it is -repeatable and never consumes output. The command is a Bash string or an argv list. Nonzero -exit codes are returned; startup/capture errors populate `error`. If output is truncated, -`await job.read(cursor=..., max_bytes=...)` reads the retained output (16 MiB per job). -Cancelling the waiting cell leaves the job running and discoverable with `shell.list()`; -`shell.get(job_id)` recovers it. Only a job handed back with `running=True` publishes a quiet -`shell.completed` inbox event (job ID, status, exit code, completeness flags and the last -4 KiB of output); quiet means it wakes the native `wait` tool but is not counted in the unread -notice, and `result()` marks it read. `await rlm.shell.setenv(NAME='value')` sets variables -for every later `run()` of the agent (`getenv()` reads the overlay; per-call `env=` wins). -The kernel and Bash jobs inherit the launching process's environment (in a sandbox, the -image's ENV) minus a blocklist: credential-looking names, values with URL-embedded -credentials, variables that would redirect the kernel's own interpreter or venv (PYTHONHOME, -UV_PYTHON, ...), and agent/daemon sockets. - -For long commands or work that should run alongside other tasks, pass `yield_after=0` and -collect with `result()`: - -```python -job = await rlm.shell.run("uv run pytest tests/", cwd="/workspace/project", yield_after=0) -# ... other work ... -res = await job.result() -print(res.exit_code, res.text) - -# Several jobs at once, and recovery of a job in a later cell: -a, b = await asyncio.gather(job_a.result(), job_b.result()) -job = await rlm.shell.get(job_id) -chunk = await job.read(cursor=0, max_bytes=16384) # continue from chunk.next_cursor -``` - -Use the native `wait` tool when there is no other work and you hold no running job; a held -job is collected with `result()`. A nonzero Bash exit code is a completed process; -startup/capture errors have status `failed`. - -`await job.cancel()` terminates its process group. Jobs survive cell completion -and lost handle variables; owner termination cancels them. Commands run in -`/bin/bash --noprofile --norc -c`, with closed stdin and combined stdout/stderr. -Relative working directories resolve against the agent's working directory. -The task environment and Git-history policy also apply to these commands. - -The supervisor permits 32 active jobs and 1,024 total jobs per session tree. -Each job retains up to 16 MiB in `/jobs//output.bin`; excess output -is drained and discarded. Reads default to 16 KiB and allow at most 64 KiB. -Cursors count bytes, and text decodes as UTF-8 with replacement. Terminal metadata -is saved alongside output in `meta.json`. Output files remain after job cleanup. - -Normal completion waits for stdout/stderr EOF. If descendants keep the pipe open -more than two seconds after Bash exits, capture stops and reports -`output_complete=False` and `output_truncated=True`. Remaining processes in the -job's process group are killed when the job ends. Keep Bash alive until its work -finishes; detached processes that escape the process group are not managed. - -There is no PTY, stdin API, or redirection of IPython's `!` / `%%bash` magics. - -### IPython crash recovery - -If the kernel exits during a cell, the harness starts a fresh kernel under the -same agent identity and resumes the conversation with a recovery notice. A dead -kernel discovered before a cell starts is also restarted; that cell is returned -unexecuted so the model can first reconstruct its Python state. - -The conversation, history, child agents, shell jobs, and inbox (including read -state) remain owned by the live supervisor. Recover handles with -`rlm.agent.list/get` and `rlm.shell.list/get`. Python variables, imports, and -in-kernel tasks are lost. The interrupted cell is never replayed automatically: -its file writes or accepted spawn/send/job requests may already have happened. -Inspect those resources before retrying an operation. - -Recovery notices are separate `kernel_recovery` history records and active -conversation messages, so truncating tool output cannot hide them. A kernel -restart required after an unresponsive timeout/interrupt produces the same notice. -A successful interrupt that preserves the kernel does not claim variables were lost. - -There are at most three recovery attempts without a subsequently completed cell. -A failed startup reports failure; a later IPython call may retry within that -budget. Exhausting the budget leaves IPython unavailable, while the agent can -still respond or use native waiting. Supervisor crash recovery is not supported. -Kernel death is detected during execution or at the next IPython call; this does -not introduce a background kernel-health subscription. - -### Subscriptions - -Watch future activity without keeping a Python cell running: - -```python -activity = await rlm.watch.agent(researcher) -output = await rlm.watch.job(job) -files = await rlm.watch.path("/workspace/results", recursive=False) - -subscriptions = await rlm.watch.list() -watch = await rlm.watch.get(subscriptions[0].id) -await watch.cancel() -``` - -Agent watches observe direct children after complete assistant/tool steps, -including final answers. Job watches observe newly captured stdout/stderr from -jobs owned by the caller. Path watches observe existing files or directories; -relative paths resolve against the agent's working directory. Hidden files are -included, and directory recursion is opt-in. Child/job completion already -notifies the owner automatically and needs no subscription. - -Events enter the same pull-based inbox and carry `subscription_id`. Their -`content` contains a `target` (agent ID, job ID, or absolute path) plus: - -| Event | References | -| --- | --- | -| `watch.agent` | `start` and exclusive `end` message indices: `(await researcher.history()).messages[start:end]` | -| `watch.job` | `start` and exclusive `end` byte cursors for `job.read(cursor=start)` | -| `watch.path` | Changed `paths` and a `truncated` flag | -| `watch.failed` | An `error` explaining why the subscription stopped | - -Subscriptions start with future activity, not historical replay. Agent and job -ranges can be coalesced across several steps/chunks; changes are batched over -200 ms. Agent ranges exclude records written before registration. Filesystem -registration waits until the backend is watching before returning the handle. -Observed removal of the watched path ends the subscription with a failure event; -register a new watch after recreating it. - -Handles and subscriptions survive kernel restart. Cancellation stops future -events and drops any unpublished batch; existing inbox events remain readable. -Owner termination cancels its subscriptions. Registration and state transitions -are journaled in the owner's `inbox.jsonl`; supervisor restart recovery remains -out of scope. - -The tree permits 64 active subscriptions and 1,024 total registrations. Changed -path lists are capped at 32 KiB of JSON with explicit truncation. A subscription -stops with a failure event when the inbox event limit is reached. There are no -model-authored callbacks, event selectors, or output predicates. - -Activity subscriptions become `completed` after their target permanently terminates, flushing pending activity and releasing their active slot. Watches of idle persistent agents remain active. Watching an already-finished target returns a completed subscription. - -### Separate tool execution - - - - Colocated, the agent loop, model credentials and the agent's tools share one machine. Split, the harness keeps the loop, credentials, supervisor and ledger, and a worker in the workspace runs the tools over one authenticated connection. - - -The ACP runtime metadata accepts `execution_command` (a trusted argv launching -`python -m rlm.execution`) and `execution_cwd` (the task workspace). For example, -an orchestrator can start the worker with `docker exec -i -/path/to/python -m rlm.execution`. Both processes must have the same nano-rlm -version installed. The orchestrator owns container provisioning and teardown. - -The engine, model credentials, supervisor, and authoritative conversation logs -remain in the harness process. Native tools, persistent IPython kernels, and builtin skills run -in the worker environment. Recursive agents inherit the execution destination, -keep separate kernels, and share the task workspace. With the stdio transport, -communication and recursive callbacks use the worker's stdio, so a task container -can have networking disabled. -The command itself defines the isolation boundary; launching a local Python -worker alone provides process separation, not filesystem isolation. - -ACP initialization advertises `ai.prime.rlm/execution-v1` with the supported -transports (`stdio`, `tcp`) and tools (`ipython`, `bash`, `edit`, `fetch`). Tool -calls cross the boundary as a name and arguments; filesystem operations happen -inside the worker. `ExecutionTransport` supplies `connect()` (reader and writer) -and `close()` independently of tool dispatch. - -For a trusted private TCP network, launch `python -m rlm.execution --listen -0.0.0.0 --port 8765` with a random `RLM_EXECUTION_TOKEN` of at least 32 characters. -With `--port 0` the kernel picks a free port, which the worker prints on stderr as -`listening on HOST:PORT` once it accepts connections. -Set `execution_tcp` to `{"host": "worker-address", "port": 8765, "token": "..."}` -instead of `execution_command`. Each connection owns a separate execution session. -The token authenticates connections; this transport does not encrypt traffic, so -use a private network or an encrypted tunnel. The worker removes its token from -the environment before starting tools. The orchestrator owns listener teardown. - -Supervisor shell jobs (`rlm.shell`), services (`rlm.services`) and path watches -(`rlm.watch.path`) run in the worker's environment, beside the kernels; the -supervisor keeps their records and inbox events, so the agent gets the same -runtime prompt and API as in local execution. The current agent's history is a -local snapshot refreshed before each cell; `await child.history()` reads a child's -ledger from the supervisor in pages. Frames are limited to 16 MiB: a command or -cell keeps the first and last 512 KiB of its output, and a long ledger syncs in -chunks. Kernel recovery preserves the existing no-replay behavior. Worker -disconnects fail the session. - - - - One agent step in split mode: the harness sends the tool call with the new ledger records, the cell runs in the workspace and calls back for rlm.* operations, and the result returns to the harness. A tool call costs one round trip, each rlm.* call one more. - - -Each tool call costs one round trip between harness and worker, each `rlm.*` -call one more, and `rlm.shell.run` two. - -A session that closes stops what is still attached to its kernels, as a closing -terminal does; daemons and bash background jobs keep running, as they do in local -execution, so the orchestrator stops what remains when the rollout ends. diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 00000000..78278806 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,38 @@ +# Architecture + +nano-rlm gives a model one place to work, a persistent IPython kernel, and lets it extend itself with sub-agents and background work from there. Everything that has to outlive a single cell, such as agents, jobs, services, inbox events and the conversation itself, belongs to processes the kernel cannot lose: the engine and the supervisor. + + + + An ACP client sends the runtime contract and prompts to rlm. The engine runs the agent loop against the model API and appends to the session ledger; its tool calls run in a persistent IPython kernel, whose rlm.* calls reach the supervisor; the supervisor owns child agents, each with its own engine, kernel and session directory. + + +## Components + +**ACP agent.** `rlm --acp` serves the [Agent Client Protocol](https://agentclientprotocol.com/) over stdio. Each ACP session owns one engine for its lifetime. The session is configured entirely by the [runtime contract](runtime-contract.md) the client sends; nothing is read from the process environment except `RLM_HOME`. + +**Engine** (`RLMEngine`). The agent loop: it sends the conversation to the model, runs the tool calls the model makes, appends every message to the ledger, and compacts the context when it runs out of room. Model credentials live here and in the supervisor, never in the kernel. + +**IPython kernel.** One persistent Jupyter kernel per agent, running in rlm's own Python. Variables, imports and loaded data persist across cells, across turns and across compaction. The `rlm` package is importable in it, and enabled built-in Python tools (`edit`, `fetch`, ...) and installed tool packages are pre-imported by name. The kernel starts from a minimal environment, so ambient credentials do not reach it ([details](tools.md#the-kernels-environment)). + +**Supervisor.** Owns everything that must outlive a cell: child agents and their inboxes, shell jobs, services, watches, muted hints, and the admission of model calls across the agent tree. The kernel reaches it through `rlm.*` calls on a session-local capability broker. The broker also runs credentialed tools, such as `search` and MCP servers, on the kernel's behalf, so their keys and URLs stay out of the kernel. + +**Child agents.** `rlm.agent.spawn(...)` asks the supervisor for a child: another engine with its own kernel and session directory, configured from the parent's contract one level deeper. Children can spawn their own children up to `max_depth`. The parent holds a handle; losing the handle does not stop the child. + +**Session directory.** Each agent writes `$RLM_HOME/sessions//`, with children nested under it as `sub-/`. Its `messages.jsonl` is the append-only ledger of the conversation, which the agent itself can read back from Python ([Context and history](context.md)). + +## One turn + +1. The client sends `session/prompt`. The engine appends the prompt to the ledger and asks the supervisor for a model-call slot (`max_concurrent_subagents` bounds calls across the whole tree). +2. The model answers with text or tool calls. For an `ipython` call the engine runs the cell in the kernel; native `bash`, `edit` and `fetch` calls run beside it. +3. While the cell runs, its `rlm.*` calls (spawn an agent, read the inbox, start a job) go to the supervisor and return into the cell. +4. The tool result enters the conversation, truncated to its first and last 10 KB when longer ([details](context.md#tool-output)). The engine loops until the model answers without tool calls, a budget runs out, or the client cancels. +5. The answer goes back to the client as an agent message, and the ACP response carries a credential-free snapshot of usage, metrics and limits. + +Between turns, the kernel and every supervisor-owned resource stay alive; the next prompt continues the same conversation. + +## Trust boundaries + +- **Credentials.** The model's provider key travels over the private ACP stdio channel and stays in the engine. Search and MCP credentials stay in the supervisor. The kernel's environment is rebuilt from an allowlist plus the contract's `kernel_env`. +- **Not containment.** In local execution the kernel shares the user, filesystem, process namespace and network with the engine. Filesystem access between agents is trusted; agent handles are orchestration controls, not isolation. +- **Real separation** comes from [split execution](split-execution.md): the tools run in a separate workspace, such as a container or VM, and only tool calls cross between them. diff --git a/docs/context.md b/docs/context.md new file mode 100644 index 00000000..df02e2d3 --- /dev/null +++ b/docs/context.md @@ -0,0 +1,64 @@ +# Context and history + +A session has no fixed length. When the conversation outgrows the model's context window, the engine compacts it into a handoff summary and carries on, while the kernel keeps every variable and the full record stays on disk, readable from Python. The model loses nothing it cannot get back. + +## Compaction + +Compaction is on by default and unlimited: `compaction: false` turns it off and `max_compactions` caps it. The default `max_total_tokens` budget still bounds the session, since each compaction spends new tokens too. + +**When.** The engine reads the model's context window from the provider's `/models` response and compacts once 16k tokens remain; small windows keep at least half. `summarize_at_tokens` pins the threshold instead. Without a known window or explicit threshold, compaction happens only when the provider reports an overflow (a 400 or 413 naming a context limit), which also triggers it with a threshold set. + +**How.** The engine asks the model for a plain-text handoff summary, without reasoning, and resumes on a fresh context seeded with it. The checkpoint prompt asks the summary to keep the task's requirements, evidence, outstanding assignments, jobs, subscriptions, pending event actions, output cursors and history references, plus the commands and edits that matter, and to name the important kernel variables so the resumed context knows what exists. An empty or tool-calling reply is resampled. A rejected checkpoint request falls back to the last state that passed a threshold check, which by definition has a full reserve of room. After `max_compaction_attempts` failures the run ends cleanly with what the conversation holds. An overflow with nothing beyond the task to drop propagates, because there is nothing to reclaim. + +The kernel keeps running through compaction, so variables, imports and in-memory data survive it. The same policy applies to every agent in the tree. + +## Tool output + +A tool result longer than 20 KB (`max_tool_output_bytes`) enters the conversation as its first and last 10 KB around a notice with the whole output's token estimate, line count and omitted bytes. The full result stays in the ledger. + +A command or cell always runs to its end, however much it prints. Its result keeps at most the first and last 512 KiB of output (bash divides that between stdout and stderr) and records how much more the whole output measured, so the notice reports the whole output's size either way. The ledger keeps the retained ends with a marker where the middle was dropped. `rlm.shell` jobs keep up to 16 MiB each in their own files ([Shell jobs](python-api.md#shell-jobs)). + +## The ledger + +Each agent appends its conversation to `messages.jsonl` in its session directory. The ledger serves the agent's own recovery: it is what `history()` reads and what compacted contexts point to. Clients that trace model calls record requests and responses at their own proxy and receive metrics and semantic edges over ACP, independent of this file. + +- **Messages.** Message records carry a stable event `id`, a zero-based `message_index` and a `message` with the role, content and tool calls. System messages and installed summaries are recorded; checkpoint prompts and raw checkpoint responses are not. A tool result truncated for the context has a separate record linking back through `source_message_index`. +- **Windows.** A `context_window` record opens a window with a zero-based `window` index, a `reason` (`start`, `compaction` or `rollback`) and the ordered `message_indices` that seed it. Later records with a `window` field append to it, and closed windows never change. A failed or cancelled prompt rolls back by opening a new window with the restored context, so addresses in the failed window stay valid. +- **Prompts and turns.** A `user` record begins a prompt attempt, and `prompt_rollback.prompt_id` marks one that failed or was cancelled; its records remain as history. The `turn` field counts loop iterations and is not an index. +- **Other records.** Lifecycle records such as `kernel_recovery`, child spawns and delivered inbox notifications are in the same file. + +Indices are local to each agent. A session opens a fresh ledger and refuses to reopen an existing `messages.jsonl` for writing. A failed write or flush makes the session refuse further prompts; rollback restores the in-memory context even if its own record cannot be written. + +## Reading history + +```python +from rlm import history + +h = await history() # this agent's ledger ($RLM_SESSION_DIR) +requests = h.user_messages() # original inputs, rolled-back attempts included +message = h.messages[3] # by message index +earlier = h.windows[0].messages # the first context window +other = await history(session_dir="/path/to/session") +message = other.windows[4].messages[2] # if that session has reached window 4 +``` + +A snapshot holds the complete records as of the read; read again to see new activity. `h.events` holds the lifecycle records, including each child's spawn prompt and rollback markers. For a child, `await child.history()` goes through the supervisor ([Agents](python-api.md#agents)). Compacted contexts point to this API, so the model can recover details without putting the transcript back in context. + +## Session directory + +Every agent writes `$RLM_HOME/sessions//`, and children nest under their parent, mirroring the call tree: + +```text +~/.rlm/sessions/abc123/ +├── meta.json +├── messages.jsonl +├── inbox.jsonl +├── jobs//{output.bin, meta.json} +├── sub-d4e5/ +│ ├── meta.json +│ ├── messages.jsonl +│ └── sub-f6g7/ +└── sub-h8i9/ +``` + +These files are meant for debugging, visualization and extracting training data. diff --git a/docs/images/overview-dark.svg b/docs/images/overview-dark.svg new file mode 100644 index 00000000..a885ac72 --- /dev/null +++ b/docs/images/overview-dark.svg @@ -0,0 +1 @@ +ACP clientverifiers, an editor,your own scriptone rlm session (rlm --acp)Model APIenginethe agent loop: model turns,tool calls, compactionRLMEngineIPython kernelvariables and imports that persistthe rlm.* namespace, edit, fetch,your Python tool packagessession directorymessages.jsonlinbox.jsonl · jobs/ · meta.jsonone per agentsupervisoragents · inbox · shell jobs · serviceswatches · model-call admissioncapability broker: search, MCPchild agentseach with its own engine, kernel and session directory, owned by the supervisorcontract, prompts →← answers, snapshotstool callsresultsappendsrlm.* callsspawns, owns diff --git a/docs/images/overview.svg b/docs/images/overview.svg new file mode 100644 index 00000000..4e0b0664 --- /dev/null +++ b/docs/images/overview.svg @@ -0,0 +1 @@ +ACP clientverifiers, an editor,your own scriptone rlm session (rlm --acp)Model APIenginethe agent loop: model turns,tool calls, compactionRLMEngineIPython kernelvariables and imports that persistthe rlm.* namespace, edit, fetch,your Python tool packagessession directorymessages.jsonlinbox.jsonl · jobs/ · meta.jsonone per agentsupervisoragents · inbox · shell jobs · serviceswatches · model-call admissioncapability broker: search, MCPchild agentseach with its own engine, kernel and session directory, owned by the supervisorcontract, prompts →← answers, snapshotstool callsresultsappendsrlm.* callsspawns, owns diff --git a/docs/processes.md b/docs/processes.md new file mode 100644 index 00000000..9d9d17c7 --- /dev/null +++ b/docs/processes.md @@ -0,0 +1,40 @@ +# Kernel and processes + +nano-rlm's workspace behaves like a Linux box an agent is logged into. A command can be stopped when it hangs, a crashed kernel comes back without losing the conversation, and what the agent deliberately leaves running keeps running after it is done, so a task that asks for a server gets graded with the server up. + +## Timeouts and cancellation + +| | Times out after | Then | +|---|---|---| +| `ipython` cell | `exec_timeout` (300 s), or the call's `timeout`, at most 600 s | The kernel is interrupted and the output so far is returned with a notice. A kernel that does not respond is restarted ([below](#kernel-crash-recovery)) | +| `bash` command | `exec_timeout` | Its process group is killed and the output so far is returned with the error | +| `rlm.shell` job | its own `timeout`, if given | Its process group is killed and `timed_out` is set | + +A cancelled turn stops a running `bash` command the same way. A cancelled `ipython` call interrupts its cell, and its kernel is recovered if needed. + +`bash` commands read an empty stdin, so a command that waits for input sees end-of-file at once rather than hanging or reading the harness's own streams. `rlm.shell` jobs have stdin closed too. + +## Which processes outlive what + +| Process | Kernel restart | Timeout or cancelled turn | Session end | +|---|---|---|---| +| Started from a cell and still attached to the kernel (`!cmd &`, `subprocess.Popen`) | stopped | the cell is interrupted | stopped | +| A `bash` tool command | unaffected | stopped, partial output kept | stopped | +| A `bash` tool command's background job (`cmd &`, `nohup cmd &`) | runs on | runs on | runs on | +| An `rlm.shell` job and the processes in its group | runs on | runs on (its own `timeout` applies) | stopped with its owner | +| An `rlm.services` service | runs on | runs on | runs on, until `stop()` | +| A daemon in a session of its own (`setsid`, `start_new_session=True`) | runs on | runs on | runs on | + +- **The kernel's session.** The kernel leads a session of its own. A restart or the end of the session stops every process still in that session, the way a closing terminal does, while processes that started a session of their own are left alone. +- **Bash commands.** Each `bash` command, whether a tool call or `await bash(...)` from a cell, leads its own process group. A timeout or cancellation kills that group while the command is still running. Once bash itself has exited, a job it put in the background outlives the call, even one that kept the call's output open until the timeout. A `bash` command run from a cell is in the kernel's session, so a restart stops it with the kernel. +- **Shell jobs and services.** The supervisor owns these. A job's leftover processes are killed when the job ends, and its owner's end ends it. A service runs in its own session and only `stop()` ends it ([Services](python-api.md#services)). + +What keeps running after a session is the orchestrator's to clean up, for example by tearing down the container or VM. + +## Kernel crash recovery + +If the kernel dies during a cell, rlm starts a fresh kernel under the same agent and resumes the conversation with a recovery notice. A dead kernel found before a cell starts is restarted too, and that cell is returned unexecuted so the model can rebuild its Python state first. A kernel restarted after an unresponsive timeout or interrupt produces the same notice; an interrupt that leaves the kernel alive does not claim variables were lost. + +Python variables, imports and in-kernel tasks are lost. The conversation, history, child agents, shell jobs, services, watches and inbox, read state included, belong to the supervisor and stay; `rlm.agent.list`/`get` and `rlm.shell.list`/`get` recover their handles. The interrupted cell is never replayed: its file writes or accepted spawns, sends and jobs may already have happened, so the model inspects them before retrying. + +Recovery notices are `kernel_recovery` records in the history and messages in the context, so truncated tool output cannot hide them. rlm makes at most three recovery attempts without a completed cell in between. A failed start is reported, and a later `ipython` call may retry within that budget; once it is spent, IPython is unavailable but the agent can still answer or wait. Kernel death is noticed during a cell or at the next `ipython` call; nothing polls the kernel in the background. A crashed supervisor is not recovered. diff --git a/docs/python-api.md b/docs/python-api.md new file mode 100644 index 00000000..2808a3aa --- /dev/null +++ b/docs/python-api.md @@ -0,0 +1,168 @@ +# Python API + +Inside the kernel, the `rlm` package is how an agent reaches beyond its own cell: it starts and steers sub-agents, runs shell jobs and services, watches for activity, and reads its own history. These are async calls to the [supervisor](architecture.md#components), so what they create outlives the cell, the variable holding it, and even a kernel crash. + +| Module | For | +|---|---| +| [`rlm.agent`](#agents) | Spawning sub-agents and steering, waiting for and cancelling them | +| [`rlm.inbox`](#inbox-and-waiting) | Reports, completions and watch events, pulled when the agent chooses | +| [`rlm.shell`](#shell-jobs) | Bash commands, short or in the background | +| [`rlm.services`](#services) | Processes that must keep running after the agent finishes | +| [`rlm.watch`](#watches) | Notifications about an agent's steps, a job's output or changes under a path | +| [`rlm.history`](#history) | Reading this agent's or a child's conversation | +| [`rlm.hints`](#hints) | Muting the supervisor's one-line usage hints | +| `rlm.tools` | Searching and loading MCP tools ([Tools](tools.md#mcp-tools)) | + +## Agents + +Agents are available when recursion is allowed (`max_depth` above the agent's depth). A Python variable holds a handle; the supervisor owns the agent, so ending a cell or losing the variable does not stop it. + +```python +researcher = await rlm.agent.spawn(task="Check authentication behavior", name="researcher") +other = await rlm.agent.spawn(task="Check login behavior", name="login") + +agents = await rlm.agent.list() # direct children, finished ones included +info = await researcher.info() # a fresh metadata snapshot +h = await researcher.history() # a fresh conversation snapshot +researcher = await rlm.agent.get("researcher") # recover a handle by name or ID +``` + +`spawn` returns once the child is registered, not when it finishes; a rejected spawn (depth, spawn or budget limits) raises before registration. Names are unique among siblings and stay reserved for the session. `list(recursive=True)` includes descendants, but only direct children can be retrieved as handles and controlled. Metadata includes the ID, name, parent, task, status, persistence, creation time, elapsed seconds, session directory and any failure. + +**Results.** + +```python +result = await researcher.result(yield_after=60) +if result.running: + print("still working, answer so far:", result.answer) +else: + print(result.answer) +``` + +`result()` waits up to `yield_after` seconds (default 300) and returns an `AgentResult` with `status`, `answer`, `usage`, `turns`, `session_dir` and `running`. `answer` is `None` while the agent works on its first answer; a persistent agent's latest answer stays available while it runs again. A failed or cancelled agent raises. `wait(timeout=30)` returns current metadata after an outcome or its timeout (0 to 300 seconds); it waits inside the cell, under the cell's own timeout. Neither cancels the agent. + +**Status** is `starting` while waiting for capacity, `running`, `waiting` during a native wait, `idle` for a persistent agent that has answered, and `completed`, `failed` or `cancelled` once it has ended. Finished agents' metadata, results and history stay readable while the session lives. + +**Lifetime.** An agent releases its kernel after answering. One spawned with `persistent=True` goes `idle` instead, keeping its conversation and kernel, and wakes on a parent instruction or a new inbox event. `await researcher.cancel()` ends an agent and its descendants and waits for cleanup; calling it again retries cleanup that did not finish, and `info.cleanup_error` reports a cleanup failure without hiding a completed result. A parent's end ends all its descendants, persistent ones included, and closing the ACP session ends the tree. Cancelling a single prompt or cell leaves its accepted children registered and recoverable. Idle persistent agents hold no inference slot but count toward `max_subagent_calls`. + +**Messages.** + +```python +await researcher.send("Also check logout") # delivered after its answer or at a native wait +await researcher.steer("Focus on login first") # delivered at its next model/tool boundary + +# inside the child: +await rlm.agent.send_to_parent("Found a missing permission check") +``` + +Parent instructions enter the child's conversation; neither kind interrupts a running model call or tool. Both wake an idle persistent child, and sending to an agent that has ended raises. `send` and `steer` return acceptance IDs, not acknowledgements, and refuse when the tree's budget is spent. An accepted instruction that cannot be delivered (the budget ran out, or the child ended) is recorded as `instruction_failed` in the child's inbox journal and reported to the parent as an `agent.delivery_failed` event with `agent_id`, `message_id` and `reason`. + +## Inbox and waiting + +Reports, completions and watch events arrive in the agent's inbox. The model sees the unread count whenever it changes and decides when to read: + +```python +for event in await rlm.inbox.list(): # metadata, unread only by default + report = await rlm.inbox.read(event["id"]) # the payload, now marked read + if report["type"] == "agent.message": + print(report["content"]["name"], report["content"]["text"]) +``` + +`list(unread_only=False)` includes events already read. A child's `agent.completed` event names the agent and its status; its answer comes from the handle's `result()`. + +The native **`wait`** tool suspends inference without holding a cell, until a new inbox event, a parent instruction or its timeout (default 300 seconds). Events already announced do not wake it again. An agent with nothing else to do waits instead of polling; a final root answer instead returns control to the client. + +Inboxes and instruction queues belong to the supervisor. Each agent's `inbox.jsonl` journals arrivals, deliveries and reads, and delivered messages and notifications also enter its `messages.jsonl`. Message payloads are limited to 65,536 JSON bytes. An inbox takes up to 10,000 events and a pending instruction queue as many; lifecycle events are always recorded. + +## Shell jobs + +`rlm.shell.run` runs a Bash command under the supervisor. A short command returns its result in the same cell: + +```python +result = await rlm.shell.run("git status --short") +print(result.text, result.exit_code) +``` + +A long one keeps running in the background while the cell moves on: + +```python +job = await rlm.shell.run("uv run pytest tests/", cwd="/workspace/project", yield_after=0) +# ... other work ... +res = await job.result() +print(res.exit_code, res.text) + +a, b = await asyncio.gather(job_a.result(), job_b.result()) # several at once +job = await rlm.shell.get(job_id) # recover in a later cell +chunk = await job.read(cursor=0, max_bytes=16384) # continue from chunk.next_cursor +``` + +`run(command, cwd=..., env=..., yield_after=10, timeout=None)` takes a Bash string or an argv list and returns a `ShellJob` snapshot: `id`, `running`, `ok`, `exit_code`, `text` (combined stdout and stderr, up to 16 KiB: the first and last 8 KiB around a marker when longer), `truncated`, `timed_out` and `error`. + +- **Yielding.** `run` waits `yield_after` seconds (default 10, 0 returns at once, at most 300) before handing back a job that is still running, with `running=True` and its output so far. Yielding never stops the job. `await job.result(yield_after=...)` waits up to 300 seconds per call and returns the finished snapshot, or the job again if it is still running; it is repeatable and never consumes output. +- **Timeout and cancel.** `timeout` kills the job's process group after that many seconds and sets `timed_out`. `await job.cancel()` terminates the process group. +- **Output.** Each job keeps up to 16 MiB in `/jobs//output.bin`, readable with `job.read(cursor, max_bytes)` (16 KiB by default, 64 KiB at most; cursors count bytes, text decodes as UTF-8 with replacement). A finished job's metadata is kept beside it. +- **Completion events.** A job handed back while still running publishes a quiet `shell.completed` event when it ends, with its ID, status, exit code, completeness flags and last 4 KiB of output. Quiet events wake `wait` but are not counted as unread, and `result()` marks them read. +- **Environment.** `await rlm.shell.setenv(NAME="value")` sets variables for every later `run` of this agent; `getenv()` reads them back, and a call's own `env=` wins. Jobs inherit the agent's process environment minus credential-looking names, URLs with embedded credentials, variables that would redirect rlm's interpreter or virtual environment, and agent sockets. +- **Execution.** Commands run in `/bin/bash --noprofile --norc -c`, with stdin closed and stdout and stderr combined. Relative directories resolve against the agent's working directory, and the Git guard applies. A nonzero exit code is a completed job; failing to start or capture is status `failed`, with `error` set. There is no PTY or stdin API, and IPython's `!` and `%%bash` do not go through `rlm.shell`. + +Jobs survive the end of their cell, lost variables and kernel restarts; `rlm.shell.list()` finds them. A job ends with its owner, and when it ends the processes left in its process group are killed. A job counts as finished when stdout and stderr close: if descendants keep them open more than two seconds after Bash exits, capture stops with `output_complete=False`. Processes that leave the job's process group are not managed; anything that must keep running is a [service](#services). A session tree allows 32 running jobs and 1,024 in total. + +## Services + +A service is a process that must keep running after the agent finishes, such as a server or daemon a task asks to leave up. + +```python +svc = await rlm.services.start("api", "python -m http.server 8000", cwd="/workspace/site") +print(await svc.tail()) # the last lines of its log +svc = await rlm.services.get("api") +await svc.stop(grace=5) # SIGTERM to its process group, SIGKILL after the grace period +``` + +`start(name, command, cwd=..., env=...)` returns at once; check that the service came up with `tail()` or by connecting to it. The service runs in a session of its own, with stdin from `/dev/null` and stdout and stderr appended to its log, and only `stop()` ends it. Names are 1 to 64 letters, digits, `.`, `_` or `-`; starting a name that is still running raises. The environment and default working directory are those of `rlm.shell.run`. `get(name)` returns a fresh snapshot (`name`, `pid`, `command`, `cwd`, `log`, `running`, `exit_code`, `started_at`) and works after kernel restarts; `list()` returns every service started in this workspace, running or not. + +## Watches + +A watch turns future activity into inbox events, without keeping a cell running: + +```python +activity = await rlm.watch.agent(researcher) # its completed steps, final answers included +output = await rlm.watch.job(job) # newly captured output +files = await rlm.watch.path("/workspace/results", recursive=False) + +subscriptions = await rlm.watch.list() +watch = await rlm.watch.get(subscriptions[0].id) +await watch.cancel() +``` + +Agent watches cover direct children; job watches cover the caller's own jobs; path watches cover an existing file or directory, hidden files included, with recursion opt-in. A child's or job's completion already notifies its owner without a watch. + +Each event carries its `subscription_id`, and its `content` names the `target` (an agent ID, job ID or absolute path) plus: + +| Event | References | +|---|---| +| `watch.agent` | `start` and exclusive `end` message indices: `(await researcher.history()).messages[start:end]` | +| `watch.job` | `start` and exclusive `end` byte cursors for `job.read(cursor=start)` | +| `watch.path` | The changed `paths` and a `truncated` flag | +| `watch.failed` | An `error` saying why the watch stopped | + +Watches see only activity after registration; there is no replay. Agent and job ranges can merge several steps or chunks, and path changes are batched over 200 ms. A path watch returns once the backend is watching; removing the watched path ends it with `watch.failed`. Watches survive kernel restarts, end with their owner, and complete once their target has ended (flushing pending activity); a watch on an idle persistent agent stays active, and one on an already finished target returns completed. Cancelling drops any unpublished batch but keeps delivered events. Registrations and state changes are journaled in the owner's `inbox.jsonl`. A tree allows 64 active watches and 1,024 registrations, changed-path lists are capped at 32 KiB of JSON, and a watch stops with `watch.failed` when its owner's inbox is full. + +## History + +An agent can read its own conversation, or a child's, as it stands: + +```python +from rlm import history + +h = await history() # this agent's ledger +requests = h.user_messages() # original inputs, rolled-back attempts included +message = h.messages[3] # by session-wide message index +earlier = h.windows[0].messages # the first context window +child = await researcher.history() # a child's, even while it runs +``` + +Each call returns a complete snapshot as of the read; call again to see new activity. `h.events` holds the lifecycle records too, such as each child's spawn prompt and rollback markers. `history(session_dir=...)` reads any session directory, also outside a session. [Context and history](context.md#the-ledger) describes the records. + +## Hints + +The supervisor attaches a one-line hint to the next turn when something worth knowing happened, for example a blocking `run()` that detached into a background job. Each hint names a tag; `await rlm.hints.mute("tag")` stops that hint for this agent, `unmute` restores it, and `muted()` lists the muted tags. diff --git a/docs/runtime-contract.md b/docs/runtime-contract.md new file mode 100644 index 00000000..ced1c820 --- /dev/null +++ b/docs/runtime-contract.md @@ -0,0 +1,103 @@ +# Runtime contract + +nano-rlm is configured by its client, once per session, through one versioned object. There is no config file and no environment fallback: a training run gets exactly the agent it asked for, and the response tells it exactly what that agent spent. + +`rlm --acp` is an [Agent Client Protocol](https://agentclientprotocol.com/) agent over stdio. There is no standalone prompt mode. + +## Handshake + +1. `initialize` returns `ai.prime.rlm/contract-v1: true` in its response `_meta`. Clients must require it. The same `_meta` advertises steering (`steering.supported`) and split execution (`ai.prime.rlm/execution-v1`, with its transports and tools). +2. `session/new` carries one complete `ai.prime.rlm/runtime-v1` object in its `_meta`. Nullable and disabled values are sent explicitly as `null` or empty collections. Missing, partial, unknown or unsupported contracts are rejected. +3. `session/prompt` runs a turn. Repeated prompts continue the same conversation and the same live kernel. Cancelling a turn leaves the session available for the next prompt. +4. `session/close` tears down the agent tree and returns the final [session snapshot](#session-snapshot). + +The agent accepts text prompts, and stdio or streamable-HTTP MCP servers (with HTTP headers) in `session/new`. It does not advertise `session/load`: a live Python kernel cannot be rebuilt after the process exits, so a client keeps the process alive for the session's lifetime. + +## The runtime object + +| Field | Type | Meaning | +|---|---|---| +| `session_id` | string | The client's ID for the session, echoed in snapshots (`[A-Za-z0-9._:-]`, at most 128 characters) | +| `model` | string | Model name sent to the provider | +| `provider` | object | `base_url`, `api_key`, `headers`, `max_retries` (default 5). `Idempotency-Key`, `x-stainless-retry-count` and the ACP tracing headers are reserved | +| `policy` | object | Budgets, recursion, compaction and timeouts, see [Policy](#policy). Omitted fields take their defaults | +| `system_prompt_path` | string or null | A file whose instructions replace the default task role | +| `append_to_system_prompt` | string or null | Instructions appended for every agent | +| `subagent_append_to_system_prompt` | string or null | Override for sub-agents that can still recurse | +| `leaf_append_to_system_prompt` | string or null | Override for agents at `max_depth` | +| `skills` | list | Built-in Python tools to pre-import: `bash`, `edit`, `fetch`, `search` ([Tools](tools.md#built-in-python-tools)) | +| `builtin_tools` | list or null | Native tools; null means `["ipython"]` ([Tools](tools.md#native-tools)) | +| `kernel_env` | object | Variables added to the kernel's minimal environment | +| `search_api_key` | string or null | Serper key for `search`, held by the supervisor | +| `network_block` | list | Domains, wildcard hosts or URL origins that `search` results must not point to | +| `execution_command` | list or null | A trusted argv starting `python -m rlm.execution`, for [split execution](split-execution.md) over stdio | +| `execution_cwd` | string or null | The tools' working directory in split execution | +| `execution_tcp` | object or null | `host`, `port` and `token` of a worker for split execution over TCP | + +Recursive children inherit their parent's configuration in memory, one level deeper; nothing is re-read. + +The only process environment variable is `RLM_HOME` (default `~/.rlm`): the root for session directories and host-supplied tool packages (`$RLM_HOME/skills`). + +## Policy + +| Field | Default | Meaning | +|---|---|---| +| `max_total_tokens` | 1,000,000 | New tokens across the whole tree: each call's completion plus uncached prompt. Once reached, every engine stops before its next model call and no more agents are spawned. The default terminator | +| `max_total_turns` | null | Model calls across the tree, compaction excluded | +| `max_depth` | 1 | How deep agents can recurse; 0 disables `rlm.agent.spawn` | +| `max_concurrent_subagents` | 4 | Concurrent model calls across the tree, root and compaction calls included | +| `max_subagent_calls` | null | Total spawns across the tree | +| `delegation_prompt` | false | Add guidance on when and how to delegate, for agents that can | +| `exec_timeout` | 300 | Seconds a tool call may run; an `ipython` call may ask for up to 600 | +| `max_tokens` | null | Completion tokens one agent may spend; reaching it ends the turn | +| `max_tool_output_bytes` | null (20 KB) | How much of one tool result enters the conversation | +| `compaction` | true | Compact the context when it runs out of room, and recover from provider overflows | +| `summarize_at_tokens` | null | Compaction threshold; null derives it from the provider's advertised context window | +| `max_compactions` | null | Compactions per session | +| `max_compaction_attempts` | 5 | Summary attempts within one compaction | +| `allow_git` | false | Lift the guard that refuses Git commands reaching beyond the current branch (other refs, branch and tag listings, `clone`, `fetch`, `pull`) | + +**Model-call admission.** `max_concurrent_subagents` bounds calls across the tree. Queued calls closer to the root go first, FIFO within a depth, and calls in flight are not preempted, so continuous shallow work can delay deeper agents. Tools, child waits and the backoff between outer retries hold no slot; the SDK's own retries stay within one. The limit is per tree: a cap across trees belongs in the runner or inference gateway. + +## Prompts + +`system_prompt_path` supplies task instructions in place of the default task role. The runtime guide is always appended, so root agents, persistent children and leaves all get the tool and API documentation and the lifecycle rules. The role's append instructions sit between the task instructions and the guide; sub-agents and leaves fall back to the next more general tier when their own is unset. + +The guide distinguishes Python state from supervisor-owned resources and explains waiting, completion, history, jobs and subscriptions. Delegation guidance appears only when IPython and recursion are both available. + +## Steering + +During a running turn, `_session/steering` (with `sessionId` and text `prompt` blocks) adds a user message at the next model step, after any tool results; running tools are not interrupted. It wakes the native `wait` tool. The request returns `injected` once the message is in the conversation, or `promptRequired` with reason `noRunningTurn` when no turn can take it; idle steering never starts a turn. Clients can set `_meta.steering.idleBehavior` to `promptRequired`. + +An optional `messageId` deduplicates retries within the session; reusing an ID with different content fails. Injected messages survive prompt rollback and stay in the user-request history that compaction keeps. Queued messages that cannot be applied stay unconsumed. + +## Session snapshot + +Prompt responses and the `session/close` response carry one credential-free snapshot under `ai.prime.rlm/session-v1`: the session ID and last stop reason, the model, turns, cumulative token usage, metrics (compactions, sub-agent calls, programmatic tool calls, ...), supervisor counters and the effective limits. The `session/close` snapshot is authoritative. + +The ACP stop reason says when a limit cut a prompt short; the snapshot's `last_stop_reason` names the exact cause: + +| `last_stop_reason` | Cause | ACP stop reason | +|---|---|---| +| `done` | The agent answered | `end_turn` | +| `max_total_turns` | The tree reached `max_total_turns` | `max_turn_requests` | +| `max_total_tokens` | The tree reached `max_total_tokens` | `max_tokens` | +| `token_budget` | The agent reached its `max_tokens` | `max_tokens` | +| `compaction_failed` | A compaction failed with the context full | `max_tokens` | +| `cancelled` | The client cancelled the prompt | `cancelled` | + +## Tracing headers + +Every model call carries a standard `Idempotency-Key` header that stays stable across SDK and outer retries (attempts differ in `x-stainless-retry-count`), so an inference proxy can deduplicate replays. + +Each call also carries a private `X-ACP-Model-Request-ID`. Responses publish sparse, labeled edges between those request IDs under `ai.prime.acp/semantic-edges-v1`: + +| Edge | Meaning | +|---|---| +| `continuation` | The same agent's next call, in causal order even where a consumer's token-prefix graph splits | +| `subagent_call` | A spawn | +| `subagent_return` | The parent retrieving a child's result | +| `agent_message` | A delivered instruction, or an inbox event the agent explicitly read; an unread count alone creates none | +| `compaction` | A compaction call | + +Clients that do not understand the extension ignore it. diff --git a/docs/split-execution.md b/docs/split-execution.md new file mode 100644 index 00000000..773b2792 --- /dev/null +++ b/docs/split-execution.md @@ -0,0 +1,66 @@ +# Split execution + +By default the agent's tools run beside the harness: the agent loop, the model credentials, the supervisor and the ledger share a filesystem and a process tree with every command the agent runs. A command can then read the credentials, kill or starve the loop, or rewrite the ledger, and the harness's own install is part of the environment being graded. + +Split execution moves the boundary to the tool call. The harness keeps the loop, the credentials, the supervisor and the authoritative conversation, and a worker in a separate workspace, such as a container or VM, runs the tools. The model sees the same tools, prompt and results. + + + + Colocated, the agent loop, model credentials and the agent's tools share one machine. Split, the harness keeps the loop, credentials, supervisor and ledger, and a worker in the workspace runs the tools over one authenticated connection. + + +## How it works + +- **Tool calls cross the boundary.** `ipython`, `bash`, `edit` and `fetch` travel as a name and arguments to `python -m rlm.execution` in the workspace, which hosts the persistent kernels, the built-in Python tools, interruption and kernel recovery. File operations happen in the workspace. Recursive agents use the same worker, each with a kernel of its own, and share the workspace. +- **`rlm.*` calls come back.** A cell's calls into the supervisor return over the same connection, scoped to the agent that made them. The supervisor's shell jobs, services and path watches run in the workspace, beside the kernels, while the supervisor keeps their records and inbox events. The agent gets the same runtime prompt and API as in local execution. +- **The ledger follows the cell.** Before each cell the worker receives the conversation's new records, in chunks that fit a frame, so `history()` reads locally in the workspace. A child's history comes from the supervisor in pages. +- **Bounded frames.** Messages are length-prefixed JSON of at most 16 MiB. A command or cell keeps the first and last 512 KiB of its output, so any result fits. + + + + One agent step in split mode: the harness sends the tool call with the new ledger records, the cell runs in the workspace and calls back for rlm.* operations, and the result returns to the harness. A tool call costs one round trip, each rlm.* call one more. + + +## Configuring it + +Both sides need the same nano-rlm version. The runtime contract takes one of two transports: + +**stdio.** `execution_command` is a trusted argv that starts the worker, and `execution_cwd` is the tools' working directory in the workspace: + +```json +{ + "execution_command": ["docker", "exec", "-i", "task-container", "/opt/rlm/bin/python", "-m", "rlm.execution"], + "execution_cwd": "/workspace" +} +``` + +Calls and callbacks share the worker's stdio, so the workspace can have networking disabled. The command defines the isolation boundary: a worker started as a local process gives process separation, not filesystem isolation. + +**TCP.** For a trusted private network, start the worker with a random token of at least 32 characters: + +```bash +RLM_EXECUTION_TOKEN=... python -m rlm.execution --listen 0.0.0.0 --port 0 +``` + +With `--port 0` the kernel picks a free port, and the worker prints `listening on HOST:PORT` on stderr once it accepts connections. Then set `execution_tcp` instead of `execution_command`: + +```json +{ + "execution_tcp": {"host": "10.0.0.5", "port": 41234, "token": "..."}, + "execution_cwd": "/workspace" +} +``` + +Each connection is a separate execution session. The token authenticates connections, and the worker removes it from its environment before starting tools. The transport does not encrypt, so use a private network or an encrypted tunnel. + +`initialize` advertises `ai.prime.rlm/execution-v1` with the supported transports (`stdio`, `tcp`) and tools (`ipython`, `bash`, `edit`, `fetch`). In code, `ExecutionTransport` supplies `connect()` (a reader and writer) and `close()`, independently of tool dispatch. + +## What it costs + +A tool call adds one round trip between harness and worker, each `rlm.*` call one more, and `rlm.shell.run` two. The worker holds a running job's new output for up to 50 ms, so a short command's output and exit arrive together, and both ends set `TCP_NODELAY`. On a cloud provider's private network a round trip is typically tens to a few hundred milliseconds. + +## Responsibilities + +The orchestrator provisions the workspace, starts the worker or listener, and tears both down. A worker that disconnects fails the session. Process lifetimes are those of [local execution](processes.md#which-processes-outlive-what): a closing session stops what is attached to its kernels, while daemons, services and `bash` background jobs keep running, so stopping what remains when a rollout ends is the orchestrator's job. Kernel recovery works as locally, including never replaying an interrupted cell. + +The prompt lists Python tool packages and `SKILL.md` skills from the harness's installation, but the agent imports and reads them in the workspace, so the workspace needs the same packages and files. Built-in Python tools work as in local execution, and MCP calls still run in the harness's supervisor. diff --git a/docs/tools.md b/docs/tools.md new file mode 100644 index 00000000..e6f03316 --- /dev/null +++ b/docs/tools.md @@ -0,0 +1,116 @@ +# Tools and skills + +nano-rlm keeps the model's tool list short and puts everything else in Python. By default the model has one native tool, `ipython`, and reaches shell commands, edits, web pages, MCP servers and installed packages as callables in its kernel. Native tool calling is available for runs that want it, and skills teach workflows on top of either. + +| Kind | Selected by | The model uses it as | +|---|---|---| +| [Native tools](#native-tools) | `builtin_tools` in the contract | a tool call | +| [Built-in Python tools](#built-in-python-tools) | `skills` in the contract | `await edit(...)` in a cell | +| [Python tool packages](#python-tool-packages) | installed by the host under `$RLM_HOME/skills` | `await websearch(...)`, or a shell command | +| [Skills](#skills) | any folder with a `SKILL.md` under `$RLM_HOME/skills` | instructions it reads when relevant | +| [MCP tools](#mcp-tools) | `mcpServers` in `session/new` | `await tool(...)` after searching the catalog | + +## Native tools + +`builtin_tools` selects the model's tools; null selects `["ipython"]`. + +| Tool | Does | +|---|---| +| `ipython` | Runs a cell in the agent's persistent kernel, with an optional `timeout` up to 600 seconds | +| `bash` | Runs one `bash -c` command in the working directory and returns stdout, stderr and a nonzero exit code | +| `edit` | Replaces one exact occurrence of a string in a file | +| `fetch` | Retrieves a URL as cleaned text | + +Whenever `ipython` is active, the model also gets the native `wait` tool ([Inbox and waiting](python-api.md#inbox-and-waiting)). [Kernel and processes](processes.md) describes timeouts, cancellation and what a command's processes outlive. + +## Built-in Python tools + +The contract's `skills` list pre-imports these into the kernel by name: + +| Name | Call | +|---|---| +| `bash` | `await bash("...")`: the `bash` tool's semantics from a cell | +| `edit` | `await edit(path=..., old_str=..., new_str=...)`: a single-occurrence replacement | +| `fetch` | `await fetch(url=...)`: a URL as cleaned text | +| `search` | `await search(query=...)`: web search through Serper, returning titles, URLs and snippets | + +`search` runs in the supervisor through the capability broker, so `search_api_key` never reaches the kernel or its subprocesses; `network_block` filters its results. The others run in the kernel. + +## Python tool packages + +A host can supply more tools as Python packages. Before running `install.sh`, it places them under `$RLM_HOME/skills//`; `install.sh` installs them beside rlm, so each is importable in the kernel and its command is on `$PATH`. The prompt lists pre-imported callables as "Python tools (pre-imported)" and packages with console commands as "Shell-enabled Python tools". + +```python +import websearch + +help(websearch) # signature and docstring +results = await websearch(queries=["latest jupyter_client release"]) +``` + +```bash +websearch --queries "latest jupyter_client release" +``` + +A package looks like this: + +```text +/ +├── SKILL.md # optional workflow instructions +├── pyproject.toml +└── src/ + └── / + ├── __init__.py + └── .py +``` + +Its one public entry point is an async `run(...)` with type-annotated parameters and a Google-style docstring (`Args:`, `Returns:`), which drive both the Python call and the CLI. The console script points at rlm's shared entry: + +```toml +[project.scripts] + = "rlm.skill:cli" +``` + +`rlm.skill:cli` reads `sys.argv[0]`, imports the matching module, and builds the CLI from `run`'s signature with [tyro](https://brentyi.github.io/tyro/): `queries: list[str]` becomes `--queries`. It prints the return value; an exception surfaces as a traceback. The directory, import name and console script are all ``, and the distribution is `rlm-skill-`; that prefix and the entry point identify a tool package, with or without a `SKILL.md`. Declare `rlm` among the package's dependencies so the entry point resolves; version conflicts between packages installed side by side are the host's to resolve. [The `say` fixture](../tests/fixtures/skills/say) is a minimal example. + +Built-in Python tools and MCP callables exist only in the kernel; tool packages also have their command. + +## Skills + +A skill is a folder holding a `SKILL.md` with `name` and `description` frontmatter: instructions for a workflow, possibly with scripts, references or a tool package. Every such folder under `$RLM_HOME/skills` gets one line in the prompt's `## Skills` section, `- name: description (file: path)`, and the agent reads the full file when its task matches. Its scripts and tools run through the agent's usual tools. A folder with both a `SKILL.md` and a tool package contributes to both listings. The contract's `skills` field selects built-in Python tools, not `SKILL.md` skills. + +## MCP tools + +A client can give a session [MCP](https://modelcontextprotocol.io) servers in `session/new`, over streamable HTTP or stdio: + +```json +{ + "mcpServers": { + "remote": {"url": "http://127.0.0.1:8000/mcp"}, + "local": {"command": "/path/to/tool-server", "args": ["--stdio"], "env": {"API_KEY": "..."}} + } +} +``` + +Programmatically, pass `mcp_servers={"tools": "http://127.0.0.1:8000/mcp"}` to `RLMEngine`, or `{"url": ..., "headers": {...}}` for a server that needs request headers. + +At startup the supervisor connects to each server and indexes its whole catalog, schemas included, without creating anything in the kernel. The model finds tools by keyword and loads only what it uses: + +```python +await rlm.tools.servers() # names, tool counts, descriptions +page = await rlm.tools.search("calendar events", limit=5, server="tools") +tool = await rlm.tools.get(page["matches"][0]["id"]) +print(tool.schema) # the full JSON input schema +help(tool) # Python signature and description +result = await tool(day="monday", title="standup") # returns text +data = json.loads(result) # when the tool returns JSON +``` + +`search` returns up to 20 `matches` per page (ID, server and description) and a `next_cursor`, which continues the same query and server filter; `None` means there are no more. Search is keyword matching, so try other words when nothing fits. Tool names are not listed in the prompt, which shows the discovery guide only when IPython is active and at least one tool was found. + +Each call connects with the configured transport, runs the tool and returns its text; a tool-reported error raises `RuntimeError`. Discovery and calls run in the supervisor through the capability broker: the kernel receives only a tool's name, description, public schema and an opaque capability, never server URLs, headers, commands or environment. A catalog is limited to 64 MiB of metadata, 1 MiB per tool and 4,096 pages per server. + +## The kernel's environment + +The kernel runs in rlm's own Python (`sys.executable`), in the same environment `install.sh` installs rlm and the tool packages into, so their imports work from a cell. To use a project's own packages, shell out to its interpreter: `!./.venv/bin/python -m pytest`. + +The kernel starts from a small platform environment (`PATH`, home, user, shell, locale, temporary directory, certificate and virtual-environment variables) plus the contract's `kernel_env`, with private Jupyter and IPython configuration directories. Nothing else from rlm's environment reaches it, which keeps credentials out of reach of the agent's code. This is not containment: in local execution the kernel shares rlm's user, filesystem, processes and network. [Split execution](split-execution.md) moves the tools out. From 970e56761f7fcdc99599aaafe6e2b7da89c5ea7e Mon Sep 17 00:00:00 2001 From: Konstantin Dunas Date: Mon, 2 Nov 2026 01:13:48 +0100 Subject: [PATCH 2/3] tests: run a hosted model end to end, local and split (#244) Co-authored-by: Claude Opus 5.5 --- .github/workflows/e2e.yml | 60 ++++++ README.md | 13 ++ pyproject.toml | 2 + tests/e2e/__init__.py | 0 tests/e2e/conftest.py | 364 +++++++++++++++++++++++++++++++++ tests/e2e/fixtures/desk_mcp.py | 25 +++ tests/e2e/test_agents.py | 67 ++++++ tests/e2e/test_session.py | 94 +++++++++ tests/e2e/test_tasks.py | 108 ++++++++++ tests/e2e/test_tools.py | 95 +++++++++ tests/test_compaction.py | 8 +- 11 files changed, 835 insertions(+), 1 deletion(-) create mode 100644 .github/workflows/e2e.yml create mode 100644 tests/e2e/__init__.py create mode 100644 tests/e2e/conftest.py create mode 100644 tests/e2e/fixtures/desk_mcp.py create mode 100644 tests/e2e/test_agents.py create mode 100644 tests/e2e/test_session.py create mode 100644 tests/e2e/test_tasks.py create mode 100644 tests/e2e/test_tools.py diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml new file mode 100644 index 00000000..ae636f84 --- /dev/null +++ b/.github/workflows/e2e.yml @@ -0,0 +1,60 @@ +name: E2E + +on: + push: + branches: [main] + paths: + - "src/**" + - "tests/**" + - "pyproject.toml" + - "uv.lock" + - ".github/workflows/e2e.yml" + pull_request: + paths: + - "src/**" + - "tests/**" + - "pyproject.toml" + - "uv.lock" + - ".github/workflows/e2e.yml" + workflow_dispatch: + +concurrency: + group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} + cancel-in-progress: true + +permissions: + contents: read + +jobs: + e2e: + name: E2E (${{ matrix.mode }}) + # Secrets do not reach pull requests from forks. + if: >- + github.event_name != 'pull_request' || + (github.event.pull_request.head.repo.full_name == github.repository && + github.event.pull_request.user.login != 'dependabot[bot]') + runs-on: ubuntu-latest + timeout-minutes: 30 + strategy: + fail-fast: false + matrix: + mode: [local, stdio, tcp] + env: + PRIME_API_KEY: ${{ secrets.PRIME_API_KEY }} + PRIME_TEAM_ID: ${{ secrets.PRIME_TEAM_ID }} + RLM_E2E_MODEL: ${{ vars.RLM_E2E_MODEL }} + steps: + - uses: actions/checkout@v4 + - name: Install uv + uses: astral-sh/setup-uv@v7 + with: + python-version: "3.12" + - name: Install dependencies + run: uv sync + - name: Check credentials + if: env.PRIME_API_KEY == '' + run: | + echo "::error::PRIME_API_KEY is not set; add it under Settings → Secrets and variables → Actions" + exit 1 + - name: Run end-to-end tests + run: uv run pytest tests/e2e -m e2e -k ${{ matrix.mode }} -v --durations=0 diff --git a/README.md b/README.md index a9c4fc1d..8a340408 100644 --- a/README.md +++ b/README.md @@ -94,3 +94,16 @@ uv run pre-commit install uv run ruff check --fix . && uv run ruff format . uv run pytest tests/ ``` + +The end-to-end suite in `tests/e2e` gives a hosted model real tasks through `rlm --acp` +and checks what the agent leaves in its workspace, with tools in the agent's process +tree, in a [split-execution](docs/split-execution.md) worker over stdio, and in one +over TCP. It runs only when asked for and needs a Prime Inference key (`PRIME_TEAM_ID` +too for a team's models): + +```bash +PRIME_API_KEY=... uv run pytest tests/e2e -m e2e +``` + +`RLM_E2E_MODEL` picks the model (default `internal/glm-5.3-fast`) and `RLM_E2E_BASE_URL` +the endpoint. diff --git a/pyproject.toml b/pyproject.toml index 1c49824d..be6daf35 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -45,6 +45,8 @@ dev = [ [tool.pytest.ini_options] asyncio_mode = "auto" testpaths = ["tests"] +addopts = "-m 'not e2e'" +markers = ["e2e: runs a hosted model end to end (opt in with -m e2e)"] [build-system] requires = ["hatchling"] diff --git a/tests/e2e/__init__.py b/tests/e2e/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/tests/e2e/conftest.py b/tests/e2e/conftest.py new file mode 100644 index 00000000..9f182685 --- /dev/null +++ b/tests/e2e/conftest.py @@ -0,0 +1,364 @@ +"""End-to-end runs against a hosted model. + +Each test starts `rlm --acp` the way an orchestrator does, gives it a task in a +throwaway workspace, and checks what the agent left there: files, its ledger and the +session snapshot. Where the model could get an answer without the feature under test, +the answer exists only behind that feature (a stamp only a tool call returns, a +codeword only a skill holds, a token only a sub-agent's history shows), so a pass proves +the feature ran. The suite runs only with `-m e2e`, needs `PRIME_API_KEY` (plus +`PRIME_TEAM_ID` for a team's models), and skips without it. `RLM_E2E_MODEL` and +`RLM_E2E_BASE_URL` choose the model. + +Every test runs in three placements: `local` (tools in the agent's process tree), +`stdio` (a worker the agent starts, over its stdin and stdout) and `tcp` (a worker +over TCP, started on its own with no credentials in its environment). The last two +are split execution. +""" + +from __future__ import annotations + +import asyncio +import contextlib +import json +import os +import re +import secrets +import subprocess +import sys +from collections.abc import AsyncIterator, Sequence +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +import pytest +from acp import PROTOCOL_VERSION, spawn_agent_process, text_block + +from rlm.acp import ( + ACP_SEMANTIC_EDGES_METADATA_KEY, + RUNTIME_METADATA_KEY, + SESSION_METADATA_KEY, +) + +MODEL = os.environ.get("RLM_E2E_MODEL") or "internal/glm-5.3-fast" +BASE_URL = os.environ.get("RLM_E2E_BASE_URL") or "https://api.pinference.ai/api/v1" +MODES = ("local", "stdio", "tcp") +PROMPT_TIMEOUT = 600 + + +def pytest_collection_modifyitems(items): + for item in items: + if "tests/e2e/" in item.nodeid.replace(os.sep, "/"): + item.add_marker(pytest.mark.e2e) + + +@pytest.fixture(scope="session", autouse=True) +def install_fixture_skills(): + """The agent sees the skills a real install has, not the unit tests' fixtures.""" + yield + + +@pytest.fixture(params=MODES) +def mode(request) -> str: + return request.param + + +@dataclass +class Turn: + stop_reason: str + answer: str + meta: dict[str, Any] + + +class _Client: + def __init__(self) -> None: + self.messages: list[str] = [] + + async def session_update(self, session_id: str, update: Any, **kwargs: Any) -> None: + content = getattr(update, "content", None) + if getattr(update, "session_update", None) == "agent_message_chunk" and content: + self.messages.append(getattr(content, "text", "")) + + +class AgentSession: + """One ACP session: prompt, steer or cancel it while it lives, then read what it + left behind.""" + + def __init__( + self, + connection: Any, + session_id: str, + client: _Client, + workspace: Path, + home: Path, + ) -> None: + self._connection = connection + self._client = client + self.session_id = session_id + self.workspace = workspace + self.home = home + self.turns: list[Turn] = [] + self.running: asyncio.Task[Turn] | None = None + self._calls_before_running = 0 + self.closed_meta: dict[str, Any] | None = None + + async def prompt(self, text: str) -> Turn: + """Run one prompt to its end.""" + self._client.messages.clear() + response = await asyncio.wait_for( + self._connection.prompt( + session_id=self.session_id, prompt=[text_block(text)] + ), + PROMPT_TIMEOUT, + ) + turn = Turn( + response.stop_reason, + "".join(self._client.messages), + response.field_meta or {}, + ) + self.turns.append(turn) + return turn + + def start(self, text: str) -> asyncio.Task[Turn]: + """Start a prompt and return while it runs.""" + self._calls_before_running = len(self.tool_calls()) + self.running = asyncio.create_task(self.prompt(text)) + return self.running + + async def called(self, name: str, timeout: float = 300) -> None: + """Wait until the running prompt's agent has called the tool `name`.""" + deadline = asyncio.get_running_loop().time() + timeout + while name not in self.tool_calls()[self._calls_before_running :]: + assert self.running is not None and not self.running.done(), ( + f"the prompt ended without calling {name}: {self.describe()}" + ) + assert asyncio.get_running_loop().time() < deadline, ( + f"no {name} call within {timeout}s: {self.describe()}" + ) + await asyncio.sleep(0.2) + + async def steer(self, text: str) -> dict[str, Any]: + return await self._connection.ext_method( + "session/steering", + {"sessionId": self.session_id, "prompt": [{"type": "text", "text": text}]}, + ) + + async def cancel(self) -> None: + await self._connection.cancel(session_id=self.session_id) + + @property + def session_dir(self) -> Path: + (session_dir,) = (self.home / "sessions").iterdir() + return session_dir + + @property + def records(self) -> list[dict[str, Any]]: + """The root agent's ledger as written so far.""" + return self.ledger(self.session_dir) + + def children(self) -> list[Path]: + """Session directories of the root agent's sub-agents.""" + return sorted( + path for path in self.session_dir.iterdir() if path.name.startswith("sub-") + ) + + @staticmethod + def ledger(session_dir: Path) -> list[dict[str, Any]]: + path = session_dir / "messages.jsonl" + if not path.exists(): + return [] + records = [] + for line in path.read_text().splitlines(): + with contextlib.suppress( + json.JSONDecodeError + ): # a line still being written + records.append(json.loads(line)) + return records + + @property + def snapshot(self) -> dict[str, Any]: + """The authoritative snapshot `session/close` returned.""" + assert self.closed_meta is not None, "the session is still open" + return self.closed_meta[SESSION_METADATA_KEY] + + @property + def answer(self) -> str: + return self.turns[-1].answer if self.turns else "" + + @property + def stop_reason(self) -> str: + return self.turns[-1].stop_reason if self.turns else "" + + def read(self, name: str) -> str: + """A file the agent left in the workspace, stripped; empty when it left none.""" + path = self.workspace / name + return path.read_text().strip() if path.exists() else "" + + def tool_calls(self) -> list[str]: + """Names of the tools the root agent called, in order.""" + return [call["function"]["name"] for call in self.tool_call_records()] + + def tool_call_records(self) -> list[dict[str, Any]]: + return [ + call + for record in self.records + if record.get("type") == "assistant" + for call in record["message"].get("tool_calls") or () + ] + + def tool_results(self) -> list[str]: + """What the root agent's tool calls returned, in order.""" + return [ + record["content"] + for record in self.records + if record.get("type") == "tool_result" + ] + + def edges(self) -> set[str]: + """Types of the semantic edges the session published.""" + metas = [turn.meta for turn in self.turns] + [self.closed_meta or {}] + return { + edge["type"] + for meta in metas + for edge in (meta.get(ACP_SEMANTIC_EDGES_METADATA_KEY) or {}).get( + "edges", () + ) + } + + def describe(self) -> str: + stops = [turn.stop_reason for turn in self.turns] + return ( + f"stops {stops}; tools {self.tool_calls()}; answer {self.answer[-500:]!r}" + ) + + +def _provider() -> dict[str, Any]: + headers = {} + if os.environ.get("PRIME_TEAM_ID"): + headers["X-Prime-Team-ID"] = os.environ["PRIME_TEAM_ID"] + return { + "base_url": BASE_URL, + "api_key": os.environ["PRIME_API_KEY"], + "headers": headers, + "max_retries": 5, + } + + +def _start_tcp_worker(workspace: Path, token: str) -> subprocess.Popen: + """A worker the way a remote workspace runs one: its own process, its own + environment, none of the agent's credentials.""" + env = { + key: value + for key, value in os.environ.items() + if not key.startswith(("PRIME_", "RLM_E2E_")) + } + env["RLM_EXECUTION_TOKEN"] = token + return subprocess.Popen( + [sys.executable, "-m", "rlm.execution", "--listen", "127.0.0.1", "--port", "0"], + cwd=workspace, + env=env, + stderr=subprocess.PIPE, + text=True, + ) + + +async def _listening_port(worker: subprocess.Popen) -> int: + line = await asyncio.wait_for(asyncio.to_thread(worker.stderr.readline), 60) + match = re.search(r"listening on \S+:(\d+)", line) + if not match: + raise RuntimeError(f"execution worker did not start: {line!r}") + return int(match.group(1)) + + +@pytest.fixture +def agent(tmp_path, mode): + """Open a session on a fresh workspace, in this test's placement. Closing it + records the final snapshot.""" + if not os.environ.get("PRIME_API_KEY"): + pytest.skip("needs PRIME_API_KEY") + workers: list[subprocess.Popen] = [] + + @contextlib.asynccontextmanager + async def open_session( + *, + files: dict[str, str] | None = None, + builtin_tools: list[str] | None = None, + skills: tuple[str, ...] = ("edit",), + skill_docs: dict[str, str] | None = None, + mcp_servers: Sequence[Any] = (), + policy: dict[str, Any] | None = None, + ) -> AsyncIterator[AgentSession]: + workspace = tmp_path / "workspace" + workspace.mkdir() + for name, text in (files or {}).items(): + (workspace / name).write_text(text) + home = tmp_path / "rlm" + for name, text in (skill_docs or {}).items(): + (home / "skills" / name).mkdir(parents=True) + (home / "skills" / name / "SKILL.md").write_text(text) + runtime: dict[str, Any] = { + "session_id": "e2e", + "model": MODEL, + "provider": _provider(), + "policy": {"max_depth": 1, "exec_timeout": 120, **(policy or {})}, + "system_prompt_path": None, + "append_to_system_prompt": None, + "skills": list(skills), + "builtin_tools": builtin_tools, + "kernel_env": {}, + "search_api_key": None, + } + if mode == "stdio": + runtime["execution_command"] = [sys.executable, "-m", "rlm.execution"] + runtime["execution_cwd"] = str(workspace) + elif mode == "tcp": + token = secrets.token_hex(32) + worker = _start_tcp_worker(workspace, token) + workers.append(worker) + port = await _listening_port(worker) + runtime["execution_tcp"] = { + "host": "127.0.0.1", + "port": port, + "token": token, + } + runtime["execution_cwd"] = str(workspace) + client = _Client() + executable = str(Path(sys.executable).parent / "rlm") + env = {**os.environ, "RLM_HOME": str(home)} + async with spawn_agent_process(client, executable, "--acp", env=env) as ( + connection, + _process, + ): + await connection.initialize(PROTOCOL_VERSION) + created = await connection.new_session( + cwd=str(workspace), + mcp_servers=list(mcp_servers), + **{RUNTIME_METADATA_KEY: runtime}, + ) + session = AgentSession( + connection, created.session_id, client, workspace, home + ) + try: + yield session + finally: + if session.running is not None and not session.running.done(): + session.running.cancel() + await asyncio.gather(session.running, return_exceptions=True) + closed = await connection.close_session(created.session_id) + session.closed_meta = closed.field_meta + + yield open_session + for worker in workers: + worker.kill() + worker.wait() + + +@pytest.fixture +def run_agent(agent): + """Run one prompt to its end in a fresh session.""" + + async def run(prompt: str, **options: Any) -> AgentSession: + async with agent(**options) as session: + await session.prompt(prompt) + return session + + return run diff --git a/tests/e2e/fixtures/desk_mcp.py b/tests/e2e/fixtures/desk_mcp.py new file mode 100644 index 00000000..0e15cd94 --- /dev/null +++ b/tests/e2e/fixtures/desk_mcp.py @@ -0,0 +1,25 @@ +"""A stdio MCP server whose results only a real call produces: `echo` stamps what it +returns, and `lookup` fails with a token in its error.""" + +from mcp.server.fastmcp import FastMCP + +ECHO_STAMP = "ok-7f3" +ERROR_TOKEN = "err-5c2" + +server = FastMCP("desk") + + +@server.tool() +def echo(message: str) -> str: + """Echo the message back, stamped so the caller can prove the tool ran.""" + return f"{message} [{ECHO_STAMP}]" + + +@server.tool() +def lookup(order_id: str) -> str: + """Look up an order by its id.""" + raise ValueError(f"Order {order_id} not found [{ERROR_TOKEN}]") + + +if __name__ == "__main__": + server.run() diff --git a/tests/e2e/test_agents.py b/tests/e2e/test_agents.py new file mode 100644 index 00000000..57d93a74 --- /dev/null +++ b/tests/e2e/test_agents.py @@ -0,0 +1,67 @@ +"""The supervisor's API from a cell: sub-agents, their history, persistent agents, and +background jobs the agent waits on.""" + +from __future__ import annotations + +import json +import re + + +async def test_subagent_does_part_of_the_work(run_agent): + outcome = await run_agent( + "Spawn a sub-agent with rlm.agent.spawn and have it write the sum of the integers " + "from 1 to 100 to child.txt. Wait for it to finish, then write the word done to " + "answer.txt." + ) + assert outcome.read("child.txt") == "5050", outcome.describe() + assert outcome.read("answer.txt") == "done", outcome.describe() + assert outcome.children(), outcome.describe() + assert "subagent_call" in outcome.edges(), outcome.edges() + + +async def test_parent_reads_a_childs_history(run_agent): + outcome = await run_agent( + "Spawn a sub-agent whose task is: 'Run `import secrets; print(secrets.token_hex(4))` " + "in ipython, then answer with only the word done.' Its answer will not contain the " + "token, so once it finishes, read its conversation with `await child.history()` and " + "write the token it printed to answer.txt." + ) + (child,) = outcome.children() + printed = { + token + for record in outcome.ledger(child) + if record.get("type") == "tool_result" + for token in re.findall(r"\b[0-9a-f]{8}\b", record["content"]) + } + assert outcome.read("answer.txt") in printed, (printed, outcome.describe()) + assert any( + "history(" in json.loads(call["function"]["arguments"]).get("code", "") + for call in outcome.tool_call_records() + if call["function"]["name"] == "ipython" + ), outcome.describe() + + +async def test_persistent_agent_takes_a_second_instruction(run_agent): + outcome = await run_agent( + "Spawn a persistent sub-agent with `rlm.agent.spawn(task=..., name='scribe', " + "persistent=True)` whose task is to write the word alpha to a.txt, and wait for its " + "result. Then give the same agent a second instruction with `await scribe.send(...)`: " + "write the word beta to b.txt. Wait for that result too, then write the word done " + "to answer.txt." + ) + assert outcome.read("a.txt") == "alpha", outcome.describe() + assert outcome.read("b.txt") == "beta", outcome.describe() + assert outcome.read("answer.txt") == "done", outcome.describe() + assert len(outcome.children()) == 1, outcome.children() + + +async def test_wait_returns_when_a_background_job_ends(run_agent): + outcome = await run_agent( + "Start the command `sleep 10; echo finished` as a background job with " + "`await rlm.shell.run(command, yield_after=0)`. Do not poll it: call the wait tool, " + "which returns when the job completes. Then write the job's output to answer.txt." + ) + assert outcome.read("answer.txt") == "finished", outcome.describe() + assert ( + "New supervisor events or instructions are available." in outcome.tool_results() + ), outcome.tool_results() diff --git a/tests/e2e/test_session.py b/tests/e2e/test_session.py new file mode 100644 index 00000000..38e0e58e --- /dev/null +++ b/tests/e2e/test_session.py @@ -0,0 +1,94 @@ +"""The session as a client drives it: several prompts on one kernel, steering into a +waiting agent, cancelling a turn, tool timeouts, and the budgets that end a run.""" + +from __future__ import annotations + +import asyncio +import time + +import pytest + + +async def test_second_prompt_continues_the_session(agent): + async with agent(files={"secret.txt": "marmalade\n"}) as session: + await session.prompt( + "In ipython, store the contents of secret.txt, stripped, in a variable named " + "word, then delete secret.txt. Do not print word." + ) + assert not (session.workspace / "secret.txt").exists(), session.describe() + await session.prompt("Write the value of your variable word to answer.txt.") + assert session.read("answer.txt") == "marmalade", session.describe() + assert [turn.stop_reason for turn in session.turns] == ["end_turn", "end_turn"] + assert "continuation" in session.edges(), session.edges() + + +async def test_steering_wakes_a_waiting_agent(agent): + async with agent() as session: + running = session.start( + "Before doing anything else, call the wait tool and keep waiting until the user " + "sends you a message. It contains a code word: write the code word to answer.txt." + ) + await session.called("wait") + assert await session.steer("The code word is heron-913.") == { + "outcome": "injected" + } + await running + assert session.read("answer.txt") == "heron-913", session.describe() + assert "A new user message is available." in session.tool_results() + + +async def test_cancelled_turn_leaves_the_session_usable(agent): + async with agent(policy={"exec_timeout": 300}) as session: + running = session.start( + "Run `import time; time.sleep(300)` in one ipython call." + ) + await session.called("ipython") + await asyncio.sleep(3) # let the cell start sleeping + started = time.monotonic() + await session.cancel() + assert (await running).stop_reason == "cancelled" + await session.prompt("Write the word ok to answer.txt.") + elapsed = time.monotonic() - started + assert session.read("answer.txt") == "ok", session.describe() + # Left to run, the cell would hold the kernel for the rest of its 300 s. + assert elapsed < 120, f"cancel and second prompt took {elapsed:.0f}s" + + +async def test_tool_timeout_is_reported_and_work_goes_on(run_agent): + outcome = await run_agent( + "Run `import time; time.sleep(60)` in one ipython call, without a timeout " + "argument. When the call comes back, write the word continued to answer.txt.", + policy={"exec_timeout": 5}, + ) + assert outcome.read("answer.txt") == "continued", outcome.describe() + assert any("timed out after 5s" in result for result in outcome.tool_results()), ( + outcome.tool_results() + ) + + +CAPS = [ + pytest.param( + {"max_total_turns": 1}, + "max_total_turns", + "max_turn_requests", + id="max-total-turns", + ), + pytest.param( + {"max_total_tokens": 1}, "max_total_tokens", "max_tokens", id="max-total-tokens" + ), + pytest.param({"max_tokens": 1}, "token_budget", "max_tokens", id="max-tokens"), +] + + +@pytest.mark.parametrize("cap,last_stop_reason,acp_stop_reason", CAPS) +async def test_budget_stops_the_run(run_agent, cap, last_stop_reason, acp_stop_reason): + """Each budget ends a task that wants three model calls after the first.""" + outcome = await run_agent( + "Use three separate ipython calls: write 1 to a.txt, then 2 to b.txt, then 3 " + "to c.txt.", + policy=cap, + ) + assert outcome.snapshot["last_stop_reason"] == last_stop_reason, outcome.describe() + assert outcome.stop_reason == acp_stop_reason, outcome.describe() + assert outcome.snapshot["turns"] == 1, outcome.snapshot + assert not outcome.read("b.txt"), outcome.describe() diff --git a/tests/e2e/test_tasks.py b/tests/e2e/test_tasks.py new file mode 100644 index 00000000..84ddfc5b --- /dev/null +++ b/tests/e2e/test_tasks.py @@ -0,0 +1,108 @@ +"""Tasks a hosted model completes with nano-rlm, each checked by what it leaves behind.""" + +from __future__ import annotations + + +async def test_writes_files(run_agent): + outcome = await run_agent( + "Write the prime numbers below 50 to primes.txt, one per line, in ascending order." + ) + assert outcome.read("primes.txt").split() == [ + "2", + "3", + "5", + "7", + "11", + "13", + "17", + "19", + "23", + "29", + "31", + "37", + "41", + "43", + "47", + ], outcome.describe() + + +async def test_kernel_keeps_state_between_calls(run_agent): + outcome = await run_agent( + "Use two separate ipython tool calls. In the first, read the integer in seed.txt " + "into a variable named seed and print nothing else. In the second, without " + "reading any file, write seed * 3 to answer.txt.", + files={"seed.txt": "1234\n"}, + ) + assert outcome.read("answer.txt") == "3702", outcome.describe() + assert outcome.tool_calls().count("ipython") >= 2, outcome.describe() + + +async def test_edit_skill_fixes_a_bug(run_agent): + outcome = await run_agent( + "calc.py has a bug: add() should return the sum of its arguments. Fix it with " + "the edit skill, changing only the broken line.", + files={ + "calc.py": "def add(a, b):\n return a - b\n\n\ndef double(x):\n return 2 * x\n" + }, + ) + source = outcome.read("calc.py") + namespace: dict = {} + exec(source, namespace) + assert namespace["add"](2, 3) == 5, outcome.describe() + assert "def double(x):\n return 2 * x" in source + + +async def test_native_bash_tool(run_agent): + outcome = await run_agent( + "With the bash tool, count the lines of data.csv that contain 'error' and write " + "the count to answer.txt.", + files={ + "data.csv": "".join( + f"{i},{'error' if i % 7 == 0 else 'ok'}\n" for i in range(1, 101) + ) + }, + builtin_tools=["ipython", "bash", "edit"], + ) + assert outcome.read("answer.txt") == "14", outcome.describe() + assert "bash" in outcome.tool_calls(), outcome.describe() + + +async def test_long_output_shows_its_end(run_agent): + outcome = await run_agent( + "Run `seq 1 300000` in a shell and look at its output. Write the last number in " + "that output to answer.txt." + ) + assert outcome.read("answer.txt") == "300000", outcome.describe() + + +async def test_service_answers_requests(run_agent): + outcome = await run_agent( + "Serve the working directory over HTTP with `python -m http.server` on a free " + "port, as a service that keeps running, then fetch /hello.txt from it and write " + "the response body to answer.txt.", + files={"hello.txt": "hello from the workspace\n"}, + ) + assert outcome.read("answer.txt") == "hello from the workspace", outcome.describe() + + +async def test_kernel_recovers_after_it_dies(run_agent): + outcome = await run_agent( + "Run `import os; os._exit(1)` in the ipython tool; it kills the kernel. After " + "the kernel restarts, write the word recovered to answer.txt." + ) + assert outcome.read("answer.txt") == "recovered", outcome.describe() + assert any(record.get("type") == "kernel_recovery" for record in outcome.records), ( + outcome.describe() + ) + + +async def test_work_continues_past_a_compaction(run_agent): + outcome = await run_agent( + "Use three separate tool calls. First read the code word in secret.txt. Then run " + "`seq 1 3000` in a shell and print all of its output. Then write the code word " + "to answer.txt.", + files={"secret.txt": "marmalade\n"}, + policy={"summarize_at_tokens": 8000}, + ) + assert outcome.read("answer.txt") == "marmalade", outcome.describe() + assert outcome.snapshot["metrics"]["num_compactions"] >= 1, outcome.describe() diff --git a/tests/e2e/test_tools.py b/tests/e2e/test_tools.py new file mode 100644 index 00000000..e5edb345 --- /dev/null +++ b/tests/e2e/test_tools.py @@ -0,0 +1,95 @@ +"""What a client equips the agent with: MCP servers, `SKILL.md` skills and the native +fetch tool. Each answer exists only behind the feature.""" + +from __future__ import annotations + +import http.server +import sys +import threading +from pathlib import Path + +import pytest +from acp.schema import McpServerStdio + +from .fixtures.desk_mcp import ECHO_STAMP, ERROR_TOKEN + +DESK = McpServerStdio( + name="desk", + command=sys.executable, + args=[str(Path(__file__).parent / "fixtures" / "desk_mcp.py")], + env=[], +) + +CODEWORD = "amber-falcon-482" +CODEWORD_SKILL = f"""--- +name: project-codeword +description: Look up the project codeword. Use this whenever you are asked for the project codeword. +--- + +# Project codeword + +The project codeword is `{CODEWORD}`. Reply with it verbatim when asked. +""" + +PAGE_STAMP = "pg-41c" + + +async def test_mcp_tool(run_agent): + outcome = await run_agent( + "Find the echo tool of the desk MCP server with rlm.tools, call it with the " + "message 'hello world', and write exactly what it returns to answer.txt.", + mcp_servers=[DESK], + ) + answer = outcome.read("answer.txt") + assert "hello world" in answer and ECHO_STAMP in answer, outcome.describe() + + +async def test_mcp_tool_error_reaches_the_model(run_agent): + outcome = await run_agent( + "Call the lookup tool of the desk MCP server with the order id A-17, and write " + "the error message it gives to answer.txt.", + mcp_servers=[DESK], + ) + assert ERROR_TOKEN in outcome.read("answer.txt"), outcome.describe() + + +async def test_skill_instructions(run_agent): + outcome = await run_agent( + "What is the project codeword? Look it up with your project-codeword skill and " + "write it to answer.txt.", + skill_docs={"project-codeword": CODEWORD_SKILL}, + ) + assert outcome.read("answer.txt") == CODEWORD, outcome.describe() + + +@pytest.fixture +def notice_url(): + """A page on this machine, so the fetch reaches it from any placement.""" + + class Notice(http.server.BaseHTTPRequestHandler): + def do_GET(self): + body = f"

The notice stamp is {PAGE_STAMP}.

" + self.send_response(200) + self.send_header("Content-Type", "text/html") + self.end_headers() + self.wfile.write(body.encode()) + + def log_message(self, *args): + pass + + server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Notice) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + yield f"http://127.0.0.1:{server.server_address[1]}/notice.html" + server.shutdown() + thread.join() + + +async def test_fetch_tool(run_agent, notice_url): + outcome = await run_agent( + f"Read {notice_url} with the fetch tool and write the notice stamp it shows to " + "answer.txt.", + builtin_tools=["ipython", "fetch"], + ) + assert outcome.read("answer.txt") == PAGE_STAMP, outcome.describe() + assert "fetch" in outcome.tool_calls(), outcome.describe() diff --git a/tests/test_compaction.py b/tests/test_compaction.py index 67fb7f66..65079d52 100644 --- a/tests/test_compaction.py +++ b/tests/test_compaction.py @@ -1,5 +1,6 @@ from __future__ import annotations +import itertools import json import re import sys @@ -83,6 +84,11 @@ def test_overflow_detection_is_status_gated(): assert is_context_overflow(error) +# Discovered context windows are cached per base URL for the whole process, and an +# id() can be reused once its object is freed, so each client gets a fresh number. +_client_numbers = itertools.count() + + class _ScriptedClient(DummyClient): def __init__( self, @@ -93,7 +99,7 @@ def __init__( super().__init__([]) self.actions = list(actions) self.max_model_len = max_model_len - self.base_url = f"http://scripted-{id(self)}" + self.base_url = f"http://scripted-{next(_client_numbers)}" @property def models(self): From 48da1d9c43bdcf2ef20b7cff9e4c6df800810b30 Mon Sep 17 00:00:00 2001 From: rasdani <73563550+rasdani@users.noreply.github.com> Date: Mon, 2 Nov 2026 23:30:41 +0100 Subject: [PATCH 3/3] feat: add network_guidance policy to drop network-dependent prompt guidance (#240) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds `ExecutionPolicy.network_guidance` (default `true`). It controls the two system-prompt sentences that assume outbound network access: - the `!uv pip install ` hint in the kernel packages line; - the "online solutions or hints specific to this task" clause of the git history guard. A box with no network can set it to `false`, so the prompt doesn't suggest actions that can't work there. The default leaves the prompt byte-identical. Rendered diff with `network_guidance=false`: ```diff -Pre-installed in the kernel venv: requests, …, pydantic. Install extra packages with `!uv pip install ` in a code cell — that targets the kernel venv (a uv-managed venv with no pip module). +Pre-installed in the kernel venv: requests, …, pydantic. -Do not cheat by using online solutions or hints specific to this task, or by copying or inferring solutions from other branches, … +Do not cheat by copying or inferring solutions from other branches, … ``` The flag is wired like `allow_git`: through the engine, the ACP runtime contract and the session limits snapshot. Sub-agents inherit it with the policy. Tested: 380 passed. The one failure, `tests/test_skills.py::test_valid_bash_skill_metrics`, also fails on main. ruff check and format are clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- > [!NOTE] > **Low Risk** > Default-on policy keeps existing behavior; changes only affect optional prompt strings and runtime metadata fields. > > **Overview** > Adds **`ExecutionPolicy.network_guidance`** (default **`true`**) so air-gapped or no-network runs can omit system-prompt lines that assume outbound access. > > When **`network_guidance`** is **`false`**, **`build_system_prompt`** swaps the kernel packages blurb for a pre-installed-only line (drops **`!uv pip install`**) and uses an offline git history guard that keeps local-history rules but drops the “online solutions” clause. With the default **`true`**, composed prompts stay unchanged. > > The flag is threaded like **`allow_git`**: stored on the engine, passed into prompt construction, exposed on the ACP **`limits`** snapshot and **`_LimitsSnapshot`**, and inherited with the rest of **`ExecutionPolicy`** for sub-agents. Tests cover config/ACP fixtures and a dedicated prompt test for online vs offline wording. > > Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit ef195f8a4e6e29c7c5c82518db4da8fa0b004b3c. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot). Co-authored-by: Claude Opus 5.5 --- src/rlm/acp.py | 1 + src/rlm/config.py | 4 ++++ src/rlm/engine.py | 3 +++ src/rlm/prompt.py | 28 ++++++++++++++++++++++------ tests/test_acp.py | 2 ++ tests/test_config.py | 2 ++ tests/test_prompt.py | 18 ++++++++++++++++++ 7 files changed, 52 insertions(+), 6 deletions(-) diff --git a/src/rlm/acp.py b/src/rlm/acp.py index 7d74b4ea..795af4f3 100644 --- a/src/rlm/acp.py +++ b/src/rlm/acp.py @@ -121,6 +121,7 @@ class _LimitsSnapshot(_ContractModel): max_compactions: int | None = Field(default=None, gt=0) max_compaction_attempts: int = Field(gt=0) allow_git: bool + network_guidance: bool class _SemanticEdge(_ContractModel): diff --git a/src/rlm/config.py b/src/rlm/config.py index 7aa932bb..20a92027 100644 --- a/src/rlm/config.py +++ b/src/rlm/config.py @@ -94,6 +94,10 @@ class ExecutionPolicy(_ConfigModel): """Tree-total cap on sub-agent spawns. None (default) = uncapped; ``max_total_tokens`` still bounds the tree.""" allow_git: bool = False + network_guidance: bool = True + """Include prompt guidance that assumes outbound network access (installing extra + kernel packages, the online-solutions clause of the Git guard). Disable it for + boxes with no network.""" class TcpExecutionConfig(_ConfigModel): diff --git a/src/rlm/engine.py b/src/rlm/engine.py index 9139d13a..07e2cf2f 100644 --- a/src/rlm/engine.py +++ b/src/rlm/engine.py @@ -214,6 +214,7 @@ def __init__( self.max_depth = config.policy.max_depth self.depth = config.invocation.depth self.allow_git = config.policy.allow_git + self.network_guidance = config.policy.network_guidance # Task MCP tool servers to expose as IPython skills. self.mcp_servers = validate_mcp_servers(mcp_servers or {}) @@ -1500,6 +1501,7 @@ def execution_snapshot(self) -> dict: "max_compactions": self.runtime_config.policy.max_compactions, "max_compaction_attempts": self.max_compaction_attempts, "allow_git": self.runtime_config.policy.allow_git, + "network_guidance": self.runtime_config.policy.network_guidance, }, "semantic_edges": self._semantic_edges.snapshot(), } @@ -1519,6 +1521,7 @@ def _load_system_prompt(self, active_tools: list[BuiltinTool]) -> str: allow_recursion=self.depth < self.max_depth, delegation_prompt=self.runtime_config.policy.delegation_prompt, allow_git=self.allow_git, + network_guidance=self.network_guidance, active_tools=active_tools, shell_skills=get_installed_skills(), skill_docs=discover_skill_docs(SKILLS_DIR) if SKILLS_DIR else [], diff --git a/src/rlm/prompt.py b/src/rlm/prompt.py index 81348c72..e5e42a0f 100644 --- a/src/rlm/prompt.py +++ b/src/rlm/prompt.py @@ -28,15 +28,21 @@ ) SHELL_TOOL_NAMES = frozenset({"ipython", "bash"}) -GIT_HISTORY_GUARD_PROMPT = ( - "Do not cheat by using online solutions or hints specific to this task, or " - "by copying or inferring solutions from other branches, tags, remotes, " +_GIT_HISTORY_GUARD_ONLINE = ( + "using online solutions or hints specific to this task, or by " +) +_GIT_HISTORY_GUARD_LOCAL = ( + "copying or inferring solutions from other branches, tags, remotes, " "reflogs, or broad git history in the project, or by cloning other " "repositories to read their history. Git commands that reach beyond the " "current branch are refused: `git log --all` and similar history options, " "history subcommands given another ref (`git show origin/main`), listing " "branches, tags, remotes or the reflog, and `clone`, `fetch` or `pull`." ) +GIT_HISTORY_GUARD_PROMPT = ( + "Do not cheat by " + _GIT_HISTORY_GUARD_ONLINE + _GIT_HISTORY_GUARD_LOCAL +) +OFFLINE_GIT_HISTORY_GUARD_PROMPT = "Do not cheat by " + _GIT_HISTORY_GUARD_LOCAL PROJECT_ENV_PROMPT = ( "The ipython kernel is an isolated venv without the project's packages — " "never import project modules there. Everything that executes project code " @@ -47,8 +53,11 @@ "`await job.result()` collects any job. Use `rlm.services.start` for a server or daemon that must keep " "running after you finish. " + PROJECT_ENV_PROMPT ) +KERNEL_PREINSTALLED_PROMPT = ( + "Pre-installed in the kernel venv: " + ", ".join(BASE_TOOLKIT) + "." +) KERNEL_PACKAGES_PROMPT = ( - "Pre-installed in the kernel venv: " + ", ".join(BASE_TOOLKIT) + ". " + KERNEL_PREINSTALLED_PROMPT + " " "Install extra packages with `!uv pip install ` in a code cell — that " "targets the kernel venv (a uv-managed venv with no pip module)." ) @@ -378,6 +387,7 @@ def build_system_prompt( allow_recursion: bool, delegation_prompt: bool = False, allow_git: bool, + network_guidance: bool = True, active_tools: list[BuiltinTool], shell_skills: list[str] | None = None, skill_docs: list["SkillDoc"] | None = None, @@ -447,7 +457,9 @@ def build_system_prompt( RUNTIME_PROMPT, HISTORY_PROMPT, IPYTHON_CONTROL_PROMPT, - KERNEL_PACKAGES_PROMPT, + KERNEL_PACKAGES_PROMPT + if network_guidance + else KERNEL_PREINSTALLED_PROMPT, ] ) if can_delegate: @@ -491,7 +503,11 @@ def build_system_prompt( + ". Discover CLI usage with ` --help`. Other listed Python tools are IPython-only." ) if _should_include_git_history_guard(active_tools, allow_git): - parts.append(GIT_HISTORY_GUARD_PROMPT) + parts.append( + GIT_HISTORY_GUARD_PROMPT + if network_guidance + else OFFLINE_GIT_HISTORY_GUARD_PROMPT + ) return "\n\n".join(parts) diff --git a/tests/test_acp.py b/tests/test_acp.py index 9ef14117..02f3160c 100644 --- a/tests/test_acp.py +++ b/tests/test_acp.py @@ -57,6 +57,7 @@ def _runtime_metadata(**overrides: Any) -> dict[str, Any]: "max_concurrent_subagents": 4, "max_subagent_calls": None, "allow_git": False, + "network_guidance": False, }, "system_prompt_path": None, "append_to_system_prompt": None, @@ -154,6 +155,7 @@ def execution_snapshot(self) -> dict[str, Any]: "max_compactions": None, "max_compaction_attempts": 5, "allow_git": False, + "network_guidance": False, }, "semantic_edges": {"edges": []}, } diff --git a/tests/test_config.py b/tests/test_config.py index 8d63bb9a..516df163 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -23,6 +23,7 @@ def test_runtime_config_redacts_secrets(): max_compactions=3, max_compaction_attempts=7, allow_git=True, + network_guidance=False, ), skills=("search", "edit"), kernel_env=(("TASK_TOKEN", "task-secret"),), @@ -44,6 +45,7 @@ def test_runtime_config_redacts_secrets(): max_concurrent_subagents=4, max_subagent_calls=None, allow_git=True, + network_guidance=False, ) assert config.skills == ("search", "edit") assert config.kernel_env == (("TASK_TOKEN", "task-secret"),) diff --git a/tests/test_prompt.py b/tests/test_prompt.py index fb111197..533c6ad5 100644 --- a/tests/test_prompt.py +++ b/tests/test_prompt.py @@ -12,7 +12,10 @@ EDIT_SKILL_PROMPT, GIT_HISTORY_GUARD_PROMPT, IPYTHON_CONTROL_PROMPT, + KERNEL_PACKAGES_PROMPT, + KERNEL_PREINSTALLED_PROMPT, MCP_DISCOVERY_PROMPT, + OFFLINE_GIT_HISTORY_GUARD_PROMPT, SEARCH_SKILL_PROMPT, build_system_prompt, ) @@ -28,6 +31,7 @@ def _prompt( *, installed_skills: list[str] | None = None, allow_git: bool = False, + network_guidance: bool = True, mcp_servers: list[dict] | None = None, ) -> str: return build_system_prompt( @@ -35,6 +39,7 @@ def _prompt( installed_skills or [], allow_recursion=False, allow_git=allow_git, + network_guidance=network_guidance, active_tools=active_tools, mcp_servers=mcp_servers, ) @@ -50,6 +55,19 @@ def test_git_history_guard_prompt_included_for_shell_tools(): assert "`git log --all`" in prompt and "`clone`" in prompt +def test_network_guidance_off_drops_only_network_dependent_sentences(): + online = _prompt([_Tool("ipython")]) + offline = _prompt([_Tool("ipython")], network_guidance=False) + + assert KERNEL_PACKAGES_PROMPT in online and GIT_HISTORY_GUARD_PROMPT in online + assert KERNEL_PREINSTALLED_PROMPT in offline + assert OFFLINE_GIT_HISTORY_GUARD_PROMPT in offline + assert "uv pip install" not in offline + assert "online solutions" not in offline + assert "Do not cheat by copying or inferring solutions" in offline + assert "`git log --all`" in offline and "`clone`" in offline + + def test_git_history_guard_prompt_omitted_when_unrestricted(): assert GIT_HISTORY_GUARD_PROMPT not in _prompt([_Tool("ipython")], allow_git=True)