feat(code): classifier-backed Auto approval mode - #4804
Merged
Conversation
Contributor
There was a problem hiding this comment.
Auto mode's _fixed_repo_command_allowed regex omits the single & shell control operator, allowing an attacker-influenced execute tool call like pytest & rm -rf . to bypass deterministic approval and run an arbitrary background command without human review.
Contributor
There was a problem hiding this comment.
The new deterministic allow logic in Auto mode unconditionally bypasses classifier review for build/test/package-manager commands (make, npm/yarn/pnpm, pytest, cargo, go, uv run) that execute arbitrary project-defined code, creating an unreviewed code-execution path for users working against a malicious or compromised repository when Auto is enabled.
…-auto-mode # Conflicts: # libs/code/deepagents_code/tui/widgets/messages.py # libs/code/tests/unit_tests/tui/test_textual_adapter.py
Mason Daugherty (mdrxy)
pushed a commit
that referenced
this pull request
Jul 17, 2026
> [!CAUTION] > Merging this PR will automatically publish to **PyPI** and create a **GitHub release**. For the full release process, see [`.github/RELEASING.md`](https://github.com/langchain-ai/deepagents/blob/main/.github/RELEASING.md). --- _Release notes preview: keep this section in sync with the package `CHANGELOG.md`. The published GitHub release body is extracted from the merged `CHANGELOG.md` by `release.yml`, not from this PR description._ --- ## [0.1.43](deepagents-code==0.1.42...deepagents-code==0.1.43) (2026-07-17) ### Features - Added classifier-backed Auto approval mode behind `DEEPAGENTS_CODE_EXPERIMENTAL=1` ([#4804](#4804)). - Added a shutdown toast for deferred exits ([#4830](#4830)). - Task descriptions that were truncated can now be expanded by clicking or pressing `Ctrl+O` ([#4811](#4811)). - Debug Console clears with `Ctrl+L` now persist after reopening ([#4812](#4812)). - Added debug logging for skill-name override collisions ([#4772](#4772)). ### Bug Fixes - Keep chat input responsive during `/restart` ([#4808](#4808)). - Fixed paste placeholders disappearing when backspacing a newline below them ([#4757](#4757)). - Made markdown `AppMessage` output selectable ([#4814](#4814)). - Fixed live tool-group counts to include only running tools ([#4809](#4809)). - Kept `task` timers monotonic across nested subagent human-in-the-loop flows ([#4771](#4771)). - Preserved goal criteria proposals when marker clearing fails ([#4785](#4785)). - Reduced repeated probing of an unreachable Ollama daemon to once per reload ([#4806](#4806)). - Quieted MCP auth-skip debug logging for known patterns ([#4805](#4805)). - Improved `/version` diagnostics for editable installs and core dependency reporting, including surfacing `langchain-quickjs` ([#4816](#4816), [#4813](#4813)). - Removed the `uv install` tip from the `/version` update hint ([#4822](#4822)). _End release notes preview._ --- > [!NOTE] > A **New Contributors** section is appended to the GitHub release notes automatically at publish time (see [Release Pipeline](https://github.com/langchain-ai/deepagents/blob/main/.github/RELEASING.md#release-pipeline), step 2). --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: langchain-oss-automated-triage[bot] <248757908+langchain-oss-automated-triage[bot]@users.noreply.github.com>
17 tasks
Mason Daugherty (mdrxy)
added a commit
to langchain-ai/docs
that referenced
this pull request
Jul 21, 2026
…4993) Documents the Manual/Auto/YOLO approval modes shipped in `deepagents-code` 0.1.42 ([langchain-ai/deepagents#4804](langchain-ai/deepagents#4804)).
Marcelo5444
pushed a commit
to Marcelo5444/deepagents
that referenced
this pull request
Jul 30, 2026
Deep Agents Code now offers an opt-in **Auto** approval mode. Auto
immediately runs a narrow, deterministic set of routine actions. For
other approval-gated actions, the active model checks whether they match
the user's literal request. It runs approved actions and blocks the
rest. After repeated denials or classifier failures, Auto sends the next
batch to the normal approval UI, then continues in Auto mode.
Auto is experimental and currently limited to interactive, local TUI
sessions using `FilesystemBackend`. Manual remains the default.
> [!WARNING]
> Auto is an authorization heuristic for a local coding agent. It is
**not** sandbox containment, an operating-system boundary, or a
guarantee that model-generated actions are safe.
## Enable Auto
1. Set `DEEPAGENTS_CODE_EXPERIMENTAL=1`.
2. Select Auto in one of these ways:
- In-app: use <kbd>Shift</kbd>+<kbd>Tab</kbd> or
<kbd>Ctrl</kbd>+<kbd>T</kbd> to toggle between Manual and Auto.
- At launch: pass `-y` or `--auto-approve`.
- By default: set `[startup].mode = "auto"` in
`~/.deepagents/config.toml`.
`-y` and `--auto-approve` now select Auto rather than unrestricted
execution. Users who want the previous unrestricted behavior must choose
`--yolo`. The removed `dangerously-auto` startup value is no longer
recognized and falls back to Manual.
---
## Why
The previous approval modes forced a binary choice:
- **Manual** preserved a human checkpoint, but repeatedly interrupted
routine coding work.
- **YOLO** removed that checkpoint entirely, including for actions that
exceeded the user's request (blanket approval).
Auto introduces a third policy: keep the low-friction path for clearly
routine work, but make authorization explicit for potentially
consequential actions.
| Mode | Policy | Entry points | Availability |
|---|---|---|---|
| **Manual** | Uses the normal human approval UI for gated actions |
Default; `[startup].mode = "manual"` | All interactive sessions |
| **Auto** | Uses narrow deterministic rules, then classifier review,
with human fallback after repeated denials or failures | `-y`,
`--auto-approve`, `[startup].mode = "auto"`, or the live keyboard toggle
| Experimental, local, unsandboxed TUI sessions |
| **YOLO** | Runs gated actions without review | `--yolo` or
`[startup].mode = "yolo"` | Interactive sessions after a one-time
acknowledgement |
## How Auto handles a proposed action
```mermaid
flowchart TD
A[Model proposes tool calls] --> B{Covered by the approval policy?}
B -->|No| X[Existing tool behavior is unchanged]
B -->|Yes| S{Is this thread still in Auto, with readable denial/failure history?}
S -->|No| J[Open the existing approval HITL widget]
S -->|Yes| C{Narrow deterministic allow?}
C -->|Yes| R[Execute without a classifier call]
C -->|No| D[Build one structured decision batch]
D --> H{Was this batch already reviewed, or do earlier denials/failures require human review?}
H -->|Yes| J
H -->|No| E[Active model reviews effects against user request/input]
E -->|Allow| R
E -->|Deny| Q{Total-denial threshold reached?}
Q -->|No| F[Return a sanitized error result]
Q -->|Yes| J
E -->|Unavailable or invalid| G[Return a compact unavailable result]
F --> I[Agent can revise its plan]
G --> I
J -->|Approve| R
J -->|Reject| K[Return a rejection result]
J -->|Switch to Manual| L[Persist Manual, then review the full gated batch]
R --> M[Reconcile the result and continue]
K --> M
I --> M
L --> M
```
In practice:
1. **Auto starts with Manual's existing human-in-the-loop (HITL)
rules.** It changes only how HITL-gated actions are reviewed.
- These built-in tools are not HITL-gated: `ls`, `read_file`, `glob`,
`grep`, `ask_user`, `get_goal`, `get_rubric`, and `update_goal`.
- MCP tools with valid read-only annotations, the optional `js_eval`
tool, and caller-supplied tools without HITL configuration continue to
run normally.
2. **Only narrow cases resolve locally.** The deterministic policy can
allow, never deny:
- A routine write such as `src/parser.py` can proceed based on its
resolved path and suffix; a sensitive target such as
`.github/workflows/ci.yml` goes to classifier review.
- A read-only Git command such as `git status` can proceed; a mutating
command such as `git commit -m change` goes to classifier review.
Explicit shell allow-list entries still apply after Auto removes broad
and wildcard rules.
- An MCP tool with a valid read-only annotation can proceed; tools
without read-only annotations, or with contradictory or malformed
annotations, go to classifier review.
<details>
<summary>How MCP read-only annotations are evaluated</summary>
MCP servers can attach standard `ToolAnnotations` when advertising a
tool. Deep Agents Code copies those annotations into the wrapped tool's
metadata and lets the tool bypass classifier review only when:
- `readOnlyHint` is the literal Boolean `true`;
- `destructiveHint` is absent, `null`, or `false`; and
- every supplied standard hint (`readOnlyHint`, `destructiveHint`,
`idempotentHint`, or `openWorldHint`) is a Boolean or `null`, rather
than a string or another type.
For example:
- `readOnlyHint: true, destructiveHint: false` proceeds without
classifier review.
- No `readOnlyHint` goes to review because the tool never explicitly
claimed to be read-only.
- `readOnlyHint: true, destructiveHint: true` goes to review because the
hints contradict each other.
- `readOnlyHint: true, destructiveHint: "false"` goes to review because
`"false"` is a string, not a Boolean.
In Auto, tools that do not pass this check enter the classifier batch.
In Manual, they use the normal approval UI; headless runtimes reject
them because they have no approval UI. The annotation is a
server-provided assertion that Deep Agents Code does not independently
verify.
</details>
3. **The classifier reviews all unresolved calls together.** Only
pre-expansion literal user text can authorize an action. Referenced
paths, summarized arguments, prior tool calls, and MCP metadata provide
effect context but cannot grant permission. Expanded files, assistant
prose, and tool results are excluded.
- **Example:** For the literal request `Research issue langchain-ai#123, update
@src/parser.py, and open a PR`, proposed `fetch_url` and `git push`
calls can be reviewed in one batch and receive separate decisions. The
referenced path and summarized call arguments help identify what each
action would affect, but the expanded file contents are not sent.
Instructions found in that file, assistant prose, or a prior tool result
therefore cannot widen the user's request or authorize another action.
4. **Auto revalidates before side effects.** The decision plan is bound
to the thread, mode, batch, and exact gated calls. Missing or invalid
state, a mode race, or a replay falls back to human review. Denied calls
are skipped and receive sanitized error results so the agent can revise
its plan.
- **Examples:** If the user switches to Manual while classifier review
is in progress, an earlier Auto decision cannot execute silently; the
normal approval UI opens instead. A plan associated with another thread,
a different set of tool-call IDs, missing decisions, or a previously
processed batch also requires human review. If Auto denies a proposed
`delete`, it returns an error result with that call's ID instead of
invoking the tool, allowing the agent to propose a safer next step.
## Limits and user-visible behavior
- The Manual approval menu can enable Auto for the current thread.
Threshold fallback can switch permanently to Manual or perform a one-off
review while leaving Auto enabled.
- Denials, classifier failures, and fallbacks appear as short sanitized
transcript events rather than raw classifier output.
- Auto requested without the experimental opt-in, or in a sandboxed
session, falls back to Manual with a warning. Explicit autonomous-mode
flags are rejected in non-interactive and ACP entry points.
- The active model is not an independent security authority. MCP
read-only annotations are trusted as a deliberate beta tradeoff.
- Parent-level Auto review does not cover actions performed inside
delegated subagents or broader explicitly configured PTC/`js_eval`
fan-out. Model providers and tracing backends may still observe
classifier inputs and outputs even though the TUI hides them.
<details>
<summary><strong>Implementation and reviewer details</strong></summary>
### Approval boundary and decision inputs
The following tools are HITL-gated:
- `execute`, `write_file`, `edit_file`, and `delete`;
- `web_search` and `fetch_url`;
- `task`;
- `start_async_task`, `update_async_task`, and `cancel_async_task`;
- `compact_conversation` when optional compaction approval is enabled;
and
- MCP tools without valid read-only annotations.
For deterministic writes and edits, the fast path uses only the resolved
target path and suffix; it does not inspect proposed content. Dependency
and lock files, sensitive names and locations, shell scripts,
unsupported file types, and paths outside the trusted root do not
qualify. Selected Git inspection commands must also pass shell-control
and outside-worktree path checks. Configured shell entries remain an
explicit user trust exception, but broad commands and wildcard entries
are ignored by Auto.
The classifier receives one typed batch containing:
- up to 20 client-attached, pre-expansion prompt rows, including literal
prompts, referenced path names, and turn IDs;
- trusted project-root and redacted-origin facts; and
- depth- and length-limited effect context from current arguments, prior
AI tool calls after the latest trusted prompt, and selected MCP
annotations.
Only `literal_user_text` is authorizing. Content-bearing write/edit
arguments are reduced to character counts. The classifier must return
exactly one unique decision for every unresolved tool-call ID; missing,
duplicated, unknown, or malformed decisions make the classifier
unavailable rather than partially authorizing the batch.
Before routing calls, Auto re-reads the live mode and validates its
private decision plan. Manual mode, invalid control state, mismatched
calls, or replayed batches require human review. Classifier review uses
a second call to the active session model with a 20-second timeout. Raw
classifier text and tool-call chunks are hidden from the transcript,
while usage accounting and tracing are retained.
Denial reasons are sanitized before persistence or display by removing
control characters, known credential values, credential-like
assignments, and URL credentials or query values. MCP tools without
valid read-only annotations are HITL-gated; headless runtimes reject
them because no approval UI is available.
### Failure and fallback behavior
| Condition | Result |
|---|---|
| Selected mode cannot be persisted | Persist Manual instead; if that
also fails, block graph execution |
| Classifier timeout, provider error, or invalid response | Block the
affected calls and return a compact unavailable result |
| Three consecutive policy denials in one user turn | Send the next
classifier-review-eligible calls to one-off human review |
| Two consecutive classifier-unavailable results | Send the next
classifier-review-eligible calls to one-off human review |
| A denial raises the thread's total denial count to 20 | Require human
review for that call immediately |
| Counter state cannot be read or reset before classification | Require
human review for every gated call in the batch |
| Counter state cannot be written after classification | Require human
review for classifier-path decisions; deterministic fast-path allows
remain eligible |
| Decision plan is missing, malformed, replayed, or bound to another
thread | Require Manual review |
| User chooses **Switch to Manual** during fallback | Persist Manual
before presenting the full gated batch for review |
### Python API and state compatibility
- Existing Boolean compatibility input retains its previous meaning:
`auto_approve=True` maps to YOLO, not Auto.
- Per-thread Store records persist the typed mode instead of a Boolean.
Missing, legacy, or malformed records fail closed.
- Startup and live mode changes take effect only after the Store write
succeeds.
- YOLO requires a versioned local acknowledgement and cannot be entered
through the live keyboard toggle.
### Reviewer guide
Suggested review order:
1. **Mode semantics and persistence** — `ApprovalMode`, Store payloads,
acknowledgement, startup resolution, and live mode writes.
2. **Authorization policy** — `AutoModeHITLMiddleware`, literal prompt
metadata, deterministic routing, structured classification,
sanitization, counters, and checkpoint validation.
3. **Graph boundary** — middleware replacement, the gated-tool
inventory, MCP annotation handling, headless rejection, PTC, and
delegated-subagent scope.
4. **Client behavior** — stream metadata, classifier-output filtering,
fallback interrupts, keyboard toggles, status presentation, and
transcript events.
Highest-risk invariants:
- every consequential tool in the supported boundary is gated or
explicitly outside scope;
- path-based writes and selected Git fast paths stay within their
documented checks;
- only literal user prompts can authorize classifier-reviewed effects;
- malformed output, Store failures, mode races, and checkpoint replay
fail closed; and
- hidden classifier chunks never create transcript content while usage
accounting remains intact.
</details>
Marcelo5444
pushed a commit
to Marcelo5444/deepagents
that referenced
this pull request
Jul 30, 2026
> [!CAUTION] > Merging this PR will automatically publish to **PyPI** and create a **GitHub release**. For the full release process, see [`.github/RELEASING.md`](https://github.com/langchain-ai/deepagents/blob/main/.github/RELEASING.md). --- _Release notes preview: keep this section in sync with the package `CHANGELOG.md`. The published GitHub release body is extracted from the merged `CHANGELOG.md` by `release.yml`, not from this PR description._ --- ## [0.1.43](langchain-ai/deepagents@deepagents-code==0.1.42...deepagents-code==0.1.43) (2026-07-17) ### Features - Added classifier-backed Auto approval mode behind `DEEPAGENTS_CODE_EXPERIMENTAL=1` ([langchain-ai#4804](langchain-ai#4804)). - Added a shutdown toast for deferred exits ([langchain-ai#4830](langchain-ai#4830)). - Task descriptions that were truncated can now be expanded by clicking or pressing `Ctrl+O` ([langchain-ai#4811](langchain-ai#4811)). - Debug Console clears with `Ctrl+L` now persist after reopening ([langchain-ai#4812](langchain-ai#4812)). - Added debug logging for skill-name override collisions ([langchain-ai#4772](langchain-ai#4772)). ### Bug Fixes - Keep chat input responsive during `/restart` ([langchain-ai#4808](langchain-ai#4808)). - Fixed paste placeholders disappearing when backspacing a newline below them ([langchain-ai#4757](langchain-ai#4757)). - Made markdown `AppMessage` output selectable ([langchain-ai#4814](langchain-ai#4814)). - Fixed live tool-group counts to include only running tools ([langchain-ai#4809](langchain-ai#4809)). - Kept `task` timers monotonic across nested subagent human-in-the-loop flows ([langchain-ai#4771](langchain-ai#4771)). - Preserved goal criteria proposals when marker clearing fails ([langchain-ai#4785](langchain-ai#4785)). - Reduced repeated probing of an unreachable Ollama daemon to once per reload ([langchain-ai#4806](langchain-ai#4806)). - Quieted MCP auth-skip debug logging for known patterns ([langchain-ai#4805](langchain-ai#4805)). - Improved `/version` diagnostics for editable installs and core dependency reporting, including surfacing `langchain-quickjs` ([langchain-ai#4816](langchain-ai#4816), [langchain-ai#4813](langchain-ai#4813)). - Removed the `uv install` tip from the `/version` update hint ([langchain-ai#4822](langchain-ai#4822)). _End release notes preview._ --- > [!NOTE] > A **New Contributors** section is appended to the GitHub release notes automatically at publish time (see [Release Pipeline](https://github.com/langchain-ai/deepagents/blob/main/.github/RELEASING.md#release-pipeline), step 2). --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: langchain-oss-automated-triage[bot] <248757908+langchain-oss-automated-triage[bot]@users.noreply.github.com>
Mason Daugherty (mdrxy)
added a commit
that referenced
this pull request
Jul 31, 2026
Closes #4519 Deep Agents Code now keeps a running estimated USD total for the active thread. The status bar shows it beside token usage, `/cost` and `/tokens` break it down on demand, and usage summaries include per-model cost. The total covers everything the thread spends — assistant turns, subagents, offload/compaction, and Auto mode's classifier — and it is restored on resume and thread switches. --- ## Where the number lives A thread's lifetime cost should be durable to support `--resume` and switching between threads. Consequently, the graph owns it and the client (e.g. TUI / headless CLI) can read it. Concretely, `_session_cost_usd` is a new channel on the agent's state schema (`CostState`, which extends `ResumeState`), which means it is persisted by the graph's checkpointer, in the same checkpoint as `messages`. > <details> > <summary>FAQ: <code>CostState</code>, and why it extends <code>ResumeState</code></summary> > > `CostState` is the [state schema](https://docs.langchain.com/oss/python/langgraph/graph-api#schema) `CostTrackingMiddleware` declares — the agent's state plus one channel: > > ```python > class CostState(ResumeState): > _session_cost_usd: Annotated[NotRequired[float], PrivateStateAttr, operator.add] > ``` > > **What `ResumeState` already holds.** It's the schema for facts that belong to the thread as a whole rather than to any one message — the things the TUI/client needs the moment you open a thread, before the agent runs again: > > - `_context_tokens` — how full the context window was after the last turn. This drives token readout — e.g., the display in the TUI's footer status bar, where the cost figure now also sits. > - `_model_spec` / `_model_params` — the model and settings the thread was using, so `-r` resumes on _that_ model instead of whatever your current [global default](https://docs.langchain.com/oss/deepagents/code/config-file#default-and-recent-model) might happen to be. > - `_goal_objective`, `_goal_rubric`, `_goal_status`, `_goal_status_note`, `_sticky_rubric`, and the pending-proposal fields — the [goal](https://docs.langchain.com/oss/deepagents/code/goals-and-rubrics) the thread is working toward (if one is set) and where progress on said goal stands. > > **Every field here is private, including the new one.** A graph has a public interface: its input and output schemas say what callers may pass in and receive back (`messages` is the obvious example). None of the fields above belong in that interface, as they aren't things a caller supplies or asks for — they're notes that dcode writes for itself so it can pick the thread back up later. > > `PrivateStateAttr` omits a field from those public input/output schemas. It does **not** make the field ephemeral or unreadable: the channel remains ordinary checkpointed state, is available to graph nodes, and is returned by `get_state`. Its writes can also appear as raw deltas in the `updates` stream; the live status bar uses a dedicated absolute-total event instead so the client does not have to reproduce the reducer (details below). > > **These values are stored rather than recomputed**, because recomputing them is expensive or impossible after the fact. Reopening a thread has to show the context gauge and the current model immediately. The context number originally came from the last response's `usage_metadata` — the provider counted it for us. If it weren't saved, the CLI would have to load the whole message history and count tokens itself to guess at it: slow, dependent on having a tokenizer that matches the provider's, and still only an approximation of what was actually billed. Cost is the same shape of problem and worse — re-deriving it would mean re-pricing every message in the thread every time you open it. > > **So why extend `ResumeState`?** Cost is the same kind of resumable thread fact, and pricing also needs the model information that `ResumeState` already carries. LangChain's `AIMessage` contract does not require a model identifier: `usage_metadata` standardizes token counts, while `response_metadata` is optional. Provider integrations commonly include a model name, but custom and third-party integrations are allowed to omit it. When a response does, pricing falls back to the `_model_spec` persisted for that turn. > > </details> Riding the checkpoint is a deliberate choice over an alternative where the client prices a turn and then pushes the new total with `aupdate_state`. (That's a second round trip that can be interrupted — e.g. `Ctrl-C`, a crash, a dropped connection — *after* the messages have already been committed.) Writing from inside the graph puts the cost in the same commit as the response that incurred it. Importantly, it also keeps an `UpdateState` run/span out of LangSmith traces if configured. Restoring is then a single `aget_state` read from the checkpoint. Two things about the channel's declaration matter downstream: - **Additive.** Its reducer is `operator.add`, so a writer returns *only the delta it just priced* — literally `{"_session_cost_usd": 0.02}`, never a replacement total. If a thread starts a turn at $1.00, `after_model` contributes $0.02 and a later rubric-grading drain contributes $0.005, the checkpoint ends at $1.025. Neither hook has to perform its own read-modify-write. - **Private.** It carries `PrivateStateAttr`, keeping cost out of the graph's public input/output contract while leaving it checkpointed and readable through `get_state`. Deep Agents also uses this marker to identify state that must not cross the main-agent/subagent boundary. The cumulative channel has one writer: the `CostTrackingMiddleware` instance belonging to the thread's main agent. Subagents run the same middleware, but theirs is constructed with `nested=True`, which switches its pricing off — a subagent's spend is charged by its parent instead, as the next section explains. To get every completed model call to that writer, the cost-tracking module installs one process-wide **Recorder**. The Recorder is an in-memory inbox for completed model calls, implemented as a LangChain callback (`BaseCallbackHandler`). When a request finishes, it files the request's usage, model/provider metadata, and message ID under that thread. It does not calculate cost or write graph state; it only holds records until the main agent's middleware drains them. ```mermaid flowchart LR subgraph calls["every model call in the run"] direction TB A["assistant"] B["subagent"] C["offload / summarization"] D["Auto mode classifier"] E["rubric grader"] end calls --> REC["in-memory model-call recorder<br/>collects usage, keyed by thread"] REC --> MW["CostTrackingMiddleware<br/>drains, prices, adds the delta"] MW --> CK[("checkpoint<br/>_session_cost_usd")] MW -. "stream event: absolute total" .-> BAR["status bar and /cost"] CK -. "resume, thread switch, end of turn" .-> BAR ``` The client is purely a reader; nothing outside the graph writes to the channel. ## Why collecting is split from pricing The obvious place to price a request is `after_model`, but that hook only ever sees the agent's own model node, and a large share of a thread's spend can happen somewhere else entirely: | Model call | Runs in | | --- | --- | | Assistant turn, [Auto mode](#4804) classifier | model node | | Offload / [summarization](https://docs.langchain.com/oss/python/deepagents/context-engineering#summarization) | `before_model` (stock) or the model wrap chain (deepagents) | | Subagent turns | tool node, in a child graph | | `compact_conversation` | tool node | | [Rubric grading](https://docs.langchain.com/oss/deepagents/code/goals-and-rubrics) | `after_agent` | The Recorder and the middleware have complementary access: - The Recorder sits at LangChain's callback layer, so it can observe model calls made by the main agent, a subagent, summarization, or another direct invocation. But a callback is not a graph node and cannot return an update to `_session_cost_usd`. - `CostTrackingMiddleware` runs inside the main agent's graph, so its hooks can return an additive update that the checkpointer persists. But those hooks do not run inside a child graph or around a bare side invocation, so they cannot discover all spend on their own. The in-memory queue bridges those two scopes. The callback places one record in the queue when a request completes; `CostTrackingMiddleware` drains the current thread's records in `after_model` and `after_agent`, prices them, and returns their sum as one state update. > <details> > <summary>Example: one turn with hidden model work</summary> > > Suppose a thread already sits at $1.00. Before the main response, summarization spends $0.004; the main response then spends $0.020. The Recorder files both completions under the thread. `after_model` drains those two records and writes `+0.024`, taking the checkpoint to $1.024. > > If rubric grading then spends $0.005 after the final model step, `after_agent` drains that later record and writes `+0.005`. The committed total is $1.029. A subagent call works the same way: its record waits in the shared thread queue until the parent's next drain. > > </details> > <details> > <summary>Why the main response is also checked in state</summary> > > Normally, a main-agent response with message ID `m1` produces a Recorder entry for `m1`. The middleware prices that entry, remembers that `m1` was charged, and skips the copy of `m1` already present in graph state. > > If a custom or third-party integration returns an `AIMessage` with usage metadata but no usable callback record was filed — for example, the callback could not associate the request with a thread — `after_model` can still price that main response from state. Comparing message IDs lets the normal and fallback paths coexist without double-counting. This fallback is limited to the main response; hidden calls are not in the parent's state and therefore still rely on the Recorder. > > </details> ## How subagent spend is counted An earlier version of this PR had subagents accumulate into their own copy of the channel and merge the total back. That silently did nothing because Deep Agents intentionally treats `PrivateStateAttr` as agent-local state: `SubAgentMiddleware` discovers those keys and strips them both when seeding a child and when merging its result. This prevents one agent's internal bookkeeping from leaking into another and avoids parallel subagents racing to write private `LastValue` channels. A parent-visible aggregate would need a separate shared/public contract; for cost, the thread-keyed recorder supplies that bridge without weakening private-state isolation. Since a subagent inherits its parent's thread ID, its completed model calls land in the same in-memory recorder queue. The parent drains and prices those records on its next `after_model` or `after_agent` hook: ```mermaid sequenceDiagram participant P as Main agent participant R as In-memory recorder participant S as Subagent participant C as Checkpoint P->>R: assistant call usage R-->>P: drain 1 record P->>C: add priced delta P->>S: task tool S->>R: nested call usage Note over S: Nested cost middleware does not<br/>price or accumulate spend S-->>P: result P->>R: next assistant call usage R-->>P: drain nested + assistant records P->>C: add priced delta ``` ## Keeping the status bar live The `updates` stream can expose the private channel's writes, but those writes are reducer inputs such as `+0.02`, not the post-reducer lifetime total. Making the client accumulate them would create a second copy of the graph's state machine, vulnerable to missed stream chunks. Instead, the writer emits a small `custom` event carrying the **absolute** total and the client applies it with `set_cost`. > <details> > <summary>How an absolute event recovers from a dropped update</summary> > > Suppose the client shows $1.00. The graph commits another $0.02 and emits `$1.02`, but that event is dropped. A later $0.005 charge emits `$1.025`. Because the payload is an absolute value, the client replaces $1.00 with $1.025 and is correct again; a delta-only stream would leave it permanently short by the missed $0.02. The end-of-turn checkpoint read provides another reconciliation point if the final event is the one that disappears. > > </details> The provisional number exists only to make the status bar react before the graph reaches its next cost-tracking hook. These two client-side fields have different jobs: - `_session_cost_usd` is the latest absolute total received from the graph. It is the durable value restored from the checkpoint. - `_provisional_cost_usd` is a temporary display estimate derived from usage on the message stream. It is never written to graph state. For a request estimated at $0.02, the display moves through this sequence: | Moment | Durable total | Provisional | Status bar | | --- | ---: | ---: | ---: | | Before the request | $1.00 | $0.00 | $1.00 | | Usage arrives on the message stream | $1.00 | $0.02 | $1.02 | | The graph emits its new absolute total | $1.02 | $0.00 | $1.02 | The last step replaces the durable total and clears the provisional bucket, so the same request is not shown twice. The displayed number stays steady at $1.02; only its source changes from an early estimate to checkpointed state. On a completed turn, the final checkpoint read provides the same reconciliation if the custom event was missed. `/cost`'s per-type and per-model rows remain client-side and stay labeled “since this thread was loaded” — only the lifetime total above them comes from the graph. ## Pricing Pricing stays in one place: `estimate_cost` is the only function that touches canonical price data, currently through `genai-prices`. For each request it needs split input/output token counts, the model, and — when the model name is available from more than one host — the provider. Provider names are normalized to the identifiers in the price catalog; if response metadata omits the model or provider, the checkpointed model spec is the fallback. Here are the main pricing paths. **Ordinary input and output.** Suppose a catalog entry charges $2 per million input tokens and $8 per million output tokens. A request with 10,000 input tokens and 2,000 output tokens is estimated as: ```text input: 10,000 / 1,000,000 × $2 = $0.020 output: 2,000 / 1,000,000 × $8 = $0.016 total: $0.036 ``` (Those rates are illustrative; `genai-prices` supplies the actual model/provider rates.) **Provider-specific lookup.** The response's provider matters when the same model family is sold through different hosts. For example, LangChain's `azure_openai` provider name is normalized to the catalog's `azure` identifier, while a direct OpenAI request uses `openai`. This prevents a recognizable model name from silently selecting the wrong host's price. **Cached input.** LangChain's `input_tokens` is inclusive of cache reads and writes. If a response reports 1,000 input tokens with 600 cache-read tokens and 100 cache-write tokens, only 300 are ordinary input: ```text ordinary input: 1,000 - 600 - 100 = 300 cache reads: 600 cache writes: 100 ``` `estimate_cost` passes all three buckets to `genai-prices`, which applies the ordinary-input, cache-read, and cache-write rates separately. Output tokens are priced independently. If malformed provider metadata says the cache buckets exceed the inclusive input count, the tracker clamps them to the input total rather than creating a negative ordinary-input count. **Anthropic cache TTLs.** When Anthropic reports its TTL breakdown, LangChain sets generic `cache_creation` to zero and exposes `ephemeral_5m_input_tokens` and `ephemeral_1h_input_tokens` instead. If those fields contain 100 and 50 tokens, the tracker passes 150 cache-write tokens rather than accidentally billing them as ordinary input. `genai-prices` currently exposes one cache-write bucket, so the two TTL buckets share that catalog estimate. The estimator declines to guess when the available data cannot support a defensible calculation: - A response with only `total_tokens=1_200` is unpriceable because those tokens could be input, output, or a mixture, and the rates usually differ. - An unknown model has no canonical catalog rate to apply. - Subscription-style access such as the Codex provider is not equivalent to per-token API billing. - Malformed or non-finite usage/price data is ignored rather than allowed to fail the model turn. In each case `estimate_cost` returns `None`; the request is omitted from the dollar total while its token usage can still appear in `/tokens`.
23 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Deep Agents Code now offers an opt-in Auto approval mode. Auto immediately runs a narrow, deterministic set of routine actions. For other approval-gated actions, the active model checks whether they match the user's literal request. It runs approved actions and blocks the rest. After repeated denials or classifier failures, Auto sends the next batch to the normal approval UI, then continues in Auto mode.
Auto is experimental and currently limited to interactive, local TUI sessions using
FilesystemBackend. Manual remains the default.Warning
Auto is an authorization heuristic for a local coding agent. It is not sandbox containment, an operating-system boundary, or a guarantee that model-generated actions are safe.
Enable Auto
DEEPAGENTS_CODE_EXPERIMENTAL=1.-yor--auto-approve.[startup].mode = "auto"in~/.deepagents/config.toml.-yand--auto-approvenow select Auto rather than unrestricted execution. Users who want the previous unrestricted behavior must choose--yolo. The removeddangerously-autostartup value is no longer recognized and falls back to Manual.Why
The previous approval modes forced a binary choice:
Auto introduces a third policy: keep the low-friction path for clearly routine work, but make authorization explicit for potentially consequential actions.
[startup].mode = "manual"-y,--auto-approve,[startup].mode = "auto", or the live keyboard toggle--yoloor[startup].mode = "yolo"How Auto handles a proposed action
flowchart TD A[Model proposes tool calls] --> B{Covered by the approval policy?} B -->|No| X[Existing tool behavior is unchanged] B -->|Yes| S{Is this thread still in Auto, with readable denial/failure history?} S -->|No| J[Open the existing approval HITL widget] S -->|Yes| C{Narrow deterministic allow?} C -->|Yes| R[Execute without a classifier call] C -->|No| D[Build one structured decision batch] D --> H{Was this batch already reviewed, or do earlier denials/failures require human review?} H -->|Yes| J H -->|No| E[Active model reviews effects against user request/input] E -->|Allow| R E -->|Deny| Q{Total-denial threshold reached?} Q -->|No| F[Return a sanitized error result] Q -->|Yes| J E -->|Unavailable or invalid| G[Return a compact unavailable result] F --> I[Agent can revise its plan] G --> I J -->|Approve| R J -->|Reject| K[Return a rejection result] J -->|Switch to Manual| L[Persist Manual, then review the full gated batch] R --> M[Reconcile the result and continue] K --> M I --> M L --> MIn practice:
ls,read_file,glob,grep,ask_user,get_goal,get_rubric, andupdate_goal.js_evaltool, and caller-supplied tools without HITL configuration continue to run normally.A routine write such as
src/parser.pycan proceed based on its resolved path and suffix; a sensitive target such as.github/workflows/ci.ymlgoes to classifier review.A read-only Git command such as
git statuscan proceed; a mutating command such asgit commit -m changegoes to classifier review. Explicit shell allow-list entries still apply after Auto removes broad and wildcard rules.An MCP tool with a valid read-only annotation can proceed; tools without read-only annotations, or with contradictory or malformed annotations, go to classifier review.
How MCP read-only annotations are evaluated
MCP servers can attach standard
ToolAnnotationswhen advertising a tool. Deep Agents Code copies those annotations into the wrapped tool's metadata and lets the tool bypass classifier review only when:readOnlyHintis the literal Booleantrue;destructiveHintis absent,null, orfalse; andreadOnlyHint,destructiveHint,idempotentHint, oropenWorldHint) is a Boolean ornull, rather than a string or another type.For example:
readOnlyHint: true, destructiveHint: falseproceeds without classifier review.readOnlyHintgoes to review because the tool never explicitly claimed to be read-only.readOnlyHint: true, destructiveHint: truegoes to review because the hints contradict each other.readOnlyHint: true, destructiveHint: "false"goes to review because"false"is a string, not a Boolean.In Auto, tools that do not pass this check enter the classifier batch. In Manual, they use the normal approval UI; headless runtimes reject them because they have no approval UI. The annotation is a server-provided assertion that Deep Agents Code does not independently verify.
Research issue #123, update @src/parser.py, and open a PR, proposedfetch_urlandgit pushcalls can be reviewed in one batch and receive separate decisions. The referenced path and summarized call arguments help identify what each action would affect, but the expanded file contents are not sent. Instructions found in that file, assistant prose, or a prior tool result therefore cannot widen the user's request or authorize another action.delete, it returns an error result with that call's ID instead of invoking the tool, allowing the agent to propose a safer next step.Limits and user-visible behavior
js_evalfan-out. Model providers and tracing backends may still observe classifier inputs and outputs even though the TUI hides them.Implementation and reviewer details
Approval boundary and decision inputs
The following tools are HITL-gated:
execute,write_file,edit_file, anddelete;web_searchandfetch_url;task;start_async_task,update_async_task, andcancel_async_task;compact_conversationwhen optional compaction approval is enabled; andFor deterministic writes and edits, the fast path uses only the resolved target path and suffix; it does not inspect proposed content. Dependency and lock files, sensitive names and locations, shell scripts, unsupported file types, and paths outside the trusted root do not qualify. Selected Git inspection commands must also pass shell-control and outside-worktree path checks. Configured shell entries remain an explicit user trust exception, but broad commands and wildcard entries are ignored by Auto.
The classifier receives one typed batch containing:
Only
literal_user_textis authorizing. Content-bearing write/edit arguments are reduced to character counts. The classifier must return exactly one unique decision for every unresolved tool-call ID; missing, duplicated, unknown, or malformed decisions make the classifier unavailable rather than partially authorizing the batch.Before routing calls, Auto re-reads the live mode and validates its private decision plan. Manual mode, invalid control state, mismatched calls, or replayed batches require human review. Classifier review uses a second call to the active session model with a 20-second timeout. Raw classifier text and tool-call chunks are hidden from the transcript, while usage accounting and tracing are retained.
Denial reasons are sanitized before persistence or display by removing control characters, known credential values, credential-like assignments, and URL credentials or query values. MCP tools without valid read-only annotations are HITL-gated; headless runtimes reject them because no approval UI is available.
Failure and fallback behavior
Python API and state compatibility
auto_approve=Truemaps to YOLO, not Auto.Reviewer guide
Suggested review order:
ApprovalMode, Store payloads, acknowledgement, startup resolution, and live mode writes.AutoModeHITLMiddleware, literal prompt metadata, deterministic routing, structured classification, sanitization, counters, and checkpoint validation.Highest-risk invariants: