feat(code): selectable model for the Auto approval classifier by mdrxy · Pull Request #5205 · langchain-ai/deepagents

Mason Daugherty (mdrxy) · 2026-07-30T20:07:51Z

Auto approval mode can now use a separate, cheaper model to review actions instead of always reusing the main agent model. Set it with --auto-classifier-model, DEEPAGENTS_CODE_AUTO_CLASSIFIER_MODEL, [models].auto_classifier, or /auto model in the TUI.

In Auto mode, any gated tool call that deterministic policy can't clear is reviewed by an LLM classifier. That review used the main agent model, so every batch paid frontier-model price and latency for a short verdict, in the middle of the turn.

The classifier decides what runs without asking you, so the behavior around it is conservative:

The default is unchanged. With nothing configured, the classifier still uses the main agent model.
Precedence: /auto model → launch flag → env var → config.toml → inherit. A blank value anywhere means "inherit".
Trusted sources only. A project .env can't set the env var, so a cloned repo can't quietly point your review at a weaker model. Shell exports, the global ~/.deepagents/.env, config.toml, the flag, and /auto model all work as usual.
A broken classifier never falls back to the main model. If the configured model can't be built (bad spec, missing credentials, missing provider package), the calls it would have reviewed are denied and don't run, and Auto starts asking you after repeated failures. The error names the model so you know what to fix. /auto model validates a spec before accepting it.
You can always tell which model is reviewing. The TUI names it when Auto turns on and in /auto model, including one set by env var or config.toml.
Main-model settings don't leak. Cache control, prompt cache keys, reasoning budgets and --model-params are provider-specific, so they only travel when the classifier is the main model.
Nothing else about Auto changed. Deterministic allow/deny, the denial counters, replay detection, control-state checks, and the fallback thresholds are all untouched.

--auto-classifier-model only works where Auto runs — interactive TUI, no sandbox — and errors out otherwise instead of being quietly ignored. THREAT_MODEL.md gains T14 for the tradeoff: a weaker classifier means a weaker review, including against prompt injection in the content it reads.

Not included: a recommended cheap model (there's no accuracy eval suite yet), classifier-specific invocation params, a configurable timeout, and cache invalidation when credentials rotate (a rotated key shows up as a named classifier failure).

Test plan

Unit tests cover precedence, blank specs, the project/global .env split, model caching, settings hygiene, fail-closed resolution and its deadline, logging and trace fields, the per-run context for both setting and clearing, the sandbox and headless guards, and the /auto model paths.

Manual: launch with --auto-classifier-model <spec>, confirm reviews use it, then /auto model clear to go back.

Made by Open SWE

References

Plan: https://openswe.vercel.app/agents/a1237a94-e79b-6c84-c74f-b69b5acfc7c4/plan

Auto mode's authorization classifier reused the main agent model for every gated action batch, so each review paid frontier price and latency on the critical path. It can now be pointed at a separate model via `--auto-classifier-model`, `DEEPAGENTS_CODE_AUTO_CLASSIFIER_MODEL`, `[models].auto_classifier`, or `/auto model`, with the per-run context taking effect without a server restart. The classifier is an authorization control, so the default is unchanged (inherit the main model), a spec that cannot be built fails closed to human approval instead of silently reverting to the main model, and primary-model settings are not forwarded to a distinct classifier. Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>

open-swe

Open SWE Review found 2 potential issues.

Open in Web • View Open SWE trace

Review follow-ups on the selectable Auto classifier model. `/auto model clear` sent a bare `None` on the run context, which the middleware could not tell apart from "this run has no preference", so a session started with `--auto-classifier-model`, the env var, or `[models].auto_classifier` kept authorizing actions with that model after the UI said reviews had moved back to the main agent model. The context now carries an explicit inherit marker, and the client no longer treats a locally unset classifier as proof that the server has none either. The classifier model selector also started its worker and refocused the chat input inside the modal's own dismiss callback; both now wait for the refresh that removes the modal, per the package's Textual contract. Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>

The Auto approval classifier authorizes gated tool calls, so choosing which model performs that review is a user-level security decision. `DEEPAGENTS_CODE_AUTO_CLASSIFIER_MODEL` is now denied from a *project* `.env` (`_PROJECT_DOTENV_DENIED_ENV_KEYS`), matching the treatment of the project-MCP trust vars. Without it, a cloned repo could commit a `.env` that silently pointed the review at a weaker model — weakening the control, including its resistance to prompt injection in the untrusted material it reads. Shell exports, the global `~/.deepagents/.env`, `config.toml`, `--auto-classifier- model`, and `/auto model` are unaffected. A blank spec now means "inherit" at every tier. `_classifier_spec` previously passed a blank construction-time value through to `create_model`, which treats an empty spec as `[models].default` and would have reviewed with a model nobody selected, while labelling it `inherited` in traces. `--auto-classifier-model` is rejected under `--sandbox`, where Auto is disabled, for the same reason the headless form is rejected. When a classifier failure coincides with a failed counter write, both are now reported. Control state tends to recover on its own; a misconfigured spec never does, so collapsing the two sent the user back for another round after fixing the disk. Invocation failures on a distinct classifier also name the spec, so a cached model built against a rotated credential points at a restart rather than another `/auth`. A blank configured value is logged instead of being dropped silently. `INHERIT_CLASSIFIER_MODEL` changes from a NUL-prefixed sentinel to plain ASCII. The context is serialized to JSON and may be persisted, and Postgres text/jsonb rejects NUL; a stripped sentinel reads as "no preference", which would leave a startup classifier authorizing actions after the UI reported the clear. Docs corrected to match behavior: an unbuildable classifier denies the batch and escalates to human approval only after `_CONSECUTIVE_UNAVAILABLE_FALLBACK` consecutive failures, rather than routing to approval immediately. Precedence is documented consistently as env then `config.toml`, and the settings-browser summary carries the review-quality caveat. Adds coverage for the project/global `.env` split, blank-spec inheritance, argument-over-env precedence, per-run context delivery for both set and clear, cache recency, instance labelling, the combined diagnostic, the unavailable log label, and the sandbox guard.

open-swe

Open SWE Review found 1 potential issue.

Open in Web • View Open SWE trace

open-swe

Open SWE Review found 1 potential issue.

Open in Web • View Open SWE trace

A misconfigured Auto classifier previously degraded in ways the user could neither see nor act on. Six fixes, all on the paths that run when the configured classifier cannot review a batch. Escalate permanently-broken classifiers instead of oscillating. Approving a human fallback resets `consecutive_unavailable`, so counting construction failures left a bad spec denying two batches for every one it asked about, indefinitely. Construction faults now latch the failing spec in `AutoModeCounters.classifier_config_failed_spec`: the first failure still denies, and every later batch escalates to human approval. Construction is still retried each batch, so fixing the setting clears the latch on the next successful review without a restart. The prompt names the spec and the commands that fix it — the approval prompt renders the batch-level `fallback_reason`, not each decision's own reason, so the diagnostic is carried there too. Report a configured-but-unusable classifier. A blank or wrong-typed `[models].auto_classifier` silently reverted authorization review to the main agent model — the agent grading its own actions — with only a debug-log line, or nothing at all for the non-string case (`resolve_scalar` coerces it to the option default, making it indistinguishable from absent). `resolve_auto_classifier_model_with_problem` returns the reason alongside the spec and the launch path prints it. Startup specs also get the cheap half of what `/auto model` validates — spec parse plus provider credentials, no `create_model`, so LangChain stays off the launch path. Say which model reviews actions, and which role it plays. The first-enable notice called the classifier the "active model", which reads as the model writing the code; with `--auto-classifier-model` the two differ, and this modal is the only place the disclosure appears. The copy now distinguishes a separate classifier from an inherited one. The interpolation anchor is a named constant and the builder raises rather than silently dropping the disclosure. Give classifier construction its own deadline. Building shared the 20s inference budget, so a cold provider import made the first review the likeliest to be denied and reported it as "the classifier did not respond" about a model that was never built. Recover from a revoked credential in-session. Construction succeeds once and the model is cached, so an invoke-time failure repeated forever — `/auth` runs in the client and cannot reach the cache. Failing models are now evicted, and `/auto model <same spec>` revalidates instead of short-circuiting. Normalize a blank `--auto-classifier-model` at parse time so the ACP, headless, and sandbox guards and the value passed to the TUI all agree it means "inherit". Also: correct two docstrings that stated the opposite of the code (a blank runtime-context spec means "no preference", not "inherit"; a blank string is not equivalent to `None` in `create_cli_agent`); log unexpected detached-construction failures instead of discarding them; hoist markdown escaping to `_markdown.py` rather than keeping a second copy in the notice widget; drop an unreachable `getattr` fallback that would have labelled classifier rows from the main-model catalog; add T14 to the THREAT_MODEL input-coverage tables and revision history; and note that `/auto model` is session-scoped. Tests cover the invoke-time diagnostic, cache eviction, the latch and its survival across a real approval, headless flag rejection, `/auto`'s queue-bypass tier on both sides, blank-flag normalization, and the notice copy. Mutation testing confirms the fail-closed path, the latch, and the prompt plumbing are each caught by a failing test. `test_live_tail_appends_new_records_incrementally` read the process-wide log buffer, so it depended on how many records earlier test files happened to emit: once retained records cross `_RECORD_LIMIT`, every poll takes the prune-and-re-render path instead of appending. It now drives a controlled buffer.

open-swe

Open SWE Review found 1 potential issue.

Open in Web • View Open SWE trace

The remediation text enumerated three ways out — `/auto model`, `/auto model clear`, and `/auth` — which is too long for a message that appears inline in an approval prompt the user reads while deciding whether to allow an action. Keep only how to switch the classifier; the spec is already named, and the other routes are discoverable from `/auto`.

…ier-model # Conflicts: # libs/code/THREAT_MODEL.md

…assifier An explicit blank flag means "review with the main agent model", but parse_args collapsed it to `None`, which downstream reads as "no flag — consult `DEEPAGENTS_CODE_AUTO_CLASSIFIER_MODEL` / `[models].auto_classifier`". A user with a configured separate classifier launching with `--auto-classifier-model ""` kept the weaker configured classifier authorizing actions instead of inheriting the main model as documented. Preserve blank-vs-absent from argparse through launch resolution by mapping the explicit blank to `INHERIT_CLASSIFIER_MODEL` — the sentinel `/auto model clear` already uses — instead of `None`: - run_textual_cli_async resolves a blank flag to the sentinel so it beats env/TOML without falling into resolve_auto_classifier_model_with_problem; a bare "" can't be used because ServerConfig.from_env collapses empty strings to None, which would re-enable a configured classifier in the server subprocess. - AutoModeHITLMiddleware maps the sentinel to inherit at construction time too (previously only per-run), so it never tries to build a model named __dcode_inherit_classifier__. - The TUI treats the sentinel as inherit in labels, the model selector, the Auto-activation notification, and the per-run context value so the raw sentinel never surfaces to the user.

@mdrxy

…de (#5271) Fixes DOC-1474 ## Summary - `approval-modes.mdx`: adds a new **"Select a classifier model"** section under "How Auto works", documenting all four configuration sources (`/auto model`, `--auto-classifier-model`, `DEEPAGENTS_CODE_AUTO_CLASSIFIER_MODEL`, `[models].auto_classifier`), precedence order, fail-closed error behavior, the project `.env` restriction, and the prompt-injection security tradeoff (T14) - `cli-reference.mdx`: adds `--auto-classifier-model` to the command-line flag table - `config-file.mdx`: documents the `[models].auto_classifier` key in the `[models]` table with a link to the full precedence description ## Links - Linear: https://linear.app/langchain/issue/DOC-1474/document-selectable-auto-approval-classifier-model-for-deep-agents - Slack: https://langchain.slack.com/archives/C09G1T60QV9/p1785791397422779 - Source PR: langchain-ai/deepagents#5205 ## Verification Not run; docs-only content change. ## Reviewers Requested review from: @mdrxy, @npentrel --------- Co-authored-by: Docs Writer Bot <brace@langchain.dev> Co-authored-by: Mason Daugherty <mason@langchain.dev> Co-authored-by: Naomi Pentrel <5212232+npentrel@users.noreply.github.com>

> [!CAUTION] > Merging this PR will automatically publish to **PyPI** and create a **GitHub release**. For the full release process, see [`.github/RELEASING.md`](https://github.com/langchain-ai/deepagents/blob/main/.github/RELEASING.md). --- _Release notes preview: keep this section in sync with the package `CHANGELOG.md`. Publish reads the merged CHANGELOG via `release.yml`, not this PR description — keep them aligned anyway so the PR stays an accurate historical record for reviewers and anyone returning later._ --- ## [0.1.52](deepagents-code==0.1.51...deepagents-code==0.1.52) (2026-08-04) ### Features - Hooks v2 is now generally available, with support for loading hooks from installed plugins. ([#5307](#5307), [#5198](#5198)) - Auto approval classifier configuration now supports selecting the classifier model and setting a review timeout. ([#5205](#5205), [#5302](#5302)) - HITL rejection reasons are now framed for the model, and the approval menu makes reject-with-feedback easier to discover. ([#5259](#5259), [#5260](#5260)) - Added a tri-state `DEEPAGENTS_CODE_ONBOARDING` environment variable. ([#5301](#5301)) - The `/model` footer Ctrl+N hint now follows the current display mode. ([#5247](#5247)) - The price catalog now refreshes hourly in the background. ([#5264](#5264)) - Updated recommendations to include DeepSeek V4 Flash 0731. ([#5244](#5244)) ### Bug Fixes - Fixed several Hooks v2 lifecycle issues: session-end teardown is now bounded, hooks refresh after cwd switches, malformed hook resumes are handled, hook stops surface without agent errors, and unused `SessionEndCause` members were removed. ([#5248](#5248), [#5249](#5249), [#5233](#5233), [#5276](#5276), [#5240](#5240)) - `PreCompact` now fires before auto-compaction. ([#5277](#5277)) _End release notes preview._ --- > [!NOTE] > A **New Contributors** section is appended to the GitHub release notes automatically at publish time (see [Release Pipeline](https://github.com/langchain-ai/deepagents/blob/main/.github/RELEASING.md#release-pipeline), step 2). --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: langchain-oss-automated-triage[bot] <248757908+langchain-oss-automated-triage[bot]@users.noreply.github.com>

github-actions Bot added dcode Related to `deepagents-code` feature New feature/enhancement or request for one internal User is a member of the `langchain-ai` GitHub organization open-swe size: XL 1000+ LOC labels Jul 30, 2026

Mason Daugherty (mdrxy) marked this pull request as ready for review July 30, 2026 20:13

open-swe Bot reviewed Jul 30, 2026

View reviewed changes

Comment thread libs/code/deepagents_code/auto_mode.py Outdated

Comment thread libs/code/deepagents_code/app.py Outdated

Mason Daugherty (mdrxy) and others added 3 commits July 31, 2026 00:25

Merge branch 'main' into open-swe/auto-classifier-model

c57e3bd

open-swe Bot reviewed Jul 31, 2026

View reviewed changes

Comment thread libs/code/deepagents_code/auto_mode.py Outdated

fix(code): retain timed-out classifier construction

f3e6c32

open-swe Bot reviewed Jul 31, 2026

View reviewed changes

Comment thread libs/code/deepagents_code/auto_mode.py Outdated

Mason Daugherty (mdrxy) added 5 commits July 30, 2026 22:01

fix(code): allow classifier switches past hung constructors

3dc3f68

feat(code): refine Auto classifier model picker

4fb8933

fix(code): clarify Auto mode notices

831f46a

fix(code): suppress redundant Auto classifier toast

1e3ed1e

open-swe Bot reviewed Aug 3, 2026

View reviewed changes

Comment thread libs/code/deepagents_code/main.py Outdated

Mason Daugherty (mdrxy) added 3 commits August 3, 2026 16:02

Merge remote-tracking branch 'origin/main' into open-swe/auto-classif…

24674e7

…ier-model # Conflicts: # libs/code/THREAT_MODEL.md

Mason Daugherty (mdrxy) merged commit 1d3feb1 into main Aug 3, 2026
54 checks passed

Mason Daugherty (mdrxy) deleted the open-swe/auto-classifier-model branch August 3, 2026 21:10

github-actions Bot mentioned this pull request Aug 3, 2026

release(deepagents-code): 0.1.52 #5228

Merged

langsmith-fleet Bot mentioned this pull request Aug 3, 2026

Document selectable Auto approval classifier model for Deep Agents Code langchain-ai/docs#5271

Merged

Navigation Menu

Uh oh!

feat(code): selectable model for the Auto approval classifier - #5205

feat(code): selectable model for the Auto approval classifier#5205
Mason Daugherty (mdrxy) merged 13 commits into
mainfrom
open-swe/auto-classifier-model

Mason Daugherty (mdrxy) commented Jul 30, 2026 •

edited

Loading

Uh oh!

open-swe Bot left a comment

Uh oh!

Uh oh!

Uh oh!

open-swe Bot left a comment

Uh oh!

Uh oh!

open-swe Bot left a comment

Uh oh!

Uh oh!

open-swe Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Mason Daugherty (mdrxy) commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

References

Uh oh!

open-swe Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

open-swe Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

open-swe Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

open-swe Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Mason Daugherty (mdrxy) commented Jul 30, 2026 •

edited

Loading