feat(code): let the rubric grader inspect working-directory files - #4835
Merged
Mason Daugherty (mdrxy) merged 18 commits intoJul 22, 2026
Conversation
The rubric/goal grader judged completion from a transcript truncated to the last 30 messages, so evidence scrolled out of view on long efforts and criteria were marked unmet. Give the grader read-only `ls`, `read_file`, `glob`, and `grep` tools scoped to the working directory (plus the existing offloaded-results read), bounded identically to the goal-criteria agent, and prompt it to verify criteria against the files. Extract the shared path-safety, containment, arg-clamping, and result-bounding logic into `_repository_bounds.RepositoryBounds` so the grader tools and `_RepositoryToolBudgetMiddleware` share one implementation. Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
github-actions
Bot
added
dcode
![corridor-security[bot]](https://avatars.githubusercontent.com/in/1177339?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
The PR exposes a symlink escape vulnerability in local (non-sandbox) mode: sandbox_contains() returns True without a canonical realpath check, allowing a malicious repository symlink to escape the working-directory boundary via the rubric grader's glob tool and the grep Python-fallback walker. The read_file tool is largely protected by O_NOFOLLOW on Linux, but glob and the Python rglob grep fallback follow symlinks without containment verification.
![open-swe[bot]](https://avatars.githubusercontent.com/in/1395372?s=60&v=4){kind=small}
github-actions
Bot
added
the
deepagents
…r-file-access # Conflicts: # libs/code/deepagents_code/agent.py # libs/code/deepagents_code/goal_rubric.py # libs/code/tests/unit_tests/test_agent.py
14 tasks
Mason Daugherty (mdrxy)
pushed a commit
that referenced
this pull request
Jul 24, 2026
> [!CAUTION] > Merging this PR will automatically publish to **PyPI** and create a **GitHub release**. For the full release process, see [`.github/RELEASING.md`](https://github.com/langchain-ai/deepagents/blob/main/.github/RELEASING.md). --- _Release notes preview: keep this section in sync with the package `CHANGELOG.md`. The published GitHub release body is extracted from the merged `CHANGELOG.md` by `release.yml`, not from this PR description._ --- ## [0.1.46](deepagents-code==0.1.45...deepagents-code==0.1.46) (2026-07-24) ### Highlights - Auto mode is now generally available. [#4957](#4957) - Added configurable Auto goal-criteria acceptance. [#4940](#4940) - Improved Auto behavior by authorizing actions from active goal/rubric directives, avoiding redundant approval prompts, showing the enable notice only on first global enable, deduplicating classifier-unavailable transcript spam, logging underlying classifier failures, and reporting classifier timeout budgets. [#5017](#5017) [#4993](#4993) [#5012](#5012) [#5013](#5013) [#5011](#5011) [#5025](#5025) - Added Hooks v2 capability snapshots and session transcripts, and hardened Hooks v2 command execution. [#4916](#4916) [#4918](#4918) [#4917](#4917) - Raised the agent recursion limit to 2000 and made it configurable. [#4994](#4994) ### Improvements and fixes - Let the rubric grader inspect working-directory files, show rubric grader defaults, and improved `/rubric` help and empty-state messaging. [#4835](#4835) [#4966](#4966) [#5015](#5015) - Unified goal activation signaling. [#4980](#4980) - Made Version, Model, and CWD copyable in the Debug Console. [#4975](#4975) - Improved `config get` output when a key is missing. [#4976](#4976) - Aborted YOLO launch on `Ctrl+C`/`Ctrl+D` and made the YOLO warning friendlier for new users. [#4953](#4953) [#4950](#4950) - Updated LangSmith handling: secret redaction is disabled by default, `/trace` now flags empty env overrides that shadow the LangSmith key, and the default US endpoint is no longer treated as a custom target. [#4970](#4970) [#4996](#4996) [#5022](#5022) - Injected OpenAI `prompt_cache_key` for any OpenAI-provider endpoint. [#4995](#4995) - Improved tool and schema presentation: finished calls stay on the live tool-group line, first-party tool schemas now include field descriptions, and `web_search`/`fetch_url` tool descriptions were trimmed. [#4927](#4927) [#5019](#5019) [#5016](#5016) - Omitted `plugins/` and `conversation_history/` from the `/agent` picker. [#4991](#4991) - Made selector modal backdrop dimming consistent. [#4990](#4990) - Restored the `"Server log preserved at:"` notice on exit. [#4999](#4999) - Used the SDK pin as the effective editable version. [#4949](#4949) _End release notes preview._ --- > [!NOTE] > A **New Contributors** section is appended to the GitHub release notes automatically at publish time (see [Release Pipeline](https://github.com/langchain-ai/deepagents/blob/main/.github/RELEASING.md#release-pipeline), step 2). --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: langchain-oss-automated-triage[bot] <248757908+langchain-oss-automated-triage[bot]@users.noreply.github.com>
Marcelo5444
pushed a commit
to Marcelo5444/deepagents
that referenced
this pull request
Jul 30, 2026
…ngchain-ai#4835) Long-running tasks can now be graded against the final repository and connected read-only state, rather than only a bounded conversation excerpt. This prevents completed work from being marked incomplete after the supporting evidence scrolls out of the grader's transcript context. --- ## Before and after **Before** The completion grader received the original request and a bounded tail of the conversation, but it could not inspect the resulting files or connected systems. - A task asks the agent to update `src/config.py` and create `docs/setup.md`. - The agent makes and verifies both changes, then continues through a long debugging conversation. - By the time grading runs, the relevant tool results are no longer in the retained transcript excerpt. - The grader cannot confirm either file and may report the criteria as unmet. **After** The completion grader can inspect the final state with bounded, read-only tools. - It uses `read_file` or `glob` to verify `src/config.py` and `docs/setup.md` inside the effective working directory. - If a criterion concerns an external result, such as a Notion update, it can use the same filtered external context available while proposing criteria: `fetch_url`, optional `web_search`, and MCP tools that are consistently annotated as read-only. - It grades the criterion from the verified final state even when the original execution evidence is no longer present in the transcript excerpt. Repository verification is limited to `ls`, `read_file`, `glob`, and `grep`; it does not give the grader shell access or the ability to run tests. Access remains subject to the configured filesystem-tool allowlist and requires a trusted local or sandbox working-directory root. Path traversal and symlink escapes are rejected, and unknown sandbox types fail closed without repository tools. External verification follows the active approval mode. Approval requests from nested grading runs pause and resume through the parent graph, using the same live runtime context as the task agent. Configurations that disable HITL remain uninterrupted. In normal server wiring, mutating or ambiguously annotated MCP tools are excluded. Each grading iteration has bounded call, web-search, and accumulated-result budgets. Repository operations also cap file size, lines read, directory entries, matches, and returned characters so verification cannot grow without limit. --------- Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
Marcelo5444
pushed a commit
to Marcelo5444/deepagents
that referenced
this pull request
Jul 30, 2026
> [!CAUTION] > Merging this PR will automatically publish to **PyPI** and create a **GitHub release**. For the full release process, see [`.github/RELEASING.md`](https://github.com/langchain-ai/deepagents/blob/main/.github/RELEASING.md). --- _Release notes preview: keep this section in sync with the package `CHANGELOG.md`. The published GitHub release body is extracted from the merged `CHANGELOG.md` by `release.yml`, not from this PR description._ --- ## [0.1.46](langchain-ai/deepagents@deepagents-code==0.1.45...deepagents-code==0.1.46) (2026-07-24) ### Highlights - Auto mode is now generally available. [langchain-ai#4957](langchain-ai#4957) - Added configurable Auto goal-criteria acceptance. [langchain-ai#4940](langchain-ai#4940) - Improved Auto behavior by authorizing actions from active goal/rubric directives, avoiding redundant approval prompts, showing the enable notice only on first global enable, deduplicating classifier-unavailable transcript spam, logging underlying classifier failures, and reporting classifier timeout budgets. [langchain-ai#5017](langchain-ai#5017) [langchain-ai#4993](langchain-ai#4993) [langchain-ai#5012](langchain-ai#5012) [langchain-ai#5013](langchain-ai#5013) [langchain-ai#5011](langchain-ai#5011) [langchain-ai#5025](langchain-ai#5025) - Added Hooks v2 capability snapshots and session transcripts, and hardened Hooks v2 command execution. [langchain-ai#4916](langchain-ai#4916) [langchain-ai#4918](langchain-ai#4918) [langchain-ai#4917](langchain-ai#4917) - Raised the agent recursion limit to 2000 and made it configurable. [langchain-ai#4994](langchain-ai#4994) ### Improvements and fixes - Let the rubric grader inspect working-directory files, show rubric grader defaults, and improved `/rubric` help and empty-state messaging. [langchain-ai#4835](langchain-ai#4835) [langchain-ai#4966](langchain-ai#4966) [langchain-ai#5015](langchain-ai#5015) - Unified goal activation signaling. [langchain-ai#4980](langchain-ai#4980) - Made Version, Model, and CWD copyable in the Debug Console. [langchain-ai#4975](langchain-ai#4975) - Improved `config get` output when a key is missing. [langchain-ai#4976](langchain-ai#4976) - Aborted YOLO launch on `Ctrl+C`/`Ctrl+D` and made the YOLO warning friendlier for new users. [langchain-ai#4953](langchain-ai#4953) [langchain-ai#4950](langchain-ai#4950) - Updated LangSmith handling: secret redaction is disabled by default, `/trace` now flags empty env overrides that shadow the LangSmith key, and the default US endpoint is no longer treated as a custom target. [langchain-ai#4970](langchain-ai#4970) [langchain-ai#4996](langchain-ai#4996) [langchain-ai#5022](langchain-ai#5022) - Injected OpenAI `prompt_cache_key` for any OpenAI-provider endpoint. [langchain-ai#4995](langchain-ai#4995) - Improved tool and schema presentation: finished calls stay on the live tool-group line, first-party tool schemas now include field descriptions, and `web_search`/`fetch_url` tool descriptions were trimmed. [langchain-ai#4927](langchain-ai#4927) [langchain-ai#5019](langchain-ai#5019) [langchain-ai#5016](langchain-ai#5016) - Omitted `plugins/` and `conversation_history/` from the `/agent` picker. [langchain-ai#4991](langchain-ai#4991) - Made selector modal backdrop dimming consistent. [langchain-ai#4990](langchain-ai#4990) - Restored the `"Server log preserved at:"` notice on exit. [langchain-ai#4999](langchain-ai#4999) - Used the SDK pin as the effective editable version. [langchain-ai#4949](langchain-ai#4949) _End release notes preview._ --- > [!NOTE] > A **New Contributors** section is appended to the GitHub release notes automatically at publish time (see [Release Pipeline](https://github.com/langchain-ai/deepagents/blob/main/.github/RELEASING.md#release-pipeline), step 2). --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: langchain-oss-automated-triage[bot] <248757908+langchain-oss-automated-triage[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Long-running tasks can now be graded against the final repository and connected read-only state, rather than only a bounded conversation excerpt. This prevents completed work from being marked incomplete after the supporting evidence scrolls out of the grader's transcript context.
Before and after
Before
The completion grader received the original request and a bounded tail of the conversation, but it could not inspect the resulting files or connected systems.
src/config.pyand createdocs/setup.md.After
The completion grader can inspect the final state with bounded, read-only tools.
read_fileorglobto verifysrc/config.pyanddocs/setup.mdinside the effective working directory.fetch_url, optionalweb_search, and MCP tools that are consistently annotated as read-only.Repository verification is limited to
ls,read_file,glob, andgrep; it does not give the grader shell access or the ability to run tests. Access remains subject to the configured filesystem-tool allowlist and requires a trusted local or sandbox working-directory root. Path traversal and symlink escapes are rejected, and unknown sandbox types fail closed without repository tools.External verification follows the active approval mode. Approval requests from nested grading runs pause and resume through the parent graph, using the same live runtime context as the task agent. Configurations that disable HITL remain uninterrupted. In normal server wiring, mutating or ambiguously annotated MCP tools are excluded.
Each grading iteration has bounded call, web-search, and accumulated-result budgets. Repository operations also cap file size, lines read, directory entries, matches, and returned characters so verification cannot grow without limit.