feat(evals): goal-tool over-eager-calling evals + gated prompt copy by mdrxy · Pull Request #4412 · langchain-ai/deepagents

Mason Daugherty (mdrxy) · 2026-07-01T18:51:29Z

Models over-eagerly call dcode's get_rubric / get_goal even when no goal or rubric was ever set, because GOAL_TOOLS_SYSTEM_PROMPT leads with an unconditional "inspect before finishing" instruction. This adds evals to pin the behavior down and rewrites the copy to gate on the precondition. Specifically:

New tool_not_called hard-fail success assertion in libs/evals/tests/evals/utils.py (the framework only had text-based success assertions + efficiency presence checks).
New libs/evals/tests/evals/test_goal_tools.py exercising the real GoalToolsMiddleware via create_agent (mirrors test_langchain_middleware_todo.py): two baseline regression gates that get_rubric / get_goal stay untouched when no goal/rubric is set (trivial task + real multi-step tool use), and a hillclimb case that a consulted rubric is still allowed (seeded via the public rubric input, since goal channels aren't part of the public graph input in an isolated harness).
Rewrote GOAL_TOOLS_SYSTEM_PROMPT to lead with "a goal/rubric is only present if one was set earlier; if none was set, do not call these tools."
Regenerated EVAL_CATALOG.md.

Made by Open SWE

References

Plan: https://openswe.vercel.app/agents/de848f56-d39d-53f9-1959-7cd5cca9eee6/plan

`dcode`'s goal-tools prompt made models call `get_rubric`/`get_goal` even when no goal or rubric was ever set. Add a `tool_not_called` hard-fail assertion and a `test_goal_tools.py` eval module (baseline gates that the tools stay untouched with no goal/rubric set, plus a hillclimb case that a consulted rubric is still allowed), and rewrite `GOAL_TOOLS_SYSTEM_PROMPT` to lead with the precondition that the tools are inert when nothing is set. Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>

github-actions · 2026-07-01T18:51:40Z

⛔ This PR title scope does not match the package directory it changes.

Title scope(s): evals

Touched package dir(s) not covered by those scopes:

package label dcode from libs/code/
package label deepagents from libs/deepagents/

This check is blocking because the PR title declares one package scope while the changed files live in a different package directory.

To resolve

Edit the PR title scope so it covers the changed package directory (for example, use fix(code): ... for libs/code/**), or move the files so they match the declared scope.

If intentional

Apply the allow-scope-mismatch label. The check re-runs and passes with an informational note.

open-swe

✅ Open SWE Review: No issues found

Open SWE reviewed this PR and found no potential bugs to report.

Open in Web • View Open SWE trace

github-actions · 2026-07-16T15:00:11Z

This PR has been open for at least 14 days.

It will be closed automatically once it has been open for at least 30 days and this warning is at least 16 days old, unless the do-not-close label is applied.

Mason Daugherty (mdrxy) · 2026-07-16T20:14:26Z

maybe instead of system prompt artificla msg when goal is added?

…ager-calling-evals # Conflicts: # libs/code/deepagents_code/goal_tools.py # libs/code/tests/unit_tests/smoke_tests/snapshots/system_prompt_interactive_local.md

…ager-calling-evals # Conflicts: # libs/code/deepagents_code/app.py

…ager-calling-evals

…ager-calling-evals # Conflicts: # libs/evals/tests/unit_tests/test_harbor_langgraph_agent.py

Mason Daugherty (mdrxy) · 2026-07-23T01:24:08Z

Superseded by #4980.

The replacement carries forward the goal-state notices, goal-tool gating, internal-message filtering, and relevant eval coverage, reconciled with #4944’s accepted-goal continuation. This avoids landing two overlapping hidden-message protocols for the same goal-activation transition.

Closing this PR in favor of #4980.

Supersedes #4412 Supersedes #4944 Accepted goals now transition into agent work through one coordinated internal protocol: a durable notice records the authoritative goal state, while a hidden continuation resumes work without replaying the objective as another user message. --- The previous PRs addressed two symptoms at the same transition. #4944 changed how work resumes after goal acceptance, while #4412 introduced persisted notices that tell the agent when goal tools are relevant. Landing them independently would create overlapping internal-message lifecycles with separate persistence, restoration, summarization, and transcript-filtering behavior. This replacement gives the two message types explicit roles: - The goal-state notice is the durable source of truth. - The continuation is a one-time trigger emitted only after that state is available. It also handles persistence failure without directing the agent to nonexistent saved state and keeps all internal goal-control messages out of user-visible and grader transcripts. Goal-specific filtering remains in `deepagents-code`; the replacement does not change the Deep Agents SDK.

Supersedes langchain-ai#4412 Supersedes langchain-ai#4944 Accepted goals now transition into agent work through one coordinated internal protocol: a durable notice records the authoritative goal state, while a hidden continuation resumes work without replaying the objective as another user message. --- The previous PRs addressed two symptoms at the same transition. langchain-ai#4944 changed how work resumes after goal acceptance, while langchain-ai#4412 introduced persisted notices that tell the agent when goal tools are relevant. Landing them independently would create overlapping internal-message lifecycles with separate persistence, restoration, summarization, and transcript-filtering behavior. This replacement gives the two message types explicit roles: - The goal-state notice is the durable source of truth. - The continuation is a one-time trigger emitted only after that state is available. It also handles persistence failure without directing the agent to nonexistent saved state and keeps all internal goal-control messages out of user-visible and grader transcripts. Goal-specific filtering remains in `deepagents-code`; the replacement does not change the Deep Agents SDK.

github-actions Bot added dcode Related to `deepagents-code` evals Evaluation suite and Harbor integration feature New feature/enhancement or request for one internal User is a member of the `langchain-ai` GitHub organization open-swe size: M 200-499 LOC labels Jul 1, 2026

Merge branch 'main' into open-swe/goal-tools-eager-calling-evals

84cc3eb

Mason Daugherty (mdrxy) marked this pull request as ready for review July 6, 2026 22:17

open-swe Bot reviewed Jul 6, 2026

View reviewed changes

Mason Daugherty (mdrxy) and others added 2 commits July 6, 2026 18:33

Merge branch 'main' into open-swe/goal-tools-eager-calling-evals

4c107d2

cr

15b62a2

github-actions Bot added size: L 500-999 LOC and removed size: M 200-499 LOC labels Jul 7, 2026

Mason Daugherty (mdrxy) and others added 5 commits July 7, 2026 13:40

Merge branch 'main' into open-swe/goal-tools-eager-calling-evals

8bba242

Merge branch 'main' into open-swe/goal-tools-eager-calling-evals

a1b5f87

merge

45839f8

cr

e7ed630

Merge branch 'main' into open-swe/goal-tools-eager-calling-evals

27eff6c

Mason Daugherty (mdrxy) added the do-not-close Bypass PR auto-deletion label Jul 16, 2026

Mason Daugherty (mdrxy) and others added 3 commits July 17, 2026 16:03

Merge remote-tracking branch 'origin/main' into open-swe/goal-tools-e…

252c01d

…ager-calling-evals # Conflicts: # libs/code/deepagents_code/goal_tools.py # libs/code/tests/unit_tests/smoke_tests/snapshots/system_prompt_interactive_local.md

Merge branch 'main' into open-swe/goal-tools-eager-calling-evals

286ab41

feat(code): support eager goal tool calls

41dcd56

github-actions Bot added deepagents Related to the `deepagents` SDK / agent harness size: XL 1000+ LOC and removed size: L 500-999 LOC labels Jul 21, 2026

Merge remote-tracking branch 'origin/main' into open-swe/goal-tools-e…

ba430ba

…ager-calling-evals # Conflicts: # libs/code/deepagents_code/app.py

Mason Daugherty (mdrxy) and others added 7 commits July 21, 2026 15:04

Merge branch 'main' into open-swe/goal-tools-eager-calling-evals

14b0eb6

Merge remote-tracking branch 'origin/main' into open-swe/goal-tools-e…

0f245ba

…ager-calling-evals

Merge remote-tracking branch 'origin/main' into open-swe/goal-tools-e…

de87437

…ager-calling-evals # Conflicts: # libs/evals/tests/unit_tests/test_harbor_langgraph_agent.py

Merge branch 'main' into open-swe/goal-tools-eager-calling-evals

9ed837d

fix(code): skip goal notice setup for criteria runs

6bfe5d6

Merge branch 'main' into open-swe/goal-tools-eager-calling-evals

baff155

Merge branch 'main' into open-swe/goal-tools-eager-calling-evals

27e001d

Mason Daugherty (mdrxy) mentioned this pull request Jul 23, 2026

fix(code,evals): unify goal activation signaling #4980

Merged

Mason Daugherty (mdrxy) closed this Jul 23, 2026

Mason Daugherty (mdrxy) mentioned this pull request Jul 23, 2026

fix(code): stop replaying accepted /goal objective as a user turn #4944

Closed

Navigation Menu

Uh oh!

feat(evals): goal-tool over-eager-calling evals + gated prompt copy - #4412

feat(evals): goal-tool over-eager-calling evals + gated prompt copy#4412
Mason Daugherty (mdrxy) wants to merge 20 commits into
mainfrom
open-swe/goal-tools-eager-calling-evals

Mason Daugherty (mdrxy) commented Jul 1, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 1, 2026 •

edited

Loading

Uh oh!

open-swe Bot left a comment

Uh oh!

github-actions Bot commented Jul 16, 2026

Uh oh!

Mason Daugherty (mdrxy) commented Jul 16, 2026

Uh oh!

Mason Daugherty (mdrxy) commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Mason Daugherty (mdrxy) commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

References

Uh oh!

github-actions Bot commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

To resolve

If intentional

Uh oh!

open-swe Bot left a comment

Choose a reason for hiding this comment

✅ Open SWE Review: No issues found

Uh oh!

github-actions Bot commented Jul 16, 2026

Uh oh!

Mason Daugherty (mdrxy) commented Jul 16, 2026

Uh oh!

Mason Daugherty (mdrxy) commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Mason Daugherty (mdrxy) commented Jul 1, 2026 •

edited

Loading

github-actions Bot commented Jul 1, 2026 •

edited

Loading