Skip to content

feat(code,evals): inject goal/rubric context, dropping get_goal/get_rubric - #5041

Merged
Mason Daugherty (mdrxy) merged 63 commits into
mainfrom
mdrxy/code/goal-read-tools-removal
Aug 24, 2026
Merged

feat(code,evals): inject goal/rubric context, dropping get_goal/get_rubric#5041
Mason Daugherty (mdrxy) merged 63 commits into
mainfrom
mdrxy/code/goal-read-tools-removal

Conversation

@mdrxy

@mdrxy Mason Daugherty (mdrxy) commented Jul 24, 2026

Copy link
Copy Markdown
Member

dcode now automatically supplies active goal objectives and acceptance criteria as hidden internal model context, so the agent can begin and continue goal-directed work without get_goal or get_rubric tool calls.


/goal still owns the persistent, user-reviewed objective and generated criteria. The checkpoint fields remain the source of truth; GoalToolsMiddleware renders a current goal-state notice from them whenever the model needs it. The notice uses a framework-owned lc_source, so it is excluded from user-facing conversation history, thread titles, and derived grading transcripts even though it travels through the model as a HumanMessage.

The notice includes:

  • An actionable goal (active or blocked): objective and any blocker/status note.
  • An active rubric: acceptance criteria, including a one-turn rubric that remains applicable while a saved goal is paused.

update_goal remains the model’s constrained status-reporting tool. Creation, amendment, pause, resume, and clearing stay user-controlled.

The middleware refreshes the notice after goal/rubric changes and re-pins it when compaction makes the saved copy invisible. It also replaces superseded notices in the model request so stale objectives do not compete with current state.

Safeguards
  • State is validated and bounded before it can become automatic model context; oversized restored state produces recovery guidance instead.
  • Embedded objective, criteria, and status text is escaped and labeled as user-provided context data, not higher-priority instructions.
  • Regression coverage verifies detailed goal context cannot become a user-visible thread title.

Remove the `get_goal` and `get_rubric` read tools from the agent tool
surface. The goal-state notice now embeds the objective and acceptance
criteria (escaped, tagged) when a goal is actionable or a rubric is
active, so the model no longer needs a read tool to know what it is
working on. Inactive states stay coarse and instruct the model not to
act on a prior goal. The constrained write-side `update_goal` tool is
retained. Rubric grading stays in `RubricMiddleware.after_agent`.

The goal-state notice message schema version is bumped so a resumed
thread holding a stale coarse notice (which named the removed tools) is
superseded by a fresh notice.

Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
@github-actions github-actions Bot added dcode Related to `deepagents-code` evals Evaluation suite and Harbor integration feature New feature/enhancement or request for one internal User is a member of the `langchain-ai` GitHub organization size: L 500-999 LOC labels Jul 24, 2026
@mdrxy Mason Daugherty (mdrxy) changed the title feat(code): inject goal/rubric context and drop get_goal/get_rubric feat(code,evals): inject goal/rubric context and drop get_goal/get_rubric Jul 24, 2026
@mdrxy
Mason Daugherty (mdrxy) marked this pull request as ready for review July 24, 2026 05:46

@open-swe open-swe Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Open SWE Review: No issues found

Open SWE reviewed this PR and found no potential bugs to report.

Open in WebView Open SWE trace

Mason Daugherty (mdrxy) and others added 2 commits July 30, 2026 23:33
The goal-state notice told the model to work toward a goal in states that
had none. Its guidance branch was keyed on `is_actionable or has_rubric`
while asserting both a goal and automatic grading, so a rubric with no
goal — or a one-shot rubric outliving a paused goal, which `app.py` allows
on purpose — rendered "Goal actionable: no" directly above "Work toward
the goal", and an actionable goal with no rubric promised a verdict
`RubricMiddleware` never produces. Guidance now names only what is live.

Restore the goal status note to the model's view. It was projected and
fingerprinted but never rendered, and `get_goal` had been its only reader,
so a blocked goal's own recorded blocker became unreachable once the
originating tool message was summarized away. It renders in a tagged
section gated on actionability, like the objective.

Bound embedded objective/criteria/note text at `EMBEDDED_TEXT_LIMIT`,
slicing before escaping so truncation cannot split an entity. The notice
is re-pinned into every request on an append-only channel, so unbounded
user text grew context on every fingerprint change. The marker states the
full text still applies, because grading reads the rubric from state and
no read tool remains to fetch the remainder.

Share `summarization_cutoff` between `app.py` and `goal_tools.py`. The
middleware predicate lacked the cutoff term its client-side twin has, so a
notice below the cutoff — present in persisted history but invisible to the
model — suppressed the durable write and left the transient re-pin to carry
the objective every turn. The cutoff is an absolute index, so it is passed
only from `before_model`; `wrap_model_call` sees an already-trimmed window
whose indices are relative to it.

Add `REMOVED_GOAL_TOOL_NAMES` so the evals still assert the model does not
call `get_goal`/`get_rubric`, and a contract test that they are never
re-registered. Shrinking `GOAL_TOOL_NAMES` had removed the only signal that
would catch the model acting on a stale schema-version-1 notice.

Replace the recomputed actionability rule in the notice byte-stability test
with an explicit expectation table, and cover what previously survived
mutation: the schema-version floor, prior-version rejection, fingerprint
sensitivity to objective/criteria/note text, the async re-pin, the cutoff
refresh, truncation, and the paused-plus-one-shot-rubric shape.

Correct docstrings that misattributed `project_goal_state` as a caller of
`coerce_goal_status` (it inlines the vocabulary and must be kept in sync by
hand), overclaimed acceptance criteria that may not exist, omitted
`update_goal`'s paused/complete rejection and its unsaved-goal path, and
overstated what escaping prevents.

Relock `libs/evals` against the local `deepagents` 0.7.1, which the
pre-commit `uv run` step regenerates.
@github-actions github-actions Bot added size: XL 1000+ LOC and removed size: L 500-999 LOC labels Jul 31, 2026

@open-swe open-swe Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Open SWE Review found 1 potential issue.

Open in WebView Open SWE trace

Comment thread libs/code/deepagents_code/goal_state_notice.py Outdated
@github-actions

This comment has been minimized.

@github-actions github-actions Bot added the pending-deletion https://github.com/langchain-ai/deepagents/pull/4998 label Aug 7, 2026
@mdrxy Mason Daugherty (mdrxy) added do-not-close Bypass PR auto-deletion and removed pending-deletion https://github.com/langchain-ai/deepagents/pull/4998 labels Aug 7, 2026

@open-swe open-swe Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Open SWE Review found 1 potential issue.

Open in WebView Open SWE trace

Comment thread libs/code/deepagents_code/app.py
…udget

`/rubric next` only checked the per-rubric 12,000-character limit, so a
rubric that overflowed the 16,000-character combined notice budget (once
the active goal's objective and status note were included) was accepted
with "Rubric set for next turn." but then silently dropped at turn time:
`validate_goal_notice_text` rejected the pair, the middleware cleared the
invocation rubric, and the fallback notice marked both goal and rubric
inactive — skipping the promised one-shot grade.

Validate the candidate rubric together with the currently actionable
objective and status note before storing it, mirroring the combined
check `/goal resume` already performs, and reject up front with a
user-facing error instead.
Mason Daugherty (mdrxy) and others added 12 commits August 19, 2026 11:20
The notice filter removed superseded goal-state messages from the model
request. The summarizer this middleware wraps reads its messages from
that filtered list, but persists the cutoff it derives as an absolute
index into `state["messages"]`, which is never filtered. Every removal
made the two coordinate systems disagree by the number dropped, so the
persisted cutoff sliced the checkpointed list too early on the next
turn: live turns vanished, and a `ToolMessage` could outlive the
`AIMessage` that called it, which the provider rejects.

The read side was already guarded by flooring the filter at the cutoff.
The write side was not, and with no prior event the floor was 0, so the
whole history was fair game.

Replace each superseded notice in place with a fixed-size stand-in. The
length, every later index, and the human/AI/tool shape now match the
checkpointed list, so any cutoff the summarizer picks is valid against
both. The stand-in keeps the original id and carries its own source, so
it cannot win `latest_goal_state_message_index` over the notice it
yields to.
A proposal that overshot the combined objective-plus-criteria budget
never reported that fact to the user. The rejection was raised inside a
pydantic `model_validator`, so pydantic turned it into a
`ValidationError`, the parser rewrapped that as a plain `ValueError`, and
the structured-output loop retried it like any other validation failure.
The model cannot see the combined budget — the schema publishes only the
two per-field `max_length` values, whose sum exceeds it — so it retried
blind until the recursion limit. The resulting `GraphRecursionError` was
logged as a context fault, spent the fallback agent on the same request,
and finally surfaced as "could not generate acceptance criteria".

The two `except GoalStateSizeError: raise` clauses meant to prevent this
could never fire: nothing in the loop raises that type, and the error
object itself is unrecoverable from the exception the agent hands to
`handle_errors` — pydantic does not chain a `ValidationError` to the
error its validator raised, and the `StructuredOutputValidationError` has
not been raised yet, so it has neither `__cause__` nor `__context__`.

Install a `handle_errors` callable that re-runs the check against the
rejected tool-call arguments and raises a genuine `GoalStateSizeError`
carrying the real limit and excess, which ends the turn. Scope it to the
case the model cannot see: when either field also overshoots its own
published `max_length`, the retry still happens, because shortening that
field often brings the total inside the budget too. Publish the combined
limit in the system prompt so the model can avoid the failure. Split
`validate_goal_application_total` out for the combined-only check.

Replace the tests that mocked a bare `GoalStateSizeError` — an exception
the real graph cannot produce — with ones that wrap an invalid proposal
through the actual parser and agent layering.
`coerce_goal_status` maps an unrecognized `_goal_status` to `None`
specifically so an unknown status is treated as "no goal status" instead
of a silently active goal. `project_goal_state` mapped the same value to
`active`, unlogged, which made it actionable and rendered "Work toward
the goal".

Since the read tools were removed, that projection is the only goal
channel the model has. A corrupt or forward-version checkpoint therefore
told the model to start working toward a goal the TUI reports as absent,
while `_warn_discarded_goal_channels` logged and notified about the very
same value on the client.

Degrade an unrecognized status to `paused` and log it. The objective
stays on record; only its ability to drive work is revoked. A *missing*
status still defaults to `active`, because goals predate the status
channel, so absence means "never recorded" rather than "corrupt".
The unsaved-criteria plumbing was only covered at the callee, by a test
that invoked `_continue_created_goal_work` with hand-written arguments.
That pins the forwarding, not any caller's choice of argument. The two
tests that drive the real caller chain asserted the objective reached the
message but never the criteria, so replacing `application.rubric` with
`None` at either call site left the whole suite green.

That mutation is the failure the plumbing exists to prevent: the save
failed, so no goal-state notice is written for the thread, and with the
read tools gone the continuation is the only channel left. Criteria
dropped there leave the model working toward a goal it cannot grade
itself against.

Assert the criteria in both persist-failure tests. Verified each caller
individually: nulling either one now fails its test.
The middleware logged a malformed `_summarization_event`, with the
reasoning that dropping it also drops the summary, so the next request
silently re-sends the whole untrimmed history — a large token and latency
cost whose only symptom is a slow turn. The client read the same corrupt
event twice and said nothing:

- `_effective_conversation` returned the full list.
- `_ensure_goal_state_notice` collapsed the cutoff to `0`, which makes the
  `latest[0] >= cutoff` freshness test trivially true, so a stale notice
  counts as visible and the durable write is skipped.

The client is the side that reads possibly-malformed remote snapshot
dicts, as its own docstring says, so it is the likelier place to meet one.

Move the logger to `goal_state_notice`, beside the cutoff validation both
sides already share, and call it from all three places. Bound the repr
while moving it: a non-Mapping event is logged whole, so a list or a
message object would have spilled into the log.
Three `_summarization_cutoff` calls in the `/offload` path omitted
`message_count`, so they trusted an out-of-bounds cutoff while
`_effective_conversation` — reading the same event a few lines away —
rejected it. The two then disagree: `messages_offloaded` inflates,
`messages_kept` collapses to 0, and the token counts still come from the
full history, so the user is shown a large offload beside roughly zero
savings, with nothing logged.

Pass the count of the list each cutoff indexes. The message counts were
already in scope at all three sites.
Four places where an illegal state was representable:

- `GOAL_NOTICE_TEXT_CHAR_LIMIT` documented that it equals
  `GOAL_APPLICATION_CHAR_LIMIT + GOAL_STATUS_NOTE_CHAR_LIMIT`, which
  `validate_goal_notice_text`'s layering depends on, but was written out
  as a literal. Derive it, and pin the equality.
- `validate_goal_status_note`'s `label` accepted the whole
  `GoalStateSizeLabel` set, so a note overflow could be reported under a
  budget with nothing to do with notes — and the message reaches the model
  verbatim. Narrow it to `GoalStatusNoteLabel`.
- `rubric_source` was `str | None` while only three values are ever
  produced, and it is hashed into the state fingerprint, so a typo would
  silently change notice identity and force a fresh notice every turn.
  Narrow it to `RubricSource`.
- `notice_text_sections` returned three same-typed optionals unpacked
  positionally at five sites, each re-passing them as keyword arguments to
  `validate_goal_notice_text`, where a swap type-checks and validates the
  wrong text against the wrong budget. Return a `NoticeTextSections`
  `NamedTuple`; unpacking still works.

Also record why `_resume_goal` bypasses `notice_text_sections` — the goal
is still paused there, so projecting would withhold the very text the
notice is about to embed — and name the status vocabulary in the
`goal_state_limits` module docstring.

Verified with `ty` that a bad `label` and a bad `rubric_source` are now
type errors.
Four sites each ran the same project-then-validate sequence:
`notice_text_sections(project_goal_state(state))` followed by
`validate_goal_notice_text(...)`. Their correctness depended on all four
projecting exactly as the renderer does — the fragility
`notice_text_sections` documents — and two of them already wrapped the
sequence in a private helper that only wanted `str(exc)`.

Add `goal_notice_size_error(state, *, criteria_override=None)` beside the
projection it depends on, and route all four through it. The override
covers `/rubric next`, which validates a candidate rubric in place of the
committed one.

The renderer keeps its own inline checks: it already holds the projection
for the status and actionability lines, and it runs a second validation
for `prior_blocker` that no caller shares.
`/rubric` help stated its character limit but `/goal` did not, so a long
objective learned its limit only from a rejection — after a full criteria
round trip in the combined case. State both limits in `/goal` help, and
the rubric limit in `--rubric` argparse help.

THREAT_MODEL:

- The header pinned commit `4a458cd3a`, a branch-local merge commit that
  will not survive a squash merge, so the pin was unresolvable for exactly
  the reader wanting to reproduce the analysis. Drop it and date the
  header to the latest changelog entry.
- T16 described `/rubric file` as persisting "the entire selected file",
  which contradicted TB12 and DF26 — both of which record the enforced
  12,000-character limit. The disclosure risk is real; the unbounded
  framing is not.
- Add a changelog entry for the fixes in this series.

Also point the three restatements of the summarization-cutoff rule at
`validated_summarization_cutoff`, which now carries the canonical
explanation, and split the longest compound sentences.
Three paths had no test, all of them reached only when something has
already gone wrong — which is when an unhelpful message costs the most:

- The goal-proposal worker-error net branches on `GoalStateSizeError` to
  keep the character limit instead of "failed unexpectedly". Only the
  turn-handler twin was covered. Verified the parametrized test fails when
  the branch is removed.
- `_hint_invalid_submission` logs instead of hinting when there is no help
  widget. `_submit_edit` has already returned by then, so Enter looks like
  a no-op; a refactor dropping the `logger.warning` would restore a fully
  silent failure. Verified the test fails when the message changes.
- An oversized legacy `_goal_status_note` trips the `update_goal` size
  guard, and the model cannot shorten the note because `update_goal` is
  the call that would do it. Recovery is user-only. Every other oversized
  test triggers on the objective or the rubric, so the case where the
  model's own prior write locks it out was untested. Record the intent
  beside the guard.

Also record why `--rubric` needs only the standalone limit check, unlike
`/rubric next`: it requires `-n`, and `run_non_interactive` takes no resume
or thread-id argument, so no checkpointed goal can coexist with it.

@open-swe open-swe Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Open SWE Review found 2 potential issues.

Open in WebView Open SWE trace

Comment thread libs/code/deepagents_code/goal_state_limits.py
Comment thread libs/code/deepagents_code/goal_state_notice.py
`validate_goal_application` checked raw character counts only, but
`build_goal_state_notice` escapes embedded text before validating the
rendered total. A raw-valid objective and criteria could therefore be
accepted — with the UI reporting success and persisting the goal — and
then render past `GOAL_NOTICE_TEXT_CHAR_LIMIT`, so the very first notice
fell back to "unavailable: do not work toward it and do not grade".

- `validate_goal_application` now also validates the escaped pair
  against `GOAL_NOTICE_TEXT_CHAR_LIMIT`. The wider aggregate (rather
  than the raw 12,000) keeps the widest ordinary accepted pair valid
  while guaranteeing a full status-note budget stays in reserve.
- `_propose_goal_rubric` gains an early objective-only escaped check,
  since the criteria model counts raw characters and no proposal could
  survive acceptance when the objective alone fills the notice.
- The criteria-agent retry handler deliberately stays raw-only: the
  schema and prompt publish raw limits, so a rendered overshoot is
  reported to the user rather than retried blind.

Addresses #5041 (comment)
…paused

A corrupt or forward-version checkpoint could carry a goal status string
that `coerce_goal_status` does not recognize. Restore dropped it to
`None`, which left the goal stuck: the model-visible notice already
projected the same value as `paused` (so the model would not work it),
but `/goal resume` requires exactly `"paused"`, so neither the model nor
the user could move the goal without clearing and recreating it.

Restore now normalizes the unrecognized status to the same recoverable
`paused` the notice shows and persists it back, so the checkpoint itself
becomes resumable and the two reads stop disagreeing.

@open-swe open-swe Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Open SWE Review found 1 potential issue.

Open in WebView Open SWE trace

Comment thread libs/code/deepagents_code/app.py
The `paused` normalization added for unrecognized persisted statuses also
caught goals whose checkpoint predates the `_goal_status` channel:
`coerce_goal_status` yields `None` for both, so resuming an old thread
silently paused its active goal until `/goal resume` was run.

Thread a `goal_status_recorded` flag through the history payload so
restore can tell the two apart: only a *recorded* but unrecognized status
normalizes to `paused`; a missing channel keeps the implicit pre-status
`active` the notice projects.
@mdrxy Mason Daugherty (mdrxy) changed the title feat(code,evals): inject goal/rubric context and drop get_goal/get_rubric feat(code,evals): inject goal/rubric context, dropping get_goal/get_rubric Aug 19, 2026
@mdrxy
Mason Daugherty (mdrxy) merged commit 2c80153 into main Aug 24, 2026
74 checks passed
@mdrxy
Mason Daugherty (mdrxy) deleted the mdrxy/code/goal-read-tools-removal branch August 24, 2026 01:59
Mason Daugherty (mdrxy) pushed a commit that referenced this pull request Aug 24, 2026
> [!CAUTION]
> Merging this PR will automatically publish to **PyPI** and create a
**GitHub release**.

For the full release process, see
[`.github/RELEASING.md`](https://github.com/langchain-ai/deepagents/blob/main/.github/RELEASING.md).

---

_Release notes preview: keep this section in sync with the package
`CHANGELOG.md`. Publish reads the merged CHANGELOG via `release.yml`,
not this PR description — keep them aligned anyway so the PR stays an
accurate historical record for reviewers and anyone returning later._

---


##
[0.1.61](deepagents-code==0.1.60...deepagents-code==0.1.61)
(2026-08-24)

### Features

- Added `google_anthropic_vertex` provider support for Claude on Vertex
AI ([#5760](#5760)).
- Enforced configured model allowlists
([#5649](#5649)).
- Injected goal and rubric context directly, replacing `get_goal` and
`get_rubric`
([#5041](#5041)).
- Made `/offload` server-owned
([#5261](#5261)).
- Added prompt clipboard support
([#5733](#5733)).
- Show Auto approval review progress
([#5729](#5729)).

### Bug Fixes

- Kept long thread resumes responsive
([#5772](#5772)).
- Render first streamed text immediately
([#5761](#5761)).
- Show the incognito shell command widget
([#5768](#5768)).
- Only highlight actionable tool rows
([#5769](#5769)).
- Warn and ignore `--auto-approve` and `--yolo` in headless mode
([#5750](#5750)).
- Sweep expired history archives at startup
([#5751](#5751)).
- Clarified auth environment setup
([#5767](#5767)).

_End release notes preview._

---

> [!NOTE]
> A **community contributors** list and a **Special thanks** section
(crediting the users who filed the issues this release's PRs closed) are
appended to the GitHub release notes automatically at publish time (see
[Release
Pipeline](https://github.com/langchain-ai/deepagents/blob/main/.github/RELEASING.md#release-pipeline),
step 3).

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: langchain-oss-automated-triage[bot] <248757908+langchain-oss-automated-triage[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dcode Related to `deepagents-code` do-not-close Bypass PR auto-deletion evals Evaluation suite and Harbor integration feature New feature/enhancement or request for one internal User is a member of the `langchain-ai` GitHub organization size: XL 1000+ LOC

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant