feat(code,evals): inject goal/rubric context, dropping get_goal/get_rubric - #5041
Merged
Mason Daugherty (mdrxy) merged 63 commits intoAug 24, 2026
Merged
Conversation
Remove the `get_goal` and `get_rubric` read tools from the agent tool surface. The goal-state notice now embeds the objective and acceptance criteria (escaped, tagged) when a goal is actionable or a rubric is active, so the model no longer needs a read tool to know what it is working on. Inactive states stay coarse and instruct the model not to act on a prior goal. The constrained write-side `update_goal` tool is retained. Rubric grading stays in `RubricMiddleware.after_agent`. The goal-state notice message schema version is bumped so a resumed thread holding a stale coarse notice (which named the removed tools) is superseded by a fresh notice. Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
get_goal/get_rubricget_goal/get_rubric
Mason Daugherty (mdrxy)
marked this pull request as ready for review
July 24, 2026 05:46
The goal-state notice told the model to work toward a goal in states that had none. Its guidance branch was keyed on `is_actionable or has_rubric` while asserting both a goal and automatic grading, so a rubric with no goal — or a one-shot rubric outliving a paused goal, which `app.py` allows on purpose — rendered "Goal actionable: no" directly above "Work toward the goal", and an actionable goal with no rubric promised a verdict `RubricMiddleware` never produces. Guidance now names only what is live. Restore the goal status note to the model's view. It was projected and fingerprinted but never rendered, and `get_goal` had been its only reader, so a blocked goal's own recorded blocker became unreachable once the originating tool message was summarized away. It renders in a tagged section gated on actionability, like the objective. Bound embedded objective/criteria/note text at `EMBEDDED_TEXT_LIMIT`, slicing before escaping so truncation cannot split an entity. The notice is re-pinned into every request on an append-only channel, so unbounded user text grew context on every fingerprint change. The marker states the full text still applies, because grading reads the rubric from state and no read tool remains to fetch the remainder. Share `summarization_cutoff` between `app.py` and `goal_tools.py`. The middleware predicate lacked the cutoff term its client-side twin has, so a notice below the cutoff — present in persisted history but invisible to the model — suppressed the durable write and left the transient re-pin to carry the objective every turn. The cutoff is an absolute index, so it is passed only from `before_model`; `wrap_model_call` sees an already-trimmed window whose indices are relative to it. Add `REMOVED_GOAL_TOOL_NAMES` so the evals still assert the model does not call `get_goal`/`get_rubric`, and a contract test that they are never re-registered. Shrinking `GOAL_TOOL_NAMES` had removed the only signal that would catch the model acting on a stale schema-version-1 notice. Replace the recomputed actionability rule in the notice byte-stability test with an explicit expectation table, and cover what previously survived mutation: the schema-version floor, prior-version rejection, fingerprint sensitivity to objective/criteria/note text, the async re-pin, the cutoff refresh, truncation, and the paused-plus-one-shot-rubric shape. Correct docstrings that misattributed `project_goal_state` as a caller of `coerce_goal_status` (it inlines the vocabulary and must be kept in sync by hand), overclaimed acceptance criteria that may not exist, omitted `update_goal`'s paused/complete rejection and its unsaved-goal path, and overstated what escaping prevents. Relock `libs/evals` against the local `deepagents` 0.7.1, which the pre-commit `uv run` step regenerates.
…tools-removal # Conflicts: # libs/code/deepagents_code/agent.py # libs/evals/uv.lock
This comment has been minimized.
This comment has been minimized.
…udget `/rubric next` only checked the per-rubric 12,000-character limit, so a rubric that overflowed the 16,000-character combined notice budget (once the active goal's objective and status note were included) was accepted with "Rubric set for next turn." but then silently dropped at turn time: `validate_goal_notice_text` rejected the pair, the middleware cleared the invocation rubric, and the fallback notice marked both goal and rubric inactive — skipping the promised one-shot grade. Validate the candidate rubric together with the currently actionable objective and status note before storing it, mirroring the combined check `/goal resume` already performs, and reject up front with a user-facing error instead.
The notice filter removed superseded goal-state messages from the model request. The summarizer this middleware wraps reads its messages from that filtered list, but persists the cutoff it derives as an absolute index into `state["messages"]`, which is never filtered. Every removal made the two coordinate systems disagree by the number dropped, so the persisted cutoff sliced the checkpointed list too early on the next turn: live turns vanished, and a `ToolMessage` could outlive the `AIMessage` that called it, which the provider rejects. The read side was already guarded by flooring the filter at the cutoff. The write side was not, and with no prior event the floor was 0, so the whole history was fair game. Replace each superseded notice in place with a fixed-size stand-in. The length, every later index, and the human/AI/tool shape now match the checkpointed list, so any cutoff the summarizer picks is valid against both. The stand-in keeps the original id and carries its own source, so it cannot win `latest_goal_state_message_index` over the notice it yields to.
A proposal that overshot the combined objective-plus-criteria budget never reported that fact to the user. The rejection was raised inside a pydantic `model_validator`, so pydantic turned it into a `ValidationError`, the parser rewrapped that as a plain `ValueError`, and the structured-output loop retried it like any other validation failure. The model cannot see the combined budget — the schema publishes only the two per-field `max_length` values, whose sum exceeds it — so it retried blind until the recursion limit. The resulting `GraphRecursionError` was logged as a context fault, spent the fallback agent on the same request, and finally surfaced as "could not generate acceptance criteria". The two `except GoalStateSizeError: raise` clauses meant to prevent this could never fire: nothing in the loop raises that type, and the error object itself is unrecoverable from the exception the agent hands to `handle_errors` — pydantic does not chain a `ValidationError` to the error its validator raised, and the `StructuredOutputValidationError` has not been raised yet, so it has neither `__cause__` nor `__context__`. Install a `handle_errors` callable that re-runs the check against the rejected tool-call arguments and raises a genuine `GoalStateSizeError` carrying the real limit and excess, which ends the turn. Scope it to the case the model cannot see: when either field also overshoots its own published `max_length`, the retry still happens, because shortening that field often brings the total inside the budget too. Publish the combined limit in the system prompt so the model can avoid the failure. Split `validate_goal_application_total` out for the combined-only check. Replace the tests that mocked a bare `GoalStateSizeError` — an exception the real graph cannot produce — with ones that wrap an invalid proposal through the actual parser and agent layering.
`coerce_goal_status` maps an unrecognized `_goal_status` to `None` specifically so an unknown status is treated as "no goal status" instead of a silently active goal. `project_goal_state` mapped the same value to `active`, unlogged, which made it actionable and rendered "Work toward the goal". Since the read tools were removed, that projection is the only goal channel the model has. A corrupt or forward-version checkpoint therefore told the model to start working toward a goal the TUI reports as absent, while `_warn_discarded_goal_channels` logged and notified about the very same value on the client. Degrade an unrecognized status to `paused` and log it. The objective stays on record; only its ability to drive work is revoked. A *missing* status still defaults to `active`, because goals predate the status channel, so absence means "never recorded" rather than "corrupt".
The unsaved-criteria plumbing was only covered at the callee, by a test that invoked `_continue_created_goal_work` with hand-written arguments. That pins the forwarding, not any caller's choice of argument. The two tests that drive the real caller chain asserted the objective reached the message but never the criteria, so replacing `application.rubric` with `None` at either call site left the whole suite green. That mutation is the failure the plumbing exists to prevent: the save failed, so no goal-state notice is written for the thread, and with the read tools gone the continuation is the only channel left. Criteria dropped there leave the model working toward a goal it cannot grade itself against. Assert the criteria in both persist-failure tests. Verified each caller individually: nulling either one now fails its test.
The middleware logged a malformed `_summarization_event`, with the reasoning that dropping it also drops the summary, so the next request silently re-sends the whole untrimmed history — a large token and latency cost whose only symptom is a slow turn. The client read the same corrupt event twice and said nothing: - `_effective_conversation` returned the full list. - `_ensure_goal_state_notice` collapsed the cutoff to `0`, which makes the `latest[0] >= cutoff` freshness test trivially true, so a stale notice counts as visible and the durable write is skipped. The client is the side that reads possibly-malformed remote snapshot dicts, as its own docstring says, so it is the likelier place to meet one. Move the logger to `goal_state_notice`, beside the cutoff validation both sides already share, and call it from all three places. Bound the repr while moving it: a non-Mapping event is logged whole, so a list or a message object would have spilled into the log.
Three `_summarization_cutoff` calls in the `/offload` path omitted `message_count`, so they trusted an out-of-bounds cutoff while `_effective_conversation` — reading the same event a few lines away — rejected it. The two then disagree: `messages_offloaded` inflates, `messages_kept` collapses to 0, and the token counts still come from the full history, so the user is shown a large offload beside roughly zero savings, with nothing logged. Pass the count of the list each cutoff indexes. The message counts were already in scope at all three sites.
Four places where an illegal state was representable: - `GOAL_NOTICE_TEXT_CHAR_LIMIT` documented that it equals `GOAL_APPLICATION_CHAR_LIMIT + GOAL_STATUS_NOTE_CHAR_LIMIT`, which `validate_goal_notice_text`'s layering depends on, but was written out as a literal. Derive it, and pin the equality. - `validate_goal_status_note`'s `label` accepted the whole `GoalStateSizeLabel` set, so a note overflow could be reported under a budget with nothing to do with notes — and the message reaches the model verbatim. Narrow it to `GoalStatusNoteLabel`. - `rubric_source` was `str | None` while only three values are ever produced, and it is hashed into the state fingerprint, so a typo would silently change notice identity and force a fresh notice every turn. Narrow it to `RubricSource`. - `notice_text_sections` returned three same-typed optionals unpacked positionally at five sites, each re-passing them as keyword arguments to `validate_goal_notice_text`, where a swap type-checks and validates the wrong text against the wrong budget. Return a `NoticeTextSections` `NamedTuple`; unpacking still works. Also record why `_resume_goal` bypasses `notice_text_sections` — the goal is still paused there, so projecting would withhold the very text the notice is about to embed — and name the status vocabulary in the `goal_state_limits` module docstring. Verified with `ty` that a bad `label` and a bad `rubric_source` are now type errors.
Four sites each ran the same project-then-validate sequence: `notice_text_sections(project_goal_state(state))` followed by `validate_goal_notice_text(...)`. Their correctness depended on all four projecting exactly as the renderer does — the fragility `notice_text_sections` documents — and two of them already wrapped the sequence in a private helper that only wanted `str(exc)`. Add `goal_notice_size_error(state, *, criteria_override=None)` beside the projection it depends on, and route all four through it. The override covers `/rubric next`, which validates a candidate rubric in place of the committed one. The renderer keeps its own inline checks: it already holds the projection for the status and actionability lines, and it runs a second validation for `prior_blocker` that no caller shares.
`/rubric` help stated its character limit but `/goal` did not, so a long objective learned its limit only from a rejection — after a full criteria round trip in the combined case. State both limits in `/goal` help, and the rubric limit in `--rubric` argparse help. THREAT_MODEL: - The header pinned commit `4a458cd3a`, a branch-local merge commit that will not survive a squash merge, so the pin was unresolvable for exactly the reader wanting to reproduce the analysis. Drop it and date the header to the latest changelog entry. - T16 described `/rubric file` as persisting "the entire selected file", which contradicted TB12 and DF26 — both of which record the enforced 12,000-character limit. The disclosure risk is real; the unbounded framing is not. - Add a changelog entry for the fixes in this series. Also point the three restatements of the summarization-cutoff rule at `validated_summarization_cutoff`, which now carries the canonical explanation, and split the longest compound sentences.
Three paths had no test, all of them reached only when something has already gone wrong — which is when an unhelpful message costs the most: - The goal-proposal worker-error net branches on `GoalStateSizeError` to keep the character limit instead of "failed unexpectedly". Only the turn-handler twin was covered. Verified the parametrized test fails when the branch is removed. - `_hint_invalid_submission` logs instead of hinting when there is no help widget. `_submit_edit` has already returned by then, so Enter looks like a no-op; a refactor dropping the `logger.warning` would restore a fully silent failure. Verified the test fails when the message changes. - An oversized legacy `_goal_status_note` trips the `update_goal` size guard, and the model cannot shorten the note because `update_goal` is the call that would do it. Recovery is user-only. Every other oversized test triggers on the objective or the rubric, so the case where the model's own prior write locks it out was untested. Record the intent beside the guard. Also record why `--rubric` needs only the standalone limit check, unlike `/rubric next`: it requires `-n`, and `run_non_interactive` takes no resume or thread-id argument, so no checkpointed goal can coexist with it.
`validate_goal_application` checked raw character counts only, but `build_goal_state_notice` escapes embedded text before validating the rendered total. A raw-valid objective and criteria could therefore be accepted — with the UI reporting success and persisting the goal — and then render past `GOAL_NOTICE_TEXT_CHAR_LIMIT`, so the very first notice fell back to "unavailable: do not work toward it and do not grade". - `validate_goal_application` now also validates the escaped pair against `GOAL_NOTICE_TEXT_CHAR_LIMIT`. The wider aggregate (rather than the raw 12,000) keeps the widest ordinary accepted pair valid while guaranteeing a full status-note budget stays in reserve. - `_propose_goal_rubric` gains an early objective-only escaped check, since the criteria model counts raw characters and no proposal could survive acceptance when the objective alone fills the notice. - The criteria-agent retry handler deliberately stays raw-only: the schema and prompt publish raw limits, so a rendered overshoot is reported to the user rather than retried blind. Addresses #5041 (comment)
…paused A corrupt or forward-version checkpoint could carry a goal status string that `coerce_goal_status` does not recognize. Restore dropped it to `None`, which left the goal stuck: the model-visible notice already projected the same value as `paused` (so the model would not work it), but `/goal resume` requires exactly `"paused"`, so neither the model nor the user could move the goal without clearing and recreating it. Restore now normalizes the unrecognized status to the same recoverable `paused` the notice shows and persists it back, so the checkpoint itself becomes resumable and the two reads stop disagreeing.
The `paused` normalization added for unrecognized persisted statuses also caught goals whose checkpoint predates the `_goal_status` channel: `coerce_goal_status` yields `None` for both, so resuming an old thread silently paused its active goal until `/goal resume` was run. Thread a `goal_status_recorded` flag through the history payload so restore can tell the two apart: only a *recorded* but unrecognized status normalizes to `paused`; a missing channel keeps the implicit pre-status `active` the notice projects.
get_goal/get_rubricget_goal/get_rubric
…tools-removal # Conflicts: # libs/code/THREAT_MODEL.md
…al' into mdrxy/code/goal-read-tools-removal
Mason Daugherty (mdrxy)
pushed a commit
that referenced
this pull request
Aug 24, 2026
> [!CAUTION] > Merging this PR will automatically publish to **PyPI** and create a **GitHub release**. For the full release process, see [`.github/RELEASING.md`](https://github.com/langchain-ai/deepagents/blob/main/.github/RELEASING.md). --- _Release notes preview: keep this section in sync with the package `CHANGELOG.md`. Publish reads the merged CHANGELOG via `release.yml`, not this PR description — keep them aligned anyway so the PR stays an accurate historical record for reviewers and anyone returning later._ --- ## [0.1.61](deepagents-code==0.1.60...deepagents-code==0.1.61) (2026-08-24) ### Features - Added `google_anthropic_vertex` provider support for Claude on Vertex AI ([#5760](#5760)). - Enforced configured model allowlists ([#5649](#5649)). - Injected goal and rubric context directly, replacing `get_goal` and `get_rubric` ([#5041](#5041)). - Made `/offload` server-owned ([#5261](#5261)). - Added prompt clipboard support ([#5733](#5733)). - Show Auto approval review progress ([#5729](#5729)). ### Bug Fixes - Kept long thread resumes responsive ([#5772](#5772)). - Render first streamed text immediately ([#5761](#5761)). - Show the incognito shell command widget ([#5768](#5768)). - Only highlight actionable tool rows ([#5769](#5769)). - Warn and ignore `--auto-approve` and `--yolo` in headless mode ([#5750](#5750)). - Sweep expired history archives at startup ([#5751](#5751)). - Clarified auth environment setup ([#5767](#5767)). _End release notes preview._ --- > [!NOTE] > A **community contributors** list and a **Special thanks** section (crediting the users who filed the issues this release's PRs closed) are appended to the GitHub release notes automatically at publish time (see [Release Pipeline](https://github.com/langchain-ai/deepagents/blob/main/.github/RELEASING.md#release-pipeline), step 3). --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: langchain-oss-automated-triage[bot] <248757908+langchain-oss-automated-triage[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
dcode now automatically supplies active goal objectives and acceptance criteria as hidden internal model context, so the agent can begin and continue goal-directed work without
get_goalorget_rubrictool calls./goalstill owns the persistent, user-reviewed objective and generated criteria. The checkpoint fields remain the source of truth;GoalToolsMiddlewarerenders a current goal-state notice from them whenever the model needs it. The notice uses a framework-ownedlc_source, so it is excluded from user-facing conversation history, thread titles, and derived grading transcripts even though it travels through the model as aHumanMessage.The notice includes:
activeorblocked): objective and any blocker/status note.update_goalremains the model’s constrained status-reporting tool. Creation, amendment, pause, resume, and clearing stay user-controlled.The middleware refreshes the notice after goal/rubric changes and re-pins it when compaction makes the saved copy invisible. It also replaces superseded notices in the model request so stale objectives do not compete with current state.
Safeguards