feat(compaction): structured summary output with template rendering - #10471
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 46819bf79b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0e9a442ada
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
0e9a442 to
fd30354
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fd3035423c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
fd30354 to
9738ede
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9738edeb72
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
9738ede to
2069edb
Compare
The compaction LLM now returns a JSON document (sections ordered by importance) that is rendered into the post-compaction context via a user-overridable minijinja template. Any parse failure falls back to keeping the raw response text.
…tim detail
Probe-eval A/B against the pre-PR baseline showed two weaknesses in the
structured summary as committed:
- Strict Vec<String> deserialization discarded the whole summary when the
model emitted list entries as objects (19/30 Haiku compactions fell back
to raw text, all from errors_and_fixes shaped as {error, fix} objects).
String fields now deserialize leniently by stringifying non-string values.
- The prompt invited paraphrase and spent most of the output budget on the
discarded analysis scratchpad. It now requires plain-string list entries,
verbatim error/panic/test output, a brief analysis with detail in the
JSON fields, and decision rationale (chosen/rejected/why).
With these changes the blind-judged fidelity delta vs the old summarizer is
within noise on both Haiku 4.5 and Sonnet 4.6 (n=120 paired probes each),
with structured parse success at 30/30 and about half the retained tokens.
2069edb to
310b5c5
Compare
A JSON object quoted inside a prose summary (schema-ignoring model or user-overridden prompt) could parse as a non-empty StructuredSummary and replace the entire raw summary with one rendered fragment. Candidates must now begin at the first non-whitespace position after their marker (json fence, </analysis>, or response start); JSON embedded mid-prose falls back to the lossless raw text.
310b5c5 to
57f2ace
Compare
|
Re-ran the compaction A/B with a small local summarizer (gpt-oss:20b via Ollama): structured beat the old prose compaction by +0.86/5 blind-judged fidelity (3.46 → 4.31, all six dimensions significant), 29/30 clean structured renders, 11% fewer retained tokens. Trace analysis shows that prose from a small model drops ~half the requested sections and paraphrases away the concrete facts (72% vs 95% ground-truth term coverage, code snippets survived in 0/30 prose summaries) while the PR introduced schema forces them to be kept. |
…nflict # Conflicts: # crates/goose/src/context_mgmt/mod.rs
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 10ae8289ca
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…JSON quotes </analysis> Summary JSON that legitimately contains the literal </analysis> (e.g. a session editing compaction prompts) made rfind slice mid-string, so no candidate matched and the structured render was lost to the raw-text fallback. Terminators are now tried last-first, and a candidate found after an earlier terminator is accepted only when it contains every later occurrence, so scratchpad examples still cannot leak through.
|
I like this. the on thing I would note though, is that it does not use the ability that a lot of models have to produce structured json natively. that gives us stronger guarantees that it will actually work, but probably fine to do in a follow up if we want to |
* origin/main: fix(ui): clear stale pending ACP connection after terminal recovery failure (#10552) Make provider smoke tests faster and more reliable (#10605) fix(acp): confirm pending steer message on queuedSteer notification (#10532) chore(deps): bump body-parser from 1.20.5 to 1.20.6 in /documentation (#10601) chore(deps): bump webpack-dev-server from 5.2.5 to 5.2.6 in /documentation (#10593) feat(compaction): structured summary output with template rendering (#10471) feat(skills): allow disabling built-in skills (#10600) fix: apply hermit env directly in node shims so a fish login shell doesn't break MCP startup (#10028) chore(release): bump version to 1.44.0 (minor) (#10597) fix(extensions): preserve command arguments through forms (#10527) fix(permissions): enforce manual approval for code mode (#10528) fix(apps): confine app file operations (#10481) fix(local-inference): preserve featured model size on delete and backfill missing sizes (#10422) refactor(acp): extract tool call handling from server (#10574)
…egory Structured compaction (#10471) turned the post-compaction summary into a template-rendered artifact with named sections, but the x-ray still reported it as one anonymous message inside "Conversation" - so the largest single block of retained context was the least legible thing in the panel. Compaction now marks its summary message, and the report gives it a `compaction_summary` category whose parts are the rendered `##` sections, each with its own token and character count. Section boundaries come from the rendered markdown rather than the parsed StructuredSummary, so the breakdown reports what the context window actually holds even when compaction_summary.md has been customized to drop or rename sections. Fenced blocks are skipped over because key_code can legally contain heading lines. Summaries that fell back to raw model output are labelled as such and keep a whole-text preview instead of invented sections, since splitting arbitrary prose on `##` would fabricate structure that isn't there. Segment totals are unchanged: the summary segment carries the same count_message_tokens value it contributed as a message part before.
2 participants
Problem
The compaction summary is an opaque prose blob: there is no way to control what survives compaction, order it by importance, or trim the least important parts without prompt surgery. The session's post-compaction context baseline also used the summarization call's raw output token count, overstating the actually retained context ~2.3x.
Implementation
<analysis>scratchpad plus one ```json block matchingStructuredSummary(`context_mgmt/structured.rs`): nine sections (user intent, technical concepts, files + key code, errors and fixes, problem solving, user messages, pending tasks, current work, next step), each ordered most-important-first.~/.config/goose/prompts/compaction_summary.md, so changing what the post-compaction context contains is a config edit, not a rebuild. Unknown JSON fields are preserved for custom prompt + template pairs.</analysis>, or at the response start, and wrong-shaped fields are stringified rather than rejecting the document. Any parse failure keeps the raw response verbatim, i.e. exactly the old behavior.code_fenceminijinja filter sizes fences past the longest backtick run so embedded fences cannot break the rendered block.Testing
goose runsessions with compaction forced between turns): task success 18/18 vs 17/18 for main, 18/18 structured renders with zero fallbacks, planted constraints survived compaction 100% in both arms, 12% fewer accumulated session tokens.