fix(kv-cache): inject subdirectory hints as tail messages

[mcelrath]

Background

LLM inference backends (llama.cpp, vLLM, etc.) cache key-value attention tensors for stable token prefixes. Any insertion or mutation before the tail of the current conversation history forces a full KV recomputation from the changed position forward. Goose has three distinct sites that trigger this on every turn.

UPDATE: Two of the three original fixes are now redundant with upstream and have been dropped from this PR:

• MOIM tail-append — superseded by Update Moim #9636 ("Update Moim", 09c8d2b), which moved MOIM to a tail block on the last user message.
• System-prompt timestamp — the rendered hourly timestamp was already removed by Platform Extension MOIM (Minus One Info Message) #5027 ("Platform Extension MOIM", 02ca6f7) back in Nov 2025; this PR only deletes the leftover vestigial current_date_timestamp field (no caching effect).

What remains is the subdirectory-hints fix: stop .goosehints/AGENTS.md discovered from tool-touched directories from permanently mutating the system prompt; inject them once per directory as a tail message instead.

=======

Three changes that together make Goose KV-cache-neutral across turns:

1. Remove timestamp from system prompt (prompt_manager.rs) The system prompt previously embedded the current date, causing it to change every hour and invalidating all cached KV tensors. The timestamp is removed entirely from the system prompt — it is no longer needed there since MOIM (below) delivers accurate time on every turn.

2. MOIM: tail-append with second-resolution timestamp (moim.rs, extension_manager.rs) inject_moim() previously inserted the working-directory context message before the last assistant message, shifting KV positions for all tail messages on every turn. Changed to messages.push() at the tail, which fix_conversation() merges with the existing last user message. Also updates the MOIM timestamp from minute-granularity ("%H:%M:00") to second-resolution ("%H:%M:%S"), replacing the timestamp removed from the system prompt with a more accurate value that is cache-neutral because it lives at the tail. [OBSOLETE by Update Moim #9636 09c8d2b]

3. Subdirectory hints as user-visible tail messages (prompt_manager.rs, agent.rs) When tools touch a new subdirectory, hints from .goosehints/AGENTS.md in that subtree were appended to system_prompt_extras, permanently mutating the system prompt for all subsequent turns. Changed to return the hint text from collect_new_subdirectory_hints() and inject it as a plain user message (with_text) at the tail. The message is visible in the UI so users can see when new subdirectory instructions are loaded. Each directory is loaded at most once per session (dedup tracker).

Also adds Message::with_agent_text() to goose-providers for agent-only content injection (used by MOIM), and HookEvent::AssistantResponse + emit_collect() to HookManager for settled-response hook support.

Summary

Testing

Related Issues

Fixes #9766
Discussion: LINK (if any)

Screenshots/Demos (for UX changes)

Before:

After:

[mcelrath]

Goose KV Cache PR: Provider Cost Analysis

What the PR Changed

Source: git show 5a89e3884 — three distinct fixes.

Fix 1: System prompt hourly timestamp removed (prompt_manager.rs)

The system prompt embedded the current date at hour granularity (2026-06-15 10:00:00).
This mutated the system prompt at each clock hour, invalidating the prefix cache at position 0
for all providers — one full cache reset per hour boundary crossed.

Fix 2: MOIM injection position (moim.rs)

Before:

let idx = messages.iter().rposition(|m| m.role == Role::Assistant).unwrap_or(0);
messages.insert(idx, Message::user().with_text(moim));

MOIM inserted before the last assistant message; fix_conversation merged it with the preceding
user message. At turn k, the sent array was:

[U1, A1, ..., U_{k-1}+MOIM_k, A_{k-1}, U_k]

After:

messages.push(Message::user().with_agent_text(moim));

MOIM appended at the tail, merged with the last user message:

[U1, A1, ..., U_{k-1}, A_{k-1}, U_k+MOIM_k]

Cache impact: The common prefix between consecutive turns k and k+1 (for k ≥ 2):

Version	Common prefix	Cached tokens
OLD	[U1..A_{k-2}] — diverges at U_{k-1} (MOIM merged in, absent next turn)	P + (k-2)·Δ
NEW	[U1..A_{k-1}] — diverges at U_k (MOIM at tail)	P + (k-1)·Δ

The PR recovers one expected Δ tokens of cached history per API call (Δ = C/N on average).
The old code was not a complete cache break — it lost one turn of cached history per request.

The per-second MOIM timestamp (%H:%M:%S) is safe with tail-append because it lives in the
last user message, which is never cached in either approach.

Fix 3: Subdirectory hints moved out of system prompt (prompt_manager.rs, agent.rs)

When goose explored a directory containing .goosehints/AGENTS.md, those hints were appended
to system_prompt_extras, permanently mutating the system prompt for all subsequent turns.
Each mutation caused a full cache reset from position 0.

This is the dominant cache invalidation source in real coding sessions. A typical session
exploring 5–15 subdirectories triggered 5–15 full cache resets. The PR injects hints as one-time
user messages at the tail instead, leaving the system prompt (and its cache entry) stable.

---

How Goose Caches Anthropic Conversations

Source: crates/goose/src/providers/formats/anthropic.rs lines 311–381.

Goose uses a sliding window strategy with up to 4 cache_control: {type: ephemeral} breakpoints
per request:

1. System prompt — always marked; establishes the base cache entry
2. Last tool spec — caches all tool definitions as a single prefix
3. Second-to-last user message — reads the growing cache from the previous turn
4. Last user message — writes a new cache entry for the next turn

The full conversation history is cached and grows each turn. At turn k WITH the PR, the cached
prefix is P + (k-1)·Δ tokens.

---

Session Parameters

Parameter	Value	Notes
Context window (C)	Model limit; capped at 200k	Upper bound; typical sessions ~30–50k
Turns (N)	100
Tool calls	50	Embedded across turns
Output per turn	1,000 tokens
System prefix (P)	8,000 tokens	System prompt + tool schemas
Turn delta (Δ)	C/N = 2,000 tokens	Average tokens added per turn

Total input billed without caching: Σ(k=1..N)(P + k·Δ) ≈ 10.9M tokens for a 200k session.

---

Provider Caching Survey

Discount figures are expressed as cost fraction of the normal input rate (lower = cheaper).
For example, Anthropic's 10% means cache reads cost $0.30/MTok when input is $3.00/MTok — a 90% saving.

Provider	Caching	Cache read cost (fraction of input rate)	TTL	Source
Anthropic	Explicit cache_control	10% (write = 125%)	5 min or 1 hr	Docs
OpenAI	Automatic prefix caching	50% (gpt-4o, o3, gpt-4.1-mini)	~1 hr	Docs
Google Gemini 2.5	Implicit automatic ⚠️	25%	Unspecified	Docs
Groq	Automatic prefix caching	50%	2 hours	Docs
Together AI	Selective (model-dependent)	11–58% on select models	Unspecified	Pricing

⚠️ Gemini implicit caching is not guaranteed per docs; treat Gemini rows as approximate. Sessions
where the accumulated prompt crosses the 200k token tier boundary will be billed at higher rates
for those turns; this analysis uses the ≤200k rate throughout.

Together AI's automatic caching applies to select models only (Qwen3.5-397B, DeepSeek-V4-Pro,
MiniMax M3). Qwen3-235B-A22B has no cached rate listed as of 2026-06-15.

Groq's 2h TTL means a session spanning >2 hours incurs one cache expiry even WITH the PR
(stable prefix, but cache evicted). Effect: ~$0.01–$0.03 extra; negligible.

---

Cost Model

WITH PR — turn 1 cold start, turns 2–N with stable cache hits:

• Hit turn k (k ≥ 2): (P + (k-1)·Δ) × read_rate + Δ × full_rate

WITHOUT PR — three additive sources of extra cost vs WITH PR:

1. MOIM 1-turn lag (every turn, k ≥ 2): cached prefix is P + (k-2)·Δ instead of P + (k-1)·Δ
2. Hourly resets (2 per 2h session): full cache miss at turns ~33 and ~67
3. Subdir hint resets (session-dependent): full cache miss at each new directory visited

At a full cache reset turn k:

• Anthropic: P × write_rate + k·Δ × full_rate
• OpenAI / Groq / Gemini: (P + k·Δ) × full_rate

---

Pricing Reference

Model	Provider	C	Input $/MTok	Output $/MTok	Cache Write	Cache Read
claude-opus-4-8	Anthropic	200k	$5.00	$25.00	$6.25	$0.50 (10%)
claude-sonnet-4-6	Anthropic	200k	$3.00	$15.00	$3.75	$0.30 (10%)
claude-haiku-4-5	Anthropic	200k	$1.00	$5.00	$1.25	$0.10 (10%)
gpt-4o	OpenAI	128k	$2.50	$10.00	—	$1.25 (50%)
o3	OpenAI	200k	$2.00	$8.00	—	$0.50 (25%)
gpt-4.1-mini	OpenAI	200k	$0.40	$1.60	—	$0.10 (25%)
gemini-2.5-pro	Google	200k	$1.25†	$10.00†	implicit	$0.3125 (25%)
gemini-2.5-flash	Google	200k	$0.30	$2.50	implicit	$0.075 (25%)
llama-4-scout-17b	Groq	128k	$0.11	$0.34	—	$0.055 (50%)
llama-3.3-70b	Groq	128k	$0.59	$0.79	—	$0.295 (50%)
Qwen3-235B-A22B	Together	200k	$0.20	$0.60	unconfirmed	—

† Gemini 2.5 Pro ≤200k tier. Sessions where total context exceeds 200k tokens are billed at
higher rates for those turns (approximately 2× for tokens beyond 200k).
Sources: Anthropic · OpenAI · Google · Groq · Together

---

Cost Tables

Session: N=100 turns, 50 tool calls, 1,000 output tokens/turn, 200k context (128k where noted).

Scenario A — Best case: 2 hourly resets, no subdir visits, MOIM lag only

Model	C	Without PR	With PR	Savings	%
claude-opus-4-8	200k	$10.73	$8.90	$1.84	17.1%
claude-sonnet-4-6	200k	$6.44	$5.34	$1.10	17.1%
claude-haiku-4-5	200k	$2.15	$1.78	$0.37	17.1%
gpt-4o	128k	$10.58	$10.25	$0.33	3.1%
o3	200k	$7.17	$6.56	$0.61	8.5%
gpt-4.1-mini	200k	$1.43	$1.31	$0.12	8.5%
gemini-2.5-pro ⚠️	200k	$4.98	$4.60	$0.38	7.6%
gemini-2.5-flash ⚠️	200k	$1.21	$1.11	$0.09	7.5%
llama-4-scout (groq)	128k	$0.46	$0.44	$0.01	3.2%
llama-3.3-70b (groq)	128k	$2.34	$2.26	$0.08	3.3%
Qwen3-235B (together)	200k	$2.24	$2.24	$0.00	0.0%

Scenario B — Realistic: 5 subdir visits + 2 hourly resets + MOIM lag

Model	C	Without PR	With PR	Savings	%
claude-opus-4-8	200k	$13.03	$8.90	$4.14	31.7%
claude-sonnet-4-6	200k	$7.82	$5.34	$2.48	31.7%
claude-haiku-4-5	200k	$2.61	$1.78	$0.83	31.7%
gpt-4o	128k	$11.00	$10.25	$0.75	6.8%
o3	200k	$7.92	$6.56	$1.36	17.1%
gpt-4.1-mini	200k	$1.58	$1.31	$0.27	17.1%
gemini-2.5-pro ⚠️	200k	$5.45	$4.60	$0.85	15.6%
gemini-2.5-flash ⚠️	200k	$1.32	$1.11	$0.20	15.4%
llama-4-scout (groq)	128k	$0.47	$0.44	$0.03	6.9%
llama-3.3-70b (groq)	128k	$2.44	$2.26	$0.18	7.2%
Qwen3-235B (together)	200k	$2.24	$2.24	$0.00	0.0%

Scenario C — Heavy: 15 subdir visits + 2 hourly resets + MOIM lag

Model	C	Without PR	With PR	Savings	%
claude-opus-4-8	200k	$17.01	$8.90	$8.12	47.7%
claude-sonnet-4-6	200k	$10.21	$5.34	$4.87	47.7%
claude-haiku-4-5	200k	$3.40	$1.78	$1.62	47.7%
gpt-4o	128k	$11.72	$10.25	$1.47	12.6%
o3	200k	$9.21	$6.56	$2.65	28.8%
gpt-4.1-mini	200k	$1.84	$1.31	$0.53	28.8%
gemini-2.5-pro ⚠️	200k	$6.26	$4.60	$1.66	26.5%
gemini-2.5-flash ⚠️	200k	$1.51	$1.11	$0.40	26.3%
llama-4-scout (groq)	128k	$0.51	$0.44	$0.06	12.8%
llama-3.3-70b (groq)	128k	$2.61	$2.26	$0.35	13.3%
Qwen3-235B (together)	200k	$2.24	$2.24	$0.00	0.0%

---

Key Findings

1. Savings are session-behavior-dependent, ranging from 3–17% (quiet session, few directories)
to 13–48% (heavy coding session with many directory visits). The dominant variable is how many
new directories the agent explores.

2. The subdirectory hints fix (Fix 3) is the most financially significant change. Hourly
resets are bounded at 2 per session. MOIM lag costs one expected Δ tokens per call. Subdir resets
are unbounded — each new directory visited caused a full reset from position 0. In a 15-directory
session, Fix 3 accounts for the majority of savings.

3. Anthropic benefits most (17–48%) because cache reads cost only 10% of the normal input
rate (a 90% saving). OpenAI and Groq have a 50% read discount, producing proportionally smaller
savings per reset event. Together AI Qwen3-235B-A22B has no confirmed billing discount for
cached tokens, so the PR produces no cost savings for that specific model.

4. For PR documentation: A conservative honest range is 17–48% cost reduction for Anthropic
users on long sessions approaching the 200k context limit. Percentage savings from Fix 3
(subdir resets) are approximately session-size-independent. Fix 1 (hourly resets) saves a fixed
absolute amount per session that represents a larger share of shorter sessions.

---

Worked Example: Subdirectory hint reset (claude-sonnet-4-6, turn 50)

This a cost breakdown example for ONE turn (turn 50).

WITHOUT PR — new directory entered, system_prompt_extras mutated:

Token category	Tokens	Rate	Cost
System prompt (cache write)†	8,000	$3.75/MTok	$0.030
Conversation history (cache miss)	100,000	$3.00/MTok	$0.300
Turn total	108,000		$0.330

† When the system prompt changes, Anthropic re-writes the cache at 125% of input rate. The
conversation history (k·Δ = 50×2,000 = 100,000 tokens) is billed at full input rate because
the system prompt change invalidated the entire prefix cache.

WITH PR — hint injected as user message, system prompt unchanged:

Token category	Tokens	Rate	Cost
Prior context (cache read)	106,000	$0.30/MTok	$0.032
New user message + hint (Δ)	2,000	$3.00/MTok	$0.006
Turn total	108,000		$0.038

Cache read covers P + (k-1)·Δ = 8,000 + 49×2,000 = 106,000 tokens.

Single reset savings at turn 50: $0.292 (8.7× cheaper).

[michaelneale]

these look good to me - especially not editing system prompt with agents/time - I think it makes sense.

[michaelneale]

I think @jamadeo and @jh-block had some thoughts/Q's to clarify, but I think overall this makes sense. I don't actually think we lose much and in happy cases save a lot. MOIM right near the tail is probably ok but could be better. I think worthwhile pushing with (still need to check that it really isn't mutating things accidentally)

[DOsinga]

I looked at this in some detail:

• removing the timestamp from the prompt manager is fine, but I'll note that it wasn't used so shouldn't have impact on the caching
• the addition to the hook system seems unrelated. let's drop
• I have concerns about the MOIM handling. it belongs to the last real user message and in my benchmarking this version (which was the previous version) hurts
• the real catch here is the discovered working dirs. I think the instinct is right, but I have questions about the implementation

[DOsinga]

I think drop this method. it is only used in MOIM where it doesn't really help and the rest of the code uses .with_visibility(false, true)

[DOsinga]

Let's drop this from the PR, it doesn't seem related to the stated change.

[DOsinga]

I think this will confuse the agent. the MOIM context message annotates the last real user message, not toolresponses. large models will probably recover from this, but if the model does something like

cd some_folder* && pwd

we will now append the message about the current folder as part of that tool response.

also we should not rely on fix conversation. we should compose correct interactions.

[DOsinga]

this does not belong in prompt_manager.rs if it doesn't make changes to the prompt

[DOsinga]

I think this is the right instinct -- these discovered hints should be part of the message that they discover, but I see some issues here:

• this gets saved to the session manager, but not the current conversation, so it won't impact the current conversation until we reload the session
• it gets added as user visible, to the user now sees a potentially very large AGENTS.md
• it doesn't check whether we already have this AGENTS.md in the conversation. so the agent will add this multiple times, one for each toolcall

how did you test this? the stats on the PR suggests this all saves tokens, but if we hit this path multiple times, I am not sure it wouldl

[mcelrath]

FYI I'm finding that the MOIM injection on every turn is confusing agents (Qwen 3.6 27B). They're reasoning about <info_msg> in their thinking. It's wasting tokens and distracting. It IS generally useful for the .goosehints though and cwd changes.

So this is no longer about prefix caching though...I now think that injecting the time and context (and TODO) usage is a bad idea here. I'm going to remove it in my build. Want me to raise a separate PR for that or incorporate it here? @DOsinga will address your comments shortly.

[michaelneale]

looking pretty close @mcelrath do you think the info is mainly for the small models, and should be dropped for them or in general? (possibly separate PR curious what @DOsinga thinks) but looks like we are getting to something better for all, nice one.

[DOsinga]

Really like the direction here — not mutating the system prompt every hour is exactly the kind of thing that quietly wastes a lot of cache, and it's great to have someone digging into the actual cost mechanics.

The thing I most want to see land is the issue you surfaced around .goosehints/AGENTS.md being picked up from the directories tool calls touch: re-injecting those on every tool call (un-deduped, and potentially a large file) is a genuine correctness and cost bug, not just a cache-invalidation one. Solving that properly is valuable in its own right and worth getting right.

Your point that the per-turn MOIM injection actively confuses smaller models (Qwen) is also a good catch — agreed that we shouldn't be relying on fix_conversation to clean up after a tail push(); better to compose the correct interaction directly.

No rush — happy to give you room to push the changes you mentioned. I'll snooze this for a few days so it comes back to us once you've had a chance to iterate.

[DOsinga]

Hey @mcelrath — friendly nudge on this one. Back on 2026-06-17 you mentioned you'd address the review comments shortly, and we snoozed to give you room, but there hasn't been any new activity since.

The core idea here — especially stopping the .goosehints/AGENTS.md re-injection from permanently mutating the system prompt — is something we really want to see land. So no worries at all if you've gotten busy.

Could you let us know whether you'd like to keep iterating on this? If you've got bandwidth in the next few days, great — we'll pick the review back up. Otherwise we may close this and take a fresh run at it ourselves so the improvement doesn't get lost, and you'd of course be credited for the original idea. Just let us know which works for you. 🙏

[mcelrath]

Yes, I'll finish it up. Have been out of town for a few days, just got home.

[DOsinga]

Thanks for the rework — this now cleanly resolves the earlier comments (MOIM/hook changes dropped, dead timestamp removed, logic moved to SubdirectoryHintTracker, hints injected agent-only into the live conversation and deduped within a turn). Two things before we can land it, one of which is a blocker.

Blocker — the dedup has to survive a session resume. loaded_dirs lives only in-memory on the Agent and is never persisted or rehydrated. On resume you get a fresh Agent with an empty set, so the next tool call that touches sub/ re-injects a hint block that's already present in the restored history — the "exactly once" guarantee only holds within a single process lifetime. We need this to be stateless: on injection, derive the already-loaded set from the conversation rather than trusting in-memory state.

Concretely (option 1 from our discussion): keep the subdir_hints:<dir> key on the injected message so it's identifiable in history, and before injecting, seed/skip against the directories already present in the conversation. A simple approach: tag the injected message with a stable marker per directory, and on each turn (including the first after resume) scan the current conversation for those markers to reconstruct loaded_dirs, instead of relying on the in-memory HashSet alone. Right now collect_new_hints throws the key away and pushes a plain user-text message, so there's nothing to match against — that needs to change.

[DOsinga]

This drops the subdir_hints:<dir> key and pushes the content as an anonymous user message, so there's no way to tell later which directories are already represented in the conversation. For the resume case we need the injected message to stay identifiable per directory — please preserve the key (e.g. as a marker line/prefix in the message, or a message-metadata field) so loaded_dirs can be reconstructed from history.

[DOsinga]

Non-blocking nit: the dedup HashSet<PathBuf> compares paths lexically and this never canonicalizes, so sub/a.txt, ./sub/b.txt, and sub/../sub/c.txt all resolve to distinct keys for the same directory and each triggers a fresh injection. Since load_hints_from_directory already gates on is_dir(), canonicalizing the resolved dir before comparison (it exists on disk) would collapse these and also handle symlinks.

[DOsinga]

Thanks for working on this. The remaining change is solving a real problem, so I captured the desired behavior and implementation constraints in #10330.

The current PR still has the unresolved blocker from the core-team review: deduplication of injected subdirectory hints needs to survive resumed sessions, not just the current in-memory Agent instance. In particular, we need a way to preserve/reconstruct a stable per-directory marker from conversation history before injecting new hints.

Since there has not been movement on that blocker after the review, I am going to close this PR for now. Please feel free to reopen it when it is ready to address the resume-safe deduplication requirement, or someone else can pick up the linked issue.