Skip to content

feat: configurable prompt_caching.cache_ttl for Anthropic TTL tier - #12659

Closed
ericnicolaides wants to merge 2 commits into
NousResearch:mainfrom
ericnicolaides:feat/configurable-cache-ttl
Closed

feat: configurable prompt_caching.cache_ttl for Anthropic TTL tier#12659
ericnicolaides wants to merge 2 commits into
NousResearch:mainfrom
ericnicolaides:feat/configurable-cache-ttl

Conversation

@ericnicolaides

@ericnicolaides ericnicolaides commented Apr 19, 2026

Copy link
Copy Markdown

What does this PR do?

Adds a prompt_caching.cache_ttl config knob so operators can opt into Anthropic's 1h prompt-cache TTL tier instead of the default 5m. The default remains 5m.

Motivation: With 5m, long interactive sessions that pause (reading tool output, context switching) often cross the TTL boundary and pay full write pricing again on the next turn. Anthropic documents a 1h tier for the same cache-control flow; exposing it via config makes longer sessions cheaper without changing caching strategy (still system_and_3 / apply_anthropic_cache_control).

Approach: AIAgent.__init__ reads load_config().get("prompt_caching", {}).get("cache_ttl", "5m"), accepts only 5m and 1h, and ignores anything else (stays 5m, no exception). DEFAULT_CONFIG and cli-config.yaml.example document the key; developer guide YAML is corrected (it previously showed model.cache_ttl, which is not wired).

Related Issue

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • run_agent.py — load prompt_caching.cache_ttl into self._cache_ttl (same call site as existing apply_anthropic_cache_control(..., cache_ttl=self._cache_ttl)).
  • hermes_cli/config.pyDEFAULT_CONFIG["prompt_caching"]["cache_ttl"].
  • cli-config.yaml.example — commented example block.
  • website/docs/developer-guide/context-compression-and-caching.md — fix YAML to prompt_caching.cache_ttl.
  • tests/run_agent/test_run_agent.py — three TestInit tests (default, 1h, invalid fallback).

How to Test

cd hermes-agent
uv run --with pytest --with pytest-asyncio --with pytest-xdist python -m pytest \
  tests/run_agent/test_run_agent.py::TestInit::test_prompt_caching_cache_ttl_defaults_without_config \
  tests/run_agent/test_run_agent.py::TestInit::test_prompt_caching_cache_ttl_custom_1h \
  tests/run_agent/test_run_agent.py::TestInit::test_prompt_caching_cache_ttl_invalid_falls_back \
  -q

Verification

  • Targeted pytest (commands above): 3 passed, 0 failed (Linux agent host, uv resolves deps per pyproject.toml).

Risks

  • Unsupported / non-Anthropic providers: unchanged — prompt caching remains gated by existing _use_prompt_caching logic; this only changes the TTL string passed when caching is already on.
  • Invalid config values (e.g. typos, "30m", wrong types): silently keep 5m — same broad try/except pattern as other optional config reads in AIAgent.__init__; no crash.
  • Very long TTL (1h): higher cache-write cost if the conversation prefix churns every turn; users opt in explicitly. Maliciously huge strings are not accepted (not in ("5m", "1h")).

Model used

Contribution prepared in a Wildcat Paperclip agent heartbeat using the Cursor harness. The exact LLM id/version is determined by the operator's Cursor product settings (not pinned in-repo).

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass (targeted tests above; full suite not run in this environment)
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: Linux (Debian-based agent runner)

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

WildCat Eng Manager added 2 commits April 19, 2026 13:46
- Load prompt_caching.cache_ttl in AIAgent (5m default, 1h opt-in)
- Document DEFAULT_CONFIG and developer guide example
- Add unit tests for default, 1h, and invalid TTL fallback

Made-with: Cursor
@ericnicolaides

Copy link
Copy Markdown
Author

Human comment here. This was a joint human-Hermes/Cursor agent work effort this morning as we were concerned about runaway model costs (~$500-1000 yesterday alone). Yes, Opus 4.7 is expensive but this was clearly a bigger issue we were dealing with. Hope this if helpful to the community and if accepted, happy to explore broadening this to other models in a more generic way

@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint area/config Config system, migrations, profiles provider/anthropic Anthropic native Messages API labels Apr 23, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Likely duplicate of #3082 — both add configurable prompt cache TTL via config.yaml for Anthropic. Also related to #5713 (multi-block tiered TTLs).

@alt-glitch

Copy link
Copy Markdown
Collaborator

Likely duplicate of #3082 — both add configurable prompt cache TTL via config.yaml for Anthropic.

@teknium1

Copy link
Copy Markdown
Contributor

Merged via #15065 — your two commits were cherry-picked onto current main with authorship preserved in git log. Added agent@wildcat.local@ericnicolaides to scripts/release.py::AUTHOR_MAP so release notes attribute correctly.

One small conflict resolved: current main refactored the Anthropic prompt-caching detection into _anthropic_prompt_cache_policy(), so we kept main's helper call and layered your TTL lookup on top.

Chose your approach over two parallel PRs (#14812 env-var only, #3082 full kwarg threading) because load_config() inside AIAgent is the lightest pattern that works and matches our 'config.yaml for behavior, .env for secrets' rule. Thanks for the well-scoped change + clear cost-motivation writeup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P3 Low — cosmetic, nice to have provider/anthropic Anthropic native Messages API type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants