feat: configurable prompt_caching.cache_ttl for Anthropic TTL tier - #12659
feat: configurable prompt_caching.cache_ttl for Anthropic TTL tier#12659ericnicolaides wants to merge 2 commits into
Conversation
- Load prompt_caching.cache_ttl in AIAgent (5m default, 1h opt-in) - Document DEFAULT_CONFIG and developer guide example - Add unit tests for default, 1h, and invalid TTL fallback Made-with: Cursor
Made-with: Cursor
|
Human comment here. This was a joint human-Hermes/Cursor agent work effort this morning as we were concerned about runaway model costs (~$500-1000 yesterday alone). Yes, Opus 4.7 is expensive but this was clearly a bigger issue we were dealing with. Hope this if helpful to the community and if accepted, happy to explore broadening this to other models in a more generic way |
|
Likely duplicate of #3082 — both add configurable prompt cache TTL via config.yaml for Anthropic. |
|
Merged via #15065 — your two commits were cherry-picked onto current main with authorship preserved in One small conflict resolved: current main refactored the Anthropic prompt-caching detection into Chose your approach over two parallel PRs (#14812 env-var only, #3082 full kwarg threading) because |
What does this PR do?
Adds a
prompt_caching.cache_ttlconfig knob so operators can opt into Anthropic's1hprompt-cache TTL tier instead of the default5m. The default remains5m.Motivation: With
5m, long interactive sessions that pause (reading tool output, context switching) often cross the TTL boundary and pay full write pricing again on the next turn. Anthropic documents a1htier for the same cache-control flow; exposing it via config makes longer sessions cheaper without changing caching strategy (stillsystem_and_3/apply_anthropic_cache_control).Approach:
AIAgent.__init__readsload_config().get("prompt_caching", {}).get("cache_ttl", "5m"), accepts only5mand1h, and ignores anything else (stays5m, no exception).DEFAULT_CONFIGandcli-config.yaml.exampledocument the key; developer guide YAML is corrected (it previously showedmodel.cache_ttl, which is not wired).Related Issue
Type of Change
Changes Made
run_agent.py— loadprompt_caching.cache_ttlintoself._cache_ttl(same call site as existingapply_anthropic_cache_control(..., cache_ttl=self._cache_ttl)).hermes_cli/config.py—DEFAULT_CONFIG["prompt_caching"]["cache_ttl"].cli-config.yaml.example— commented example block.website/docs/developer-guide/context-compression-and-caching.md— fix YAML toprompt_caching.cache_ttl.tests/run_agent/test_run_agent.py— threeTestInittests (default,1h, invalid fallback).How to Test
cd hermes-agent uv run --with pytest --with pytest-asyncio --with pytest-xdist python -m pytest \ tests/run_agent/test_run_agent.py::TestInit::test_prompt_caching_cache_ttl_defaults_without_config \ tests/run_agent/test_run_agent.py::TestInit::test_prompt_caching_cache_ttl_custom_1h \ tests/run_agent/test_run_agent.py::TestInit::test_prompt_caching_cache_ttl_invalid_falls_back \ -qVerification
uvresolves deps perpyproject.toml).Risks
_use_prompt_cachinglogic; this only changes the TTL string passed when caching is already on."30m", wrong types): silently keep5m— same broadtry/exceptpattern as other optional config reads inAIAgent.__init__; no crash.1h): higher cache-write cost if the conversation prefix churns every turn; users opt in explicitly. Maliciously huge strings are not accepted (not in("5m", "1h")).Model used
Contribution prepared in a Wildcat Paperclip agent heartbeat using the Cursor harness. The exact LLM id/version is determined by the operator's Cursor product settings (not pinned in-repo).
Checklist
Code
fix(scope):,feat(scope):, etc.)pytest tests/ -qand all tests pass (targeted tests above; full suite not run in this environment)Documentation & Housekeeping
docs/, docstrings) — or N/Acli-config.yaml.exampleif I added/changed config keys — or N/ACONTRIBUTING.mdorAGENTS.mdif I changed architecture or workflows — or N/A