fix(caching): honor prompt_caching.cache_ttl disable in config - #33555
fix(caching): honor prompt_caching.cache_ttl disable in config#33555BB-light wants to merge 1 commit into
Conversation
Config `prompt_caching.cache_ttl: false` was silently ignored -- upstream only accepted "5m" or "1h". Cache writes continued even when the operator intended to disable caching, incurring write costs that bill against "extra usage" on OAuth subscription. Now accepts falsy values (false, null, "off", "false", "disabled", "no", "none") to disable prompt caching entirely. Sets both _use_prompt_caching and _use_native_cache_layout to False, and _cache_ttl to None for consistent internal state. Also updates the inline doc in hermes_cli/config.py and adds parametrized tests for all seven falsy variants.
teknium1
left a comment
There was a problem hiding this comment.
Thanks for identifying a real current gap: main documents prompt caching as always-on, and cache_ttl currently honors only "5m" and "1h" (website/docs/user-guide/configuration.md:915-924; agent/agent_init.py:643-650).
Problems
- The PR disables caching only during initialization (
agent/agent_init.pyPR line 497). Current model switching re-derives both flags atagent/agent_runtime_helpers.py:2019-2027, and fallback activation does the same atagent/chat_completion_helpers.py:1596-1604; either path can re-enable markers. - The added test covers only initialization (
tests/run_agent/test_run_agent.pyPR lines 934-955), not those re-derivation paths. - The config surface needs deliberate resolution. The related
prompt_caching.enabledimplementation was reverted in #56126 for broader evaluation; current docs still definecache_ttlonly as a"5m"/"1h"tier selector.
Suggested changes
- Put the chosen opt-out in the shared cache-policy decision path and test init, model-switch, and fallback behavior.
- Document one selected configuration contract rather than treating TTL aliases as a second disable API.
Automated hermes-sweeper review.
| or _ttl is None | ||
| or str(_ttl).lower() in ("off", "false", "disabled", "no", "none") | ||
| ): | ||
| agent._use_prompt_caching = False |
There was a problem hiding this comment.
This only affects initial construction. Current main overwrites both flags when switching models (agent/agent_runtime_helpers.py:2019-2027) and when activating a fallback (agent/chat_completion_helpers.py:1596-1604), so cache_ttl: false can be lost mid-session. Put the selected opt-out in the shared policy path and cover those re-derivations.
Setting prompt_caching.cache_ttl to a falsy value (false, null, off, disabled, no, none) now fully disables prompt caching instead of being silently ignored. The disable propagates through anthropic_prompt_cache_policy() (early return when _cache_disabled flag is set) and restore_primary_runtime() (override after snapshot restore), so it survives /model switches and fallback re-derivation — the gap that caused NousResearch#56105 to be reverted in NousResearch#56126. Salvage of NousResearch#33555 by @BB-light, with model-switch/fallback survival gap fixed on top. Co-authored-by: BB-light <BB-light@users.noreply.github.com>
Setting prompt_caching.cache_ttl to a falsy value (false, null, off, disabled, no, none) now fully disables prompt caching instead of being silently ignored. The disable propagates through anthropic_prompt_cache_policy() (early return when _cache_disabled flag is set) and restore_primary_runtime() (override after snapshot restore), so it survives /model switches and fallback re-derivation — the gap that caused #56105 to be reverted in #56126. Salvage of #33555 by @BB-light, with model-switch/fallback survival gap fixed on top. Co-authored-by: BB-light <BB-light@users.noreply.github.com>
|
Merged via PR #76015 — your fix was cherry-picked with authorship preserved. The model-switch/fallback survival gap (the same issue that caused #56105 to be reverted in #56126) was fixed on top by gating anthropic_prompt_cache_policy() on a _cache_disabled flag set during init. Thanks for the contribution! |
…ry_runtime) Whole-function owner tests for turn-scoped primary restore: NousResearch#20465 index reset, rate-limit + reset-aware skip gates (future/already-logged/exception-fallthrough), wire-aware rebuild (moa-facade NousResearch#53802 / anthropic-nulled / openai), _cache_disabled survival (NousResearch#33555), transport-cache clear, pool rebind (mismatch-reload / prefetched-reuse / reload-exception-cleared), credential re-select NousResearch#25205 (match / mismatch / no-key / select-None) incl. custom:<name> disambiguation NousResearch#56885 (match/exception both blocks), reasoning restore, fallback reset, and rebuild fail-safe. Function (1449-1732) 100%. NOTE: an earlier grep-artifact hid the custom-provider blocks as false-covered; caught via re-roast, added the missing cases, switched to a python range filter.
Setting prompt_caching.cache_ttl to a falsy value (false, null, off, disabled, no, none) now fully disables prompt caching instead of being silently ignored. The disable propagates through anthropic_prompt_cache_policy() (early return when _cache_disabled flag is set) and restore_primary_runtime() (override after snapshot restore), so it survives /model switches and fallback re-derivation — the gap that caused NousResearch#56105 to be reverted in NousResearch#56126. Salvage of NousResearch#33555 by @BB-light, with model-switch/fallback survival gap fixed on top. Co-authored-by: BB-light <BB-light@users.noreply.github.com>
Summary
Allows operators to fully disable Anthropic prompt caching by setting
cache_ttlto a falsy value inconfig.yaml.The problem
The existing
prompt_caching.cache_ttlconfig only accepts"5m"or"1h". Any other value (includingfalse,null,"off") is silently ignored, and caching remains enabled at the"5m"tier. This means:cache_ttl: falseexpecting to disable caching see no effectThe fix
After the existing
"5m"/"1h"check, a newelifbranch catches falsy values:When detected, sets:
agent._use_prompt_caching = Falseagent._use_native_cache_layout = Falseagent._cache_ttl = NoneChanges
agent/agent_init.py— falsy-value detection after existing TTL parsinghermes_cli/config.py— updated inline doc commenttests/run_agent/test_run_agent.py— parametrized test covering all 7 falsy variantsTest plan
pytest tests/run_agent/test_run_agent.py -k cache_ttlpasses (existing + new tests)cache_ttl: "5m"still works (existing test)cache_ttl: "1h"still works (existing test)cache_ttl: "30m"(invalid) still falls back to "5m" (existing test)cache_ttl: falsedisables caching (new test, 7 variants)