Skip to content

fix(caching): honor prompt_caching.cache_ttl disable in config - #33555

Closed
BB-light wants to merge 1 commit into
NousResearch:mainfrom
BB-light:bb/cache-ttl-disable
Closed

fix(caching): honor prompt_caching.cache_ttl disable in config#33555
BB-light wants to merge 1 commit into
NousResearch:mainfrom
BB-light:bb/cache-ttl-disable

Conversation

@BB-light

Copy link
Copy Markdown
Contributor

Summary

Allows operators to fully disable Anthropic prompt caching by setting cache_ttl to a falsy value in config.yaml.

The problem

The existing prompt_caching.cache_ttl config only accepts "5m" or "1h". Any other value (including false, null, "off") is silently ignored, and caching remains enabled at the "5m" tier. This means:

  • Operators who set cache_ttl: false expecting to disable caching see no effect
  • Cache writes continue, incurring write costs (1.25x for 5m tier) that bill against "extra usage" on OAuth subscription
  • There is no config lever to disable prompt caching without code changes

The fix

After the existing "5m" / "1h" check, a new elif branch catches falsy values:

# config.yaml
prompt_caching:
  cache_ttl: false    # also accepts: null, "off", "disabled", "no", "none"

When detected, sets:

  • agent._use_prompt_caching = False
  • agent._use_native_cache_layout = False
  • agent._cache_ttl = None

Changes

  • Modified: agent/agent_init.py — falsy-value detection after existing TTL parsing
  • Modified: hermes_cli/config.py — updated inline doc comment
  • Modified: tests/run_agent/test_run_agent.py — parametrized test covering all 7 falsy variants

Test plan

  • pytest tests/run_agent/test_run_agent.py -k cache_ttl passes (existing + new tests)
  • cache_ttl: "5m" still works (existing test)
  • cache_ttl: "1h" still works (existing test)
  • cache_ttl: "30m" (invalid) still falls back to "5m" (existing test)
  • cache_ttl: false disables caching (new test, 7 variants)

Config `prompt_caching.cache_ttl: false` was silently ignored --
upstream only accepted "5m" or "1h". Cache writes continued even when
the operator intended to disable caching, incurring write costs that
bill against "extra usage" on OAuth subscription.

Now accepts falsy values (false, null, "off", "false", "disabled",
"no", "none") to disable prompt caching entirely. Sets both
_use_prompt_caching and _use_native_cache_layout to False, and
_cache_ttl to None for consistent internal state.

Also updates the inline doc in hermes_cli/config.py and adds
parametrized tests for all seven falsy variants.
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint area/config Config system, migrations, profiles provider/anthropic Anthropic native Messages API labels May 28, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for identifying a real current gap: main documents prompt caching as always-on, and cache_ttl currently honors only "5m" and "1h" (website/docs/user-guide/configuration.md:915-924; agent/agent_init.py:643-650).

Problems

  • The PR disables caching only during initialization (agent/agent_init.py PR line 497). Current model switching re-derives both flags at agent/agent_runtime_helpers.py:2019-2027, and fallback activation does the same at agent/chat_completion_helpers.py:1596-1604; either path can re-enable markers.
  • The added test covers only initialization (tests/run_agent/test_run_agent.py PR lines 934-955), not those re-derivation paths.
  • The config surface needs deliberate resolution. The related prompt_caching.enabled implementation was reverted in #56126 for broader evaluation; current docs still define cache_ttl only as a "5m"/"1h" tier selector.

Suggested changes

  • Put the chosen opt-out in the shared cache-policy decision path and test init, model-switch, and fallback behavior.
  • Document one selected configuration contract rather than treating TTL aliases as a second disable API.

Automated hermes-sweeper review.

Comment thread agent/agent_init.py
or _ttl is None
or str(_ttl).lower() in ("off", "false", "disabled", "no", "none")
):
agent._use_prompt_caching = False

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only affects initial construction. Current main overwrites both flags when switching models (agent/agent_runtime_helpers.py:2019-2027) and when activating a fallback (agent/chat_completion_helpers.py:1596-1604), so cache_ttl: false can be lost mid-session. Put the selected opt-out in the shared policy path and cover those re-derivations.

@teknium1 teknium1 added sweeper:risk-caching Sweeper risk: may break/degrade prompt caching or cache-key stability (invariant) sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-massive Sweeper blast radius: massive — everyone, every turn (invariant surface) labels Jul 13, 2026
kshitijk4poor pushed a commit to kshitijk4poor/hermes-agent that referenced this pull request Aug 1, 2026
Setting prompt_caching.cache_ttl to a falsy value (false, null, off,
disabled, no, none) now fully disables prompt caching instead of
being silently ignored.

The disable propagates through anthropic_prompt_cache_policy() (early
return when _cache_disabled flag is set) and restore_primary_runtime()
(override after snapshot restore), so it survives /model switches and
fallback re-derivation — the gap that caused NousResearch#56105 to be reverted in
NousResearch#56126.

Salvage of NousResearch#33555 by @BB-light, with model-switch/fallback survival
gap fixed on top.

Co-authored-by: BB-light <BB-light@users.noreply.github.com>
kshitijk4poor pushed a commit that referenced this pull request Aug 1, 2026
Setting prompt_caching.cache_ttl to a falsy value (false, null, off,
disabled, no, none) now fully disables prompt caching instead of
being silently ignored.

The disable propagates through anthropic_prompt_cache_policy() (early
return when _cache_disabled flag is set) and restore_primary_runtime()
(override after snapshot restore), so it survives /model switches and
fallback re-derivation — the gap that caused #56105 to be reverted in
#56126.

Salvage of #33555 by @BB-light, with model-switch/fallback survival
gap fixed on top.

Co-authored-by: BB-light <BB-light@users.noreply.github.com>
@kshitijk4poor

Copy link
Copy Markdown
Collaborator

Merged via PR #76015 — your fix was cherry-picked with authorship preserved. The model-switch/fallback survival gap (the same issue that caused #56105 to be reverted in #56126) was fixed on top by gating anthropic_prompt_cache_policy() on a _cache_disabled flag set during init. Thanks for the contribution!

sijav added a commit to sijav/sijav-agent that referenced this pull request Aug 6, 2026
…ry_runtime)

Whole-function owner tests for turn-scoped primary restore: NousResearch#20465 index reset,
rate-limit + reset-aware skip gates (future/already-logged/exception-fallthrough),
wire-aware rebuild (moa-facade NousResearch#53802 / anthropic-nulled / openai), _cache_disabled
survival (NousResearch#33555), transport-cache clear, pool rebind (mismatch-reload /
prefetched-reuse / reload-exception-cleared), credential re-select NousResearch#25205 (match /
mismatch / no-key / select-None) incl. custom:<name> disambiguation NousResearch#56885
(match/exception both blocks), reasoning restore, fallback reset, and rebuild
fail-safe. Function (1449-1732) 100%.

NOTE: an earlier grep-artifact hid the custom-provider blocks as false-covered;
caught via re-roast, added the missing cases, switched to a python range filter.
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
Setting prompt_caching.cache_ttl to a falsy value (false, null, off,
disabled, no, none) now fully disables prompt caching instead of
being silently ignored.

The disable propagates through anthropic_prompt_cache_policy() (early
return when _cache_disabled flag is set) and restore_primary_runtime()
(override after snapshot restore), so it survives /model switches and
fallback re-derivation — the gap that caused NousResearch#56105 to be reverted in
NousResearch#56126.

Salvage of NousResearch#33555 by @BB-light, with model-switch/fallback survival
gap fixed on top.

Co-authored-by: BB-light <BB-light@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists provider/anthropic Anthropic native Messages API sweeper:blast-massive Sweeper blast radius: massive — everyone, every turn (invariant surface) sweeper:risk-caching Sweeper risk: may break/degrade prompt caching or cache-key stability (invariant) sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants