Skip to content

feat(provider): add prompt-cache policy hook - #79621

Open
zjin1126 wants to merge 1 commit into
NousResearch:mainfrom
zjin1126:feat/provider-prompt-cache-policy
Open

zjin1126 wants to merge 1 commit into
NousResearch:mainfrom
zjin1126:feat/provider-prompt-cache-policy

Conversation

@zjin1126

@zjin1126 zjin1126 commented Aug 5, 2026

Copy link
Copy Markdown

What does this PR do?

Adds a narrow ProviderProfile.prompt_cache_policy() hook so model-provider plugins can declare request-scoped explicit prompt-cache support without adding plugin IDs or endpoint allowlists to Hermes core.

The hook returns (enabled, native_layout) or None. None, invalid values, and exceptions preserve the existing core detection path. An explicit operator cache disable still wins, and MoA still resolves the acting aggregator before consulting its provider profile.

This extends the existing provider-profile abstraction rather than adding a second strategy manager or any core model-tool surface. The system prompt and conversation cache prefix are unchanged.

Prior art checked:

Related Issue

Fixes #79602

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • providers/base.py: add the optional, fail-safe prompt_cache_policy() provider hook and document its contract.
  • agent/agent_runtime_helpers.py: resolve plugin policy after operator-disable and MoA-provider resolution, before legacy core rules.
  • tests/providers/test_provider_profiles.py: add the default contract and a fresh-process $HERMES_HOME provider-plugin discovery test.
  • tests/run_agent/test_anthropic_prompt_cache_policy.py: cover request context, legacy fallback, malformed returns, exceptions, and operator-disable precedence.
  • website/docs/developer-guide/model-provider-plugin.md: document the new hook for external provider authors.

How to Test

  1. Run the focused policy and discovery suite:
    scripts/run_tests.sh \
      tests/providers/test_provider_profiles.py \
      tests/agent/test_prompt_caching.py \
      tests/run_agent/test_anthropic_prompt_cache_policy.py
  2. Run the wider cache/fallback suite:
    scripts/run_tests.sh \
      tests/providers/test_provider_profiles.py \
      tests/agent/test_prompt_caching.py \
      tests/agent/test_moa_aggregator_cache_control.py \
      tests/agent/test_cache_disabled_on_stubs.py \
      tests/run_agent/test_anthropic_prompt_cache_policy.py \
      tests/run_agent/test_background_fallback_cache_parity.py \
      tests/run_agent/test_conversation_fallback_state.py \
      tests/run_agent/test_run_agent.py
  3. Run lint and whitespace validation:
    uvx --from ruff==0.15.10 ruff check \
      agent/agent_runtime_helpers.py providers/base.py \
      tests/providers/test_provider_profiles.py \
      tests/run_agent/test_anthropic_prompt_cache_policy.py
    git diff --check origin/main...HEAD

Local results:

  • Focused suite: 72 passed, 0 failed.
  • Wider cache/fallback suite: 334 passed, 0 failed.
  • Ruff and git diff --check: passed.
  • The full canonical suite was also attempted in an isolated [all,dev] environment: 25,618 passed, 69 failed in 27 untouched optional-dependency/platform-specific files. Representative failures require non-[all] lazy dependencies such as the Anthropic SDK, Parallel SDK, wake-word runtime, and audio backends. The changed policy/discovery tests passed in that run; CI remains authoritative for the repository-wide matrix.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: macOS 26.6 arm64

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — ProviderProfile docstring and model-provider authoring guide
  • I've updated cli-config.yaml.example if I added/changed config keys — N/A, no config keys
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A, this extends the existing ProviderProfile hook surface
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — platform-neutral Python path; fresh-process discovery test uses no shell-specific behavior
  • I've updated tool descriptions/schemas if I changed tool behavior — N/A, no model tools or schemas changed

Screenshots / Logs

No UI changes.

AI assistance was used to help draft and review the implementation and tests; all commands and validation reported above were run in the contributor checkout.

@zjin1126
zjin1126 force-pushed the feat/provider-prompt-cache-policy branch from 9d51af3 to 0cbc2af Compare August 5, 2026 19:16
@alt-glitch alt-glitch added type/feature New feature or request comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have sweeper:risk-caching Sweeper risk: may break/degrade prompt caching or cache-key stability (invariant) labels Aug 5, 2026
@zjin1126

zjin1126 commented Aug 5, 2026

Copy link
Copy Markdown
Author

Concrete external consumer Draft: oliver-mee/hermes-alibaba-token-plan#18

It implements the proposed ProviderProfile.prompt_cache_policy() hook for the documented qwen3.8-max-preview Chat Completions cache contract. The companion remains Draft and explicitly depends on this core PR; released Hermes versions are tested to continue loading the plugin without consuming the unknown hook.

zjin1126 added a commit to zjin1126/hermes-alibaba-token-plan that referenced this pull request Aug 5, 2026
Use the optional ProviderProfile prompt-cache hook proposed in NousResearch/hermes-agent#79621. Scope the OpenAI-wire cache marker declaration to qwen3.8-max-preview on Chat Completions and leave every other model and API mode on Hermes' existing fallback.

The QwenCloud context-cache guide lists qwen3.8-max-preview (Token Plan only) as explicit-cache capable and documents cache_control markers for the OpenAI-compatible Chat API.

Standalone and supported-Hermes compatibility suites pass without network access. AI-assisted: Hermes Agent (GPT-5.6 Sol); the author reviewed the implementation, evidence, diff, and test results.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have sweeper:risk-caching Sweeper risk: may break/degrade prompt caching or cache-key stability (invariant) type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: let ProviderProfile declare prompt-cache policy

2 participants