Skip to content

feat(background_review): add configurable background review routing - #15508

Closed
hcshen0111 wants to merge 1 commit into
NousResearch:mainfrom
hcshen0111:feat/backend_preview_model
Closed

feat(background_review): add configurable background review routing#15508
hcshen0111 wants to merge 1 commit into
NousResearch:mainfrom
hcshen0111:feat/backend_preview_model

Conversation

@hcshen0111

@hcshen0111 hcshen0111 commented Apr 25, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds configurable runtime routing for background memory/skill review runs.

Hermes currently runs background review using the primary AIAgent runtime. This PR adds a user-facing agent.review_model configuration and /review-model cli slash command so users can route background review to a cheaper or specialized model.

It also supports advanced scoped overrides via memory.review and skills.review, allowing memory and skill reviews to use different review runtimes when needed. Empty scoped fields inherit from agent.review_model, which then falls back to the primary agent runtime.

Reliability is preserved by falling back to the primary runtime if the configured review runtime fails.

Notes

This incorporates the core routing idea from #13647 while keeping this PR's existing CLI UX and unified agent.review_model config as the default user-facing path.

Related Issue

Fixes #

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • Added agent.review_model defaults in hermes_cli/config.py.
  • Added /review-model CLI command in hermes_cli/commands.py and cli.py.
  • Added review-model selection, reset, custom endpoint, and persistence under agent.review_model.*.
  • Updated AIAgent._spawn_background_review() in run_agent.py to try the configured review runtime first and fall back to the primary
    runtime on failure.
  • Added tests for command registration, config defaults, /review-model persistence, provider validation, and background review fallback
    behavior.

How to Test

  1. Run targeted tests:
    scripts/run_tests.sh tests/cli/test_review_model_command.py tests/run_agent/test_background_review_summary.py tests/hermes_cli/

test_commands.py tests/hermes_cli/test_config.py

  1. Start Hermes CLI and configure a review model:

    /review-model --provider

  2. Reset review routing back to the primary runtime:

    /review-model default

Checklist

Code

Documentation & Housekeeping

Screenshots / Logs

Targeted test run:

scripts/run_tests.sh tests/cli/test_review_model_command.py tests/run_agent/test_background_review_summary.py tests/hermes_cli/test_commands.py
tests/hermes_cli/test_config.py
180 passed

@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard area/config Config system, migrations, profiles labels Apr 25, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Likely duplicate of #13647 — both add configurable models for background review. #13647 uses memory.review./skills.review. config paths while this uses agent.review_model.*.

@hcshen0111

Copy link
Copy Markdown
Contributor Author

Likely duplicate of #13647 — both add configurable models for background review. #13647 uses memory.review./skills.review. config paths while this uses agent.review_model.*.

Good catch. I checked #13647 and there is real overlap in the background-review runtime routing.

I’m happy to rework this so it doesn’t compete with that PR. The useful delta from my side is mostly the /review-model CLI UX, picker/custom endpoint handling, and tests. I can rebase on #13647 and wire those pieces into its memory.review.* / skills.review.* config shape, or adapt to whichever config shape maintainers prefer.

@hcshen0111 hcshen0111 changed the title feat(background_review): Add configurable background review model feat(background_review): add configurable background review routing Apr 25, 2026
@hcshen0111

Copy link
Copy Markdown
Contributor Author

Likely duplicate of #13647 — both add configurable models for background review. #13647 uses memory.review./skills.review. config paths while this uses agent.review_model.*.

Thanks again for pointing me at #13647. I’ve updated this PR to incorporate that idea without keeping two competing implementations.

The PR now keeps agent.review_model and /review-model as the default user-facing path, but also supports the scoped override shape from #13647:

  • memory.review.*
  • skills.review.*
  1. primary agent runtime
  2. agent.review_model
  3. memory.review / skills.review scoped fields for the specific review type

So users get a simple CLI/config path by default, while advanced users can still route memory and skill review differently. The fallback behavior remains: if the configured review runtime fails, background review retries with the primary runtime.

@teknium1

Copy link
Copy Markdown
Contributor

Thanks for the background-review routing work. This automated hermes-sweeper review found that current main already provides the requested routing through the canonical auxiliary-task configuration surface.

  • 87c4a5ebb8a9f8122197a908288cc0abc7cef6b0 (feat(background-review): aux-model selector for the self-improvement review, merged as feat(background-review): aux-model routing + context digest + adaptive cadence to cut self-improvement cost #49252) added auxiliary.background_review.
  • hermes_cli/config.py:1712 defines the background-review provider/model runtime settings.
  • agent/background_review.py:46 resolves the configured runtime, and agent/background_review.py:683 constructs the review fork with those resolved values.
  • website/docs/user-guide/features/memory.md:273 documents the user-facing configuration, and tests/run_agent/test_background_review_cost_controls.py:40 covers inheritance, routing, and resolution fallback.

This supersedes the parallel agent.review_model routing implementation proposed here.

@teknium1 teknium1 closed this Jul 12, 2026
@teknium1 teknium1 added the sweeper:implemented-on-main Sweeper: behavior already present on current main label Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard P3 Low — cosmetic, nice to have sweeper:implemented-on-main Sweeper: behavior already present on current main type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants