Skip to content

fix: keep insights model metadata in sync after /model switch - #22282

Closed
LeonSGP43 wants to merge 1 commit into
NousResearch:mainfrom
LeonSGP43:fix/3988-insights-switch-runtime
Closed

fix: keep insights model metadata in sync after /model switch#22282
LeonSGP43 wants to merge 1 commit into
NousResearch:mainfrom
LeonSGP43:fix/3988-insights-switch-runtime

Conversation

@LeonSGP43

Copy link
Copy Markdown
Contributor

Summary

  • overwrite persisted session runtime metadata when an active session explicitly switches models
  • keep /insights aligned with the live provider/base URL instead of preserving stale first-write metadata forever
  • add a regression test that reproduces the stale Gemini-vs-GLM provider report and asserts the corrected insights output

Testing

  • ./scripts/run_tests.sh tests/run_agent/test_switch_model_context.py tests/run_agent/test_switch_model_fallback_prune.py -q
  • git diff --check
  • uv sync --frozen --extra all
  • uv run --frozen ruff check .
  • repo lint-diff semantics (ruff clean; ty advisory signal recorded in proof)
  • clean env -i pytest attribution against latest origin/main (30 head failures, 27 reproduced on base, remaining 3 isolated and passed on retry/base)

Fixes #3988.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard labels May 9, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Thanks for the focused regression case. This is now implemented on current main by a later, stronger attribution design. Automated hermes-sweeper review confirmed that the reported /insights behavior is covered.

  • hermes_state.py:2593-2660 records each API-call usage delta under the live model and billing route in session_model_usage, rather than rewriting a single mixed-session summary row.
  • agent/insights.py:524-534 and agent/insights.py:578-647 aggregate those per-model rows, so a /model switch reports usage for each model that actually processed it.
  • tests/agent/test_insights.py:323-350 verifies a DeepSeek-to-Claude mid-session switch produces separate correct totals.
  • This shipped in cb7f6bbb2e5696bae2c595ca8a3a52cefb6a4a7e (feat(agent): track per-model token usage for mid-session model switches), which is an ancestor of current main.

The PR's one-row overwrite would instead collapse pre- and post-switch usage into the latest model, while current main preserves both routes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard P2 Medium — degraded but workaround exists sweeper:implemented-on-main Sweeper: behavior already present on current main type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: /insights shows wrong model name — displays Gemini instead of actual active model (GLM-5.1 via zai provider)

3 participants