feat(agent): track per-model token usage for mid-session model switches

[tcconnally]

What does this PR do?

The sessions table records only the initial (model, billing_provider) for a session. When a user switches models mid-session (via /model or programmatically), every token — including the switched model's — is attributed to the first model. Insights/billing reports then hide the new model's cost entirely.

Real case from #51607: a session that started on deepseek-v4-pro and switched to claude-opus-4.8 reported 100% of tokens on deepseek, making ~$49 of opus spend invisible.

This adds per-model usage attribution so token/cost metrics stay accurate across any number of mid-session switches, while leaving the sessions summary row (and its "latest model" display) unchanged.

Related Issue

Fixes #51607

(A prior attempt, #28842, was closed without merging and never landed — confirmed no session_model_usage/usage_by_model symbol exists on main.)

Type of Change

• ✨ New feature (non-breaking change that adds functionality)

Changes Made

• hermes_state.py
  • New session_model_usage table keyed (session_id, model, billing_provider) (+ token/cost counters, api_call_count, first_seen/last_seen) and two indexes.
  • update_token_counts() now upserts each per-API-call delta into the table under the model active at that call — the single chokepoint every platform's per-call delta flows through (CLI / gateway / cron / delegated / codex). Recording happens only on the incremental path: the gateway's absolute=True summary overwrite is intentionally skipped, since cumulative totals can't be split per model and would double-count. When a call omits the model, it falls back to the session's recorded model (matches the existing COALESCE-from-session summary behaviour, so current callers that pass no model keep working).
  • SCHEMA_VERSION 16 → 17 with an idempotent backfill (INSERT OR IGNORE) seeding one usage row per existing token-bearing session from its aggregate totals.
• agent/insights.py
  • _compute_model_breakdown aggregates tokens & cost from session_model_usage, so a switched session splits across the models it used. Defensive fallback to the per-session aggregate for any session lacking usage rows (totals never regress). Tool-call counts remain attributed to the session's recorded model (tool calls aren't tied to a specific API invocation).
• Tests — tests/test_hermes_state.py and tests/agent/test_insights.py.

How to Test

1. pytest tests/test_hermes_state.py tests/agent/test_insights.py -q
2. New coverage:
   • test_mid_session_switch_splits_per_model_usage — pre/post-switch deltas land on the correct model; summary row keeps combined totals + latest model.
   • test_model_breakdown_splits_mid_session_switch — insights report shows both models with correctly-attributed tokens/cost.
   • test_per_model_usage_falls_back_to_session_model, test_absolute_update_does_not_record_per_model, test_v17_backfill_seeds_existing_session_usage.
3. Migration validated against real data: ran the v17 backfill against a copy of a 1.3 GB production state.db — all 1,257 token-bearing sessions migrate losslessly (per-model token sums reconcile exactly with the sessions aggregate; distinct-session count matches).

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this feature
• I've added tests for my changes
• I've run pytest tests/ -q and all tests pass — ran the relevant test_hermes_state.py + test_insights.py suites (344 pass) and validated the migration on production data; full suite runs in CI
• I've tested on my platform: Windows 11 (Python 3.14) + Linux (Docker) for the production-DB migration check

Documentation & Housekeeping

• Documentation — N/A (internal schema/aggregation; no user-facing config or tool changes)
• cli-config.yaml.example — N/A (no config keys added)
• CONTRIBUTING.md / AGENTS.md — N/A
• Cross-platform impact considered (stdlib sqlite3 + time only)
• Tool descriptions/schemas — N/A

[tonydwb]

Code Review Summary

Verdict: Approved

Well-implemented per-model token usage tracking that fixes issue #51607. The design is solid:

• New session_model_usage table with composite PK (session_id, model, billing_provider) and proper indexes
• Schema migration v17 with idempotent INSERT OR IGNORE backfill for legacy sessions
• Only incremental path records here (absolute=True calls excluded to avoid double-counting)
• Clean fallback: sessions without per-model rows fall back to their aggregate so totals never regress
• Comprehensive test coverage: single-model usage, mid-session switch, insights breakdown, and schema migration
• Good documentation in docstrings explaining the design rationale

No security concerns. The COALESCE fallback for sessions without per-model data is a nice safety net.

---

Reviewed by Hermes Agent

[tcconnally]

Rebased onto latest main to clear the merge conflict. The v17 migration was renumbered to v20 (upstream added v18 gateway-metadata + v19 FTS5 backfill in the interim); the per-model usage backfill logic is unchanged and now runs after those. tests/test_hermes_state.py + tests/agent/test_insights.py pass locally (401 passed). Ready to merge.

[teknium1]

Merged via PR #62610 after salvaging your per-model usage design onto current main and preserving your authorship in git history. The landing also adds route identity, migration/deletion safety, cumulative-total reconciliation, stored cost attribution, and fallback coverage found during review. Thanks for building the core accounting path and its initial state/Insights tests.

#62610