fix(dashboard): query session_model_usage for accurate model analytics (#71778)

[JonthanaHanh]

Summary

Fixes #71778

The Models dashboard page (/api/analytics/models) queried the sessions table with GROUP BY model, billing_provider. The sessions table only records the final billing_provider after a mid-session /model switch, so all tokens were attributed to a single provider even when multiple providers were used during the session.

This PR switches the main query to session_model_usage which tracks every API call individually with its own model+provider attribution. This matches the approach already used by the Insights engine (_compute_model_breakdown).

Changes

• hermes_cli/web_server.py: Replace FROM sessions query in _get_models_analytics() with FROM session_model_usage u JOIN sessions s
• Adds try/except fallback to old sessions query for older DBs without the session_model_usage table
• Uses COUNT(DISTINCT u.session_id) instead of COUNT(*) for accurate session count
• Filters u.task = '' to exclude auxiliary usage rows (already handled separately by _aux_usage_rows)

Test Plan

• python3 -m py_compile hermes_cli/web_server.py
• Dashboard Models page shows correct per-provider attribution for sessions that switched models mid-conversation
• Older DBs without session_model_usage table still work (fallback query)

[PRATHAMESH75]

Reviewed against #71778. The root cause is right and the core fix is correct: _get_models_analytics() now aggregates from session_model_usage (per-API-call model+provider attribution) instead of the sessions table (which holds only the final billing_provider after a /model switch), so mid-session switches stop being misattributed to one provider. WHERE u.task = '' correctly scopes to main-agent usage so it doesn't double-count the auxiliary rows added below. COUNT(DISTINCT u.session_id) and MAX(u.last_seen) are the right shapes. I verified every queried column exists on the table (hermes_state.py:1158-1178).

Two things worth tightening before merge:

1. tool_calls is now always 0. session_model_usage has no tool_call_count column (schema at hermes_state.py:1158-1178), so 0 as tool_calls hardcodes the Models-page tool-call count to zero for every card — the old query summed the real sessions.tool_call_count. That column is displayed on the page, so this is a visible regression. Tool calls aren't tracked per (model, provider), so exact per-card attribution isn't available, but a per-session tool_call_count sum joined back on session_id would at least preserve a meaningful total instead of a flat 0 (or, if that's out of scope, calling out the 0 explicitly so a reviewer signs off on dropping the column).

2. The except Exception fallback is too broad. The only legitimate reason to fall back to the old sessions query is a pre-session_model_usage DB, i.e. sqlite3.OperationalError: no such table. A bare except Exception also swallows a genuine bug in the new query and silently resurrects the exact misattribution this PR is fixing — the failure mode would be invisible. Narrowing to except sqlite3.OperationalError (or a pragma_table_info existence check like the migrations use at hermes_state.py:3405) keeps the old-DB safety net while letting real errors surface.

Neither blocks the core fix; both are about not masking a regression. Thanks for tracing this to the right table.

[PRATHAMESH75]

Reviewed against #71778 — the root-cause diagnosis and the core fix are right. sessions.billing_provider only holds the final provider after a /model switch, so aggregating there misattributes every token to one provider; moving the aggregation to session_model_usage (which is keyed per model, billing_provider, task per API call) is the correct source. Filtering u.task = '' to keep main-agent usage while the auxiliary rows are added separately below is also right — it avoids double-counting aux.

One concrete regression to flag before merge:

• tool_calls is now hardcoded to 0. session_model_usage has no tool_call_count column (hermes_state.py:1158), so the query can't source it there. But the frontend still renders it: web/src/pages/ModelsPage.tsx:520 shows entry.tool_calls when > 0, so with this change the per-model tool-call stat disappears from every model card — a silent data loss for a field that previously worked. Per-model tool-call attribution after a mid-session switch is genuinely ambiguous (same reason the tokens were wrong), so dropping it may be the right call — but it should be an explicit decision, e.g. keep a session-level SUM(tool_call_count) joined from sessions, or drop the tool_calls UI row rather than leave it permanently hidden. Either way worth a line in the PR body so a maintainer knows the stat is going away.

Minor: avg_tokens_per_session is now AVG(input+output) over session_model_usage rows, not over sessions, so for sessions spanning multiple models it averages per model-row rather than per session — the label slightly overstates what it measures now.

Neither blocks the core correctness fix; the provider/token attribution is the important part and this gets it right.

[teknium1]

Thanks for tracing the Models dashboard to the per-route accounting table. Current main still aggregates this endpoint from sessions (hermes_cli/web_server.py:13952-13969), while the write path records the active model/provider per API delta in session_model_usage (hermes_state.py:4224-4290), so the core direction is correct.

Problems

• hermes_cli/web_server.py:16835 hardcodes tool_calls to 0. session_model_usage has no such column (hermes_state_common.py:218-238), but the dashboard renders positive entry.tool_calls values (web/src/pages/ModelsPage.tsx:520-524), so this removes the stat from every main-agent card.
• hermes_cli/web_server.py:16845 catches every exception and silently falls back to the known-inaccurate sessions query. The analogous Insights fallback is limited to sqlite3.OperationalError (agent/insights.py:525-541).
• The updated test only changes an existing zero-API-call expectation; it does not cover a single session switching between two model/provider routes.

Suggested changes

• Preserve or explicitly redesign session-level tool-call attribution rather than returning zeros.
• Narrow the fallback to the expected SQLite compatibility error.
• Add an endpoint regression test for two model/provider rows written under one session.

Automated hermes-sweeper review.

[teknium1]

session_model_usage has no tool-call field, so this makes every main-agent Models card report zero and hides the UI statistic at ModelsPage.tsx:520. Please preserve a deliberate session-level attribution (as Insights does) or remove/redesign the displayed metric explicitly.

[teknium1]

Please narrow this fallback to the expected SQLite compatibility failure. Catching every exception hides defects in this new query and silently reverts users to the inaccurate sessions aggregation; Insights limits its analogous fallback to sqlite3.OperationalError.

[GottZ]

This was generated by AI during triage.

Summary

Seven PRs address this issue complex across three layers: persisting the latest model switch, recording per-route usage, and consuming that accounting in Insights and the Models dashboard. #35256 landed latest-model persistence, #62610 landed per-route accounting and Insights aggregation, while #71802 is the remaining dashboard-consumer fix but has documented gaps.

Related pull requests

• Per-model token tracking for mid-session model switches #28842 [closed] best fix — (+497/-91) — n/a: Adds broad per-model token and cost tracking through a sessions.usage_by_model JSON field and updates usage/Insights consumers, directly addressing the accounting loss in Per-model token usage is lost during mid-session model switches #28637. It remains relevant as the recorded best fix for Per-model token usage is lost during mid-session model switches #28637, but was closed as oversized for the dashboard-staleness fix and its accounting role was later implemented through the normalized design in fix(insights): attribute usage across model switches #62610.
• fix(state): persist mid-session model switch to database #35181 [closed] fixes — (+43/-0) — n/a: Adds update_session_model() and invokes it at both gateway switch paths, fixing stale sessions.model but not splitting usage between models. Closed because its commit was salvaged into fix(state): persist mid-session model switch to database (#34850) #35256 with attribution, guards, and a regression test.
• fix(state): persist mid-session model switch to database (#34850) #35256 [merged] best fix — (+69/-0) — n/a: Merged reference implementation for persisting the latest model after gateway switches through update_session_model(), resolving Bug: Model switch mid-session not persisted to database (dashboard shows stale model) #34850 without changing the token-update hot path. It is the recorded best fix for Bug: Model switch mid-session not persisted to database (dashboard shows stale model) #34850, but it does not provide per-model token attribution.
• fix(state): use COALESCE(?, model) so mid-session model switches are persisted #49682 [closed] fixes — (+2/-2) — n/a: Reverses COALESCE arguments so every token update can overwrite sessions.model, targeting the stale-model symptom but conflicting with the dedicated canonical writer introduced by fix(state): persist mid-session model switch to database (#34850) #35256. Closed as already resolved; the contributor review documents that this diff could clobber the persisted switch with a caller-provided model.
• feat(agent): track per-model token usage for mid-session model switches #51634 [closed] fixes — (+438/-20) — n/a: Introduces session_model_usage and makes Insights aggregate per-model/provider deltas, directly addressing Session model/billing fields don't reflect mid-session model switches → incorrect metrics #51607 rather than merely storing the latest model. Closed after its core design was salvaged and hardened in merged fix(insights): attribute usage across model switches #62610.
• fix(insights): attribute usage across model switches #62610 [merged] best fix — (+710/-28) — n/a: Merged per-route accounting and Insights implementation, extending feat(agent): track per-model token usage for mid-session model switches #51634 with route identity, cost reconciliation, migration/deletion safety, fallback handling, and tests. It is the recorded best fix for both Per-model token usage is lost during mid-session model switches #28637 and Session model/billing fields don't reflect mid-session model switches → incorrect metrics #51607 and is the storage foundation required by fix(dashboard): query session_model_usage for accurate model analytics (#71778) #71802.
• fix(dashboard): query session_model_usage for accurate model analytics (#71778) #71802 best fix — (+56/-20) — n/a: Correctly moves Models analytics from stale sessions aggregates to session_model_usage, filters main-agent rows, and counts distinct sessions, matching the recorded best fix for Dashboard Models page (/api/analytics/models) still queries stale sessions table instead of session_model_usage — provider badges and token attribution remain incorrect after mid-session /model switches #71778. The contributor keep_open review identifies three concrete remaining gaps in the diff: tool_calls is forced to zero, the fallback catches every exception, and no switched-route regression test exercises the reported failure.

Duplicates

#35181 is the source implementation salvaged by #35256; #51634 is the core per-route design salvaged and hardened by #62610. #28842 overlaps #51634/#62610 on per-model accounting but uses a different JSON storage design, while #49682 overlaps the stale-model symptom addressed correctly by #35256; #71802 is not a duplicate because it fixes the downstream Models dashboard consumer.

Suggested consolidation

Keep #71802 open with a salvage path, consistent with the contributor keep_open review: retain its session_model_usage aggregation and u.task = '' scoping, narrow the fallback to sqlite3.OperationalError, deliberately preserve or redesign tool-call attribution, and add a regression test with one session switching across two model/provider routes. Treat #35256 and #62610 as merged reference implementations, and keep #28842, #35181, #49682, and #51634 closed under the explicit supersession chains above.

Complex graph

flowchart LR
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I28637(["issue #28637 (closed)"])
    I51607(["issue #51607 (closed)"])
    I71778(["issue #71778 (open)"])
    P71802["PR #71802 (open)"]
    P71802 -.->|partial| I28637
    P71802 -.->|partial| I51607
    P71802 -->|best fix| I71778
    class I28637 closed
    class I51607 closed
    class I71778 open
    class P71802 open
    class P71802 best
    class P71802 target
    click I28637 "https://github.com/NousResearch/hermes-agent/issues/28637"
    click I51607 "https://github.com/NousResearch/hermes-agent/issues/51607"
    click I71778 "https://github.com/NousResearch/hermes-agent/issues/71778"
    click P71802 "https://github.com/NousResearch/hermes-agent/pull/71802"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 7 pull requests and 4 issues in this complex. Each diff was read against this issue; Assessment working set: 76 kB of PR diffs, 28 kB of issue/PR text, 15 kB of discussion (22 comments), 34 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.