Skip to content

fix(dashboard): aggregate analytics across Hermes profiles - #21773

Draft
LeonSGP43 wants to merge 1 commit into
NousResearch:mainfrom
LeonSGP43:mainline/HER-61-20260508T065013Z
Draft

fix(dashboard): aggregate analytics across Hermes profiles#21773
LeonSGP43 wants to merge 1 commit into
NousResearch:mainfrom
LeonSGP43:mainline/HER-61-20260508T065013Z

Conversation

@LeonSGP43

Copy link
Copy Markdown
Contributor

Summary

Closes #21705.

The analytics dashboard was reading only the active profile's state.db, which
undercounted token usage, sessions, costs, and API calls for multi-profile
users. This change aggregates analytics across the default profile and named
profile databases for the usage and models endpoints.

What Changed

  • discover all profile state.db files once for dashboard analytics
  • merge per-profile usage totals into the /api/analytics/usage response
  • merge per-profile model totals into the /api/analytics/models response
  • add regression coverage proving multi-profile aggregation for both endpoints

Scope Boundary

  • no UI selector or profile filter was added
  • no broader dashboard refactor was included
  • no unrelated CLI, gateway, or tool behavior was changed

Verification

  • targeted analytics regressions passed
  • uv run --frozen python -m py_compile hermes_cli/web_server.py tests/hermes_cli/test_web_server.py
  • git diff --check
  • uv sync --frozen --extra all
  • uv run --frozen ruff check .
  • lint-diff proof showed no net-new diagnostics in touched files
  • clean-env CI-equivalent pytest rerun remained red only in unrelated suite areas; preserved same-base attribution evidence marks the residual failures as preexisting_unrelated

Quality Firewall Proof

  • local_proof_sha: ca2574cef1d1077e3596b4e8cf87d466b0f05b1e
  • github_pr_head_sha: TBD after push
  • worktree_path: /private/tmp/hermes-pr-HER-64-0ec0c564-6e80b26b
  • branch: fix/HER-64-issue-21705
  • base_origin_main_sha: faa13e49f81480771ceeb55991bb0c27edf1a5fb
  • latest_fetch_at: 2026-05-08T08:09:34Z
  • commands: see proof-input-ca2574cef1d1077e3596b4e8cf87d466b0f05b1e.json
  • exit_codes: see proof-input-ca2574cef1d1077e3596b4e8cf87d466b0f05b1e.json
  • started_at: 2026-05-08T08:07:33Z
  • completed_at: 2026-05-08T08:16:52Z
  • log_artifacts_or_inline_summary: see proof-input-ca2574cef1d1077e3596b4e8cf87d466b0f05b1e.json
  • gatekeeper_audit_id: her61-20260508T080207Z-attribution-summary-11db70159
  • baseline_attribution: preexisting_unrelated
  • repair_mode: false

Submission Note

Do not open or push this PR draft until Gatekeeper either:

  1. rebinds the preserved green proof from 11db7015986440698b6609e79fd08f9e41e49c2c to the current identical tree at ca2574cef1d1077e3596b4e8cf87d466b0f05b1e, or
  2. instructs submission from the preserved proof-bearing branch/worktree instead.

@alt-glitch alt-glitch added type/bug Something isn't working comp/cli CLI entry point, hermes_cli/, setup wizard P3 Low — cosmetic, nice to have labels May 8, 2026
@upicat

upicat commented May 8, 2026

Copy link
Copy Markdown
Contributor

Thanks for picking this up! Happy to test on my multi-profile setup once it lands.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes dashboard analytics undercounting by aggregating usage and model analytics across the default Hermes profile and named profile databases.

Changes:

  • Adds profile state.db discovery for analytics.
  • Aggregates usage totals, daily rows, model totals, and skill summaries across profile databases.
  • Adds regression coverage for multi-profile usage and model analytics aggregation.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
hermes_cli/web_server.py Updates analytics endpoints to iterate over discovered profile databases and merge results.
tests/hermes_cli/test_web_server.py Adds helpers and regression tests for analytics aggregation across default and named profiles.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread hermes_cli/web_server.py
Comment on lines +2911 to +2912
for db_path in _list_analytics_db_paths():
db = SessionDB(db_path=db_path)
Comment thread hermes_cli/web_server.py
Comment on lines +3043 to +3044
for db_path in _list_analytics_db_paths():
db = SessionDB(db_path=db_path)
@alt-glitch alt-glitch added comp/dashboard Web dashboard / control panel UI (dashboard/, landing) and removed comp/cli CLI entry point, hermes_cli/, setup wizard labels Jun 27, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Thanks for the focused multi-profile analytics work. Current main took a different profile-scoped direction, so this is not safe to salvage directly.

Problems

  • PR fix(dashboard): scope sessions and analytics to selected profile #45598 added selected-profile analytics: hermes_cli/web_server.py:13841-13845 and 13911-13919 accept profile, while web/src/lib/api.ts:459-465 sends the management profile. tests/hermes_cli/test_web_server.py:1252-1291 verifies that a worker profile excludes default-profile usage. The PR's aggregate-all replacement does not preserve that contract.
  • Current /api/analytics/usage also returns tools (hermes_cli/web_server.py:13903-13905), consumed by desktop Skills at apps/desktop/src/app/skills/index.tsx:88-95; the PR response drops it.
  • Main now collapses session-only model rows into their accounted provider row (hermes_cli/web_server.py:13942-14004, tested at tests/hermes_cli/test_web_server.py:4604-4652), which a current-main aggregation implementation must retain.

Suggested changes

  • If aggregate reporting is still desired, implement an explicit aggregate mode on current main while retaining selected-profile behavior, tool counts, and model-row deduplication.

The linked issue was resolved by #25438 (f7ad2f1115eb370798abe1aca4802d96fe889795) through default-hidden lower-bound token analytics, so the aggregate direction needs a maintainer decision. This is an automated hermes-sweeper review.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform area/profiles Multi-profile isolation, HERMES_HOME scoping labels Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/profiles Multi-profile isolation, HERMES_HOME scoping comp/dashboard Web dashboard / control panel UI (dashboard/, landing) P3 Low — cosmetic, nice to have sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Analytics dashboard only counts sessions from the active profile, undercounting total token usage for multi-profile users

5 participants