feat(dashboard): component-level health, unhandled-ASGI-exception tracking, authenticated self-test - #68662
Merged
Conversation
alt-glitch
added
type/feature
Contributor
૮ >ﻌ< ა ci reviewrunning on 7cc57ea looks good to me! |
The dashboard's own liveness surface could report healthy while every
authenticated request 500'd (e.g. wedged state DB) — /api/status carried
gateway-only fields, no storage/dashboard signal, and no middleware
counted unhandled exceptions.
- DashboardHealth state holder: rolling 5-min deque of unhandled-error
timestamps + last self-test result. last_error_type/last_error_path
are internal-only diagnostics — snapshot() exports counts/enums/
timestamps exclusively (PUBLIC_API_PATHS no-secrets contract).
- Outermost @app.middleware('http') (registered last) wraps call_next
in try/except: records + re-raises unhandled exceptions, and records
responses with status >= 500.
- /api/status gains 'components' {gateway, storage, dashboard,
platforms} + top-level 'overall' ok|degraded. storage reuses the
gateway readiness state_db probe (read-only, 1s-bounded) in an
executor; platforms derive ok/degraded from existing
gateway_platforms states.
- Authenticated self-test task started in the lifespan: every 60s an
in-process httpx ASGITransport GET of /api/sessions?limit=1 with the
real _SESSION_TOKEN, feeding the dashboard component. Skips cleanly
when httpx is unavailable and while the OAuth gate is engaged (the
legacy token is not honoured there).
Tests: middleware increments on raising route and on 5xx, window
expiry, components shape + overall, storage degraded when the state_db
probe fails, dashboard degraded after an error, no secret-bearing
fields in the public payload, self-test pass/fail recording (mocked
client) + a real ASGI round trip.
teknium1
force-pushed
the
fix/dashboard-component-health
branch
from
July 21, 2026 19:55
0407d7e to
7cc57ea
Compare
randlee
pushed a commit
to randlee/hermes-agent
that referenced
this pull request
Aug 11, 2026
…search#68662) The dashboard's own liveness surface could report healthy while every authenticated request 500'd (e.g. wedged state DB) — /api/status carried gateway-only fields, no storage/dashboard signal, and no middleware counted unhandled exceptions. - DashboardHealth state holder: rolling 5-min deque of unhandled-error timestamps + last self-test result. last_error_type/last_error_path are internal-only diagnostics — snapshot() exports counts/enums/ timestamps exclusively (PUBLIC_API_PATHS no-secrets contract). - Outermost @app.middleware('http') (registered last) wraps call_next in try/except: records + re-raises unhandled exceptions, and records responses with status >= 500. - /api/status gains 'components' {gateway, storage, dashboard, platforms} + top-level 'overall' ok|degraded. storage reuses the gateway readiness state_db probe (read-only, 1s-bounded) in an executor; platforms derive ok/degraded from existing gateway_platforms states. - Authenticated self-test task started in the lifespan: every 60s an in-process httpx ASGITransport GET of /api/sessions?limit=1 with the real _SESSION_TOKEN, feeding the dashboard component. Skips cleanly when httpx is unavailable and while the OAuth gate is engaged (the legacy token is not honoured there). Tests: middleware increments on raising route and on 5xx, window expiry, components shape + overall, storage degraded when the state_db probe fails, dashboard degraded after an error, no secret-bearing fields in the public payload, self-test pass/fail recording (mocked client) + a real ASGI round trip.
prmartinow
pushed a commit
to prmartinow/hermes-agent
that referenced
this pull request
Aug 26, 2026
…search#68662) The dashboard's own liveness surface could report healthy while every authenticated request 500'd (e.g. wedged state DB) — /api/status carried gateway-only fields, no storage/dashboard signal, and no middleware counted unhandled exceptions. - DashboardHealth state holder: rolling 5-min deque of unhandled-error timestamps + last self-test result. last_error_type/last_error_path are internal-only diagnostics — snapshot() exports counts/enums/ timestamps exclusively (PUBLIC_API_PATHS no-secrets contract). - Outermost @app.middleware('http') (registered last) wraps call_next in try/except: records + re-raises unhandled exceptions, and records responses with status >= 500. - /api/status gains 'components' {gateway, storage, dashboard, platforms} + top-level 'overall' ok|degraded. storage reuses the gateway readiness state_db probe (read-only, 1s-bounded) in an executor; platforms derive ok/degraded from existing gateway_platforms states. - Authenticated self-test task started in the lifespan: every 60s an in-process httpx ASGITransport GET of /api/sessions?limit=1 with the real _SESSION_TOKEN, feeding the dashboard component. Skips cleanly when httpx is unavailable and while the OAuth gate is engaged (the legacy token is not honoured there). Tests: middleware increments on raising route and on 5xx, window expiry, components shape + overall, storage degraded when the state_db probe fails, dashboard degraded after an error, no secret-bearing fields in the public payload, self-test pass/fail recording (mocked client) + a real ASGI round trip.
melon-xf
added a commit
to melon-xf/hermes-agent
that referenced
this pull request
Sep 3, 2026
…search#68662) The dashboard's own liveness surface could report healthy while every authenticated request 500'd (e.g. wedged state DB) — /api/status carried gateway-only fields, no storage/dashboard signal, and no middleware counted unhandled exceptions. - DashboardHealth state holder: rolling 5-min deque of unhandled-error timestamps + last self-test result. last_error_type/last_error_path are internal-only diagnostics — snapshot() exports counts/enums/ timestamps exclusively (PUBLIC_API_PATHS no-secrets contract). - Outermost @app.middleware('http') (registered last) wraps call_next in try/except: records + re-raises unhandled exceptions, and records responses with status >= 500. - /api/status gains 'components' {gateway, storage, dashboard, platforms} + top-level 'overall' ok|degraded. storage reuses the gateway readiness state_db probe (read-only, 1s-bounded) in an executor; platforms derive ok/degraded from existing gateway_platforms states. - Authenticated self-test task started in the lifespan: every 60s an in-process httpx ASGITransport GET of /api/sessions?limit=1 with the real _SESSION_TOKEN, feeding the dashboard component. Skips cleanly when httpx is unavailable and while the OAuth gate is engaged (the legacy token is not honoured there). Tests: middleware increments on raising route and on 5xx, window expiry, components shape + overall, storage degraded when the state_db probe fails, dashboard degraded after an error, no secret-bearing fields in the public payload, self-test pass/fail recording (mocked client) + a real ASGI round trip.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/api/status can no longer stay green while the dashboard itself is broken: the payload now carries a
componentsdict (gateway / storage / dashboard / platforms +overall), backed by an ASGI middleware that counts unhandled exceptions and 5xx responses, and a 60s in-process self-test of an authenticated DB-touching route.Changes
hermes_cli/web_server.py: outermost middleware records exceptions/5xx into a 5-min windowedDashboardHealthholder (counts/enums only — no messages or paths in the public payload)componentsin /api/status: storage reuses the gateway readiness state_db probe; platforms derived from existing gateway_platforms states;overall: ok|degraded/api/sessions?limit=1with the real session token every 60s, feeding the dashboard component; skips cleanly without httpx or under the OAuth gateValidation
New tests: middleware increments on raising route, components shape, storage degraded on probe failure, self-test failure recording, no-secrets payload assertion. Targeted web_server suite green.
Infographic