fix(dashboard): back off corrupt usage database reads

[YuYigeng]

Summary

I prevent a damaged state.db from turning /api/analytics/usage into an unbounded database-open and traceback loop. Usage reads now fail closed with a bounded, profile-scoped circuit instead of retrying SQLite on every request.

Fixes #96591

Root cause

The usage endpoint sent _get_usage_analytics() to a worker thread but allowed every sqlite3.DatabaseError to escape. A persistent corrupt-image error therefore reopened the same file on every request, generated a full server traceback each time, and offered callers no retry guidance.

Changes

• classify usage-read failures with the existing persistence error classifier
• back off corrupt databases for 300 seconds and transient SQLite failures for 5 seconds
• key the circuit by profile so one damaged profile does not suppress another
• return an explicit HTTP 503 with Retry-After while the circuit is open
• emit at most one concise diagnostic per profile per five minutes
• clear the circuit automatically after a successful read
• leave the damaged database untouched and direct users to hermes doctor

I intentionally did not rename, delete, recreate, or repair state.db from the live dashboard request path. Recovery remains with the existing explicit repair tooling.

Tests

• scripts/run_tests.sh tests/hermes_cli/test_web_server.py -q — 176 passed
• scripts/run_tests.sh tests/hermes_cli/test_web_server_usage_db_circuit.py -q — 1 passed
• Ruff — passed
• Windows footgun diff scan — passed

The regression uses an actual non-SQLite state.db and verifies that two requests produce one database attempt, one concise log, a 503/Retry-After response, and no modification to the damaged bytes.

[Enough1122]

AI code review — automated review for reference, author can ignore or act on any point.

Well-placed circuit breaker for the hot _get_usage_analytics path. Persistently corrupted state.db previously re-opened the DB on every GET /api/analytics/usage and emitted a full FastAPI traceback per request. The per-profile Dict[key, (retry_at, detail)] plus monotonic clock, 300s corrupt / 5s transient split via classify_persistence_error, 503 with Retry-After, and 300s log throttling turn a noisy tight loop into a bounded, operator-actionable signal (hermes doctor, no mutation of the damaged file).

Points:

1. Profile key: _usage_db_circuit_key(profile or "<default>") isolates circuits per multiplexed profile — a corrupt profile-DB doesn't park the others. Confirm that get_usage's profile parameter is the same key used here (the gateway's profile param vs. default) so the right circuit is checked.
2. Monotonic vs wall clock: time.monotonic() for retry_at is correct; wall-clock time.time() would be skew-sensitive under NTP jumps.
3. Logging: _log.error(..., exc) includes the original sqlite3.DatabaseError but the test asserts Traceback not in log_calls[0] — the test stubs error as a plain lambda; in production exc_info handling will still log the exception. That's fine; the throttle (_usage_db_last_error_log) is what actually suppresses noisy repeats.
4. Recovery: _clear_usage_db_circuit on success correctly opens the circuit after a single successful to_thread result; transient blips thus self-heal at 5s cadence without manual intervention.

[YuYigeng]

Withdrawing this PR as superseded for the reported #96591 incident by merged #108130. I verified the current upstream implementation returns the structured state_db_corrupt 503 response for both usage and model analytics, throttles warnings per store, and leaves the database untouched; its corrupt-store regression passes. My additional database-reopen backoff is not part of that upstream fix, but does not justify replacing the shared response contract or continuing a competing fix for the now-closed issue. I am preserving the branch history for reference.