Fix Matrix sync halting permanently on transient 5xx outages - #66878
Fix Matrix sync halting permanently on transient 5xx outages#66878gmoranxyz wants to merge 1 commit into
Conversation
The sync loop classified any exception whose string contained "401"/"403" as a permanent auth failure and stopped for good. A homeserver blip returns a 502 whose HTML body embeds an SVG with the path coordinate "40.4302" — the substring "403" — so a passing outage permanently killed Matrix sync until a manual gateway restart. Classify on the real HTTP status / errcode instead: only genuine 401/403 (or auth errcodes like M_UNKNOWN_TOKEN) stop the loop; every 5xx/429/connection error retries. Add regression coverage including the exact 502-with-"403"- coordinate payload. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XcKQjcEM1oEamxUmiZ9nAY
alt-glitch
added
type/bug
|
|
Thanks for isolating a real Matrix sync failure mode. Current main still uses substring matching in Problems
Suggested changes
Automated hermes-sweeper review. |
teknium1
added
the
sweeper:blast-moderate
SummaryTwo open PRs address the Matrix sync-loop false positive: #57375 replaces exception-text substring matching with structured mautrix status/errcode fields, while #66878 adds a broader classifier with structured fields, bounded textual fallbacks, and direct tests for the reported 502/SVG payload. Related pull requests
Duplicates#57375 and #66878 substantially duplicate the replacement of broad Suggested consolidationKeep #66878 open with a salvage path: add the requested loop-level regression, explicitly resolve the bounded unstructured-text fallback policy, and consider the competing review's Complex graphflowchart LR
classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
classDef best stroke-width:3px,stroke:#b45309
classDef target stroke-width:3px,stroke:#4338ca
subgraph Dup57375 ["PRs duplicating each other"]
P57375["PR #57375 (open)"]
P66878["PR #66878 (open)"]
end
class P57375 open
class P66878 open
class P66878 target
click P57375 "https://github.com/NousResearch/hermes-agent/pull/57375"
click P66878 "https://github.com/NousResearch/hermes-agent/pull/66878"
Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label). Cross-PR triage: Reviewed 2 pull requests and 0 issues in this complex. Each diff was read against this issue; Assessment working set: 8 kB of PR diffs, 3 kB of issue/PR text, 2 kB of discussion (3 comments), 0 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch. |
|
I hit this issue and it is quite frustrating that small network blips intermittently take out the matrix communication. To help move things along, I created a new PR that addresses @teknium1's feedback and credits @gmoranxyz for the great work on the original fix. #80532 |
fix(matrix): classify sync auth failures by status and errcode, not substring (salvage NousResearch#66878)
5 participants
Problem
The Matrix sync loop classified any exception whose string form contained
401/403as a permanent auth failure and stopped for good, requiring a manual gateway restart.A homeserver blip (e.g. Synapse restarting behind a reverse proxy) surfaces as a 502 whose HTML body embeds an SVG — and that SVG's path coordinate
40.4302contains the substring403. So"403" in str(exc)matched, and a transient outage permanently killed Matrix sync.Observed in production: sync stopped at a 502 and never retried; the process kept running but went silent on Matrix until restarted.
Fix
Replace the substring check with
_is_permanent_matrix_auth_error(), which classifies on the real HTTP status / errcode:http_status/status/errcode) when present."<status>: <body>"string form — a known status is authoritative.401/403(or auth errcodes likeM_UNKNOWN_TOKEN/M_MISSING_TOKEN/M_FORBIDDEN) stop the loop; every5xx/429/connection error retries.Tests
Adds
TestMatrixPermanentAuthClassifier, including the exact 502-with-403-coordinate payload that triggered the outage, plus 5xx/connection-error retry cases and genuine 401/403/errcode stop cases.🤖 Generated with Claude Code