fix(homeassistant): bound every wedgeable await and watchdog the listener

[Sora-bluesky]

What does this PR do?

The Home Assistant gateway adapter could go silently deaf for hours after a single transient network failure (#67470): the reconnect ladder stalls, the process stays active(running), and nothing ever notices. This is the same bug class the Telegram adapter fixed in 3391e639f (bounded drain) and c2cb37532 (cause-agnostic watchdog); this PR ports that pattern to the HA adapter's session/WS pair, plus one additional wedge point found while verifying the report.

Four defects fixed in plugins/platforms/homeassistant/adapter.py:

1. Session leak on failed connect. self._session was assigned before ws_connect() awaited, so a raised connect left the just-created session dangling. The connect path now builds into locals, closes on raise, and wires self._session/self._ws only once the socket is usable.
2. Unbounded teardown awaits. ws.close() / session.close() (and disconnect()'s REST close) could block forever on a wedged CLOSE-WAIT socket, hanging the reconnect ladder. Every teardown await is bounded via _bounded_close (asyncio.wait_for + 5s _DRAIN_TIMEOUT), per-step, so one wedged close can't skip the other resource.
3. Unbounded auth handshake. The receive_json()/send_json() calls in the auth ladder had no timeout — a server that accepts the socket but never responds froze _ws_connect() forever (this matches the reported "storm stops with no further log lines"). Each is bounded by _HANDSHAKE_TIMEOUT, and any handshake failure — timeout, client error, or cancellation — tears the connection down in place.
4. No cause-agnostic watchdog. A _watchdog_loop task checks a _last_progress heartbeat (bumped per listen-loop pass and per received frame) and force-cancels + respawns a listener that has made no progress for _LISTEN_STUCK_TIMEOUT (300s).

Two subtleties worth calling out for review:

• Quiet ≠ wedged. aiohttp answers WS heartbeat PINGs internally — they never reach _read_events' async for — so a healthy-but-quiet HA (no state changes) is indistinguishable from a wedged socket by progress alone. Before respawning, the watchdog sends an HA-protocol ping: the pong arrives as a normal frame, bumps progress via the reader (preserving the single-reader invariant), and a live listener is spared the spurious reconnect.
• asyncio.wait, not wait_for, for task cancellation. wait_for's timeout path awaits the cancellation completing, so a task that swallows CancelledError would hang it — the very stall being fixed. _cancel_task_bounded observes with a deadline via asyncio.wait and abandons (with an error log) a zombie that won't die; disconnect() and the watchdog always return.

Related Issue

Fixes #67470

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

• plugins/platforms/homeassistant/adapter.py — the four fixes above; watchdog started alongside the listener in connect() and cancelled first in disconnect() (so it can't respawn mid-teardown).
• tests/gateway/test_homeassistant_network_reconnect.py — new, mirroring test_telegram_network_reconnect.py conventions: session-leak on failed connect, bounded hanging closes in cleanup/disconnect(), bounded auth handshake, handshake send-failure cleanup, watchdog respawn of a wedged listener, ping-probe sparing a quiet-but-healthy listener, ping-probe failure respawning, bounded abandonment of an uncancellable task, and clean watchdog exit.

How to Test

1. scripts/run_tests.sh tests/gateway/test_homeassistant_network_reconnect.py tests/gateway/test_telegram_network_reconnect.py -q
2. Mechanism-level repro of the report: with the adapter connected, drop the network to the HA host → reconnect warnings begin; previously a wedged close/handshake ended all adapter log output while the gateway stayed active(running). With this patch every such await is bounded and the watchdog recovers the listener within _LISTEN_STUCK_TIMEOUT + one interval, with an explicit error log naming the stall.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits
• I searched for existing PRs to make sure this isn't a duplicate (topic search + issue timeline: none; the Telegram fixes Telegram polling reconnect ladder stalls mid-way — gateway alive but silently dead, Restart=always never fires #66377 are the pattern source, different adapter)
• My PR contains only changes related to this fix/feature
• I've run the HA + Telegram reconnect suites — 56/56 pass on my platform, branch rebased onto current main
• I've added tests for my changes (all fail against the previous adapter except the clean-exit test)
• I've tested on my platform: Windows 11. Honest caveat: I don't run a Home Assistant deployment, so the fix is verified at the mechanism level (mocked ws/session per the Telegram suite's conventions), not against a live HA instance. The defects themselves are code-verified against the report's line references.

Documentation & Housekeeping

• Docstrings document each timeout constant and the two subtleties above — or N/A
• cli-config.yaml.example — N/A (no config surface added)
• CONTRIBUTING.md / AGENTS.md — N/A
• Cross-platform impact — pure asyncio; no OS-touching code. scripts/check-windows-footguns.py --diff is clean.
• Tool descriptions/schemas — N/A

Screenshots / Logs

$ pytest tests/gateway/test_homeassistant_network_reconnect.py -q   (Windows 11, after rebase)
10 passed

$ pytest tests/gateway/test_telegram_network_reconnect.py -q        (regression)
44+ passed

# Against the unfixed adapter (regression proof):
6 failed  (original suite; the review-driven tests likewise fail pre-fix)

🤖 Generated with Claude Code

[egilewski]

suggesting changes

_ws_connect() still leaks the newly created ClientSession when the pending WebSocket connection is cancelled. The session remains local until ws_connect() returns, but its cleanup handler catches only Exception; asyncio.CancelledError derives from BaseException, so cancellation from the gateway's outer connect deadline or shutdown bypasses _bounded_close(). Because neither adapter field has been assigned yet, the subsequent defensive disconnect() also cannot reach the session. A direct PR-head probe that timed out a hanging mocked connect and then disconnected observed zero close awaits and no retained session reference. Please close the local session in an explicit cancellation handler, re-raise the cancellation, and add a regression for that path.

The 56 focused Home Assistant and Telegram reconnect tests otherwise passed, including bounded handshake and teardown, healthy-listener ping handling, failed-ping recovery, and cancellation-resistant task abandonment.

Security evidence:

• trust boundary: an unreliable Home Assistant peer and gateway cancellation of an outbound WebSocket connection attempt
• source/sink/invariant: every created ClientSession must become adapter-owned or be closed on every exit path, including cancellation
• current-main reproduction: current main retains the original unbounded handshake and teardown paths, while its pre-connect field assignment at least leaves a cancelled session reachable by defensive disconnect
• PR-head or patch-replay validation: the focused suites pass, but an outer-timeout-plus-disconnect probe at the PR head leaves the local session unclosed and unreachable
• positive/negative cases: all 56 existing focused cases passed; the missing pre-assignment cancellation case reproduced the ownership leak
• residual bypass search: exception, timeout, and cancellation paths before and after session assignment were checked; pre-assignment cancellation is the uncovered exit
• reviewer validation: the refreshed-main comparison and focused suites passed, while the cancellation probe observed session.close awaited zero times after timeout and disconnect

Signed: GPT-5.6-sol-xhigh in Codex

[Sora-bluesky]

Good catch. CancelledError is a BaseException, so the except Exception never saw it, and self._session isn't assigned yet at that point so disconnect() couldn't reach the session either. Fixed with an explicit except (asyncio.CancelledError, Exception) that closes the session and re-raises.

While validating it I found the naive handler still leaks if a second cancellation lands while the close is in flight (the watchdog respawn and disconnect() can both cancel the same listen task). Wrapped the close in a tracked task + asyncio.shield so it runs to completion instead of getting interrupted, and kept a strong ref so asyncio doesn't GC the shielded task mid-close. The same gap was in _cleanup_ws() and disconnect(), so those close as one shielded unit now too.

Added regressions that cancel _ws_connect()/_cleanup_ws()/disconnect() mid-teardown and assert close() is still awaited. They fail on the current head (close awaited 0 times) and pass after the fix.

[egilewski]

suggesting changes

_bounded_close() is still not actually bounded when the underlying close() suppresses cancellation. It uses asyncio.wait_for, whose timeout cancels the inner awaitable and then waits for that cancellation to finish—the exact pitfall this patch already avoids in _cancel_task_bounded().

With _DRAIN_TIMEOUT=0.05, a direct PR-head probe using a close coroutine that catches CancelledError found _cleanup_ws() still pending after 0.20 seconds, and the subsequent session close had not started. The positive control, whose hanging close accepted cancellation, returned within the bound and closed the session once. This leaves the core reconnect/disconnect invariant bypassable: one cancellation-resistant close can still wedge the shielded teardown task indefinitely and prevent later resources from closing.

Please observe the close task with a deadline mechanism that does not wait for cancellation completion (for example, a separately tracked task plus asyncio.wait), abandon it after the deadline, and add the cancellation-resistant close case to the regression suite.

Security evidence:

• trust boundary: Home Assistant/aiohttp teardown can block or resist cancellation while gateway recovery and shutdown must remain bounded
• source/sink/invariant: every WebSocket/session close must return control within _DRAIN_TIMEOUT so one resource cannot block later cleanup
• current-main reproduction: the original direct close await remained pending and never reached the session close
• PR-head or patch-replay validation: cooperative hanging close was bounded, but cancellation-resistant close remained pending at four times _DRAIN_TIMEOUT and blocked the next close
• positive/negative cases: cooperative timeout closed the session once; cancellation-resistant timeout left its close count at zero
• residual bypass search: _cleanup_ws(), _cancel_safe_close(), and _full_teardown() all rely on the same _bounded_close() helper
• reviewer validation: all 62 focused HA/Telegram tests passed, but none covers a close coroutine that suppresses cancellation

Signed: GPT-5.6-sol-xhigh in Codex

[Sora-bluesky]

Thanks for catching that, it turned out to be bigger than the wait_for patch.

Went through the whole close path and the cancellation-suppression bug was one case of a wider pattern. The bounded close no longer uses wait_for (it cancels then waits for that cancellation to finish, so a close that swallows CancelledError just hangs it forever). It now runs the close as its own task and watches it with asyncio.wait(timeout=...) (the mechanism is in _run_bounded_close), so on deadline it returns and leaves the close running in the background rather than waiting on it.

Also fixed while in there:

• send() and _standalone_send() each had an ad-hoc ClientSession with no bound or tracking at all. Both go through the same close path now.
• a close() that raises CancelledError on its own used to abort the whole cleanup sequence and skip everything after it. Now it's isolated so the rest still runs.
• added a generation check so a connect() still in flight when disconnect() fires can't publish its session afterward, and closed a same-shape race between two overlapping connect() calls.

One thing to be upfront about: if close() is genuinely broken and never returns, no wrapper can prove the socket actually closed. This bounds how long the caller waits and keeps the orphaned close from getting lost, but it's not a guarantee close() itself works. That was already true before, just flagging it's not something this fix claims to solve.

[egilewski]

too large to review safely

This PR changes 699 production lines before tests and docs. Please split it or add a focused justification if it should stay together.

Signed: GPT-5.6-terra-low in Codex

[Sora-bluesky]

You're right, it grew past what's reviewable in one pass. I've split it.

This PR is now just the #67470 fix: bound the wedgeable teardown awaits, watchdog the listen loop, and close the WS session when a connect is cancelled. That's the reported "goes silently deaf after a network blip" bug on its own.

The bounded-abandon rework I described above -- making the teardown bound hold even when a close() suppresses its own cancellation (your finding), plus the connect/disconnect generation races that surfaced while fixing it -- moves to a follow-up PR. I'll open that against the merged result once this one lands, so each is a focused diff instead of one ~700-line pile.

The follow-up work is already on a branch so nothing's lost; I just didn't want to hand you all of it to review at once.

[egilewski]

suggesting changes

The smaller split restores head 57d0220, but the cancellation-resistant close blocker still belongs in this PR's stated core invariant rather than a follow-up. _bounded_close() still uses asyncio.wait_for(), so a close() coroutine that catches CancelledError keeps _cleanup_ws() pending instead of returning at _DRAIN_TIMEOUT, and the following session close never starts.

With _DRAIN_TIMEOUT=0.05, a direct probe against this head still had _cleanup_ws() pending after 0.20 seconds. The 62 existing Home Assistant and Telegram reconnect tests pass because they cover a cooperatively cancelled hanging close, not one that suppresses cancellation. Deferring the generation-race work is reasonable, but please keep the asyncio.wait()/tracked-abandon correction and its cancellation-resistant regression here so the PR's claimed bounded-teardown guarantee actually holds.

Security evidence:

• trust boundary: Home Assistant/aiohttp teardown may resist cancellation while reconnect and shutdown must remain bounded
• source/sink/invariant: every teardown step must return control within _DRAIN_TIMEOUT so one close cannot block later resource cleanup
• current-main reproduction: the exact current-main adapter remained pending on the cancellation-resistant close and never reached the session close
• PR-head or patch-replay validation: the exact PR head remained pending four times past _DRAIN_TIMEOUT for the same close
• positive/negative cases: all 62 focused tests passed; the missing cancellation-resistant close case failed
• residual bypass search: _cleanup_ws() and _full_teardown() still sequence later closes through _bounded_close()
• reviewer validation: the probe imported the adapter from head 57d0220 in the managed worktree and reproduced deterministically

Signed: GPT-5.6-sol-xhigh in Codex

[Sora-bluesky]

Good call, this belongs here. Put the bounded-abandon back.

_bounded_close now runs the close as its own retained task and watches it with asyncio.wait(timeout=...) instead of wait_for, so a close() that suppresses its cancellation gets abandoned at _DRAIN_TIMEOUT instead of keeping _cleanup_ws pending. Your probe reproduces here too: on the old head that close stayed pending 0.20s past a 0.05 _DRAIN_TIMEOUT; it returns at the deadline now. A self-raised CancelledError is also swallowed per close so it can't abort the rest of the teardown.

Added three regressions for that: the cancellation-suppressing abandon, plus the session and rest-session closes still running after a ws close that raises CancelledError on its own. _cleanup_ws / disconnect / _full_teardown all go through _bounded_close so they inherit it.

Left the connect/disconnect generation-race work out of this one, since you said that part can be a follow-up.

[egilewski]

too large to review safely

This PR changes 524 production lines before tests and docs. Please split it or add a focused justification if it should stay together.

Signed: GPT-5.6-terra-low in Codex

[Sora-bluesky]

Fair call on the numbers: 474 changed production lines in adapter.py plus 976 of tests.

Why it is one piece: the bug class is any await on the HA websocket that can wedge the adapter, and the fix bounds all of them (send, receive, auth, subscribe, reconnect backoff). A partial split would land some bounds while the remaining awaits still wedge the adapter, so the user-visible stall survives until the last piece merges. Most of the line count is tests covering each timeout path.

The size predates the discipline my recent PRs follow (150-250 lines). If one review-sized unit is not worth the cost here, deprioritizing it is a fair outcome. Taking your second option: justification over split.

[teknium1]

Thanks for the focused recovery coverage; the current main adapter still has the unbounded websocket lifecycle paths this PR targets (plugins/platforms/homeassistant/adapter.py:147-195, 219-247).

Problems

• plugins/platforms/homeassistant/adapter.py:632-640 abandons a cancellation-resistant listener, calls _cleanup_ws(), and overwrites self._listen_task. The abandoned _listen_loop() can later resume and follow its existing reconnect path (adapter.py:578-591), which operates on the same self._ws and self._session as the replacement. No generation/ownership guard prevents that old task from closing or reconnecting the replacement's connection.
• The generic abandonment test in tests/gateway/test_homeassistant_network_reconnect.py:393-424 proves the helper returns, but does not exercise a resumed abandoned _listen_loop() after replacement.

Suggested changes

• Add a listener generation/ownership guard around cleanup and reconnect, and retain abandoned listeners until completion.
• Add a regression where an old read suppresses cancellation then returns after the watchdog replacement; verify it cannot mutate the new connection.

Automated hermes-sweeper review.

[teknium1]

If _cancel_task_bounded() returned with the previous listener still pending, this overwrites the only active-task reference while that old _listen_loop() can later resume and run its own _cleanup_ws()/_ws_connect() against shared adapter fields. Please add generation ownership (and a resumed-zombie-listener regression) so an abandoned incarnation cannot touch the replacement connection.

[Sora-bluesky]

Both paths were real — verified before fixing. The abandoned loop could resume once its swallowed cancel consumed the one task.cancel() the bounded helper issues, and from there it could re-enter the shared cleanup/reconnect path against the replacement's connection.

Fixed in f862558:

• The respawn sequence is now _respawn_listener(), and it revokes the generation BETWEEN the bounded cancel and the cleanup await. Revoking after cleanup leaves a resume window inside that await.
• _read_events(gen) checks the generation per frame, before the progress write and the dispatch, so a late frame from the old socket can neither mask a wedged new reader by forging _last_progress nor dispatch through the old plumbing. The check sits before the frame-type branches, so CLOSED/ERROR frames are covered too, not just TEXT.
• Two new regressions (the second parametrized over TEXT/CLOSED/ERROR) run the real _listen_loop/_read_events over a cancellation-resistant fake socket, plus a respawn-increments-generation check. Kill power measured by mutation: moving the revoke back after cleanup fails the first; deleting the per-frame check fails the second; narrowing the check into the TEXT branch fails its CLOSED/ERROR parametrizations. 24 passed on the file.

On the retained-until-completion suggestion: I kept abandonment (the #67470 design — staying deaf is worse than leaking one stuck task) and made the abandoned instance inert instead, which I think is the property the retention idea was after.

[egilewski]

too large to review safely

This PR changes 573 production lines before tests and docs. Please split it or add a focused justification if it should stay together.

Signed: GPT-5.6-terra-low in Codex

[Sora-bluesky]

Fair, and the number checks out: 520 added and 53 removed in the adapter across 7 hunks, with 13 new functions. That is not a small review.

Two mechanisms are bundled here. Bounding the awaits — the close and teardown helpers plus their call sites — is the actual fix for the wedge that was reported. The watchdog, the listener respawn and the generation guard are a second mechanism layered on top; they share the bounded helpers but nothing forces them into the same change. They landed together because I wrote them in one sitting, which is not a reason for you to review them in one sitting.

I'll split it so the bounded-await hardening stands alone here and the watchdog goes to a follow-up that builds on it. I would rather do that than ask you to carry 573 lines at once.

[GottZ]

This was generated by AI during triage.

Summary

Fifty-four PRs address or reference this reliability complex, spanning Telegram connection retries, polling-conflict recovery, dead-poller detection, startup deadlines, and the Home Assistant listener wedge targeted by #68540. For #67470, #68540 is the only listed HA implementation: its diff bounds connection lifecycle awaits and adds listener watchdog/generation ownership, but its review size remains blocking.

Related pull requests

• fix(telegram): retry on transient TLS failures during connect and send #1527 [closed] fixes — (+51/-27) — n/a: Original transient Telegram TLS retry cut, closed after its work was salvaged into fix(telegram): retry on transient TLS failures during connect and send #1535; it does not address the HA listener wedge.
• fix(telegram): retry on transient TLS failures during connect and send #1535 [merged] fixes — (+60/-27) — n/a: Merged reference implementation for transient Telegram connect/send retries, relevant as early network-resilience work but separate from [Bug]: Home Assistant gateway adapter goes silently deaf after transient network failure (same class as Telegram #66377) #67470.
• fix(gateway): retry Telegram 409 polling conflicts before giving up #2297 [closed] fixes — (+134/-6) — n/a: Original bounded Telegram 409-recovery cut, closed after cherry-pick into fix(gateway): retry Telegram 409 polling conflicts before giving up #2312; it established the conflict ladder rather than HA recovery.
• fix: Telegram 409 polling conflict treated as permanently fatal #2298 [closed] fixes — (+13/-2) — n/a: Narrow first-409 retry proposal, closed in favor of the more complete fix(gateway): retry Telegram 409 polling conflicts before giving up #2297 with explicit polling restart and tests.
• fix(gateway): retry Telegram 409 polling conflicts before giving up #2312 [merged] fixes — (+134/-6) — n/a: Merged reference implementation of bounded Telegram 409 retries and updater restart, separate from the HA WebSocket failure path.
• fix(telegram): auto-reconnect polling after network interruption #2477 [closed] fixes — (+82/-4) — n/a: Telegram network-interruption recovery cut, closed after salvage into fix(telegram): auto-reconnect polling after network interruption #2517; relevant as a reconnect-ladder precedent.
• fix(telegram): recover polling after network interruption #2478 [closed] fixes — (+635/-38) — n/a: Broader alternative Telegram reconnect implementation, closed without becoming the canonical fix and unrelated to HA-specific awaits.
• fix(telegram): auto-reconnect polling after network interruption #2517 [merged] fixes — (+82/-4) — n/a: Merged Telegram network-error reconnect ladder, relevant as precedent for bounded recovery but not an HA fix.
• fix(telegram): self-reschedule reconnect when start_polling fails after network error #3177 [closed] fixes — (+178/-1) — n/a: Original self-rescheduling Telegram reconnect cut, closed after salvage into fix(telegram): self-reschedule reconnect when start_polling fails after 502 #3268; it prevents a dead callback chain.
• fix(telegram): self-reschedule reconnect when start_polling fails after 502 #3268 [merged] fixes — (+162/-1) — n/a: Merged reference implementation for rescheduling failed Telegram polling restarts, conceptually adjacent but outside the HA adapter.
• fix(telegram): probe polling liveness after reconnect to detect wedged Updater #18088 [closed] fixes — (+244/-0) — n/a: Telegram post-reconnect liveness-probe cut, closed after salvage into fix(telegram): probe polling liveness after reconnect to detect wedged Updater (salvage #18088) #18751; it covers a sibling alive-but-deaf state.
• fix(telegram): probe polling liveness after reconnect to detect wedged Updater (salvage #18088) #18751 [merged] fixes — (+244/-0) — n/a: Merged Telegram liveness probe for a wedged Updater, relevant as watchdog precedent for fix(homeassistant): bound every wedgeable await and watchdog the listener #68540.
• fix(update): warn when gateway is running before hermes update #23793 [closed] fixes — (+29/-0) — n/a: Closed as implemented through newer automatic update/restart handling and Telegram conflict recovery rather than its proposed pre-update warning.
• fix(telegram): make polling conflict retryable with longer backoff #23806 [closed] fixes — (+8/-8) — n/a: Longer Telegram conflict backoff cut, closed in favor of the broader fix(telegram): recover from post-update polling conflict without entering limbo #25630/fix(telegram): recover from post-update polling conflict without entering limbo (closes #23783) #28486 recovery path.
• fix(gateway): route conflict-retry start_polling failure to reconnect ladder #25232 [closed] related — (+52/-2) — n/a: Closed as implemented on main by f260aa6, which reschedules failed conflict recovery.
• fix(telegram): route conflict-retry start_polling failure into network reconnect ladder #25285 [closed] fixes — (+187/-2) — n/a: Duplicate of fix(gateway): route conflict-retry start_polling failure to reconnect ladder #25232 with additional tests, closed after the same recovery route was selected.
• fix: deregister stale Telegram webhook, store bg task results #25420 [closed] fixes — (+22/-0) — n/a: Closed mixed-scope cut combining Telegram queue deletion with unrelated background-result state; contributor review required splitting.
• fix(telegram): recover from post-update polling conflict without entering limbo #25630 [closed] fixes — (+86/-23) — n/a: Source cut for broader post-update conflict recovery, closed after its work was cherry-picked into merged fix(telegram): recover from post-update polling conflict without entering limbo (closes #23783) #28486.
• fix(telegram): recover when a polling-conflict retry fails #27099 [closed] related — (+95/-2) — n/a: Parallel failed-conflict-retry recovery cut, closed in favor of the earlier and broader fix(telegram): recover from post-update polling conflict without entering limbo #25630.
• fix(telegram): recover from post-update polling conflict without entering limbo (closes #23783) #28486 [merged] related — (+92/-28) — n/a: Merged reference implementation for post-update Telegram 409 recovery, retry rescheduling, and restart grace.
• fix(telegram): eliminate 409 polling race between PTB retry loop and conflict handler #30158 [closed] fixes — (+118/-1) — n/a: Closed alternative guarding concurrent Telegram conflict recovery; it targets the PTB/Hermes retry race, not HA.
• fix(telegram): drop pending updates on conflict recovery (#40691) #40744 [closed] fixes — (+18/-2) — n/a: Closed as superseded by queue-preserving mainline recovery; its drop_pending_updates=True approach was not adopted.
• fix(gateway): recover Telegram polling after conflicts (#40691) #40827 [closed] fixes — (+73/-15) — n/a: Closed as superseded by stronger conflict recovery and pending-update wedge detection without deleting queued messages.
• Fix #40691: Reset polling error task guard after successful conflict retry #40986 [closed] related — (+123/-0) — n/a: Closed as implemented through later callback-guard and wedged-consumer recovery work on main.
• fix(telegram): kill 409 polling conflict loop by disarming PTB retry synchronously #53941 [merged] fixes — (+181/-0) — n/a: Merged reference implementation synchronously disarming PTB retry before Telegram conflict recovery, eliminating self-overlap.
• fix(telegram): recover when polling updater stops while process stays alive (#55769) #55789 [closed] fixes — (+109/-9) — n/a: Original stopped-Updater watchdog fix, closed after salvage into merged fix(telegram): recover when polling updater stops while process stays alive (#55769) #55905.
• fix(telegram): recover when polling updater stops while process stays alive (#55769) #55905 [merged] related — (+109/-9) — n/a: Merged reference implementation detecting a fully stopped Telegram poller and routing it into recovery.
• fix(telegram): recover when polling updater stops while process stays alive (#55769) #55921 [merged] related — (+109/-9) — n/a: Duplicate salvage of fix(telegram): recover when polling updater stops while process stays alive (#55769) #55789, merged but subsequently identified as redundant with fix(telegram): recover when polling updater stops while process stays alive (#55769) #55905.
• fix(gateway): guard NoneType in Telegram reconnect after network error (#55992) #56028 [closed] fixes — (+12/-0) — n/a: Narrow Telegram None guard, closed after broader fix(telegram): close reconnect races that leave the adapter half-destroyed (#55992) #56200 fixed the underlying reconnect races and silent-limbo path.
• fix(telegram): close reconnect races that leave adapter half-destroyed #56036 [closed] fixes — (+173/-9) — n/a: Original root-cause reconnect-race fix, closed after its commits were preserved in merged fix(telegram): close reconnect races that leave the adapter half-destroyed (#55992) #56200.
• fix(telegram): close reconnect races that leave the adapter half-destroyed (#55992) #56200 [merged] fixes — (+183/-10) — n/a: Merged canonical fix for Telegram adapter teardown/recovery races, stable app references, and stale fatal notifications.
• fix(telegram): close reconnect races that leave adapter half-destroyed (salvage #56036) #56224 [closed] related — (+173/-9) — n/a: Closed salvage of fix(telegram): close reconnect races that leave adapter half-destroyed #56036 because its commits were already on main through fix(telegram): close reconnect races that leave the adapter half-destroyed (#55992) #56200, leaving an empty rebased diff.
• fix(telegram): use wall deadline for init timeout #58250 [closed] fixes — (+137/-1) — n/a: Original wall-deadline Telegram initialization fix, closed after salvage and leak hardening in fix(telegram): wall-deadline init timeout + shut down abandoned init app #58293.
• fix(telegram): wall-deadline init timeout + shut down abandoned init app #58293 [merged] fixes — (+332/-1) — n/a: Merged reference implementation using abandon-don't-await initialization deadlines and cleanup of abandoned PTB transports.
• fix(telegram): keep polling conflict counter across successful retries #58492 [closed] fixes — (+38/-1) — n/a: Closed conflict-counter fix whose premise was carried into merged fix(telegram): require getUpdates progress before polling is healthy #64361 with reset gated on real getUpdates progress.
• fix(telegram): diagnose blocked-loop init hangs, unbind DoH discovery from system DNS #63345 [closed] fixes — (+195/-6) — n/a: Original blocked-loop diagnostics and bounded DNS-discovery cut, closed after salvage into merged fix(telegram): diagnose blocked-loop init hangs, unbind DoH discovery from system DNS #64370.
• fix(telegram): escalate persistent 409 Conflict to fatal via wall-clock gate #64064 [closed] fixes — (+139/-3) — n/a: Closed as implemented by direct polling-progress accounting on main rather than its proposed wall-clock conflict gate.
• fix(telegram): break infinite 409 conflict loop by not resetting counter on polling success (#63724) #64082 [closed] fixes — (+0/-0) — n/a: Closed competing cooldown-based 409-counter design; the visible diff is empty and main selected progress-gated reset semantics.
• fix(telegram): diagnose blocked-loop init hangs, unbind DoH discovery from system DNS #64370 [merged] related — (+195/-6) — n/a: Merged Telegram blocked-loop diagnostics and system-DNS unbinding, relevant to startup hangs but not HA WebSocket recovery.
• fix(telegram): wrap Application.initialize() in wait_for to fix connect hang (#63309) #64486 [closed] related — (+148/-3) — n/a: Closed as redundant and structurally insufficient because asyncio.wait_for cannot fire while the event-loop thread is blocked.
• fix(telegram): instrument getUpdates request via subclass re-tag, not slotted attr (#64482) #64506 [closed] fixes — (+120/-27) — n/a: Original slots-safe Telegram request-instrumentation fix, closed after salvage into merged fix(telegram): slots-safe getUpdates instrumentation for Python 3.13 (#64482) #64574.
• fix(telegram): slots-safe getUpdates instrumentation for Python 3.13 (#64482) #64574 [merged] fixes — (+124/-27) — n/a: Merged reference implementation replacing the read-only instance monkey-patch with a slots-safe subclass re-tag.
• fix(telegram): attach polling instrumentation without an instance attribute #64639 [closed] fixes — (+76/-6) — n/a: Closed duplicate of fix(telegram): instrument getUpdates request via subclass re-tag, not slotted attr (#64482) #64506/fix(telegram): slots-safe getUpdates instrumentation for Python 3.13 (#64482) #64574 using the same slots-safe subclass re-tag mechanism.
• fix(telegram): catch AttributeError from PTB python-3.11 __slots__ conflict with actionable message #64773 [closed] fixes — (+14/-0) — n/a: Closed as redundant because fix(telegram): slots-safe getUpdates instrumentation for Python 3.13 (#64482) #64574 removed the underlying read-only do_request failure rather than merely improving its error.
• fix(telegram): bound drain awaits so a wedged httpx pool cannot freeze the reconnect ladder (#66377) #66492 [closed] fixes — (+62/-4) — n/a: Bounded Telegram drain-await cut, closed after salvage into broader merged fix(telegram): cause-agnostic wedged-recovery watchdog + bounded drain so the reconnect ladder can't freeze silently (#66377, supersedes #66492) #67241 despite its keep_open review requesting stronger task-level coverage.
• fix(telegram): cause-agnostic wedged-recovery watchdog + bounded drain so the reconnect ladder can't freeze silently (#66377, supersedes #66492) #67241 [merged] fixes — (+171/-4) — n/a: Merged reference implementation combining bounded Telegram drain operations with a cause-agnostic wedged-recovery watchdog.
• fix(homeassistant): bound every wedgeable await and watchdog the listener #68540 fixes — (+1692/-53) — n/a: The open diff ports bounded teardown/handshake handling plus watchdog and generation ownership to Home Assistant, but contributor reviews block safe review at its current size and the author agreed to split bounded-await hardening from watchdog machinery.
• fix(telegram): require initial polling readiness #69240 [closed] fixes — (+125/-26) — n/a: Closed after its generation-scoped Telegram cold-start readiness work landed on main with a maintainer race fix.
• fix(telegram): prevent connect hang with retry watchdog and fresh app per attempt (#67498) #70546 [closed] fixes — (+207/-2) — n/a: Original fresh-app and total-watchdog Telegram connect fix, closed after salvage into merged fix(telegram+tools): prevent connect hang with retry watchdog and fresh app per attempt (#67498, #68915) #70884.
• fix(telegram+tools): prevent connect hang with retry watchdog and fresh app per attempt (#67498, #68915) #70884 [merged] related — (+208/-2) — n/a: Merged reference implementation rebuilding Telegram applications across failed connect attempts and bounding the retry loop.
• fix(telegram): require proven polling readiness on cold start #71118 [merged] fixes — (+349/-36) — n/a: Merged Telegram cold-start readiness gate requiring current-generation getUpdates progress before publishing connected state.
• fix: bound DoH fallback-IP discovery with wall-clock deadline #74260 [closed] best fix — (+21/-1) — n/a: Recorded best fix for [Bug]: Telegram gateway hangs at "Connecting to Telegram (attempt 1/8)" — fix #64370 confirmed present, still reproduces #74140 bounds fallback discovery before initialize, but it is closed after the automated verdict cited e16743b and existing bounded resolver tests; the reporter states the cited mainline path still reproduces.
• fix(telegram): clear stale polling sessions on conflict retry #75073 [closed] partial — (+5/-2) — n/a: Closed withdrawn Telegram polling conflict: self-perpetuating retry loop from drop_pending_updates=False #75017 cut because drop_pending_updates=True risks deleting queued messages without proving stale-session termination.
• fix(telegram): preserve conflict retry backoff after transient progress #75096 best fix — (+49/-1) — n/a: Recorded best fix for Telegram polling conflict: self-perpetuating retry loop from drop_pending_updates=False #75017 and still open; its diff preserves queued updates while suppressing only the first recovery-generation progress reset, matching the keep_open high-salvageability verdict.

Duplicates

Salvage/duplicate chains are #1527→#1535, #2297→#2312 with #2298 narrower, #2477→#2517, #3177→#3268, #18088→#18751, #25630→#28486 with #23806/#27099 superseded, #55789→#55905/#55921, #56036→#56200/#56224, #58250→#58293, #63345→#64370, and #64506→#64574 with #64639 duplicate. #75073 and #75096 address #75017 through materially different mechanisms; #75073 is withdrawn, while #75096 preserves the queue contract.

Suggested consolidation

Despite the keep_open review on #68540: the latest contributor reviews identify its 573-line production scope as too large to review safely, and the author explicitly agreed that bounded-await hardening and watchdog/generation recovery are separable. Author action: split out the bounded-await/session-leak fix that can merge, then carry the watchdog and generation-ownership machinery in a focused follow-up; keep #75096 open for separate review as the recorded best fix for #75017, with #75073 already closed.

Complex graph

flowchart LR
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I66377(["issue #66377 (closed)"])
    I67470(["issue #67470 (open)"])
    P68540["PR #68540 (open)"]
    P68540 -->|fixes| I66377
    P68540 -->|fixes| I67470
    class I66377 closed
    class I67470 open
    class P68540 open
    class P68540 target
    click I66377 "https://github.com/NousResearch/hermes-agent/issues/66377"
    click I67470 "https://github.com/NousResearch/hermes-agent/issues/67470"
    click P68540 "https://github.com/NousResearch/hermes-agent/pull/68540"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 54 pull requests and 36 issues in this complex. Diffs were read for 3 of 54 PRs (rest unavailable); Assessment working set: 89 kB of PR diffs, 285 kB of issue/PR text, 121 kB of discussion (155 comments), 5 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.

[Sora-bluesky]

Split done. The bounded-await half is up as #80372, rebased on current main with all 15 of its tests carried over and mutation-checked. The watchdog + stale-generation half is ready on a follow-up branch. It builds on the teardown primitives, so it goes up once the first half lands.

I'll keep this PR open for reference until both halves are in, unless you'd rather close it sooner.