fix(gateway): offload handoff-watcher SQLite calls to avoid blocking the async heartbeat (#40695)

[kshitijk4poor]

Summary

Closes #40695 — the Discord gateway heartbeat is blocked by synchronous handoff SQLite polling (discord.gateway: Shard ID None heartbeat blocked for more than N seconds).

_handoff_watcher (an async def background task) polls SessionDB synchronously every 2s on the platform asyncio loop. A slow/contended SQLite call stalls the loop and starves Discord heartbeats.

Verified still live on current main (gateway/run.py:5950): no to_thread/run_in_executor anywhere in the watcher.

Fix

Offload all four blocking SessionDB calls inside the watcher loop via asyncio.to_thread so they run off the event loop:

• list_pending_handoffs(), claim_handoff(...), complete_handoff(...), fail_handoff(...)

These four are the only sync self._session_db.* calls inside the watcher loop (verified). The maybe_auto_prune_and_vacuum call runs in __init__ before the loop starts, so it is not a heartbeat hazard and is intentionally left alone.

Salvage / attribution

Salvaged from #40782 (@r266-tech), whose asyncio.to_thread approach was correct but based on an older line location; _handoff_watcher has since moved to gateway/run.py:5950. Re-applied to the current location, authorship preserved. (Supersedes the #40695 cluster: #40853, #40974. Note: #43504's handoff_state DB index is not the fix here — the query already plans fast; the hazard is event-loop blocking under contention — so it's not part of this PR.)

Tests

tests/gateway/test_handoff_watcher_async_db.py — a recording SessionDB asserts each call runs on a worker thread (not the event-loop thread), plus a to_thread spy, plus the error path (fail_handoff). Mutation-checked (reverting the list_pending_handoffs wrap fails the off-loop assertions). 3 pass.

Fixes #40695

[github-actions]

🔎 Lint report: salvage/40695-handoff-watcher-async vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 11181 on HEAD, 11178 on base (🆕 +3)

🆕 New issues (3):

Rule	Count
unresolved-attribute	2
unresolved-import	1

First entries

run_agent.py:2984: [unresolved-attribute] unresolved-attribute: Object of type `Self@get_credits_spent_micros` has no attribute `_credits_session_start_micros`
tests/gateway/test_handoff_watcher_async_db.py:20: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
tests/run_agent/test_credits_notices_toggle.py:76: [unresolved-attribute] unresolved-attribute: Unresolved attribute `_credits_session_start_micros` on type `AIAgent`

✅ Fixed issues (1):

Rule	Count
invalid-assignment	1

First entries

tests/run_agent/test_credits_notices_toggle.py:76: [invalid-assignment] invalid-assignment: Object of type `None` is not assignable to attribute `_credits_session_start_micros` of type `int`

Unchanged: 5904 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

[alt-glitch]

This was generated by AI during triage.

Related: salvage of #40782 (@r266-tech), re-applied to the current gateway/run.py location. Competes with the open #40695 offload cluster (#40782/#40853/#40974, same asyncio.to_thread fix) and #43504 (different mechanism: a handoff_state index). Maintainer should pick one -- this one preserves authorship and targets the current line location.