fix(gateway): wait for Telegram adapter degraded state before startup notifications

[kevin-lucifer]

What

Fixes a race condition where Telegram startup notifications fail with send_path_degraded after planned gateway restarts.

Root Cause

The Telegram adapter's _send_path_degraded flag is set during polling initialization and only cleared after the first successful getUpdates. On networks requiring fallback IPs (DoH discovery + retry), this can take >1 second. The gateway only waited 1.0s before sending startup notifications, so the send was rejected.

Fix

After the initial 1.0s settle wait, poll all adapters for the _send_path_degraded flag and wait until it clears (max 10s) before sending startup notifications.

Testing

• 26/27 restart notification tests pass
• The 1 failing test (test_restart_command_uses_detached_without_systemd) is environment-related (systemd detection in containers) and fails both before and after this change

Related Issues

• Fixes Telegram startup notification fails with send_path_degraded after planned restart — race between _send_path_degraded clear and _send_home_channel_startup_notifications #66589
• Related: Telegram: /restart command sends no user feedback before gateway shutdown #65057 (same root cause, shutdown phase)

[alt-glitch]

This was generated by AI during triage.

Implements #66589. This bounded pre-dispatch degraded-state wait overlaps the restart-readiness work in #65709 and generic lifecycle retry in #64613, but changes a different notification stage and has broader adapter scope; linked as related rather than duplicate for maintainer selection.

[tonydwb]

Code Review Summary

Verdict: Comment

Critical

• None

Warnings

• None

Assessment

Fixes startup notifications being rejected with "send_path_degraded" (#66589) by waiting up to 10s for Telegram's degraded send path to clear before sending startup notifications. The wait loops over adapters with _send_path_degraded attribute, checking every 500ms up to 20 iterations.

Looks Good

• Correct fix: waits for the adapter to finish its first successful getUpdates before sending notifications
• Reasonable timeout (10s) with graceful break
• Targets only adapters that actually use degraded send gate

---

Reviewed by Hermes Agent

[teknium1]

Thanks for targeting a real Telegram restart-notification race. The current approach needs strengthening before it reliably closes that race.

Problems

• gateway/run.py:7584 caps the new wait at 10 seconds, but Telegram only clears the gate after getUpdates progress (plugins/platforms/telegram/adapter.py:2028-2039), and its verifier permits 60 seconds (plugins/platforms/telegram/adapter.py:2623-2639). A longer startup still reaches the failed send path and _send_restart_notification() removes the marker in gateway/run.py:15455.
• The diff contains no regression test. Existing notification-helper tests do not exercise GatewayRunner.start() with delayed adapter readiness.

Suggested changes

• Use readiness handling aligned with the Telegram polling lifecycle, or defer/retry an explicitly retryable lifecycle send while retaining a redelivery path.
• Add a startup-path regression test for readiness arriving after the one-second settle delay and for the bounded-not-ready case.

Automated hermes-sweeper review.

[teknium1]

This 10-second cap is shorter than Telegram's own 60-second getUpdates-progress verifier (plugins/platforms/telegram/adapter.py:2623-2639). If the flag remains set after this loop, the following lifecycle send still returns send_path_degraded and _send_restart_notification() removes its marker. Please use readiness/retry handling that preserves a delivery path when startup takes longer.

[kevin-lucifer]

Review feedback addressed in fc38137fc:

1. Timeout too short (10s vs verifier's 60s)
Reworked the wait to be event-driven and bounded at 65s (_STARTUP_LIFECYCLE_READY_TIMEOUT), aligned just above the Telegram polling progress verifier's 60s budget (_POLLING_PROGRESS_TIMEOUT). Instead of fixed 0.5s polling, it now waits on the adapter's _polling_progress_event — re-fetched each round in short (2s) slices, since Telegram recreates that event every polling generation — with a 0.5s polling fallback for adapters exposing the flag without the event. A slow-but-recovering first getUpdates now beats the deadline, and a genuinely stuck adapter still can't stall startup indefinitely (warning logged, notifications sent anyway).

2. No regression test
Added four tests in tests/gateway/test_restart_notification.py:

• readiness arriving after the 1s settle window is picked up promptly (not at the deadline)
• bounded wait when the adapter never becomes ready
• waiter follows the recreated _polling_progress_event across polling generations instead of sleeping on a stale event
• adapters without a degraded-send gate are skipped

Also: the wait now only runs when a lifecycle notification is actually pending (.restart_notify.json / .restart_pending.json), so normal startup pays nothing for it.

[arcabotai]

Independent production reproduction on current main (d7b36070e) from a Linux host running three Hermes gateway profiles under systemd user services.

Before patching, all three profiles followed the same sequence after a planned restart:

[Telegram] Discovering Telegram API fallback IPs via DNS-over-HTTPS…
[Telegram] Connecting to Telegram (attempt 1/8)…
Home-channel startup notification failed for telegram:<redacted>: send_path_degraded

The gateways were otherwise healthy and accepted Telegram messages after polling recovered. This confirms the issue is specifically lifecycle delivery racing the first successful getUpdates, not a broken bot token or home-channel configuration.

I tested a narrower dispatch-boundary variant locally: retry only when SendResult.error == "send_path_degraded", because that result guarantees the adapter did not attempt the send. Non-degraded failures remain one-shot. Focused verification:

49 passed in 17.92s

That suite covered tests/gateway/test_restart_notification.py and tests/gateway/test_gateway_shutdown.py. Two real profile restarts then completed active, consumed their .restart_pending.json markers, and emitted no startup-delivery failure warning.

One remaining reliability gap in the current PR head (fc38137fc): after the 65-second readiness timeout, _wait_for_degraded_send_paths() logs a warning and continues to send. If the adapter is still degraded, adapter.send() returns send_path_degraded, and the restart notification marker is still removed by the existing cleanup path. The longer readiness wait substantially reduces the race, but does not fully preserve the promised notification under the bounded-not-ready case.

I suggest pairing the readiness wait with one of these terminal guarantees:

1. Retain the marker when the lifecycle send returns a retryable/degraded result, so a later recovery path can deliver it.
2. Retry or use the registered standalone delivery fallback specifically for send_path_degraded.

The event-driven wait is useful. The key is not deleting the only durable notification record when its final send was explicitly rejected before transmission.

[kevin-lucifer]

Follow-up addressed in 5ebb7de1d, adopting @arcabotai's option 1 (retain the marker) paired with a bounded in-process redelivery path:

Marker retention on refused sends

• _send_restart_notification() now keeps .restart_notify.json when adapter.send() returns SendResult(success=False, error="send_path_degraded") — a refusal, not a failure: the message never left the process, so the only durable record of the promised notification survives (retain_marker_on_degraded=True by default).
• _send_home_channel_startup_notifications() counts degraded rejections; start() keeps .restart_pending.json when nothing was delivered and at least one send was refused.

Deferred redelivery

• After startup, _schedule_deferred_lifecycle_retry() spawns a background task (only when a marker actually survived) that waits for the degraded send paths to clear — same event-driven mechanism, bounded at 120s — then retries delivery once per marker.
• The retry is last-chance (retain_marker_on_degraded=False): markers are cleared afterwards either way, so a stale file can't produce a spurious "gateway restarted" message after a future unrelated restart. Bounded loss with a warning beats a permanent marker.
• Non-degraded failures (Chat not found, exceptions) keep the existing cleanup behavior.

Tests — 8 new cases in tests/gateway/test_restart_notification.py: marker retained on refusal / cleared on last-chance retry / cleared on real failure, home-channel degraded counting, deferred retry delivering both marker types after recovery, second-refusal cleanup, and the no-marker no-op path. tests/gateway/test_restart_notification.py + test_gateway_shutdown.py: 59 passed, 1 pre-existing environment-dependent failure (test_restart_command_uses_detached_without_systemd, fails identically on unmodified HEAD).

[bjcdeshu]

Independent reproduction confirming this race and the need for the
retained-marker retry in the second commit.

Environment:

• Hermes Agent v0.19.0
• Linux, system-scoped systemd service (Restart=always)
• Telegram DM Topic mode
• Python 3.11.15

Observed lifecycle:

• /restart was recorded at 14:05:08 UTC
• the old gateway exited as expected with 75/TEMPFAIL
• systemd started the replacement gateway at 14:05:17 UTC
• Telegram entered fallback discovery / polling recovery
• the restart completion send ran at 14:05:31 UTC and returned
  send_path_degraded
• the replacement gateway remained active and ordinary Telegram
  messaging recovered normally afterward

Source inspection of the installed tree also confirmed that
_send_restart_notification() unconditionally removed
.restart_notify.json in finally, even though the failed
SendResult was marked retryable, and there was no restart-notification
retry watcher.

This independently validates that the readiness wait alone only narrows
the race, while commit 5ebb7de — retaining the marker on
send_path_degraded and performing a bounded deferred retry — is needed
for durable behavior.

[kevin-lucifer]

Rebased onto current main (was ~8.7k commits behind) and integrated with the _await_startup_boot_sends() refactor from #91969 — marker retention now lives in the boot-send finally, and _schedule_deferred_lifecycle_retry() runs right after the bounded boot sends.

Two follow-ups from the triage thread are addressed in the same push:

1. Multi-target marker gap
The planned-restart marker now survives when ANY home-channel send is rejected with send_path_degraded, even if another target succeeded (previously one success cleared the marker despite another refusal). To keep partial retention duplicate-free, _send_home_channel_startup_notifications() records the delivered targets (_last_home_notify_delivered) and the deferred retry passes them as skip_targets, so only refused targets are re-sent.

2. Boot-send race guard
When boot-path sends outlive the restore-gate drain timeout and continue in the background, the deferred lifecycle retry now waits (bounded) for the boot task before touching markers, and bows out with a warning if it is still running — an in-flight boot send owns the markers, so the retry can't race a duplicate send against it.

Tests: 3 new cases (partial degraded refusal retains marker + records delivered targets; deferred retry skips delivered targets; deferred retry waits out an in-flight boot send). tests/gateway/test_restart_notification.py + test_gateway_shutdown.py + test_restart_resume_pending.py: 73 passed.