fix(gateway): expire same-epoch orphaned drain markers past a max-age (#85433)

[PRATHAMESH75]

What does this PR do?

Fixes #85433: a .drain_request.json marker orphaned without a machine restart wedges the gateway in draining forever — every inbound message on every platform is bounced with "⏳ This agent is draining for a maintenance action…" until someone removes the marker by hand.

Root cause

The NS-570 fix (PR #53050) stamps the marker with an instantiation epoch, so a marker that survives a machine restart on the durable HERMES_HOME volume is recognised as stale. But that defence rests on the assumption that every drain-gated lifecycle action restarts the machine. When a maintenance action finishes without recreating the container and its writer never clears the marker (e.g. a crash between "action done" and "cancel drain"), the epoch still matches, _marker_epoch_is_stale() correctly returns False, and the 1s watcher honours the orphan indefinitely. In the reported incident a Hermes Cloud instance bounced every Telegram message for ~3 days.

The marker already carries requested_at, but no reader ever consulted it — there was no TTL/max-age fallback, so a same-epoch orphan had unbounded lifetime.

The fix

Add a lenient max-age fallback (_DRAIN_MARKER_MAX_AGE_SECONDS = 60 min) alongside the epoch check, consumed by both drain_requested() and drain_notification_suppressed() via a shared _marker_is_inactive():

• a marker whose requested_at is present, parseable, and older than the bound reads as stale, with a once-per-marker warning log (memoised so the 1s watcher doesn't spam);
• a marker with no requested_at, a non-string one, or an unparseable one stays honoured — the same fail-safe-toward-quiescing leniency the epoch check already uses for legacy/corrupt markers;
• a naive (tz-less) timestamp is assumed UTC, matching the writer;
• a deliberately long drain keeps a sanctioned keep-alive: write_drain_request() is documented idempotent, and re-writing refreshes requested_at.

60 min is deliberately generous — drain-gated actions (auto-update / image migrate / env edit / profile change) complete in minutes. This mirrors the requested_at staleness guard the neighbouring .restart_notify.json marker already applies in gateway/run.py, precisely so "a legitimately old marker should not swallow a fresh action."

The writer-side clean-up (the cloud control plane clearing the marker in a finally) is out of scope for this repo, as the issue notes — the gateway should defend itself regardless, since any writer crash reproduces this.

Tests

tests/gateway/test_external_drain_control.py::TestRequestedAtMaxAge — the same-epoch aged-out orphan reading as absent (the regression), a just-within-bound marker staying active, missing/unparseable requested_at staying active (leniency), naive-timestamp handling, and an aged orphan no longer suppressing the shutdown broadcast. Full test_external_drain_control.py and the neighbouring drain suites pass locally; ruff + windows-footgun gates clean.

Fixes #85433

[alt-glitch]

This was generated by AI during triage.

Duplicate of #85444: both implement the same one-hour requested_at expiry for same-epoch orphan drain markers in gateway/drain_control.py. #85444 was opened first.

[PRATHAMESH75]

Closing this as superseded. Upstream landed a maintainer fix for the same issue (#85433) in 885576671 — "fix(gateway): expire orphaned drain markers past a max-age so a leaked marker can't wedge the gateway (#85433)" — which is functionally identical to this PR: it adds a requested_at max-age fallback (DRAIN_REQUEST_MAX_AGE_SECONDS = 3600s) consulted by both drain_requested() and drain_notification_suppressed(), with the same fail-safe leniency for missing/unparseable timestamps and the keep-alive refresh contract.

Redundancy verified: the merged TestMarkerMaxAge suite (7 tests covering the same-epoch aged-out orphan, the just-within-bound marker, and missing/unparseable leniency) passes on plain upstream/main with none of this PR's code. Nothing here remains to land, so closing in favour of the merged fix.