fix(gateway): expire orphaned drain markers past a max-age so a leaked marker can't wedge the gateway

[kshitijk4poor]

Summary

A .drain_request.json marker orphaned without a machine restart wedges the gateway in draining indefinitely — every inbound message on every platform is bounced with "⏳ This agent is draining for a maintenance action…" until someone manually deletes the marker. Observed in the field: a Hermes Cloud instance refused all Telegram turns for ~3 days (marker written for a maintenance action that completed without recreating the container; the writer never cancelled the drain).

The NS-570 epoch stamp (#53050) only clears markers that survive a restart — it bakes in the assumption that every drain-gated action ends in one. A same-epoch orphan passes the epoch check forever, and the marker's requested_at was write-only: no reader ever looked at it. Compare the neighbouring .restart_notify.json marker, which gateway/run.py guards with a 5-minute requested_at staleness check for exactly this reason.

Fixes #85433.

Changes

• gateway/drain_control.py:
  • New _marker_is_expired(): a marker whose requested_at parses and is older than DRAIN_REQUEST_MAX_AGE_SECONDS (1h) reads as stale. Same leniency contract as the epoch check — a missing/unparseable timestamp still reads as drain-active (fail-safe toward quiescing), a future-dated timestamp (clock skew) is honoured, a tz-naive timestamp is treated as UTC.
  • New _marker_is_stale() composes epoch-mismatch OR expiry; drain_requested() and drain_notification_suppressed() both route through it (all external callers — the 1s gateway watcher, the shutdown broadcast gate, and the dashboard status endpoint — go through these two readers, so every path is covered).
  • Expiry logs a warning once per marker (keyed by requested_at), not once per 1s poll tick (~86k/day otherwise). A refreshed marker that expires again warns again.
  • Long drains keep a sanctioned keep-alive: write_drain_request() is idempotent and re-writing refreshes requested_at (docstring updated to say so).

Before / after

Before: marker written at 20:04 for a no-restart maintenance action, never cleared → gateway bounces every message with the draining text indefinitely (same epoch → honoured on every 1s tick).

After: the same orphan is honoured for at most 1h, then reads as absent; the watcher flips the gateway back to running on the next tick and logs one warning naming the marker's timestamp, age, and principal.

Validation

• tests/gateway/test_external_drain_control.py: 7 new tests — expired marker reads as absent (drain state AND suppression flag), missing timestamp honoured, unparseable timestamp honoured, tz-naive timestamp expires, keep-alive re-write revives an expired drain, expiry warning fires once across repeated polls and re-fires for a refreshed marker.
• Full file: 18 passed. Wider -k "drain or restart" slice over tests/gateway/: 168 passed, 3 skipped.
• Mutation check: reverting gateway/drain_control.py to the pre-fix version makes the 4 new-behavior guard tests fail (the leniency tests correctly still pass — they assert behavior that predates the fix); restoring goes green.
• ruff check clean on both files.