fix(gateway): alert operator when hermes-home auto-pull is wedged by a dirty live tree

[sahilm-ti]

Problem

gateway/hermes_home_puller.py fast-forwards the live ~/.hermes checkout hourly. _pull_once() previously did branch != main → skip, then not _is_clean() → DEBUG skip (silent), then counted behind. The live tree is almost always dirty (continuous curator/worker skill-edit churn), so the dirty-skip path fired constantly and the auto-pull never landed incoming merges. PRs #54–#58 (~5 merges) piled up undelivered on disk with zero alerting — only noticed when a cron broke. t_1adf6618 manually reconciled them, but the failure recurs within days.

This is candidate (1) of t_db5a6d17.

Change

1. Distinguish wedged from up-to-date. _pull_once now counts _commits_behind() before the clean check. State machine:

   • not main → silent DEBUG skip
   • main, behind==0 → silent DEBUG skip (up-to-date)
   • main, behind>0, clean → fast-forward
   • main, behind>0, dirty → WEDGED: log WARNING + invoke optional notify(behind) callback

2. Notification surface (reused, not invented). GatewayRunner._send_operator_alert(message) pushes to every connected home channel via adapter.send(...) with _thread_metadata_for_target(...) — the exact same home-channel delivery path the shutdown/startup notifications use. make_operator_alert_callback(loop) wraps it in a thread-safe Callable[[str], None] that schedules onto the gateway event loop via safe_schedule_threadsafe (leak-safe on loop-close races). start_gateway threads this callback into start_hermes_home_puller(notifier=...), the same way the cron ticker / code-watcher receive their gateway handles.

3. Throttle. HermesHomePuller tracks _last_alert_ts (monotonic) and suppresses repeat wedged-alerts within alert_throttle_secs (default 6h). The throttle clock only advances once the notifier accepts the message — a failed send retries on the next tick instead of being silently swallowed for 6h.

4. Silent paths preserved. Up-to-date and feature-branch stay DEBUG-only. Only main + behind>0 + dirty pages the operator.

5. Scope note (fix(kanban): stop self-deleting worker scratch workspace on complete_task #5 in the card): branch != main is left as the existing silent DEBUG skip — a feature branch on the live checkout is deliberate operator intent, not a wedged auto-pull, so it doesn't warrant a page. The primary target (dirty+behind on main) is covered.

Out of scope (per card)

• Candidate (2) auto commit-sweep to main — explicitly deferred; no committer built.
• The orchestrator's live_checkout_hygiene_watchdog.sh cron backstop is separate; this PR is the in-process gateway auto-pull alert.

Tests

tests/gateway/test_hermes_home_puller.py extended 30 → 39 tests, all green via scripts/run_tests.sh:

• wedged (main, behind>0, dirty) → notify fired once with behind count; suppressed within throttle window; fires again after window elapses
• clean+behind → fast-forwards, no alert
• up-to-date → no alert, no pull, clean-check never reached
• dirty+up-to-date → not wedged, no notify
• wedged without notifier → non-fatal; notifier exception → non-fatal + throttle clock does not advance
• start_hermes_home_puller(notifier=...) threads the callback onto the instance

Summary by CodeRabbit

• New Features
  • Detects "wedged" repository states when auto-update is blocked by uncommitted changes.
  • Configurable operator notifications for wedged states with per-instance throttling (default ≈6 hours).
  • Sends operator alerts to connected channels via a best-effort delivery mechanism.
• Tests
  • Added/updated unit tests covering wedged detection, notifier behavior, throttling, and start-up wiring.

[coderabbitai]

Warning

Review limit reached

@sahilm-ti, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 27 minutes and 18 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 8c957648-7bff-4b44-baea-2a91c5ac5c5c

📥 Commits

Reviewing files that changed from the base of the PR and between 8fbc23ce49d6458106488da80d5a900c748cafca and cdbf2dc.

📒 Files selected for processing (3)

• gateway/hermes_home_puller.py
• gateway/run.py
• tests/gateway/test_hermes_home_puller.py

📝 Walkthrough

Walkthrough

When the Hermes home auto-puller finds the checkout is behind origin/main but the working tree is dirty (blocking fast-forward), it logs a WARNING and may invoke an optional operator notifier with the behind-commit count; per-instance notifications are monotonic-time throttled (default 6 hours). The gateway provides a callback that delivers alerts to connected home channels.

Changes

Wedged-state alerting for Hermes home puller

Layer / File(s)	Summary
Type definitions and pull-once control flow gateway/hermes_home_puller.py	Introduces WedgedNotifier type alias and _ALERT_THROTTLE_SECS, expands module docstring, and refactors _pull_once to compute commits-behind before cleanliness check so it distinguishes up-to-date vs. clean+behind (fast-forward) vs. wedged (dirty+behind), calling notify(behind) for wedged with exception safety.
HermesHomePuller throttling and public API gateway/hermes_home_puller.py	HermesHomePuller.__init__ gains keyword-only notifier and alert_throttle_secs, adds _last_alert_ts and _now()/_on_wedged(behind) to throttle notifications by monotonic time, wires the throttled handler into the periodic pull loop, and start_hermes_home_puller accepts and forwards the new params with updated docstring.
Operator alert delivery in gateway gateway/run.py	Adds _send_operator_alert async method to deliver alert messages to connected home channels with best-effort error handling and delivered-count tracking, adds make_operator_alert_callback to return a thread-safe sync callback that schedules async delivery on the gateway loop, and passes this callback into start_hermes_home_puller.
Test coverage for wedged-state and throttling tests/gateway/test_hermes_home_puller.py	Reworks TestPullOnce to verify the updated decision ordering and wedged behavior (notify invoked, no fast-forward), adds TestWedgedAlertThrottle to validate throttling semantics with a fake monotonic clock (initial alert, suppression, resume, notifier failure/no-op handling), and adds a test ensuring start_hermes_home_puller forwards the notifier into the puller instance.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐰 I sniff the git tree, it's wedged and slow,
I tap the keepers gently—just so you know.
Six hours between beeps, no frantic alarm,
Fast-forward when tidy, else I sound the charm.
Hop, whisper, and throttle—operator calm.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 34.38% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: detecting and alerting when hermes-home auto-pull encounters a wedged state (behind with dirty working tree).
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch kanban/t_e0cb5cf0

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

🔎 Lint report: kanban/t_e0cb5cf0 vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 9761 on HEAD, 9760 on base (🆕 +1)

🆕 New issues (1):

Rule	Count
unresolved-attribute	1

First entries

tests/gateway/test_hermes_home_puller.py:425: [unresolved-attribute] unresolved-attribute: Attribute `_notifier` is not defined on `None` in union `HermesHomePuller | None`

✅ Fixed issues: none

Unchanged: 5053 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@gateway/hermes_home_puller.py`:
- Around line 213-226: The user-facing log and alert strings in
hermes_home_puller.py currently hardcode "~/.hermes"; replace those literals
with the profile-aware display_hermes_home() from hermes_constants so messages
show the actual HERMES_HOME location. Locate the logger.info and logger.warning
calls (and the similar messages around the later block at lines ~306-312) and
interpolate or format display_hermes_home() into the logged strings instead of
the "~/.hermes" literal, keeping the rest of the message unchanged.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 3b6afebd-f613-4d88-8b6c-8a7deea548fe

📥 Commits

Reviewing files that changed from the base of the PR and between 62a2d5886d1b0dd0b76d3589066dd560dd03df3f and 06b68c491cd7e06962e30c913bd26bfc5cefb40c.

📒 Files selected for processing (3)

• gateway/hermes_home_puller.py
• gateway/run.py
• tests/gateway/test_hermes_home_puller.py

[sahilm-ti]

auto-review: APPROVED -> human_review (awaiting Sahil's merge + approve).

Matrix checks (U1-U6, C1-C5)

• U1 in-scope: PASS. diff touches only gateway/hermes_home_puller.py, gateway/run.py, tests/gateway/test_hermes_home_puller.py -- all within AC scope.
• U2 no out-of-scope deletions: PASS. only the dirty-skip early-return removal (the change itself).
• U3 no secrets: PASS (grep clean).
• U4 AC coverage: PASS. all 5 card specifics covered (reorder behind-before-clean, reused notifier surface, 6h throttle, silent up-to-date/feature-branch, branch!=main left silent per fix(kanban): stop self-deleting worker scratch workspace on complete_task #5).
• U5 mergeable: MERGEABLE / UNSTABLE (non-required shard failures, see caveat).
• U6 UI: N/A -- no UI-emitter paths in diff.
• C1 CI: see caveat below -- all PR-relevant checks green; 2 unrelated shards red.
• C2 type-discipline: PASS. no new type: ignore / cast( / bare Any.
• C4 tests touched: PASS. test file extended 30->39, covers wedged / clean+behind / up-to-date / feature-branch / notifier-exception / throttle-window.
• C5 worker identity: PASS. sole commit 06b68c49 authored + committed as 266772320+sahilm-ai.

Code-quality judgment (role-reviewer)

Verdict: APPROVED. No blockers.

• Notifier surface is genuine reuse, not invention: _send_operator_alert is a faithful copy of the _send_home_channel_startup_notifications pattern (adapter.send + _thread_metadata_for_target, snapshot-adapters, best-effort per-channel swallow). make_operator_alert_callback wraps it via the existing safe_schedule_threadsafe helper -- verified present at agent/async_utils.py:34.
• Throttle is correct: clock advances only after a successful notifier handoff (_last_alert_ts = now after the try), so a failed send retries next tick -- and test_notifier_exception_does_not_advance_throttle locks that in.
• No fallback-on-required-value, no silent degradation introduced; wedged state is loud (WARNING) even without a notifier.
• Behind-before-clean reorder cleanly distinguishes WEDGED from up-to-date; _is_clean/_fast_forward short-circuit on up-to-date preserved.

Caveat for human merge -- CI shard failures are PRE-EXISTING and UNRELATED

test (4) and test (5) failed, but only on tests this PR does not touch:

• tests/hermes_cli/test_kanban_default_assignee.py (2 failures: assert len(res.spawned) == 1 -> got 0)
• tests/hermes_cli/test_kanban_core_functionality.py

PR #45 modifies only gateway/ files; the kanban-dispatcher source and these test files are byte-identical to base (62a2d5886). I reproduced both failures locally on this branch -- they are independent of the PR's code (pre-existing main breakage in the kanban dispatcher's dispatch_once default-assignee path). The PR's own suite passes 39/39 locally, and all PR-relevant CI is green (lint ruff+ty, nix macos/ubuntu, e2e, shards 1/2/3/6).

Recommend: merge is safe w.r.t. this PR; suggest a separate card to fix the test_kanban_default_assignee / test_kanban_core_functionality breakage on main (out of scope here).

[sahilm-ti]

auto-review: approved, awaiting human merge + kanban_approve.

Matrix checks (U1–U6, C1–C6): all pass.

• U1 in-scope ✓ U2 no out-of-scope deletions ✓ U3 no secrets ✓ U4 AC covered ✓ U5 mergeStateStatus CLEAN ✓ U6 no UI-emitter path ✓
• C1 CI green (all 6 test shards incl test(3)+test(4)) ✓ C2 no new type:ignore/cast/Any ✓ C3 ruff+ty green ✓ C4 regression tests added ✓ C5 worker identity sahilm-ai ✓ C6 no UI files ✓
• CodeRabbit: no unresolved actionable threads.

Code-quality judgment (role-reviewer): APPROVED. Single commit, clean rebase onto current main, black-box tests at public boundaries, dependencies injected (activity-state dict / notifier+clock), defensive guards sit on optional liveness/result paths (not required-value fallbacks).