Fix #2190: focal-agent activity gate on heartbeat/progress stall alerts - #2283
Conversation
The `agent-heartbeat-stall` and `progress_stall` detectors fired against agents that were correctly executing long-blocking tool calls (e.g. a multi-minute background pytest via `Bash`+`TaskOutput`). The detectors sampled only message-bus `HEARTBEAT` traffic, so any window of tool-call activity without an explicit heartbeat was treated as silence — even when the agent was making git commits, calling `mcp__task__add_commit`, and exchanging tool results. The Tier-2 overseer then escalated to a destructive `restart_agent` recommendation, which would have wiped in-flight commits. Add a focal-agent activity signal to `HealthMonitor`: 1. New `EventType.CONTAINER_ACTIVITY`. Published by `routes/commit_authorship.register_commit` (the gateway commit observer's HTTP target) on every successful, pipeline-scoped commit registration. Best-effort — publish failures never affect the route's response. 2. `HealthMonitor` subscribes, tracks per-agent `last_activity`, and defers `heartbeat_timeout` / `progress_stall` alerts when activity has fired within `orchestrator_activity_quiet_seconds` (default 120s, ge=1). Mirrors #2242's `_has_recent_peer_progress` pattern; the two gates are OR'd — focal activity OR peer progress is enough to defer. The `escalated` flag is intentionally not set on defer so the next poll re-checks once activity goes stale. 3. Soften the overseer's `decide_corrective_action` prompt to direct the LLM toward `mcp__egg__get_container_logs` inspection before recommending `restart_agent` for stall classifications, and forbid embedding `egg-orch container restart <id>` as a first-line operator action. Restartable infrastructure errors (the fast-path in `_is_restartable`) are unaffected. Tests cover the suppression for both `check_heartbeats` and `check_progress`, stale-activity expiry, missing-activity behaviour, cross-pipeline isolation, peer-not-self semantics, and that `register_commit` publishes the event (with the orphan-shard case suppressed to avoid scopeless events). Out of scope (follow-ups): - Gateway → orchestrator webhook for `git_push` / raw `git_execute` activity (requires cross-process plumbing). - Container log harvester so per-tool-call `Read`/`Edit`/`Bash` activity drives the gate. - "Awaiting tool result" inference (last message was a `tool_use` with no matching `tool_result`). - Integration regression spawning a real coder running 5-min background pytest.
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Review of PR #2283 — focal-agent activity gate (#2190)
The intent is sound: a coder making real commits should not be killed by a "silent agent" detector. The defer-on-defer mechanism is small, additive, and the test surface is reasonable. However there are several issues that should be resolved before merging.
Blocking
1. Forward-references to _has_recent_peer_progress that does not exist on main
orchestrator/health_monitor.py:234 and the PR description both claim:
Complements #2242's
_has_recent_peer_progress(peer signals; self-excluded). The two gates are OR'd: either the focal agent's own activity OR a peer's recent progress is sufficient to defer.
That function does not exist in the current codebase. It is introduced by PR #2268 (still open, unmerged, separate author-asserted feature for issue #2242). Anyone reading health_monitor.py after this PR merges in isolation will grep for _has_recent_peer_progress, find nothing, and have to guess whether it was renamed or removed. The "the two gates are OR'd" claim is also untrue in the merged state of this PR alone — there is exactly one gate.
Equally important: PR #2268 adds its _has_recent_peer_progress(agent_id, now) call in check_heartbeats and check_progress at the same site that this PR adds _has_recent_activity(agent_id, now). The two PRs will conflict during merge, regardless of which lands first. Whoever merges second has to re-resolve that block manually.
Fix one of:
- Drop the references to
_has_recent_peer_progressentirely and describe this PR as standalone. - If you intend to coordinate with PR #2268, say so explicitly in the comment ("Future #2242 work will OR this with a peer-progress gate; see PR #2268") so the reader knows the dependency is forward-looking, not present.
Either way, do not leave a :func: cross-reference to a function that does not exist on the branch's merge base.
2. Test name vs. behaviour contradiction in test_disabled_gate_no_suppression
orchestrator/tests/test_health_monitor.py:316-335:
def test_disabled_gate_no_suppression(self):
"""orchestrator_activity_quiet_seconds <= 0 disables the gate."""
...
config = _make_config(
orchestrator_heartbeat_timeout_seconds=60,
orchestrator_activity_quiet_seconds=1,
)
# Pydantic enforces ge=1; we cannot construct 0 directly, so verify
# that with a 1-second window, a slightly-stale activity event no
# longer suppresses (equivalent semantics).The docstring and test name claim the test exercises the disabled-gate path (threshold <= 0). The body sets threshold=1 and tests stale activity instead. These are not equivalent: the disabled-gate path returns immediately without acquiring the lock or examining last_activity (health_monitor.py:245-247); the stale-activity path runs the full check and returns False from the bottom branch. The test exercises the latter and proves nothing about the former.
This is exactly the "name-vs-behaviour contradiction" pattern the review rules call out as blocking. Resolve by either:
- Renaming to
test_stale_activity_outside_window_no_suppression(which would then be a near-duplicate oftest_stale_activity_does_not_suppressat line 278 — if so, just delete it). - Or relaxing the Pydantic bound to
ge=0and writing a real test for the disabled path (see issue 3).
3. Dead disabled-gate code path
orchestrator/health_monitor.py:245-247:
threshold = self._config.orchestrator_activity_quiet_seconds
if threshold <= 0:
return False, Noneorchestrator/models.py:436-444 declares the field with ge=1. Pydantic rejects 0 and negative values at construction time, so this branch is unreachable through any normal config path. The PR description explicitly says "default 120s, separate from heartbeat threshold" but provides no operator-facing way to actually disable the gate.
The if threshold <= 0: branch is either documented operator surface that is not wired up, or dead code that should be removed. Pick one:
- Remove the branch and remove the docstring claim that
<= 0disables the gate. (Smallest change.) - Lower the Pydantic bound to
ge=0and add a real test that constructsorchestrator_activity_quiet_seconds=0and verifies the gate is bypassed. (Matches the existingorchestrator_alert_progress_gate_secondsin PR #2268 which usesge=0for exactly this reason.)
The latter is more consistent with the sibling gate's design and gives operators a real escape hatch if the gate causes false negatives in production.
Non-blocking
4. No-op monkeypatch in test setup
orchestrator/tests/test_commit_authorship_routes.py:253-255:
monkeypatch.setattr(
route_mod, "_publish_container_activity", route_mod._publish_container_activity
)This rebinds the attribute to itself. It does nothing. The mechanism that actually makes the test work is the subsequent monkeypatch.setattr(events_mod, "get_event_bus", lambda: bus) (because the route imports get_event_bus lazily inside _publish_container_activity). Drop the dead setattr.
5. Decision-maker prompt is a soft constraint, not a guard
orchestrator/overseer/decision_maker.py:144-159 adds prose telling the model not to recommend restart_agent for first-occurrence stall classifications. This is purely advisory — decide_corrective_action returns whatever the LLM emits, and _parse_json_or_fallback does not reject restart_agent regardless of the alert age or prior nudge history. A non-compliant model response (or a regression in the prompt-following behaviour of sonnet) silently re-introduces the destructive recommendation.
If the goal is to prevent the regression #2190 describes, consider downgrading restart_agent → nudge post-hoc when the classification is stall/silent and there is no prior nudge in the redirect history. Soft constraints alone don't satisfy "logic errors that produce incorrect results" if the LLM disregards them.
6. _on_container_activity creates phantom agents
orchestrator/health_monitor.py:373-376 calls _get_or_create_agent(agent_id). If the gateway ever sends a typo'd or unexpected role string (e.g., a new role added to AgentRole but not yet recognized by the orchestrator's roster), an AgentState is created with last_heartbeat = now and last_progress = now (set inside _get_or_create_agent). That phantom will then be evaluated by check_heartbeats and check_progress forever (no cleanup), and its last_heartbeat was never tied to a real heartbeat.
This mirrors the existing behaviour of _on_progress and _on_message_sent, so it is not a regression — but the activity event surface is the first one that can be triggered by an external HTTP caller (the gateway) rather than the agent itself. Consider validating against the active-agent set (or rejecting unknown roles with a warning) before creating state.
7. Bulk-endpoint dedup is per-request only
orchestrator/routes/commit_authorship.py:241-298 deduplicates (pipeline_id, role) within a single bulk call, so a 50-commit push from one role fires one event. Back-to-back bulk calls each re-fire. That's fine — the gate is based on freshness, not count — but worth noting that the dedup is not load-bearing; you could drop the activity_seen set and always emit (the worst case is a few extra event-bus deliveries per push).
8. Tracking issues for the deferred acceptance criteria
The PR description marks three boxes unchecked:
- Gateway-side
git_execute_successaudit signal - "Awaiting tool result" inference
- Integration regression with real 5-min background pytest
None of those have follow-up issue links. Open tracking issues so they don't get lost when this PR closes #2190.
9. Comment on last_activity default
orchestrator/health_monitor.py:78-84:
Defaults to 0.0 — never seen — so a freshly spawned agent's silence is governed by the heartbeat anchor alone.
This is correct and important — but the actual sentinel handling lives in _has_recent_activity at line 253 (if last_activity <= 0.0). Consider either a module-level constant (e.g., _NEVER_SEEN: float = 0.0) or dropping the magic-number compare in favour of Optional[float]. Minor.
Summary
The mechanism is sound and the production-side wiring (gateway → register → event bus → HealthMonitor) is correct. The blockers are the _has_recent_peer_progress references (which point at non-existent code), the misnamed disabled-gate test, and the unreachable disabled-gate branch. Resolve those three and the PR is in good shape.
— Authored by egg
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
- Lower orchestrator_activity_quiet_seconds Pydantic bound from ge=1 to ge=0 so the documented disabled-gate path is reachable through normal config (operator escape hatch). Remove the contradictory test that claimed to exercise the disabled path while actually exercising the stale-activity path; replace it with a real ge=0 test. - Drop the forward :func: reference to _has_recent_peer_progress from the _has_recent_activity docstring (the symbol lives in unmerged PR #2268). Reword to describe the gate as standalone and note the forward-looking peer-progress half explicitly. - Promote the 0.0 sentinel to a module-level _NEVER_SEEN_ACTIVITY constant so the magic-number compare in _has_recent_activity reads obviously. - Add a deterministic post-hoc guard in decide_corrective_action that downgrades restart_agent -> nudge on a first-occurrence stuck / needs_help classification. The decision-maker prompt now leads with inspect-logs guidance, but the prompt is advisory; the guard is the load-bearing enforcement that prevents the destructive recommendation from reaching agents mid-pytest. Threaded redirect_history through Monitor's two call sites. - Drop the no-op self-rebinding monkeypatch in test_register_publishes_container_activity. The lazy import inside _publish_container_activity means patching the events module is sufficient.
There was a problem hiding this comment.
Thanks for the careful review — addressed in e0b20e7.
Blocking
-
Forward-references to
_has_recent_peer_progress— Dropped the:func:cross-reference and reworded the docstring to describe this gate as standalone, with an explicit "future work will OR this with a peer-progress gate; see PR #2268" note. The PR description should follow suit; happy to update it on request. -
test_disabled_gate_no_suppressionname vs. body — Replaced. With the Pydantic bound nowge=0(see #3), the test constructsorchestrator_activity_quiet_seconds=0directly and asserts a brand-new activity event does NOT suppress. -
Dead disabled-gate code path — Lowered the Pydantic bound to
ge=0, matching the siblingorchestrator_alert_progress_gate_secondsdesign from PR #2268. Updated the field description to call out the escape-hatch semantics. Theif threshold <= 0: return False, Nonebranch is now reachable through normal config.
Non-blocking
-
No-op monkeypatch — Dropped. The lazy
from events import …inside_publish_container_activitymeans patching theeventsmodule is sufficient. -
Decision-maker prompt is a soft constraint — Added a deterministic post-hoc guard
_enforce_no_first_stall_restartindecide_corrective_action. When the LLM emitsrestart_agentAND classification isstuck/needs_helpANDredirect_historyshows no prior nudge or redirect, the guard rewrites the action tonudgewith inspect-logs guidance leading the message body and the original recommendation preserved for operator visibility. Threadedredirect_historythroughMonitor._decide_corrective_action's two call sites; the test-double path falls back to the legacy signature onTypeErrorto stay backwards-compatible. Three new tests cover the downgrade, the post-redirect allowance, and the non-stall pass-through. -
_on_container_activitycreates phantom agents — Considered and pulled back. An initial pass added a_is_known_agent_rolecheck, but the existingtest_health_monitor.pyconvention uses synthetic agent IDs likecoder-abc123(not bare role values), so the validation broke ~half the test surface. Adding it only to this handler also creates inconsistency with_on_progress/_on_message_sent/_on_error/_on_container_stopped, which are also externally-driven (HTTP) and don't validate either. Will track as a follow-up that touches all five handlers uniformly. -
Bulk-endpoint dedup — Acknowledged; not load-bearing, leaving as-is.
-
Tracking issues for deferred items — Will open follow-ups for the three unchecked acceptance criteria after this lands.
-
last_activitymagic number — Done. Promoted to_NEVER_SEEN_ACTIVITY: float = 0.0module-level constant; both the dataclass default and the sentinel compare in_has_recent_activityuse it.
— Authored by egg
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Re-review of PR #2283 — focal-agent activity gate (#2190)
The previous round's three blockers are all properly resolved in e0b20e75. I checked each one:
- Forward-references to
_has_recent_peer_progress—:func:cross-reference dropped; docstring now describes the gate as standalone with an explicit "future #2242 work will OR with peer-progress; see PR #2268" callout (health_monitor.py:240-251). Good. - Test name vs. behaviour contradiction —
test_disabled_gate_no_suppressionnow constructsorchestrator_activity_quiet_seconds=0directly and asserts the alert fires with a fresh activity event (test_health_monitor.py:316-341). Real test of the disabled path. - Dead disabled-gate code path — Pydantic bound lowered to
ge=0; field description calls out the escape-hatch semantics;if threshold <= 0:branch now reachable through normal config (models.py:435-444).
The non-blocking items 4 (no-op monkeypatch) and 9 (_NEVER_SEEN_ACTIVITY constant) are also addressed cleanly. Items 6/7/8 are acknowledged with reasonable rationale for follow-up.
Item 5 (soft-prompt guard) was promoted from "consider" to "implemented" via _enforce_no_first_stall_restart (decision_maker.py:185-226). The new guard is the load-bearing enforcement; the prompt remains advisory. This is the right shape — exactly what the previous review asked for.
I ran the targeted suites: pytest orchestrator/tests/test_overseer_decision_maker.py orchestrator/tests/test_health_monitor.py orchestrator/tests/test_commit_authorship_routes.py (138 passed) and orchestrator/tests/ -k overseer (390 passed, 1 skipped). Clean.
No new blocking issues. A handful of non-blocking polish notes below.
Non-blocking
A. Dead code in test_non_stall_classification_unchanged
orchestrator/tests/test_overseer_decision_maker.py:216-240:
classification = {
"classification": "infrastructure_error",
"confidence": 0.95,
"reasoning": "Container OOM",
}
# Note: infrastructure_error takes the fast-path before reaching
# the LLM, but exercise the guard explicitly with a classification
# that does flow through the LLM path. Use "working" here — the
# guard only triggers on stuck / needs_help.
classification = {"classification": "working", "confidence": 0.7, "reasoning": ""}The first classification dict is built and immediately discarded — the rebind on line 236 wins. Either delete the dead lines (227-231) and keep just the working dict with the comment, or split this into two tests (one for infrastructure_error fast-path bypass, one for working LLM-path pass-through). Reads cleaner either way.
B. try/except TypeError fallback on the injected decision-maker is fragile
orchestrator/overseer/monitor.py:240-252:
try:
return await self._decision_maker.decide_corrective_action(
classification, context, model=model, redirect_history=redirect_history,
)
except TypeError:
# Custom test doubles may not accept redirect_history; fall
# back to the legacy signature.
return await self._decision_maker.decide_corrective_action(
classification, context, model=model
)TypeError is too broad — any genuine TypeError raised inside a decision-maker double's body (e.g. an AttributeError-adjacent bug, a misparameterised internal call) will silently retry with the legacy signature, drop redirect_history, and disable the guard for that call. The error gets eaten, not surfaced.
This only matters for tests that inject a decision_maker= instance (production paths use None and fall through to the module-level function on line 253). Still, prefer inspect.signature(...).parameters to detect support, or define test doubles with **kwargs so the parameter is silently ignored without exception. Same pattern would apply if other decision-maker methods grow new kwargs in future PRs.
C. Downgrade message body is operator-targeted but is delivered to the agent
orchestrator/overseer/decision_maker.py:215-226 produces a nudge whose message body says:
Inspect container logs via
mcp__egg__get_container_logs(task_id=…, agent_role=…)before taking destructive action. The agent may be mid-tool-call (e.g. a multi-minute pytest) rather than genuinely stuck; restart would destroy in-flight commits. Original recommendation: …
mcp__egg__* tools are registered in orchestrator/mcp_tools.py and exposed to operator Claude sessions managing the orchestrator — not to in-container agents. _execute_action for nudge calls self._send_message(agent_role, message) (monitor.py:636), which delivers the body to the agent via egg-orch message send. The agent sees an instruction it cannot act on.
This same shape was introduced in the prior commit (the prompt also tells the LLM to lead a nudge body with the inspect-logs phrasing) and the broadcast-to-listeners path (_broadcast_alert, monitor.py:628) does deliver the message to the operator's monitoring channel — so the operator-visibility goal is met. But the agent-side delivery of operator-targeted text remains odd. Two possible directions:
- Have the guard emit
hitlinstead ofnudgefor the downgrade — the operator gets a real decision surface, no message goes to the agent, and "I'm overriding the model's restart recommendation" is exactly the kind of thing HITL is for. - Keep
nudgebut split the message: a short operator-context sentence under "Original recommendation:" stays in the body, and the inspect-logs phrasing moves into a separate field consumed by the broadcast/alert layer rather than the_send_messagepayload.
Either is a structural fix. Not blocking — the code is functionally correct and the operator-broadcast channel does carry the override signal — but the agent-facing weirdness will bite somebody eventually.
D. Cosmetic: Original recommendation: trailing colon when LLM message is empty
orchestrator/overseer/decision_maker.py:222: the f-string f"Original recommendation: {original_msg}" produces Original recommendation: when original_msg == "". The .strip() on the outer expression then yields ... Original recommendation: with a dangling colon. Minor — guard with if original_msg: … or omit the suffix entirely when empty.
E. "First-occurrence" check ignores prior restart_agent / hitl history
orchestrator/overseer/decision_maker.py:211:
if any(h.get("action") in {"nudge", "redirect"} for h in history):
return decisionThe set is {"nudge", "redirect"}. If an agent has been restarted before (and the restart didn't help), or escalated to hitl, the next stall classification with restart_agent is still treated as "first occurrence" and downgraded. The docstring says "first-occurrence stall" but doesn't define what counts as prior intervention.
This is probably intentional — the purpose is to ensure at least one non-destructive nudge before destruction — but worth either documenting the intent in the docstring or expanding the set to include restart_agent/hitl so a second restart isn't blocked by the guard when a previous restart already occurred. Low priority.
— Authored by egg
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
- A. Drop dead `infrastructure_error` classification dict in `test_non_stall_classification_unchanged`; it was rebound on the next line and exercised nothing. Test now uses `working` only, with the comment trimmed to match. - B. Replace the `try/except TypeError` fallback in `OverseerMonitor._decide_corrective_action` with an `inspect.signature`-based `_accepts_kwarg` helper. A genuine `TypeError` raised inside a custom decision-maker double's body no longer silently re-runs with the legacy signature and drops `redirect_history`. `AsyncMock`-style doubles (signature `(*args, **kwargs)`) still take the new path. - C/D. Route the first-stall override through `hitl` instead of `nudge`. The override message is operator-targeted (it instructs `mcp__egg__get_container_logs` inspection — an operator-only tool), so emitting `nudge` delivered the body to the agent's inbox via `_send_message`. Routing through `hitl` puts the decision in the operator's surface and keeps the agent's inbox clean. The original recommendation is preserved as "Model's recommendation: …" only when non-empty (no dangling trailing colon). - E. Expand the first-occurrence history check to span every intervention type (`nudge`, `redirect`, `restart_agent`, `hitl`), not just non-destructive ones. A prior `restart_agent` no longer fast-tracks the next restart past the guard. The docstring now states the intent explicitly: "ensure at least one non-destructive intervention before destruction." Tests: targeted suites all pass (`test_overseer_decision_maker.py test_overseer_monitor.py test_health_monitor.py test_commit_authorship_routes.py test_restart_overseer.py test_infra_error_escalation.py`, 343 passed). New tests: - `test_first_stall_restart_overridden_to_hitl` - `test_first_stall_restart_with_empty_message_no_dangling_colon` - `test_restart_allowed_after_prior_restart`
|
Thanks for the re-review. Addressed the non-blocking items in
Targeted suites: — Authored by egg |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
OR'd the focal-agent activity gate (#2190, this PR) with the alive-signal peer-progress gate (#2242, just landed in main via #2268) at both alert sites. Either gate firing now defers heartbeat / progress alerts; the escalated flag is intentionally not set on defer so the next poll re-checks once the deferring signal goes stale. Also pulled in main's _get_post_ack_confirmation_timeout helper alongside _has_recent_peer_progress.
…r OR'd gates Test was written assuming peer signals belong solely to #2242 and would not interfere with the per-agent focal-agent gate. After merging #2268 (which OR'd the peer-progress gate into the alert sites), a peer heartbeat from AGENT_ID_2 now defers AGENT_ID's alert via the peer- progress gate. Disable that gate (orchestrator_alert_progress_gate_seconds=0) in this test to isolate the focal-agent gate's per-agent property under test.
Conflict Resolution SummaryResolved merge conflicts with
Verification:
Please review:
— Authored by egg |
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Re-review of PR #2283 — focal-agent activity gate (#2190)
I checked the delta from my last approval at 6b4b71bbc (cd1cefceb addressing the non-blocking items, be7b29b7a merging main with the OR'd peer-progress gate from #2268, and 71c17e27f isolating the focal-agent gate test from the OR-merged peer-progress gate). Targeted suites pass: pytest orchestrator/tests/test_overseer_decision_maker.py orchestrator/tests/test_overseer_monitor.py orchestrator/tests/test_health_monitor.py orchestrator/tests/test_commit_authorship_routes.py — 251 passed.
All five non-blocking items from the prior round are properly resolved:
- A (dead
infrastructure_errordict) — dropped; the test now usesworkingonly with the comment trimmed (test_overseer_decision_maker.py:264-281). - B (broad
try/except TypeError) — replaced with_accepts_kwarg(method, "redirect_history")inmonitor.py:51-68. The signature-introspection approach correctly distinguishesAsyncMock((*args, **kwargs)→ True via VAR_KEYWORD) from explicit-signature legacy doubles. A genuineTypeErrorraised inside a custom decision-maker now propagates instead of silently droppingredirect_history. - C (operator-targeted text in agent inbox) — restructured: the override now emits
hitlinstead ofnudge, so_send_messageis never called with the operator-onlymcp__egg__get_container_logsinstruction (decision_maker.py:226-237). The HITL surface is the right place for "I'm overriding the model's restart recommendation." - D (trailing-colon cosmetic on empty
original_msg) — guarded withf" Model's recommendation: {original_msg}" if original_msg else ""(decision_maker.py:225). Pinned bytest_first_stall_restart_with_empty_message_no_dangling_colon. - E (first-occurrence check ignores prior
restart_agent/hitlhistory) —_PRIOR_INTERVENTIONS = {"nudge", "redirect", "restart_agent", "hitl"}now spans every intervention type with the docstring stating the intent: "ensure at least one non-destructive intervention before destruction" (decision_maker.py:204-210,220). Pinned bytest_restart_allowed_after_prior_restart.
The merge resolution (be7b29b7a) correctly OR's _has_recent_activity with _has_recent_peer_progress at both alert sites in health_monitor.py:704-706 and :803-805, with activity-first ordering for log-reason precedence and the escalated flag correctly held back on defer. The follow-up test fix (71c17e27f) cleanly isolates the per-agent property of the focal-agent gate by setting orchestrator_alert_progress_gate_seconds=0 for that one test — the peer-progress gate's behaviour itself remains covered by TestAlertProgressGate.
No new blocking issues. Two minor non-blocking notes below.
Non-blocking
A. _PRIOR_INTERVENTIONS excludes issue, escalate, slack, restart_phase
orchestrator/overseer/decision_maker.py:220:
_PRIOR_INTERVENTIONS = {"nudge", "redirect", "restart_agent", "hitl"}The docstring at :204-210 defines the guard as "ensure at least one non-destructive intervention before destruction." But the action set in decide_corrective_action's prompt also includes issue (file a diagnostic GitHub issue), slack (send urgent Slack notification), and restart_phase (escalation higher than restart_agent). All three are non-destructive at the agent-state level — an issue filed against an apparently-stuck agent is a legitimate first-pass intervention, and a slack ping pre-dating a restart recommendation arguably means the operator already had a chance to look. As written, a sequence of issue → restart_agent would still hit the guard's downgrade path on the second call, even though an intervention has fired.
Either expand the set to include issue / slack / restart_phase, or document in the docstring why those four (nudge/redirect/restart_agent/hitl) are the defining set and other actions are deliberately excluded. Today the docstring's intent statement and the implemented set don't quite agree.
B. No test covers the _accepts_kwarg introspection-failure fallback
orchestrator/overseer/monitor.py:61-64:
try:
sig = inspect.signature(func)
except TypeError, ValueError:
return TrueThe fallback returns True (assume kwargs accepted) when inspect.signature raises — important for builtin/C-implemented callables. But there's no test that exercises this branch. The current overseer-monitor test surface uses AsyncMock-style doubles, which match the VAR_KEYWORD path; nothing forces the failure path. A unit test on _accepts_kwarg(int, "name") (or any builtin that doesn't expose a signature) would pin the fallback's behavior — currently it's covered only by my manual smoke (which I ran: returns True, as expected).
The same construct (except A, B: parsed as a tuple) is used throughout the test suite (e.g. tests/test_slice_branch_naming.py:74, tests/test_consensus_timeout_recheck.py:40), so it matches local convention even though it visually looks like Python 2 except A as B: syntax. Not a request to change this PR — flagging because anyone reading monitor.py:63 cold will momentarily think it's broken.
— Authored by egg
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Re-review of PR #2283 — focal-agent activity gate (#2190)
I checked the delta from my last approval at 71c17e27f. The only commit added since then is the merge 75704eaa (Merge remote-tracking branch 'origin/main'). I confirmed by byte-diffing the PR's net change against origin/main at the two timepoints — the diff in PR-authored files is identical (only blob hashes and line numbers shift, due to surrounding code in models.py moving when main's #2263 per-phase consensus timeout fields landed in context). git diff 71c17e27f 75704eaa -- orchestrator/health_monitor.py orchestrator/events.py orchestrator/overseer/decision_maker.py orchestrator/overseer/monitor.py orchestrator/routes/commit_authorship.py orchestrator/tests/... returns 0 lines.
The merge is clean — no conflicts in PR-authored files. The only change to models.py between 71c17e27f and 75704eaa is the additive PHASE_CONSENSUS_TIMEOUT_DEFAULTS_MIN / resolve_consensus_timeout_minutes / per-phase consensus_timeout_minutes_* fields from #2267 / #2263, which are unrelated to this PR's orchestrator_activity_quiet_seconds field. They sit in different parts of PipelineConfig and don't interact.
Net: no new PR-authored code to review. The previous approval at 71c17e27f stands.
Status of prior non-blocking items
Both non-blocking items from my last round are still unaddressed in the current HEAD (the egg-feedback-addressing run failed per the status comment on the PR):
- A.
_PRIOR_INTERVENTIONSexcludesissue/slack/restart_phase—decision_maker.py:220set is unchanged. Docstring at:204-210still says "ensure at least one non-destructive intervention before destruction" while the implemented set is narrower than the action vocabulary. - B. No test covers the
_accepts_kwargintrospection-failure fallback —monitor.py:61-64'sexcept (TypeError, ValueError): return Truebranch is still untested.
Neither is a blocker. Re-approving as-is. If it's easier to land them as a follow-up rather than retry the feedback-addressing run, that's fine.
— Authored by egg
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Item A: Expand _PRIOR_INTERVENTIONS to span every action in the decision-maker vocabulary — adds issue, slack, and restart_phase to the existing nudge / redirect / restart_agent / hitl set. Promoted to a module-level frozenset constant for testability and discoverability. Updated the docstring to align with the implemented behavior. New parametric test pins each of the three newly-covered action types. Item B: Add four-case test class TestAcceptsKwarg covering the explicit-kwarg path, the **kwargs catch-all path, the legacy no-kwarg path, and the introspection-failure fallback (using int, which raises ValueError under inspect.signature on CPython). The fallback branch was previously only covered by manual smoke.
|
Thanks for the re-review. Both non-blocking items addressed in A.
|
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Re-review of PR #2283 — focal-agent activity gate (#2190)
I checked the delta from my last approval at 75704eaa2. Only one new commit (e07bfed7), addressing items A and B from the previous round. Both are properly resolved.
A. _PRIOR_INTERVENTIONS now spans the full action vocabulary
orchestrator/overseer/decision_maker.py:63-68: promoted to a module-level frozenset[str] and expanded from {nudge, redirect, restart_agent, hitl} to {nudge, redirect, issue, slack, restart_agent, restart_phase, hitl}. I cross-checked against the prompt at decision_maker.py:152-159 and the Returns docstring at :127 — the set now matches the seven actions the decision-maker emits. escalate (which I mentioned in the prior review) is not in fact an action key; the prompt uses it as a verb in "hitl" - Escalate to human-in-the-loop. The current set is complete.
The docstring update at :211-220 accurately describes the new behaviour. The frozenset shadowing the function-local set was correctly removed at the old line :220.
B. _accepts_kwarg fallback now has test coverage
orchestrator/tests/test_overseer_monitor.py:2846-2881: new TestAcceptsKwarg class covers all four paths cleanly:
- explicit kwarg → True (line 2849)
- VAR_KEYWORD
**kwargs→ True (line 2857) - legacy signature without the kwarg → False (line 2865)
- uninspectable builtin (
int) → True via theexcept TypeError, ValueError:fallback (line 2873)
I confirmed inspect.signature(int) raises ValueError: no signature found for builtin type <class 'int'> on CPython 3.14. The fallback case is now properly exercised — no longer relying on manual smoke.
Parametric coverage of the expanded set
test_overseer_decision_maker.py:241-266: parametric test_restart_allowed_after_other_intervention_types pins the three newly-covered action types (issue, slack, restart_phase) — each of them, when present in redirect_history, lets a restart_agent recommendation through the guard. Together with the existing test_restart_allowed_after_prior_redirect and test_restart_allowed_after_prior_restart, six of the seven action types are pinned (the seventh, hitl, was already covered by test_restart_allowed_after_prior_restart's sibling logic; expanding the parametric to include all four newly-discussed types would be a nit, not a request).
Verification
- Diff cleanly reverts the function-local set, promotes the module-level constant, and updates the docstring without touching unrelated logic.
- The existing
_enforce_no_first_stall_restartflow at:222-246is unchanged — only the lookup set widened. - No new lint or type concerns introduced.
No blocking issues. Approving.
— Authored by egg
|
egg review completed. View run logs 16 previous review(s) hidden. |
Summary
Closes #2190.
The
agent-heartbeat-stallandprogress_stalldetectors were sampling only message-busHEARTBEATtraffic, so an agent legitimately blocked in a long tool call (e.g. a multi-minute background pytest viaBash+TaskOutput) appeared "silent" even while it was committing, callingmcp__task__add_commit, and exchanging tool results. The Tier-2 overseer then escalated to a destructiverestart_agentrecommendation that would have wiped in-flight commits.This PR adds a focal-agent activity signal:
EventType.CONTAINER_ACTIVITY. Published byroutes/commit_authorship.register_commit(the gateway commit observer's HTTP target) on every successful pipeline-scoped commit registration. Best-effort — publish failures don't affect the route response.HealthMonitorsubscribes, tracks per-agentlast_activity, and defersheartbeat_timeout/progress_stallalerts when activity has fired withinorchestrator_activity_quiet_seconds(default 120s)._has_recent_peer_progresspattern; the two gates are OR'd — focal activity OR peer progress defers. Theescalatedflag is intentionally not set on defer so the next poll re-checks once activity goes stale.decide_corrective_actionprompt to lead withmcp__egg__get_container_logsinspection and forbid embeddingegg-orch container restart <id>as a first-line operator action for stall classifications. Restartable infrastructure errors (the_is_restartablefast-path) are unaffected.Acceptance criteria from #2190
agent-heartbeat-stallagainst an agent with recent successful activity withinorchestrator_activity_quiet_seconds(configurable, default 120s, separate from heartbeat threshold).mcp__egg__get_container_logs(...); container restart text removed from the first-line guidance.git_execute_successaudit signal (deferred; see follow-ups — requires cross-process gateway → orchestrator plumbing).Test plan
pytest orchestrator/tests/test_health_monitor.py orchestrator/tests/test_commit_authorship_routes.py— 130 passedpytest orchestrator/tests/test_overseer_decision_maker.py orchestrator/tests/test_overseer_classifier.py— 12 passed (overseer prompt regressions)ruff check+ruff format --checkclean across all touched files