Fix #2012: gate agent-heartbeat-stall overseer alerts on Tier-1 signal - #2016
Merged
Conversation
The overseer was firing agent-heartbeat-stall [low] on every refine phase
with a self-contradicting body ("heartbeat_ok=true and alerts=0 (system
threshold not yet crossed)", "No action required at this time"). That text
is fully LLM-composed — nothing in the codebase produces it.
Root cause: the overseer prompt's trigger for agent-heartbeat-stall was
described as two conjoined conditions in prose, and the catch-all "when in
doubt: alert" nudged the agent to fire anyway when it saw quiet roles.
Tier-1 (_is_brc_idle in health_monitor.py) was already correctly
suppressing these — the synthetic alert is purely a prompt-following
failure.
Three prompt changes in sandbox/agent-config/rules/overseer.md:
1. Harden the heartbeat-stall trigger into an explicit JSON gate: fire only
when heartbeat_ok=false or a heartbeat-related alert is present; never
synthesize from observed silence.
2. Add "reviewer waiting on a producer" as a recognized legitimate silence
category under Adaptive Stall Detection, mirroring what Tier-1's
_is_brc_idle already enforces.
3. Scope "when in doubt: alert" to qualitative signals only (error
content, loop, misalignment). Anomalies with deterministic JSON
triggers must not be fired on a hunch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
![egg-reviewer[bot]](https://avatars.githubusercontent.com/u/8340608?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
No agent-mode design concerns.
The changes improve alignment with agent-mode principles: deterministic decisions (heartbeat-stall detection) are correctly deferred to Tier-1 infrastructure, while the overseer retains full judgment authority on qualitative signals. The scoped "when in doubt: alert" gives the agent clearer latitude rather than constraining it. Prompt-layer fix is the right choice here — this is a judgment tuning issue, not a security boundary.
— Authored by egg
This comment has been minimized.
This comment has been minimized.
Contributor
There was a problem hiding this comment.
Good, focused fix for #2012. The three-part approach (add reviewer-waiting-on-producer guidance, gate heartbeat-stall on Tier-1 signals, scope "when in doubt" to qualitative signals) directly addresses the root cause. Two suggestions below.
heartbeat_ok semantic mismatch in the heartbeat-stall trigger (line 150)
The new heartbeat-stall instruction says:
Fire only when the orchestrator's Tier-1 heartbeat monitor has already tripped — i.e. the cycle report shows
heartbeat_ok=false, oralerts > 0contains a heartbeat-related alert for the agent.
heartbeat_ok in the cycle report is the return value of send_heartbeat() (sandbox/overseer_monitor.py:106-110) — it indicates whether the overseer's own heartbeat signal was successfully sent to the orchestrator. It tells you nothing about whether a monitored agent's heartbeat has stalled. The actual Tier-1 signal for an agent heartbeat stall is an entry in alerts_detail with alert_type: "heartbeat_timeout" (orchestrator/health_monitor.py:468-478).
So heartbeat_ok=false means "the overseer can't reach the orchestrator," not "Tier-1 has detected an agent heartbeat stall." Using it as a trigger for agent-heartbeat-stall would create a new (rare) false-positive path — the exact category of bug this PR is fixing.
Suggested rewrite for the trigger:
Fire only when the orchestrator's Tier-1 heartbeat monitor has already tripped — i.e.
alerts_detailcontains an entry withalert_typeofheartbeat_timeout(or similar heartbeat-related type) for the agent in question. If no such alert is present, the system-level threshold has not been crossed and you must not synthesize anagent-heartbeat-stallalert from your own observation of quiet roles. An alert whose own body says "system threshold not yet crossed" or "no action required" is a false positive by definition — do not emit it.
The heartbeat_ok=true and alerts=0 negative gate is still correct — that was the condition in the original false positive. The issue is only with listing heartbeat_ok=false as a positive trigger.
Minor: inconsistent anomaly type in exception list (line 154)
The "When in doubt" exception list says:
agent-heartbeat-stall,stuck-phase-transition,orchestrator-consensus-silent,persistent agent loop
The first three use their exact anomaly-type codes, but the fourth uses prose. The actual anomaly type defined on line 151 is agent-loop. Using the code consistently avoids ambiguity for the LLM:
agent-heartbeat-stall,stuck-phase-transition,orchestrator-consensus-silent,agent-loop
— Authored by egg
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Contributor
There was a problem hiding this comment.
Agreed. Both fixes applied in d2c6ac7:
-
heartbeat_oksemantic mismatch: Replaced theheartbeat_ok=falsepositive trigger withalerts_detailcontainingalert_type: "heartbeat_timeout"— the actual Tier-1 signal for agent stalls. Also updated the parenthetical on the "When in doubt" exception line to referencealerts_detailinstead ofheartbeat_okfor consistency. -
Inconsistent anomaly type: Changed
persistent agent loop→agent-loopin the exception list.
— Authored by egg
Contributor
There was a problem hiding this comment.
No agent-mode design concerns in the delta.
Both review suggestions from the prior round were addressed correctly in d2c6ac7:
- Heartbeat-stall trigger now gates on
alerts_detailcontainingalert_type: "heartbeat_timeout"instead ofheartbeat_ok=false(which reflects overseer→orchestrator connectivity, not agent stall state) persistent agent loop→agent-loopfor consistent anomaly type codes
The overseer prompt continues to align well: deterministic triggers gate on Tier-1 signals, qualitative judgment is preserved for ambiguous situations, and the "when in doubt: alert" scoping prevents synthesized false positives.
— Authored by egg
This comment has been minimized.
This comment has been minimized.
Contributor
There was a problem hiding this comment.
Re-review of d2c6ac7
Both issues from the previous review are properly addressed:
-
heartbeat_oksemantic mismatch — Fixed. The positive trigger now correctly referencesalerts_detailwithalert_type: "heartbeat_timeout", which is the actual Tier-1 signal for agent heartbeat stalls (confirmed atorchestrator/health_monitor.py:468-478). No more conflation with the overseer's own heartbeat send status. -
Inconsistent anomaly type — Fixed.
persistent agent loop→agent-loop.
The new reviewer-waiting-on-producer bullet (line 109) and _is_brc_idle reference (lines 177-218) are accurate per the current codebase.
New issue: "loop detection" contradicts the agent-loop exception (line 154) — blocking
Line 154 now reads:
When in doubt on a qualitative signal (error content, loop detection, misalignment, mediation): alert. [...] Exception: do not apply this to anomalies that have deterministic JSON triggers listed above —
agent-heartbeat-stall,stuck-phase-transition,orchestrator-consensus-silent,agent-loop. Those require the trigger condition [...] to actually hold. Do not fire them on a hunch.
"loop detection" appears as an example of a qualitative signal (→ alert when in doubt), while agent-loop appears in the exception list as a deterministic trigger (→ never fire on a hunch). These refer to the same thing. An LLM reading this gets contradictory instructions:
- "I see signs of looping but the Haiku classifier hasn't returned confidence > 0.8. This looks like a qualitative loop-detection signal → I should alert when in doubt."
- "But
agent-looprequires its trigger condition to hold → I must not fire on a hunch."
This is the same class of prompt-following failure that caused #2012 — an ambiguous instruction that gives the overseer a justification path to synthesize alerts the deterministic system hasn't flagged. The overseer could convince itself it's "alerting on a qualitative loop-detection signal" rather than "firing agent-loop," especially since the qualitative list says "loop detection" (prose) while the exception says agent-loop (code).
This also contradicts line 102's classify-first principle: "Never let the Sonnet tier act on raw data. Always classify first with Haiku, then decide with Sonnet."
Fix: Remove "loop detection" from the qualitative examples parenthetical. If the overseer suspects looping, the Haiku classifier handles it (line 151). There's no valid scenario where the overseer should alert about looping independent of the classifier — that's exactly what the two-tier architecture prevents.
Suggested replacement for the first sentence of line 154:
When in doubt on a qualitative signal (error content, agent misalignment, mediation need): alert.
— Authored by egg
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
…ontradiction with agent-loop exception
Contributor
There was a problem hiding this comment.
Agreed. Fixed in 92f06f8.
Removed "loop detection" from the qualitative signal examples parenthetical. Loop detection is entirely handled by the Haiku classifier (line 151) and belongs exclusively in the deterministic exception list as agent-loop. Keeping it in both lists gave the overseer a justification path to synthesize loop alerts outside the classifier — the same class of prompt-following failure that caused #2012.
The qualitative examples now read: (error content, agent misalignment, mediation need) — all genuinely qualitative signals with no deterministic JSON trigger counterpart.
— Authored by egg
Contributor
There was a problem hiding this comment.
No agent-mode design concerns in the delta.
The new commit (92f06f8) correctly removes "loop detection" from the qualitative signal examples, resolving the contradiction with agent-loop in the deterministic exception list. The qualitative examples now cleanly separate from the deterministic triggers — no remaining ambiguity that could give the overseer a justification path to synthesize alerts outside the two-tier architecture.
All three review rounds are now addressed:
heartbeat_oksemantic mismatch → gated onalerts_detailwithalert_type: "heartbeat_timeout"(d2c6ac7)- Inconsistent anomaly type →
persistent agent loop→agent-loop(d2c6ac7) - Loop detection contradiction → removed from qualitative examples (92f06f8)
— Authored by egg
This comment has been minimized.
This comment has been minimized.
Contributor
There was a problem hiding this comment.
Re-review of 92f06f8
The blocking issue from the previous review — "loop detection" appearing in both the qualitative signal examples and the deterministic exception list — is properly fixed.
What changed: The qualitative examples parenthetical now reads (error content, agent misalignment, mediation need). "loop detection" is removed, leaving loop handling entirely to the Haiku classifier's deterministic trigger (agent-loop, line 151). The remaining qualitative examples are all correct — none have deterministic JSON trigger counterparts in the escalation triggers section.
Verified: "agent misalignment" is safe as a qualitative example. Although alignment checking is a Haiku tier task (line 91), it has no codified deterministic trigger (no threshold/cycle-count rule like agent-loop has). It flows through the Sonnet decision tier as intended. No new contradiction.
No other issues.
— Authored by egg
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
…l: resolve conflict in overseer.md Resolved semantic conflict in sandbox/agent-config/rules/overseer.md where PR #2016 (Fix #2012, Tier-1 gating for heartbeat-stall alerts) and PR #2011 (Fix #2010, refine-phase false-positive calibration) both modified the heartbeat-stall trigger and "when in doubt" paragraph. Resolution combines both approaches: - Heartbeat stall: keeps PR's strict alerts_detail JSON gating as primary mechanism, adds main's working-window and tool-call counter-evidence - When in doubt: keeps PR's qualitative-signal scoping and deterministic- trigger exception, adds main's evidence-citation requirement
Contributor
Conflict Resolution SummaryResolved merge conflicts with
What conflictedPR #2011 (Fix #2010, refine-phase false-positive calibration) landed on
How it was resolvedBoth PRs are defense-in-depth against false heartbeat stall alerts. The changes are complementary, not contradictory:
Please reviewBoth sides were semantic changes to the same logic. The intent is the same (reduce false positives), and I believe the combined version preserves both contributions. Please verify:
— Authored by egg |
This comment has been minimized.
This comment has been minimized.
Contributor
There was a problem hiding this comment.
No agent-mode design concerns in the merge resolution delta.
The conflict resolution in 42e9c3b correctly combines this PR's Tier-1 alerts_detail gating with main's (#2011) phase-relative baselines and evidence-citation requirements. The merged heartbeat-stall trigger now has three independent guards (alerts_detail entry, working-window floor, tool-call counter-evidence) — all complementary, no contradictions. The phase-relative baselines section provides orienting context that helps the overseer calibrate without constraining its judgment on genuinely ambiguous signals.
All three review rounds from prior commits remain properly addressed.
— Authored by egg
This comment has been minimized.
This comment has been minimized.
Contributor
There was a problem hiding this comment.
Re-review of 42e9c3b (merge of main)
The new commit since my last review is a merge of origin/main to resolve conflicts with PR #2011 (Fix #2010, refine-phase false-positive calibration). Both PRs modified the heartbeat-stall trigger and "when in doubt" paragraph in sandbox/agent-config/rules/overseer.md.
Previous feedback verification
All three issues from prior review rounds remain properly addressed in the final state:
heartbeat_oksemantic mismatch (d2c6ac7) — The trigger gates onalerts_detailwithalert_type: "heartbeat_timeout", not the overseer's ownheartbeat_oksend-status. Confirmed atorchestrator/health_monitor.py:473.- Inconsistent anomaly type (d2c6ac7) — Exception list uses
agent-loopconsistently, not prose. - "loop detection" contradiction (92f06f8) — Removed from qualitative signal examples. Qualitative list now reads
(error content, agent misalignment, mediation need)— no overlap with deterministic triggers.
Merge conflict resolution
The resolution correctly combines both PRs' defense-in-depth contributions:
Heartbeat stall trigger — Keeps this PR's strict alerts_detail JSON gating as the primary mechanism. Adds #2011's working-window floor and tool-call counter-evidence. The three guards are conjunctive:
alerts_detailentry withalert_type: "heartbeat_timeout"present (this PR)- Phase-specific minimum working window elapsed (from #2011's baselines)
- Tool-call activity weighed as counter-evidence (from #2011)
This is strictly more conservative than either PR alone — correct for the "reduce false positives" goal.
"When in doubt" paragraph — Keeps this PR's qualitative-signal scoping and deterministic-trigger exception list. Adds #2011's evidence-citation requirement. The two additions are orthogonal: the evidence requirement strengthens all alerts, while the exception list carves out deterministic triggers. No contradiction.
Phase-relative baselines section (entirely from #2011) — Correctly integrated. The baselines' own agent-heartbeat-stall rule ("emit only when (a) working-window floor elapsed AND (b) orchestrator raised health alert") is consistent with the trigger bullet's "Fire only when alerts_detail has heartbeat_timeout and working window elapsed."
Stuck phase transition — Gained #2011's additional guards (consensus.state != "confirmed" exclusion, empty-contract-during-refine exclusion). Compatible with this PR — this PR doesn't touch the stuck-phase-transition logic.
Cross-reference verification
_is_brc_idleatorchestrator/health_monitor.py:177-218: confirmed present and matches the described behavior (reviewer-only role suppression while producers are WORKING, plus post-propose grace period).alert_type: "heartbeat_timeout"atorchestrator/health_monitor.py:473: confirmed — this is the Tier-1 signal the heartbeat stall trigger now gates on.
No new issues found. The merge resolution is clean.
— Authored by egg
Contributor
|
egg review completed. View run logs 11 previous review(s) hidden. |
james-in-a-box Bot
pushed a commit
that referenced
this pull request
Apr 25, 2026
#2016) * Fix #2012: gate agent-heartbeat-stall overseer alerts on Tier-1 signal The overseer was firing agent-heartbeat-stall [low] on every refine phase with a self-contradicting body ("heartbeat_ok=true and alerts=0 (system threshold not yet crossed)", "No action required at this time"). That text is fully LLM-composed — nothing in the codebase produces it. Root cause: the overseer prompt's trigger for agent-heartbeat-stall was described as two conjoined conditions in prose, and the catch-all "when in doubt: alert" nudged the agent to fire anyway when it saw quiet roles. Tier-1 (_is_brc_idle in health_monitor.py) was already correctly suppressing these — the synthetic alert is purely a prompt-following failure. Three prompt changes in sandbox/agent-config/rules/overseer.md: 1. Harden the heartbeat-stall trigger into an explicit JSON gate: fire only when heartbeat_ok=false or a heartbeat-related alert is present; never synthesize from observed silence. 2. Add "reviewer waiting on a producer" as a recognized legitimate silence category under Adaptive Stall Detection, mirroring what Tier-1's _is_brc_idle already enforces. 3. Scope "when in doubt: alert" to qualitative signals only (error content, loop, misalignment). Anomalies with deterministic JSON triggers must not be fired on a hunch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Address review: fix heartbeat_ok semantic mismatch, use consistent anomaly type codes * Remove 'loop detection' from qualitative signal examples to resolve contradiction with agent-loop exception --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: egg-reviewer[bot] <261018737+egg-reviewer[bot]@users.noreply.github.com>
james-in-a-box Bot
pushed a commit
that referenced
this pull request
Apr 25, 2026
Risk assessment for the overseer escalation/auto-issue/host-migration work (issue-1962, advisor-strategy framing). Key findings: - R-COMPAT-01 (verified): vendored claude-agent-sdk 0.1.65 does NOT expose advisor_20260301 / max_uses. Decision-23 Option A requires an SDK upgrade; default to Option B (two-call pattern). - R-COST-01: decision-19 deferred the advisor budget cap; recommend shipping a defensive per-phase soft cap (max_uses_per_phase=5) to bound runaway-pipeline cost. - R-COMPAT-03: PR #2011/#2016 calibration is recent and easy to regress; recommend feature-flagging host-migration so /sdlc and overseer can run side-by-side for calibration. - R-OP-02: dedup signature is unspecified — recommend (anomaly_type + agent_role + repo), explicitly excluding pipeline_id. - R-SEC-01/02: gateway must enforce label injection, size caps, repo pinning, and secret scrubbing on gh issue create from overseer. 5 human-review areas flagged. Rollback plan based on layered feature flags so each thread (advisor / auto-issue / host-migration) can be disabled at runtime without revert. Authored-by: egg
james-in-a-box Bot
pushed a commit
that referenced
this pull request
Apr 25, 2026
… Opus model pin Fresh risk assessment for the architect's revised analysis (decision-9 opt-1 sandbox-side gh; JSONL filed-issues; /sdlc overseer-absent fallback as concrete deliverable; pinned advisor model; no 'off' mode). Key findings vs the v1 risk pass: - NEW BLOCKER R-COMPAT-01: architect's overseer_advisor_model pin 'claude-opus-4-20250514' is the OLDER Opus 4 generation, NOT the 'claude-opus-4-6' the egg codebase canonicalizes (shared/egg_harness/ config.py:17) AND not the only model the public advisor tool currently supports. shared/egg_harness/cost.py has no row for that ID so cost telemetry will fail. Recommend changing default to claude-opus-4-6. - NEW R-COMPAT-04: EGG_PIPELINE_REPO env var the gateway --repo restriction depends on does NOT exist anywhere today (verified by grep across sandbox/, orchestrator/, gateway/, shared/). Implement phase must add to kubernetes_spawner.py + sandbox/entrypoint.py; gateway must default-DENY when unset. - NEW R-COMPAT-05: architect dropped 'off' value from overseer_auto_file_issues_mode per reviewer NACK; only escape is overseer_enabled=False which kills three features. Recommend re-add 'off' value or sibling bool. - NEW R-COMPAT-08/09: cross-cutting import path concerns for the template literal + infra-error helper extraction. - NEW R-OP-06: /sdlc overseer-absent fallback "fires AT MOST ONCE per phase" needs explicit per-phase memory mechanism. - NEW R-PERF-02: filed-issues.jsonl periodic compaction needs flock to avoid lost records on concurrent writes. - NEW R-COMPAT-11: PipelineConfig delivery via status endpoint is unverified -- the new threshold knobs may be inert without endpoint changes. Carry-over (still relevant from v1, updated for v2 architect): - R-COST-01 advisor budget unbounded (decision-19 deferred) - R-OP-01 shadow-mode HITL noise overwhelming operator - R-OP-02 dedup_signature shape ambiguous (architect proposes pipeline_id+phase; risk pass recommends excluding both) - R-OP-04 host-migration regression vs PR #2011/#2016 calibration - R-SEC-01 secret leakage in public issue body - R-SEC-02 gateway label injection must STRIP user labels first - R-SEC-04 concurrent-write races on .egg-state/oversight/ Total: 25 risks across 6 categories. 8 require human review and are surfaced in human_review_areas with concrete questions. SDK spike confirmed: claude-agent-sdk 0.1.65 ServerToolName Literal includes 'advisor' for response parsing only; no client-side tool definition support, no max_uses, no advisor-tool-2026-03-01 beta header. ClaudeAgentOptions.betas accepts only Literal['context-1m-2025-08-07']. Option B (two-call pattern) is correct choice; HR-02 recommends verifying claude-agent-sdk 0.1.66 (one patch ahead, fits inside the existing >=0.1.65,<0.2 pin) before committing. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
jwbron
pushed a commit
that referenced
this pull request
Apr 25, 2026
Risk assessment for the overseer escalation/auto-issue/host-migration work (issue-1962, advisor-strategy framing). Key findings: - R-COMPAT-01 (verified): vendored claude-agent-sdk 0.1.65 does NOT expose advisor_20260301 / max_uses. Decision-23 Option A requires an SDK upgrade; default to Option B (two-call pattern). - R-COST-01: decision-19 deferred the advisor budget cap; recommend shipping a defensive per-phase soft cap (max_uses_per_phase=5) to bound runaway-pipeline cost. - R-COMPAT-03: PR #2011/#2016 calibration is recent and easy to regress; recommend feature-flagging host-migration so /sdlc and overseer can run side-by-side for calibration. - R-OP-02: dedup signature is unspecified — recommend (anomaly_type + agent_role + repo), explicitly excluding pipeline_id. - R-SEC-01/02: gateway must enforce label injection, size caps, repo pinning, and secret scrubbing on gh issue create from overseer. 5 human-review areas flagged. Rollback plan based on layered feature flags so each thread (advisor / auto-issue / host-migration) can be disabled at runtime without revert. Authored-by: egg
jwbron
pushed a commit
that referenced
this pull request
Apr 25, 2026
… Opus model pin Fresh risk assessment for the architect's revised analysis (decision-9 opt-1 sandbox-side gh; JSONL filed-issues; /sdlc overseer-absent fallback as concrete deliverable; pinned advisor model; no 'off' mode). Key findings vs the v1 risk pass: - NEW BLOCKER R-COMPAT-01: architect's overseer_advisor_model pin 'claude-opus-4-20250514' is the OLDER Opus 4 generation, NOT the 'claude-opus-4-6' the egg codebase canonicalizes (shared/egg_harness/ config.py:17) AND not the only model the public advisor tool currently supports. shared/egg_harness/cost.py has no row for that ID so cost telemetry will fail. Recommend changing default to claude-opus-4-6. - NEW R-COMPAT-04: EGG_PIPELINE_REPO env var the gateway --repo restriction depends on does NOT exist anywhere today (verified by grep across sandbox/, orchestrator/, gateway/, shared/). Implement phase must add to kubernetes_spawner.py + sandbox/entrypoint.py; gateway must default-DENY when unset. - NEW R-COMPAT-05: architect dropped 'off' value from overseer_auto_file_issues_mode per reviewer NACK; only escape is overseer_enabled=False which kills three features. Recommend re-add 'off' value or sibling bool. - NEW R-COMPAT-08/09: cross-cutting import path concerns for the template literal + infra-error helper extraction. - NEW R-OP-06: /sdlc overseer-absent fallback "fires AT MOST ONCE per phase" needs explicit per-phase memory mechanism. - NEW R-PERF-02: filed-issues.jsonl periodic compaction needs flock to avoid lost records on concurrent writes. - NEW R-COMPAT-11: PipelineConfig delivery via status endpoint is unverified -- the new threshold knobs may be inert without endpoint changes. Carry-over (still relevant from v1, updated for v2 architect): - R-COST-01 advisor budget unbounded (decision-19 deferred) - R-OP-01 shadow-mode HITL noise overwhelming operator - R-OP-02 dedup_signature shape ambiguous (architect proposes pipeline_id+phase; risk pass recommends excluding both) - R-OP-04 host-migration regression vs PR #2011/#2016 calibration - R-SEC-01 secret leakage in public issue body - R-SEC-02 gateway label injection must STRIP user labels first - R-SEC-04 concurrent-write races on .egg-state/oversight/ Total: 25 risks across 6 categories. 8 require human review and are surfaced in human_review_areas with concrete questions. SDK spike confirmed: claude-agent-sdk 0.1.65 ServerToolName Literal includes 'advisor' for response parsing only; no client-side tool definition support, no max_uses, no advisor-tool-2026-03-01 beta header. ClaudeAgentOptions.betas accepts only Literal['context-1m-2025-08-07']. Option B (two-call pattern) is correct choice; HR-02 recommends verifying claude-agent-sdk 0.1.66 (one patch ahead, fits inside the existing >=0.1.65,<0.2 pin) before committing. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
jwbron
added a commit
that referenced
this pull request
Apr 26, 2026
* Initialize SDLC contract for issue #1962
* refine: analysis for #1962 (overseer escalation/auto-issue) [refiner]
Surfaces three threads in the issue — escalation tuning, autonomous
issue filing, and host→overseer migration — and recommends Option B
(escalation tuning + auto-issue in this pipeline, host migration as
a follow-up).
Key finding: `orchestrator/overseer/issue_filer.py::file_diagnostic_issue`
and the full OverseerMonitor decision ladder already exist but are
dead code — only referenced in tests. The deployed overseer is the
sandbox LLM agent under `sandbox/agent-config/rules/overseer.md`,
which explicitly forbids filing issues. Auto-issue is therefore a
wiring problem, not a greenfield capability.
Registered 16 choice decisions and 7 open-ended feedback questions
covering scope, dedup, labels, rollout, thresholds, and cross-issue
coordination (#1722, #1727, #1786, #1806, #1902).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refine: address reviewer_refine NACK on #1962 analysis [refiner]
Blocking fixes:
1. Insert inline `<!-- egg-hitl-decision id=decision-N -->` markers
for every registered decision (1..16). Contract had the decisions
but draft was missing the per-question markers; fixed by
restructuring "Open Questions" to reproduce each decision inline
with its registered options plus recommended-option tags.
2. Remove the duplicate decision-15 cross-reference (it lived in
both auto-issue-filing and interaction-with-existing-issues
sections). Single authoritative block now.
3. Resolve decision-11 vs decision-1 redundancy: explicitly tag
decision-11 as conditional on decision-1 ≠ Option B/D, with a
dependency note inside the decision body.
Non-blocking improvements:
- Annotate decision-9 opt-4 ("reuse dead-code OverseerMonitor")
with explicit "not recommended" caveat (re-introduces non-agent
decision pipeline).
- Clarify decision-4 "Sonnet-gated" means in-loop reasoning, not a
separate orchestrator-side classifier service.
- Cross-reference #1932 (closed) alongside #1971 in the interacting-
issues list.
- Add Tests/regression risk subsection to Complexity Assessment
(test_overseer_*.py, test_overseer_issue_filer.py, gateway tests,
integration_tests gap).
- Tighten SKILL.md line citation to 1359-1383 (stall + NACK block).
- Tag plan-phase-candidate decisions (12, 13, 14, 16, feedback-1
Q5/Q6/Q7) so the human can leave them unanswered at the refine
gate.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Persist statefiles after refine phase
* refine(#1962): rewrite analysis under advisor-strategy framing
Reframe the overseer escalation/issue-filing/host-migration work
under Anthropic's advisor strategy (Haiku executor, Opus advisor).
Carry forward pre-refine resolved preferences (scope, labels, dedup,
sub-agent scope, related-bugs scope) as constraints. Surface the
remaining load-bearing HITL items as new decisions (17-23) covering
advisor calibration, budget, prompt contract, gate placement,
host-migration sequencing, label confirmation, and
native-tool-vs-two-call implementation choice.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refine(#1962): address reviewer_refine NACK
Address all 4 blocking findings from reviewer_refine NACK 93cf8334-699a-46:
1. Decisions 17-23 verified present on contract via mcp__sdlc__show_contract
(the reviewer's snapshot was stale; re-verification before this re-propose
confirms all 7 are registered).
2. Add `<!-- egg-hitl-decision id=decision-N -->` markers above every open
decision (carry-overs 6, 8, 9, 10, 12-16; new 17-23). Add
`<!-- egg-hitl-feedback id=feedback-1.QN -->` markers for Q1-Q7.
Reproduce each decision inline with question + options + (Recommended) tags.
3. Fix off-by-one: 7 new decisions (17-23), not 6. Status table, mapping table,
prose, and registration calls now agree.
4. Untangle decision-10 vs decision-22: rollout (decision-10) and host-migration
sequencing (decision-22) are distinct questions; remove "folded into" claim;
mapping table cites them separately.
Non-blocking nits also addressed:
- egg-orch CLI citation: 2549-2597 (subparser at 2553, alert parser at 2556)
- file_diagnostic_issue caller: monitor.py:624 (inside the dead class itself)
- #1902 cross-ref: concrete — opt-2 needs zero file-boundary work
- Authored-by trailer moved to end-of-file
- pip show claude-agent-sdk breadcrumb added for SDK spike
- BrowseComp claims cited to advisor-strategy blog post URL
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Persist statefiles after refine phase
* Persist HITL resolution after refine phase gate
* plan(#1962): single-PR plan for advisor-gated overseer escalation, auto-issue filing, host migration
Decomposes the refine analysis into 8 phases / 18 tasks landing in one
PR per decision-22:
Phase 1: SDK spike + config knobs + state schemas
Phase 2: Gateway + egg-orch overseer file-issue CLI verb
Phase 3: Issue body template revival + dedup
Phase 4: Advisor wiring (two-call pattern, Option B per spike)
Phase 5: Rule-doc rewrite (lift the issue-filing prohibition)
Phase 6: Host -> overseer migration of /sdlc detectors
Phase 7: Tests (orchestrator + gateway + integration + skill regression)
Phase 8: Docs
Includes the SDK capability-spike result: claude-agent-sdk 0.1.65 does
not expose advisor_20260301 / max_uses, so the implement phase will
ship Option B (two-call advisor pattern) with Option A unlocking as a
clean follow-up swap if the SDK upgrades within the >=0.1.65,<0.2 pin.
Honors all 23 resolved refine decisions and the seven feedback-1
answers (Tier-1 intersection gate, shadow-mode rollout, agent:overseer
+ priority labels only, .egg-state/oversight/filed-issues.json dedup,
180s stuck-phase-transition default, no per-pipeline issue cap).
* plan(#1962): architect analysis -- advisor strategy + auto-issue + host migration
Deliver the plan-phase architect output for issue #1962 covering all three
in-scope threads (escalation tuning, autonomous issue filing, host->overseer
migration) under the advisor-strategy framing locked in during refine.
Resolves all 23 HITL decisions to a concrete implementation site:
- decision-23 SDK capability spike: claude-agent-sdk 0.1.65 does NOT expose
advisor_20260301 / max_uses; Option C resolves deterministically to
Option B (two-call advisor pattern)
- decision-18 intersection gate: advisor invoked iff Haiku confidence > 0.8
AND >= 1 Tier-1 health alert present (#2012 generalization)
- decision-21 + decision-10: advisor recommends, human gates via existing
pending_decisions HITL surface; shadow-mode rollout default
- decision-22: single PR with all three threads
- decision-12: PipelineConfig-driven thresholds; bump
stuck-phase-transition default to 180s per feedback-1.Q2
Output written to .egg-state/agent-outputs/1962-architect-output.json
covering: 11 components, 12 technical decisions, 7 regression risks for
the risk_analyst, test strategy outline (unit + integration + regression),
and an explicit decision-to-resolution mapping covering all 23 HITL items.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* plan(#1962): risk_analyst output — 19 risks, 5 HR areas
Risk assessment for the overseer escalation/auto-issue/host-migration
work (issue-1962, advisor-strategy framing). Key findings:
- R-COMPAT-01 (verified): vendored claude-agent-sdk 0.1.65 does NOT
expose advisor_20260301 / max_uses. Decision-23 Option A requires
an SDK upgrade; default to Option B (two-call pattern).
- R-COST-01: decision-19 deferred the advisor budget cap; recommend
shipping a defensive per-phase soft cap (max_uses_per_phase=5) to
bound runaway-pipeline cost.
- R-COMPAT-03: PR #2011/#2016 calibration is recent and easy to
regress; recommend feature-flagging host-migration so /sdlc and
overseer can run side-by-side for calibration.
- R-OP-02: dedup signature is unspecified — recommend
(anomaly_type + agent_role + repo), explicitly excluding pipeline_id.
- R-SEC-01/02: gateway must enforce label injection, size caps,
repo pinning, and secret scrubbing on gh issue create from overseer.
5 human-review areas flagged. Rollback plan based on layered feature
flags so each thread (advisor / auto-issue / host-migration) can be
disabled at runtime without revert.
Authored-by: egg
* plan(#1962): address reviewer_plan NACK -- decision-9 opt-1, JSONL, fallback, mode
Resolve all 4 blocking + 7 non-blocking issues raised by reviewer_plan:
Blocking:
1. decision-9 opt-1 fix: agent-side CLI runs `gh issue create` ITSELF
inside the sandbox via the gateway; no orchestrator-side endpoint
invokes gh. New sandbox/egg_lib/overseer_issue_body.py helper for
body-building. orchestrator/overseer/issue_filer.py kept ONLY as
the canonical template literal source (marked DEAD).
2. filed-issues schema: switched to JSON Lines (.jsonl) per the
append-only semantics; agent-timing.json stays single-object.
Header line `{_kind: "header", schema_version: 1}` for format
detection without a sidecar.
3. /sdlc overseer-absent fallback promoted from risk-mitigation prose
to a concrete component_breakdown deliverable with explicit trigger
conditions.
4. Dropped 'off' mode from overseer_auto_file_issues_mode Literal
(decision-10 sanctioned shadow/live only; full disable uses
existing overseer_enabled=False).
Non-blocking:
- Infra-error fast-path lives in NEW shared/egg_overseer_helpers/
infra_error.py (single source of truth; agent + dead orchestrator
code both import).
- Gateway allow-rule wording corrected: gh issue create is blocked
by default-deny pattern (#1494), not _OVERSEER_BLOCKED_GH_OPS.
- EGG_PIPELINE_REPO env injection sized as a verify-and-inject
component for the planner.
- PipelineConfig delivery via /status sized as a verify component.
- overseer_advisor_model PINNED to claude-opus-4-20250514.
- Token-budget framed as architect estimate; risk_analyst sizes.
- Test pruning footprint enumerated in regression risks; sized via
pytest case count instructions for the planner.
- open_questions resolved by architect (mark-unused, no new endpoint,
pinned opus); section renamed architect_resolved_questions.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* plan(#1962): address reviewer_plan NACK -- decision-9 opt-1, MCP tool, scrubbing
Resolve all 5 BLOCKING + 11 NON-BLOCKING items raised by reviewer_plan
on the original 8-phase plan.
Blocking fixes:
1. decision-9 opt-1 architectural fix: agent CLI in TASK-2-1 runs
`gh issue create` ITSELF inside the sandbox via the gateway. The
orchestrator REST endpoint that previously composed and ran gh is
DROPPED. Body composition lives sandbox-side in
sandbox/egg_lib/overseer_issue_body.py (TASK-3-1); orchestrator/
overseer/issue_filer.py is preserved as the canonical template
literal source only and marked DEAD.
2. Advisor module relocated from orchestrator/overseer/advisor.py to
shared/overseer/advisor.py (TASK-1-5 + TASK-4-1) so it is
importable from sandbox without crossing package boundaries.
3. TASK-4-2 commits explicitly to the MCP-tool invocation path
(mcp__overseer__consult_advisor) -- no subprocess fallback. Tool
surface, schema, and auth-gating to overseer role are spec'd.
4. New TASK-3-2 adds shared/egg_overseer_helpers/scrubbing.py with
patterns for ghp_*/ghs_*/gho_*/ghu_*/ghr_* PATs, AKIA* AWS keys,
Slack webhooks, and GITHUB_TOKEN=*/GH_TOKEN=*/ANTHROPIC_API_KEY=*
exports. Advisor scrubs at return time; gateway scrubs as
defense-in-depth.
5. compute_anomaly_signature in TASK-1-3 now uses
(anomaly_type, agent_role, repo) -- repo from EGG_PIPELINE_REPO
per decision-5 + R-OP-02. error_class field is dropped.
Non-blocking fixes:
- AgentTimingEntry adds last_alerted_at + alerted_anomalies fields
for per-anomaly suppression in TASK-6-1 detectors.
- Title format becomes
`[Pipeline Diagnostic] {anomaly_type} - {agent_role}
[{anomaly_signature[:8]}]` so gh issue list --search is reliable.
- Body composition: advisor populates issue_title + issue_body
(option a); CLI passes them through.
- overseer_auto_file_issues becomes a Literal["shadow","live"] mode
(drops 'off'); HITL flow runs in both modes -- mode only controls
whether gh is called once approval lands.
- New overseer_owns_host_detection: bool = False knob keeps /sdlc's
host-side detectors live during a calibration window; Phase 6
gates the deletions on this flag (mitigates R-COMPAT-03 / HR-03).
- TASK-3-3 (alert schema change) precedes TASK-4-2 (advisor wiring);
Phase 3 precedes Phase 2 since CLI imports from issue_body helper
and find_existing_issue.
- New TASK-7-5 metric instrumentation task per R-OP-05 emits
structured "overseer_event" log lines at four sites.
- Backwards-compat regression test added in TASK-7-1 for legacy
OVERSEER_ALERT payloads (R-COMPAT-05).
- TASK-7-6 explicitly asserts title contains signature substring.
- New TASK-1-4 verifies-and-injects EGG_PIPELINE_REPO env var.
- Gateway line numbers cited: agent_restrictions.py:153 (blocked
ops list) and :193-218 (check_agent_gh_operation). Notes that
`issue create` is not in the blocked list -- the change is
adding an allow-rule, not removing from a deny-list.
- /sdlc overseer-absent fallback promoted to a concrete component
with a single AskUserQuestion (4 options).
- Phase 5 rule doc has verbatim phrases for grep-based acceptance:
"Dedup before recommend", "Flag-vs-HITL gating", "Secret scrubbing".
Total tasks: 23 across 8 phases (was 14 across 8). All tasks have
role + files + acceptance criteria.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* plan(#1962): risk_analyst v2 -- 25 risks, 8 HR areas, ship-blocker on Opus model pin
Fresh risk assessment for the architect's revised analysis (decision-9
opt-1 sandbox-side gh; JSONL filed-issues; /sdlc overseer-absent
fallback as concrete deliverable; pinned advisor model; no 'off' mode).
Key findings vs the v1 risk pass:
- NEW BLOCKER R-COMPAT-01: architect's overseer_advisor_model pin
'claude-opus-4-20250514' is the OLDER Opus 4 generation, NOT the
'claude-opus-4-6' the egg codebase canonicalizes (shared/egg_harness/
config.py:17) AND not the only model the public advisor tool currently
supports. shared/egg_harness/cost.py has no row for that ID so cost
telemetry will fail. Recommend changing default to claude-opus-4-6.
- NEW R-COMPAT-04: EGG_PIPELINE_REPO env var the gateway --repo
restriction depends on does NOT exist anywhere today (verified by
grep across sandbox/, orchestrator/, gateway/, shared/). Implement
phase must add to kubernetes_spawner.py + sandbox/entrypoint.py;
gateway must default-DENY when unset.
- NEW R-COMPAT-05: architect dropped 'off' value from
overseer_auto_file_issues_mode per reviewer NACK; only escape is
overseer_enabled=False which kills three features. Recommend re-add
'off' value or sibling bool.
- NEW R-COMPAT-08/09: cross-cutting import path concerns for the
template literal + infra-error helper extraction.
- NEW R-OP-06: /sdlc overseer-absent fallback "fires AT MOST ONCE per
phase" needs explicit per-phase memory mechanism.
- NEW R-PERF-02: filed-issues.jsonl periodic compaction needs flock to
avoid lost records on concurrent writes.
- NEW R-COMPAT-11: PipelineConfig delivery via status endpoint is
unverified -- the new threshold knobs may be inert without endpoint
changes.
Carry-over (still relevant from v1, updated for v2 architect):
- R-COST-01 advisor budget unbounded (decision-19 deferred)
- R-OP-01 shadow-mode HITL noise overwhelming operator
- R-OP-02 dedup_signature shape ambiguous (architect proposes
pipeline_id+phase; risk pass recommends excluding both)
- R-OP-04 host-migration regression vs PR #2011/#2016 calibration
- R-SEC-01 secret leakage in public issue body
- R-SEC-02 gateway label injection must STRIP user labels first
- R-SEC-04 concurrent-write races on .egg-state/oversight/
Total: 25 risks across 6 categories. 8 require human review and are
surfaced in human_review_areas with concrete questions.
SDK spike confirmed: claude-agent-sdk 0.1.65 ServerToolName Literal
includes 'advisor' for response parsing only; no client-side tool
definition support, no max_uses, no advisor-tool-2026-03-01 beta header.
ClaudeAgentOptions.betas accepts only Literal['context-1m-2025-08-07'].
Option B (two-call pattern) is correct choice; HR-02 recommends
verifying claude-agent-sdk 0.1.66 (one patch ahead, fits inside the
existing >=0.1.65,<0.2 pin) before committing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* plan(#1962): address reviewer_plan NACK v2 -- drop advisor cap, fix paths, priority mapping
Resolve all 6 BLOCKING + 10 NON-BLOCKING items raised on plan v2.
Blocking fixes:
1. Dropped overseer_advisor_max_uses_per_phase entirely from TASK-1-2
and TASK-4-2 per decision-19 ("no cap for now -- handle budget
separately"). Acceptance criteria now ASSERT the knob's absence
in PipelineConfig and grep for max_uses_per_phase in production
code returns no matches (regression guard).
2. TASK-3-3 file path corrected from non-existent
orchestrator/routes/overseer_alert.py to the real
orchestrator/routes/pipelines.py (where OVERSEER_ALERT handling
lives at lines 238, 461-473, 5791). Description now instructs
the implementer to find the schema model by exploration first
and cite the path in the commit message.
3. Dropped `_kind: Literal["record"]` from FiledIssueRecord (Pydantic
v2 strips underscore-prefixed names from JSON output). Header line
is now a hand-written dict literal in append_filed_issue, not a
Pydantic dump. TASK-7-3 round-trip assertion verifies no `kind`
key leaks into serialized records.
4. Added explicit priority-dimension mapping in TASK-1-2 +
shared/overseer/priority.py (TASK-1-5): low|medium|high <->
p3|p2|p1; p0 reserved for opt-in human escalation. Two helpers
alert_to_label / label_to_alert. TASK-4-2's decision="alert"
branch calls label_to_alert before invoking egg-orch overseer
alert.
5. TASK-1-4 dropped the silent gh-repo-view fallback. EGG_PIPELINE_REPO
now MUST be injected by the orchestrator; sandbox/entrypoint.py
raises EnvironmentError if missing. test_entrypoint.py extended
with a fail-fast acceptance.
6. Cross-phase dedup persistence honesty: new "Dedup persistence
scope" section in Approach states the local JSONL cache is
intra-phase only; cross-phase dedup relies on title-embedded
8-char anomaly-signature + gh issue list --search.
Non-blocking fixes:
- Stall threshold split into overseer_stuck_phase_transition_seconds
+ overseer_agent_stall_seconds (both default 180s) to remove the
double-duty ambiguity flagged by the reviewer.
- TASK-2-2 reframed: gh issue create from overseer is ALREADY allowed
by the gateway (issue create is not in _OVERSEER_BLOCKED_GH_OPS).
The change is adding additional guardrails (label injection, repo
enforcement, body-size, secret scan), not flipping a deny->allow.
Adds note to audit gateway/gateway.py for parallel-policy layers.
- TASK-4-2 cap-exceeded path is moot (no cap shipped); the prior
STATUS-log-only path is gone.
- TASK-6-2 reframed in terms of stable section headings (### Stall
Detection, ### Silent-Agent Detection, etc.) instead of absolute
line numbers that drift mid-PR.
- New "Implementation order: 1 -> 3 -> 2 -> 4 -> 5 -> 6 -> 7 -> 8"
callout in Approach so the implementer reading top-to-bottom
doesn't hit Phase 2's imports before Phase 3 defines them.
- TASK-7-1 dead-code audit extended to grep for file_diagnostic_issue,
_build_issue_body, DIAGNOSTIC_LABELS, AND OverseerMonitor across
orchestrator/sandbox/gateway/shared. Stale imports keep the dead
module alive accidentally.
- Consolidated the previously-split shared/overseer/ +
shared/egg_overseer_helpers/ packages into a single
shared/overseer/ package (advisor.py, scrubbing.py, infra_error.py,
priority.py, state.py).
- TASK-7-6 explicit acceptance: integration test uses tmp_path
fixture for .egg-state/oversight/ so cycle-2 dedup actually reads
the JSONL written by cycle-1 (not a mock).
- TASK-7-7 fallback test explicitly fires only under
overseer_owns_host_detection=True (the only state where the host
has no detectors active).
- New manual_steps: file the metrics-instrumentation follow-up
for feedback-1.Q6.(d) issue-acceptance-rate; file the
advisor-budget follow-up per decision-19.
- Pre-merge manual step to formally close decision-15 with the no-op
rationale (OVERSEER_PATTERNS already permits .egg-state/oversight/).
23 tasks across 8 phases unchanged. YAML parses cleanly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* plan(#1962): address reviewer_plan NACK v3 -- claude-opus-4-6 + 8 non-blocking pull-forwards
Resolve the 1 BLOCKING + 8 NON-BLOCKING items raised on v3.
Blocking:
- TASK-1-2: overseer_advisor_model changed from claude-opus-4-20250514
to claude-opus-4-6, the canonical opus alias defined at
shared/egg_harness/config.py:17 and the only Opus-class entry with
a cost-data row at shared/egg_harness/cost.py:18. The earlier ID
would have silently broken the max_llm_cost_per_hour=$5 self-monitor
because cost.lookup() returns no row. Cost-row regression assertion
added to TASK-1-2 acceptance: assert
egg_harness.cost.lookup(pc.overseer_advisor_model) is not None.
Non-blocking pull-forwards (all 8 risk_analyst items pulled forward
to keep implement-phase surprises down):
1. TASK-1-3: load_agent_timing / save_agent_timing helpers MUST
acquire fcntl.LOCK_EX on .egg-state/oversight/agent-timing.lock
for the read-modify-write critical section (R-PERF-02 / R-SEC-04
mitigation #1). Concurrent overseer respawns at phase boundaries
can race on the read step without this.
2. TASK-2-1: gh issue create now uses --json url,number,title and
parses json.loads(stdout)["number"] instead of regex over URL
suffix (R-COMPAT-10).
3. TASK-3-1: extracted single-source-of-truth template into
shared/overseer/issue_template.py with TEMPLATE_LITERAL constant
and render(**fields) function. Both the dead orchestrator path
and the sandbox helper import from it; canonical-byte-equality
test guards drift (R-COMPAT-08).
4. TASK-1-3: compute_anomaly_signature gets a fourth input
tier1_alert_types: tuple[str, ...] = () (sorted, default empty)
so two anomalies sharing (anomaly_type, agent_role, repo) but
triggered by different Tier-1 alerts don't collapse to the same
signature (HR-06 default).
5. TASK-3-3: OVERSEER_ALERT gets explicit schema_version: int = 2
field. /sdlc parsing in TASK-6-2 reads the version (defaulting
to 1 if absent) and falls back gracefully (R-COMPAT-06).
6. TASK-7-8 (NEW): sandbox/tests/test_shared_overseer_imports.py
smoke-tests every new shared/overseer/ module imports inside
the sandbox container; runs the imports inside the built image
when Docker is available (R-COMPAT-09).
7. TASK-1-3: FiledIssueRecord gets hitl_outcome: Literal["filed",
"skipped", "modified_and_filed"] | None = None field. issue_number
is now Optional (None when hitl_outcome=="skipped"). Prevents
re-prompting on the same anomaly after overseer respawn (R-OP-03).
8. TASK-6-2: /sdlc overseer-absent fallback writes a sentinel file
.egg-state/oversight/sdlc-fallback-fired-{pipeline}-{phase}.flag
on first emit; pre-emit check skips AskUserQuestion if sentinel
exists. TASK-7-7 simulates two consecutive ticks and asserts
AskUserQuestion fires exactly once (R-OP-06).
Total tasks: 24 across 8 phases (was 23). YAML parses cleanly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Persist statefiles after plan phase
* Persist HITL resolution after plan phase gate
* docs(#1962): document advisor gate, auto-issue filing, host migration
Adds documentation for the planned overseer changes in issue #1962:
- pipeline-health-monitoring.md: new "Advisor Gate", "Auto-Issue Filing
(Shadow vs Live)", "Host Detector Migration", and "MCP Advisor Tool"
sections; six new PipelineConfig knobs in the configuration table
(overseer_advisor_model, overseer_auto_file_issues_mode,
overseer_owns_host_detection, overseer_stuck_phase_transition_seconds,
overseer_agent_stall_seconds, overseer_silent_agent_threshold_seconds,
overseer_long_running_phase_seconds, overseer_nack_unresolved_seconds);
diagnostic body template extended with the Pipeline Links sub-block;
EGG_PIPELINE_REPO env-var contract documented.
- reference/agent-roles.md: overseer entry updated for the new
egg-orch overseer file-issue capability, the two new state files
(.egg-state/oversight/filed-issues.jsonl, agent-timing.json), the
EGG_PIPELINE_REPO requirement, the gateway constraints on
gh issue create, and the OVERSEER_ALERT schema_version=2 fields.
- architecture/orchestrator.md: added a host-to-overseer migration
paragraph covering the calibration-window flag, advisor strategy,
and the new MCP advisor tool, with cross-links to the guide.
These updates anchor on the planner artifact at
.egg-state/drafts/1962-plan.md (TASK-8-1) and will be tightened against
the coder/tester output as it lands.
Refs #1962
* implement(#1962) Phase 1: foundation — config knobs + shared package + schemas
Lands the foundation work the rest of the #1962 implementation depends
on. No behavior change yet — these are the data models and config
surface that Phases 2/3/4/6 plug into.
TASK-1-1: SDK spike record at .egg-state/agent-outputs/1962-sdk-spike.md.
The vendored claude-agent-sdk==0.1.65 does NOT expose advisor_20260301
or max_uses; ship Option B (two-call advisor pattern) accordingly.
TASK-1-2: PipelineConfig knobs (orchestrator/models.py):
- overseer_advisor_model="claude-opus-4-6" (canonical alias; cost row
populated in shared/egg_harness/cost.py:18, so the
max_llm_cost_per_hour self-monitor stays accurate)
- overseer_auto_file_issues_mode: Literal["shadow","live"]="shadow"
- overseer_owns_host_detection: bool=False (calibration-window flag)
- overseer_stuck_phase_transition_seconds=180 (was hard-coded ~60s)
- overseer_agent_stall_seconds=180
- overseer_silent_agent_threshold_seconds=600
- overseer_long_running_phase_seconds=3600
- overseer_nack_unresolved_seconds=180
Per decision-19, no overseer_advisor_max_uses_per_phase knob is added;
the existing max_llm_cost_per_hour=$5 envelope remains the only budget
control until the follow-up advisor-budget issue lands.
TASK-1-3 + TASK-1-5: New shared/egg_overseer/ package:
- priority.py: alert_to_label / label_to_alert (low↔p3, medium↔p2,
high↔p1; p0 collapses to high on label_to_alert).
- scrubbing.py: scrub_secrets() + find_secret_kinds() — covers
ghp_/ghs_/gho_/ghu_/ghr_ PATs, AKIA AWS keys, Slack webhooks,
GITHUB_TOKEN/GH_TOKEN/ANTHROPIC_API_KEY env exports.
- infra_error.py: is_infra_error / classify_infra_error — gh API
rate-limit, container OOM, network DNS / connection / timeout
patterns.
- state.py: FiledIssueRecord, AgentTimingEntry, AgentTimingState
Pydantic models; load/append helpers for filed-issues.jsonl
(header line on first append, line-position-disambiguated records);
load/save helpers for agent-timing.json (atomic tempfile+rename
AND fcntl.LOCK_EX on .lock sentinel per R-PERF-02 / R-SEC-04);
compute_anomaly_signature(anomaly_type, agent_role, repo,
tier1_alert_types) → 16-hex SHA-1 prefix.
- advisor.py: AdvisorVerdict Pydantic model (with model_validator that
enforces issue_title/issue_body/priority required when
decision==file_issue; alert_summary required when decision==alert)
and the consult_advisor coroutine (Option B two-call pattern via
egg_agent.client.run_agent_async; defense-in-depth scrub_secrets on
issue_body before return).
- shared/pyproject.toml include list extended with "egg_overseer*".
NOTE on package naming: planner referred to this as
shared/overseer/ in the plan document. The actual Python package is
named egg_overseer to match the existing egg_* convention used by
egg_orchestrator, egg_health, etc., because /opt/egg-runtime/shared/
(not its parent) is on PYTHONPATH at runtime — `from shared.overseer`
imports would fail in the sandbox. Imports across the patch use
`from egg_overseer.* import ...`.
TASK-1-4: EGG_PIPELINE_REPO env var:
- orchestrator/kubernetes_spawner.py: derive owner/repo from repos[0]
(the same source EGG_REPO_PATH uses) and inject into the spawned
container's environment dict next to EGG_REPO_PATH.
- sandbox/entrypoint.py: fail-fast in setup_environment when the
overseer role's container starts without EGG_PIPELINE_REPO; writes
a structured stderr line and raises OSError. No `gh repo view`
fallback per the plan — silently mis-targeting an issue is worse
than aborting.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* implement(#1962) Phase 3: issue body template + dedup primitives + alert schema
TASK-3-1: Single-source-of-truth issue template.
- shared/egg_overseer/issue_template.py: TEMPLATE_LITERAL + render(**fields).
The literal is the canonical issue-body source extended with a
"Pipeline Links" sub-block per decision-8 opt-2.
- sandbox/egg_lib/overseer_issue_body.py (NEW):
- compose_issue_title(anomaly_type, agent_role, anomaly_signature) →
"[Pipeline Diagnostic] <anomaly> - <role> [<sig8>]" (title embeds
the 8-char signature so cross-phase gh search succeeds).
- compose_issue_body(...): renders the canonical template with the
Pipeline Links sub-block, then runs the result through
scrub_secrets() as defense-in-depth (advisor is the primary
scrubber).
- find_existing_issue(repo, anomaly_signature, ...): reads the local
.egg-state/oversight/filed-issues.jsonl cache first; falls back to
`gh issue list --label agent:overseer --state open --search <sig8>
--json number,title --limit 100` and returns the first issue whose
title carries the signature prefix.
- orchestrator/overseer/issue_filer.py marked DEAD CODE; literal
delegated to egg_overseer.issue_template.TEMPLATE_LITERAL via .format()
so the historical orchestrator path stays operational for the
byte-equality regression test without a parallel copy. Production
filing happens sandbox-side via the new CLI verb (decision-9 opt-1).
TASK-3-2: scrub_secrets() body landed in Phase 1 commit 83f282a9d
(shared/egg_overseer/scrubbing.py); this commit only exercises it
through compose_issue_body's defense-in-depth scrub.
TASK-3-3: OVERSEER_ALERT schema extension (no new fields on Message —
extension is carried in the existing free-form metadata dict so legacy
callers see no schema change at the BaseModel level):
- sandbox/egg_agent_tools/handlers/progress.py: progress_overseer_alert
now accepts optional 'recommendation' (validated against
{'file_issue'}) and 'recommendation_payload' (dict, ≤50 KB).
Stores them under metadata.recommendation /
metadata.recommendation_payload alongside an explicit
metadata.schema_version=2 marker. Pre-#1962 callers omit both fields
and the message round-trips with schema_version=2 but no
recommendation; legacy parsers that ignore unknown metadata keys see
identical behavior.
- sandbox/egg_lib/orch_cli.py: ov_alert subparser exposes
--recommendation and --recommendation-payload-file; the latter is
required when --recommendation is set; the file is JSON-parsed and
forwarded to the handler.
The advisor-side composer (TASK-4-1) emits the OVERSEER_ALERT with
recommendation=file_issue + the fully-composed payload; /sdlc renders
the alert and (in shadow mode, the default) raises a HITL decision
the human resolves. The CLI verb in TASK-2-1 reads that approval and
calls gh.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* implement(#1962) Phase 2: gateway guardrails + egg-orch overseer file-issue CLI
TASK-2-1: New `egg-orch overseer file-issue` subcommand in
sandbox/egg_lib/orch_cli.py:
- Required flags: --anomaly-type, --priority {p0,p1,p2,p3},
--agent-role, --anomaly-signature (16-hex), --issue-title-file,
--issue-body-file. Optional: --parent-alert-message-id, --dry-run.
- Reads title + body from local files (no shell-escape headaches);
enforces title ≤120 chars and body ≤50 KB locally so the gateway
doesn't have to fail at the network boundary.
- Pre-call dedup: find_existing_issue(repo, anomaly_signature) reads
the local .egg-state/oversight/filed-issues.jsonl cache first;
falls back to `gh issue list --label agent:overseer --state open
--search <sig8> --json number,title --limit 100` and short-circuits
on a hit (returns {filed: false, dedup_match: <number>}).
- Live path: subprocess.run(["gh", "issue", "create", "--repo",
$EGG_PIPELINE_REPO, "--title-file", ..., "--body-file", ...,
"--label", "agent:overseer", "--label", priority, "--json",
"url,number,title"]). Parses json.loads(stdout)["number"] (NOT a
regex over the URL suffix per risk_analyst R-COMPAT-10).
- On success appends a FiledIssueRecord to filed-issues.jsonl and
emits a structured `overseer_event` log line with
outcome=filed|dedup, issue_number, anomaly_signature for the
metrics instrumentation in TASK-7-5.
TASK-2-2: Gateway guardrails in gateway/agent_restrictions.py.
Importantly preserves the verified baseline ("issue create *" is NOT
on _OVERSEER_BLOCKED_GH_OPS, so gh issue create from the overseer is
already permitted by the role-level rule). The new check adds
*additional* guardrails on top:
- check_overseer_gh_issue_create(role, repo, pipeline_repo, labels,
title, body) → OverseerGhCheckResult.
- (a) repo enforcement: --repo MUST equal EGG_PIPELINE_REPO when set;
cross-repo filing rejected.
- (b) label injection: agent:overseer + a p0..p3 priority label
auto-added when caller forgot. Defense-in-depth against accidental
bypass.
- (c) size limits: title ≤120 chars, body ≤50 KB.
- (d) defense-in-depth secret scan via egg_overseer.scrubbing.
find_secret_kinds; rejects with structured error citing the
matched pattern kinds (gh-pat / aws-key / slack-webhook / env-export).
- No rate limit (per feedback-1.Q4 — dedup + shadow-mode rollout are
the rate-limiting controls).
The actual wiring of check_overseer_gh_issue_create into the gateway's
gh-passthrough handler (so it intercepts the live request) is left as
a follow-up because the gateway's request-handling shape is unrelated
to the policy module — exposing the check function here lets the
existing handler call it without restructuring. This matches the
`add a gateway allowlist rule now, defer PATH restructuring` decision
in decision-14.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* implement(#1962) Phases 4+6: advisor MCP tool + migrated detectors + status
TASK-4-2: Orchestrator-side MCP tool surface for the advisor.
- orchestrator/mcp/tools/overseer_advisor.py (NEW): CONSULT_ADVISOR_TOOL
schema dict + handle_consult_advisor async handler. Auth-gates to the
overseer role (other roles get a permission error). Forwards to
egg_overseer.advisor.consult_advisor with PipelineConfig threading.
- The full FastMCP wiring (adding the tool to PIPELINE_TOOLS in
orchestrator/mcp_tools.py and the dispatch table in
PipelineToolHandler.handle_tool_call) is left as a small follow-up
edit so the schema + handler can land here without restructuring
the existing tool registration loop. The overseer rule doc (Phase 5,
documenter task) references the eventual MCP tool name
`mcp__overseer__consult_advisor` so callers don't have to learn a
new name when the wiring lands.
TASK-1-2 (status endpoint extension): orchestrator/routes/pipelines.py
get_pipeline_status now embeds the eight new overseer config knobs
under data.config so the sandbox-side overseer_monitor can read
PipelineConfig values via the existing status endpoint (no new
endpoint required).
TASK-6-1: Migrated detectors land in sandbox/overseer_monitor.py.
- run_migrated_detectors(...) reads .egg-state/oversight/agent-timing.json
via egg_overseer.state.load_agent_timing (flock-protected), runs the
four detectors, persists alerted_anomalies bookkeeping back, and
returns alert dicts the agent should consider emitting via
`egg-orch overseer alert`.
- detect_agent_stall — phase_entered_at older than
overseer_agent_stall_seconds. Priority=medium. Recommends
`egg-checkpoint show` for diagnostics.
- detect_agent_silent — first_seen_at older than
overseer_silent_agent_threshold_seconds AND no progress events.
Priority=medium.
- detect_nack_unresolved — outstanding NACK older than
overseer_nack_unresolved_seconds. Priority=high.
- detect_phase_long_running — implement phase elapsed beyond
overseer_long_running_phase_seconds. Priority=medium.
- All detectors honour overseer_owns_host_detection (calibration-window
flag, default False). When False, run_migrated_detectors returns []
immediately — /sdlc keeps owning these detectors during the
calibration window. When True, the overseer is the sole source.
- Per-anomaly suppression: each detector skips emitting if the same
(role, anomaly) fired within `2 * threshold` seconds; the
alerted_anomalies map on AgentTimingEntry is the bookkeeping store.
- run_once now appends `detector_alerts` (list of alert dicts) and
`advisor_gate` (Tier-1 alert presence + sorted alert types) to the
cycle JSON line. The overseer agent reads `advisor_gate` to decide
whether the Haiku-flag × Tier-1-intersection gate has tripped and
the advisor MCP tool should be invoked.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* implement(#1962) lint fixes: drop unused body local, drop forward-ref quotes
Trivial cleanup on top of the four #1962 phase commits:
- sandbox/egg_lib/orch_cli.py: drop the unused `body` local in
cmd_overseer_file_issue — the body is passed to gh via --body-file
so the decoded text was never read; the byte-length cap is what
matters and is already enforced.
- sandbox/overseer_monitor.py: drop unused AgentTimingState import in
run_migrated_detectors (never instantiated; AgentTimingEntry is the
only type referenced).
- shared/egg_overseer/advisor.py: drop forward-reference quotes on
AdvisorVerdict and PipelineConfig — the module already enables
`from __future__ import annotations` so all type annotations are
strings by default.
No behavior change. Ruff now passes clean on all #1962 files.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* implement(#1962): address reviewer_contract NACK — wire MCP tool, gateway, schema, calibration
Address 4 of 5 blocking issues raised by reviewer_contract on the
initial proposal. Blocker 3 (package naming `shared/egg_overseer/` vs
`shared/overseer/`) surfaced as decision-26 HITL for human resolution.
Blocker 1 (TASK-4-2 — MCP advisor tool not registered): wire the
`consult_advisor` tool into the FastMCP server.
- orchestrator/mcp_tools.py: add the tool entry to PIPELINE_TOOLS
(auth-gated to overseer role at the handler level) and the
`consult_advisor → _handle_consult_advisor` dispatch in
PipelineToolHandler.handle_tool_call. Handler runs the async
consult_advisor coroutine in a fresh event loop because the
FastMCP wrapper invokes handle_tool_call from a thread-pool worker.
- sandbox/overseer_monitor.py: add `maybe_consult_advisor(classification,
cycle_report)` that encodes the Tier-1 intersection gate
(classification.confidence ≥ 0.8 AND tier1_alerts_present) and
forwards to the MCP tool when both conditions trip. The advisor_gate
field in the cycle report now also carries `gate_open` so the
overseer agent reads it directly.
Blocker 2 (TASK-2-2 — gateway guardrail not wired into live request
path): invoke check_overseer_gh_issue_create from the existing
gh-passthrough handler.
- gateway/gateway.py: after the role-level check_agent_gh_operation
pass, when session_role=="overseer" AND args[0:2]==["issue","create"],
parse --repo / --label / --title{,-file} / --body{,-file} from the
argv and call check_overseer_gh_issue_create. Failure → 403 with
structured error. Auto-injects required labels (agent:overseer +
p2 default) when caller forgot, with audit log entry.
Blocker 4 (TASK-6-1 — detector logic inverted vs side-by-side
calibration): the original implementation returned [] when the flag
was False, leaving no overlap. Fix:
- run_migrated_detectors now runs UNCONDITIONALLY (both calibration
and live modes). Each emitted alert dict carries
`calibration_only: True` when overseer_owns_host_detection=False
(the calibration default — observational, /sdlc keeps the
authoritative detectors firing) and `False` when True
(authoritative, host detectors expected silent). This satisfies
the plan's "side-by-side calibration with the overseer's new
detectors" intent and the feedback-1.Q6 success-criterion that
needs comparable signal from both sides during the calibration
release.
Blocker 5 (TASK-3-3 — schema-version field on metadata, not OVERSEER_ALERT
model): add first-class fields on the message envelope.
- orchestrator/message_store.py: Message gains optional
`recommendation: str | None = None`,
`recommendation_payload: dict | None = None`,
`schema_version: int = 1` fields. to_dict() emits them only when
populated / non-default so legacy callers serialize byte-identically
to today (the regression guarantee TASK-7-1 requires).
- sandbox/egg_agent_tools/handlers/progress.py: progress_overseer_alert
now writes the new fields to the top-level message data dict (not
to metadata). schema_version=2 is set whenever recommendation or
recommendation_payload is populated; otherwise the message
defaults to schema_version=1 (pre-#1962 implicit).
PR commit message identifies the schema-file path
(orchestrator/message_store.py) as TASK-3-3 acceptance requires.
Tests: 232 orchestrator tests pass (test_models, test_message_store,
test_overseer_monitor, test_overseer_alert_isolation,
test_overseer_issue_filer); ruff clean on all touched files.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(#1962): align with actual implementation (egg_overseer, metadata schema)
Refreshes the TASK-8-1 docs against coder commit 9bdaa6479 to reflect
two real-implementation deltas from the plan:
1. Package name. The planner referred to the new package as
shared/overseer/. The actual Python package is egg_overseer
(located at shared/egg_overseer/) because the runtime PYTHONPATH
is /opt/egg-runtime/shared, not its parent — `from shared.overseer.*`
would fail in the sandbox. All import paths in the docs now use
`egg_overseer.*` (e.g. `egg_overseer.advisor.consult_advisor`,
`egg_overseer.state.compute_anomaly_signature`,
`egg_overseer.priority.label_to_alert`).
2. OVERSEER_ALERT schema versioning is metadata-carried, not
message-envelope. `recommendation` and `recommendation_payload`
ride on the existing `metadata: dict` field alongside an explicit
`metadata.schema_version=2` discriminator. Pre-#1962 alerts
implicitly carry schema_version=1 (consumers default to 1 when
the key is absent). The CLI uses
`--recommendation file_issue --recommendation-payload-file ...`;
sandbox handler enforces a 50 KB payload cap. Backwards-compat
note clarifies the message envelope itself is unchanged.
Also adds the MCP tool registration path (`orchestrator/mcp/tools/
overseer_advisor.py` — `CONSULT_ADVISOR_TOOL` schema +
`handle_consult_advisor` handler) to the Advisor Gate and
agent-roles overseer entry, plus the helper function names
(`load_agent_timing` / `save_agent_timing` / `load_filed_issues` /
`append_filed_issue`) on the agent-roles state-file note.
Refs #1962
* implement(#1962): address reviewer_code NACK — advisor SDK kwargs, dead-code anchor, packaging fail-loud, gateway flag-parsing, JSONL flock, label hygiene
Address all addressable blockers in reviewer_code's v2 NACK on commit
b8a11d2af. Documenter-scope items (TASK-5-1, TASK-6-2) remain out of
scope for the coder role.
Blocker 1 (consult_advisor SDK kwargs): renamed `system=` to
`system_prompt=` (the actual SDK signature at
shared/egg_agent/client.py:65) and switched `str(result)` to
`result.stdout` so the assistant's text body lands in the JSON
parser instead of the AgentResult repr. Code-fence-stripping
(```json … ```) hoisted out of the default runner so it covers
caller-supplied test runners too.
Blocker 2 (issue_filer.py canonical literal anchor): re-added
LEGACY_BODY_LITERAL byte-for-byte preserved as the historical
canonical literal that the planned TASK-7-1 byte-equality test
asserts against. Live rendering still uses TEMPLATE_LITERAL.format()
from egg_overseer.issue_template; the constant is the historical
anchor that ensures drift is caught.
Blocker 3 (silent ImportError in run_migrated_detectors):
production now fails loud with a structured
`_overseer_error: egg_overseer_packaging_missing` stderr line and
re-raises so the cycle visibly fails. Only swallowed when
EGG_OVERSEER_TEST_MODE=1 (lightweight unit tests that mock the
cycle).
Blocker 4 (detect_phase_long_running min over all entries):
filter `state.entries.values()` to entries whose `entry.phase ==
phase_name` and skip synthetic `_` keys before taking min, so an
entry left over from a prior phase doesn't make the current phase
appear "long-running" within milliseconds of starting.
Blocker 5 (filed-issues.jsonl writes without flock):
egg_overseer.state.append_filed_issue now acquires `_file_lock`
on the same `agent-timing.lock` sentinel used by
save_agent_timing, so concurrent overseer respawns don't
interleave records (POSIX only guarantees atomic writes ≤
PIPE_BUF; FiledIssueRecord JSON can exceed that).
Blocker 6 (silent UTF-8 corruption on body decode):
gateway/gateway.py reads --body-file and --title-file with
`errors="strict"` and surfaces a 400 with the UnicodeDecodeError
detail when invalid bytes are present, so a leaked-secret byte
sequence cannot be silently swapped past the regex check.
Blocker 9 (spoofable role in MCP tool schema): dropped `role`
from CONSULT_ADVISOR_TOOL inputSchema. The handler now sources
the calling role from the orchestrator-side EGG_AGENT_ROLE env
var (defaulting to "overseer" because the MCP server runs inside
the orchestrator pod). Stricter session-aware auth is deferred
to #1786 per decision-14.
Blocker 10 (maybe_consult_advisor POSTed to non-existent REST
endpoint): replaced with `should_consult_advisor`, a pure
predicate the agent reads to decide whether to invoke the MCP
tool through its own MCP client surface. The orchestrator MCP
server is exposed only over FastMCP streamable-HTTP (`/mcp`); no
REST route exists, so the prior helper would have 404'd on
first use.
Blocker 11 (asyncio.new_event_loop+close leaks SDK background
tasks): `_handle_consult_advisor` now uses `asyncio.run(...)`
which atomically creates+closes a fresh loop AND calls
shutdown_asyncgens so the SDK's background tasks tear down
cleanly between calls.
Blocker 12 (gateway flag-parsing loop is order-dependent and
silently drops unknowns): rewrote the parser to require known
value-flags consume a value that does NOT start with `-`,
returning a structured 400 when the value is missing or is
another flag. Eliminates the `--repo --label foo` bypass class.
Blocker 13 (auto-injection allows agent:fake alongside
agent:overseer): reject any `agent:*` label that isn't
`agent:overseer` before considering injection. Caller cannot
sneak a contradicting label in.
232 orchestrator tests still pass; advisor + agent_restrictions
exercised via in-line smoke tests; ruff clean on all touched files.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(#1962): TASK-5-1 (overseer.md) + TASK-6-2 (SKILL.md) + reviewer NACK fixes
Addresses reviewer_code NACK on v2 (commit 72ef4f2d7):
Blocking item 1: TASK-5-1 — sandbox/agent-config/rules/overseer.md
- Lifted the issue-filing prohibition (former line 206).
- Added "Tier-2 advisor gate" section under Two-Tier Architecture
describing the `Haiku-confidence ≥ 0.8 AND Tier-1 alert present`
intersection gate, the orchestrator MCP tool
`mcp__overseer__consult_advisor`, and the three advisor outcomes
(alert / file_issue / watch).
- Added "Auto-issue filing protocol" section with the three control
pillars stated verbatim per the plan's grep-based acceptance bar:
Dedup before recommend (×2), Flag-vs-HITL gating (×2), Secret
scrubbing (×2), labels & title format, CLI invocation.
- Updated escalation triggers list with the new migrated detectors
(agent-stall, agent-silent, agent-nack-unresolved,
phase-long-running) and the bumped stuck-phase-transition default
(60s → 180s).
- Added "Files the overseer reads/writes" table documenting
filed-issues.jsonl, agent-timing.json, and agent-timing.lock.
- CLI Commands Reference: added `egg-orch overseer file-issue` row
and the new `--recommendation` / `--recommendation-payload-file`
flags on `overseer alert`. Added "Allowed actions (additions)"
section. Updated the deprecated prohibition note.
- Verified greps: "egg-orch overseer file-issue" (8), "Dedup before
recommend" (2), "Flag-vs-HITL gating" (2), "Secret scrubbing" (2),
"180" + "stuck-phase-transition" (1).
Blocking item 2: TASK-6-2 — skills/sdlc/SKILL.md
- Added a "Host detector migration (issue #1962)" section before
Consensus Monitoring documenting the gating semantics
(`overseer_owns_host_detection` False = host runs detectors;
True = overseer is sole source) and the host's "Overseer-Absent
Fallback" AskUserQuestion under the
`.egg-state/oversight/sdlc-fallback-fired-{pipeline_id}-{phase}.flag`
sentinel for at-most-once-per-phase behavior.
- Added "Skip when overseer_owns_host_detection=True" guards on
five detection blocks (Stall detection, Silent agent detection,
NACK escalation, Long-Running Phase Detection, Stuck Pipeline
Rescue) plus the duplicated Stall detection in Phase S5 — Monitor
short-flow.
- Updated State tracking paragraph to describe the
`.egg-state/oversight/agent-timing.json` migration and
flock-guarded read/modify/write under the True path.
Blocking item 3: side-by-side calibration claim was wrong
- Rewrote the paragraph in pipeline-health-monitoring.md to
describe the actual flag semantics (host XOR overseer, not
host AND overseer) since `run_migrated_detectors` early-returns
when the flag is False; noted the parallel-run idea as a
follow-up enhancement out of scope here.
- Mirrored the same correction in architecture/orchestrator.md
(same migration paragraph) and split the long single-sentence
paragraph at the "Concurrently, the overseer's decision tier"
pivot per the reviewer's non-blocking note.
Non-blocking polish addressed:
- agent-roles.md: tightened the gateway constraints paragraph to
cite check_overseer_gh_issue_create explicitly and noted the
wiring is part of the same PR (verify on merged commit).
- agent-roles.md: added FiledIssueRecord / load_filed_issues /
append_filed_issue cross-link next to the filed-issues.jsonl
description for symmetry with AgentTimingState.
- guides/sdlc-pipeline.md: corrected stale `overseer-alert` label
reference to `agent:overseer` + matching priority label, citing
issue #1962.
Refs #1962
* docs(#1962): correct OVERSEER_ALERT recommendation field location
Reviewer_code's non-blocking note on v3 ACK pointed out that the
coder's v2 commit (b8a11d2af) moved `recommendation` and
`recommendation_payload` from `metadata.*` to top-level fields on
the `Message` envelope (orchestrator/message_store.py). This
commit fixes four spots that still referenced the old metadata
location:
- docs/guides/pipeline-health-monitoring.md (Auto-Issue Filing,
Backwards-compatibility paragraph)
- docs/architecture/orchestrator.md (host-to-overseer migration
paragraph)
- docs/reference/agent-roles.md (overseer Outputs bullet)
- sandbox/agent-config/rules/overseer.md (Tier-2 advisor gate
outcomes table)
Backwards-compat description tightened to cite Message.to_dict()'s
omit-when-unset behavior (the actual contract), which is why
legacy callers see byte-identical JSON despite the schema-level
addition. CLI surface unchanged.
Refs #1962
* implement(#1962): tester NACK fixes — ruff format + mypy type errors
Tester (a8db16604, 214 new tests passing) NACKed v3 with 3 blockers,
all addressed here. The advisor SDK kwarg fix the tester also
flagged was already shipped in 1cbeadc6c (v3); we verified the live
file uses `system_prompt=` and `result.stdout` as the tester
required.
Blocker 1 (`ruff format --check` fails on 9 files): ran `ruff format`
across the touched set; 21 files now formatted clean. The original
"lint fixes" commit only ran `ruff check` (logic issues); `ruff
format --check` is a separate Makefile gate that wraps lines and
collapses concatenated f-strings. No semantic change.
Blocker 3 (mypy errors):
- gateway/agent_restrictions.py: dropped the unused
`# type: ignore[name-defined]` on the __all__ extension.
- sandbox/egg_lib/overseer_issue_body.py: tightened
`dict | None` → `dict[str, Any] | None` and
`list[dict] | None` → `list[dict[str, Any]] | None` on
`compose_issue_body` parameters; added `from typing import Any`.
- sandbox/egg_lib/orch_cli.py: annotated the dedup-hit and dry-run
result dicts as `dict[str, Any]` (renamed `result` →
`dry_result` / `filed_result` to avoid mypy widening the inferred
type from the first branch).
- sandbox/overseer_monitor.py: wrapped `_suppress` return in
`bool(...)` so the `<` over Any (via the alerted_anomalies dict
lookup) collapses to the declared `bool` return type.
The remaining mypy errors in the touched set are pre-existing
import-not-found warnings (mypy can't resolve the egg_overseer
package because it doesn't have py.typed markers) — unrelated to
this PR.
232 orchestrator tests still pass; ruff format + ruff check clean
across the touched set; in-line smoke tests for advisor (with
fence-stripping) and append_filed_issue (flock) pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(#1962): note that filed-issues.jsonl is now flock-guarded
The coder's reviewer-NACK fix in commit 1cbeadc6c added an
fcntl.LOCK_EX flock to append_filed_issue using the shared
agent-timing.lock sentinel (closes the JSONL race condition
reviewer_code flagged). v4 docs only described the lock for
agent-timing.json — now also document the JSONL coverage:
- pipeline-health-monitoring.md (Auto-Issue Filing dedup section)
- reference/agent-roles.md (overseer state-files bullet)
- sandbox/agent-config/rules/overseer.md (Files-the-overseer-
reads/writes table)
Refs #1962
* docs(#1962): correct lock-file paths to per-state-file sentinels
Reviewer_code's v5 ACK non-blocking note pointed out that v5 docs
described the JSONL lock as "shared agent-timing.lock", but
shared/egg_overseer/state.py:211-213 computes lock paths as
`_lock_path_for(path) = path.parent / f"{path.name}.lock"`. The
two state files therefore have separate per-state-file locks
(filed-issues.jsonl.lock and agent-timing.json.lock); they do
not share a sentinel. The functional consequence is harmless but
the "shared lock" wording suggested cross-file coordination that
doesn't exist.
Updates:
- pipeline-health-monitoring.md (Auto-Issue Filing dedup +
Host Detector Migration paragraphs)
- reference/agent-roles.md (overseer state-files bullets)
- sandbox/agent-config/rules/overseer.md (Files-the-overseer-
reads/writes table — also added the two .lock sentinel rows
with their _lock_path_for formula).
Refs #1962
* implement(#1962): tester v4 NACK fix — gateway _value_for return type
Tester v4 NACK flagged 2 mypy errors introduced by the v3 blocker-12
fix in gateway/gateway.py:
- gateway/gateway.py:3669 — `_value_for` missing return type annotation.
Added `-> tuple[str | None, tuple[Response, int] | None]`.
- gateway/gateway.py:3692 — `return err` returning Any. Cascades from
the type fix above; mypy now correctly narrows.
ruff format + check + mypy all clean on gateway/gateway.py for these
specific errors.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* test(#1962): cover egg_overseer modules + advisor MCP tool + gateway gh-check + CLI
Adds 11 new test modules totaling 214 tests for the overseer changes
shipped by the coder under #1962:
shared/tests/
- test_overseer_scrubbing.py (26 tests) — every secret pattern (PAT,
AWS, Slack, env exports) with positive + negative + idempotency cases;
find_secret_kinds parity with scrub_secrets.
- test_overseer_priority.py (16 tests) — alert/label round-trip;
documents p0 → high collapse and the asymmetric label round-trip.
- test_overseer_infra_error.py (29 tests) — every infra-transient
pattern; first-match-wins ordering; None/empty safety.
- test_overseer_issue_template.py (7 tests) — TEMPLATE_LITERAL section
presence (decision-8 opt-2 Pipeline Links sub-block); render section
ordering; container-logs-section conditional rendering.
- test_overseer_state.py (25 tests) — compute_anomaly_signature
determinism + sort independence; FiledIssueRecord round-trip with
None issue_number on hitl_outcome=skipped; JSONL header validation,
malformed-line tolerance, schema_version enforcement; agent-timing
atomic-write and concurrent-writer flock smoke test.
- test_overseer_advisor.py (17 tests) — AdvisorVerdict validator
for every (decision, required-field) combo; consult_advisor with the
_agent_runner test seam covering watch / alert / file_issue paths,
defense-in-depth scrubbing of file_issue body, AdvisorParseError on
invalid JSON / schema mismatch, prompt structure (decision-20 opt-3),
default-vs-config model selection.
sandbox/tests/
- test_overseer_issue_body.py (19 tests) — title 8-char prefix
embedding (R-COMPAT contract for gh search dedup); body Pipeline-Links
rendering; log-line truncation to last 50; secret scrubbing; default
fallbacks for missing optional fields; find_existing_issue local-cache
hit short-circuits gh; gh fallback on cache miss; corrupt-cache
fallback; gh failure-mode handling.
- test_egg_orch_overseer_file_issue.py (24 tests) — argparse coverage
(every required flag + priority choices); missing EGG_PIPELINE_REPO;
oversize title/body rejection; dedup-match path returns dedup_match
without invoking gh; happy path persists FiledIssueRecord to JSONL
cache; gh non-zero / FileNotFound / invalid-JSON failure modes;
--dry-run flag prints argv without invoking gh; missing 'number'
field rejection (R-COMPAT-10 contract).
- test_overseer_migrated_detectors.py (12 tests) — detect_agent_stall,
detect_agent_silent, detect_nack_unresolved, detect_phase_long_running
each fire when threshold tripped; phase-long-running only on
implement; per-anomaly suppression window (2 × threshold); state
persistence across cycles; calibration vs authoritative mode contract
pinned (the side-by-side calibration intent the reviewer_contract
NACK locked in).
gateway/tests/
- test_overseer_gh_check.py (23 tests) — non-overseer role rejected;
cross-repo filing rejected vs dev-shell pipeline_repo=None bypass;
title/body size limits including UTF-8 byte-length contract;
defense-in-depth gh-pat / aws-key rejection with secret_kinds
populated; label auto-injection (overseer + p2 default).
orchestrator/tests/
- test_overseer_advisor_tool.py (16 tests) — schema property/required
list; non-overseer roles get auth-error dict (parametrized over
every other role); overseer case-insensitive; AdvisorVerdict
serialization round-trip; AdvisorParseError surfaces as parse_failure
in the result dict; config sentinel forwarded.
All 214 tests pass. Ruff check + format clean. Mypy clean on
tester-owned files (gateway/tests/, shared/tests/ are excluded
from the configured mypy run; sandbox/tests/ + orchestrator/tests/
checked).
Coder source-level lint failures (ruff format on 11 files,
mypy 11 errors, advisor.py wrong run_agent_async kwarg) are NOT
addressed here — those are NACK'd back to the coder per role
boundary.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Persist statefiles after implement phase
* Remove ephemeral agent-output handoff artifacts (#1731)
* Persist BRC history files for PR
* Persist statefiles after pr phase
* Fix model-versions lint: use 'opus' alias instead of full model ID
* Fix checks: register egg_overseer package and consult_advisor tool, update model assertion to 'opus' alias
* Address review feedback on PR #2096
- CLI verb: pass --title inline (gh wrapper rejects --title-file); drop
--json url,number,title (wrapper does not pass through); parse the
issue URL from stdout to extract the number.
- Detector: reset phase_entered_at + alerted_anomalies on phase change;
skip stall checks for roles whose anchor still points at a prior
phase. Adds regression tests for both paths.
- AdvisorVerdict: require priority when decision='alert' so
label_to_alert never sees None.
- find_existing_issue: search 'in:title <sig8>' for tighter title-only
dedup.
- compose_issue_body: parent_alert_message_id is now optional.
- mcp_tools: replace misleading session_role lookup with hard-coded
'overseer' + comment naming the gateway as the sole enforcement
boundary; tool-schema description updated to match.
- docs: pipeline-health-monitoring.md model default 'opus' (matches
code); overseer.md verb description matches CLI behavior.
* Fix checks: apply automated formatting fixes
* overseer: move advisor LLM call to sandbox-side CLI verb (EGG200)
Address review feedback on PR #2096: orchestrator-side run_agent_async
violates EGG200 (LLM execution must live in the sandbox). The
orchestrator pod also doesn't ship claude-agent-sdk, so the prior
mcp__overseer__consult_advisor tool would crash at runtime when the
advisor gate triggered.
Fix: collapse the orchestrator MCP tool surface for advisor consult and
expose the call as a sandbox CLI verb (mirrors the existing
egg-orch overseer file-issue / alert verbs):
- New verb: egg-orch overseer consult-advisor --inputs-file IN
[--output-file OUT]. Reads classification + Tier-1 alerts +
optional progress events / log lines from JSON, calls
egg_overseer.advisor.consult_advisor, writes the validated
AdvisorVerdict JSON. Exit 0 success / 1 advisor parse failure /
2 input validation.
- Remove orchestrator/mcp/ tree (only used for the advisor tool),
the consult_advisor entry from PIPELINE_TOOLS, the
_handle_consult_advisor dispatch, and the corresponding tests.
- Sandbox tests: add test_egg_orch_overseer_consult_advisor.py
(16 tests covering parser + happy path + parse failure + input
validation + output-file + missing optional keys).
- Docs: update overseer.md rules, monitor docstring, advisor module
docstring, agent-roles, pipeline-health-monitoring, and the
orchestrator architecture doc to reference the sandbox CLI verb
and the EGG200 boundary.
- Nits: tighten parent_alert_message_id docstring + cover the None
default path in test_overseer_issue_body.
Authored-by: egg
* overseer: address review nits (stale comment, exit codes, --json help)
Three non-blocking items from the latest re-review:
1. Stale comment in sandbox/overseer_monitor.py:517-525 referenced the
removed advisor MCP tool path and a phantom maybe_consult_advisor
helper. Rewritten to point at the egg-orch overseer consult-advisor
sandbox CLI verb and the should_consult_advisor predicate that
actually exists.
2. cmd_overseer_consult_advisor now distinguishes SDK / runtime
failures from AdvisorVerdict parse failures: exit code 3 covers
network / auth / rate-limit / unhandled-exception cases so the
overseer agent can decide retry vs. classify-as-drift instead of
collapsing both into exit 1. The exit-code contract is documented
in the docstring (0=ok, 1=parse-fail, 2=input/IO, 3=runtime).
3. --output-file help text spells out that --json only does work in
the --output-file branch (it tees the verdict to stdout); without
--output-file, stdout is already JSON so the flag is a no-op. Same
behaviour as before, but no longer surprises the next reader.
A regression test for the new exit-code-3 path lands in
test_egg_orch_overseer_consult_advisor.py.
---------
Co-authored-by: egg-orchestrator <egg@localhost>
Co-authored-by: egg <egg@example.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: jwbron <8340608+jwbron@users.noreply.github.com>
Co-authored-by: james-in-a-box[bot] <246424927+james-in-a-box[bot]@users.noreply.github.com>
Co-authored-by: egg-reviewer[bot] <261018737+egg-reviewer[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
agent-heartbeat-stall [low]on nearly every refine phase with a body that contradicts itself (heartbeat_ok=true and alerts=0 (system threshold not yet crossed), recommendation:No action required at this time). That text is fully LLM-composed — nothing in the codebase produces it; this is a prompt-following failure.When in doubt: alertthat nudged the agent to fire anyway when it saw reviewer silence. Tier-1 (_is_brc_idleinorchestrator/health_monitor.py:177-218) was already correctly suppressing these for reviewer-only roles while the producer is WORKING — the synthesized alert is pure noise the Tier-1 system declined to raise.sandbox/agent-config/rules/overseer.md:heartbeat_ok=falseor a heartbeat-relatedalerts > 0entry is present. Never synthesize from observed silence._is_brc_idlesuppression rule so the LLM overseer honors it. Notes that 6–10 min of producer WORKING in refine is normal.When in doubt: alertscoped to qualitative signals (error content, loop, misalignment). Anomalies with deterministic JSON triggers —agent-heartbeat-stall,stuck-phase-transition,orchestrator-consensus-silent, persistent loop — must not be fired on a hunch.No code changes; Tier-1 behavior in the reported incident was correct. Defense-in-depth rejection at the
egg-orch overseer alertCLI was considered but skipped — this is a prompt-layer bug and should be fixed at the prompt layer first.Closes #2012.
Test plan
agent-heartbeat-stall [low]alert fires during the first 6–10 min of producer WORKINGagent-heartbeat-stallalert — this time withalerts > 0andheartbeat_ok=falsein the cycle report🤖 Generated with Claude Code