overseer: advisor-gated escalation + auto-issue filing (#1962)

[james-in-a-box]

Closes #1962. The deployed agent-side overseer is "pretty good" at
escalating to humans via OVERSEER_ALERT but does not always do so when
appropriate, and it cannot file GitHub issues even when the underlying
plumbing exists. At the same time, the host /sdlc skill is doing
mid-pipeline debugging work (stall detection, NACK escalation, agent
nudging, long-running-phase rescue) that the issue says belongs in the
overseer so the host can be a thin reporter / HITL surface.

This PR ships three coupled threads under the
advisor-strategy framing:

1. Advisor-gated escalation tuning. Haiku 4.5 keeps driving the
   overseer cycle (max_turns=1); when it flags an anomaly AND a
   Tier-1 health alert has tripped, an Opus 4.6 advisor is invoked
   (via a new orchestrator-exposed MCP tool — see "Implementation
   choices" below) to decide alert composition / priority / whether
   to recommend a GitHub issue. New config knobs
   (overseer_advisor_model, overseer_auto_file_issues_mode)
   expose the advisor; no advisor cap is added in this PR
   per decision-19's "no cap for now" resolution — the
   existing max_llm_cost_per_hour=$5 envelope remains the
   budget control, with a follow-up issue tracking advisor-
   specific budget. The gate matches the precedent shipped in
   Overseer agent-heartbeat-stall alert fires prematurely on every refine phase #2012. The advisor
   itself lives in shared/overseer/advisor.py so it is importable
   from both the orchestrator-side MCP-tool handler and any future
   orchestrator-side caller without crossing the sandbox/orchestrator
   package boundary.
2. Auto-issue filing via a new egg-orch overseer file-issue
   CLI verb. The CLI verb runs gh issue create itself, inside
   the sandbox, mediated by the gateway — there is no
   orchestrator-side endpoint that runs gh (this is decision-9
   opt-1; the previously sketched orchestrator REST endpoint that
   would have invoked gh is dropped). The dead-code template at
   orchestrator/overseer/issue_filer.py:86-107 is preserved as
   the canonical literal source and marked DEAD; the live body
   composition lives sandbox-side in
   sandbox/egg_lib/overseer_issue_body.py. Issues land with the
   existing agent:overseer label plus the matching priority
   (p0/p1/p2/p3) — no new labels are created. The advisor
   returns recommendation=file_issue inside an OVERSEER_ALERT
   so the human gates the actual filing via the existing
   pending_decisions HITL flow (shadow-mode rollout). Dedup
   searches per-repo by agent:overseer + an anomaly signature
   computed from (anomaly_type, agent_role, repo) and caches its
   findings in .egg-state/oversight/filed-issues.jsonl
   (append-only JSON Lines, header line
   {_kind: "header", schema_version: 1}). The title carries the
   8-char signature prefix so the gh issue list --search …
   fallback is reliable. Bodies are run through a new
   shared/overseer/scrubbing.py pass that removes
   ghp_* / ghs_* / gho_* / ghu_* / ghr_* PATs, AKIA*
   AWS keys, Slack webhook URLs, and GITHUB_TOKEN=* exports
   before the advisor returns its verdict; the gateway does the
   same scan as defense-in-depth.
3. Host → overseer migration with a side-by-side calibration
   window. Stall / silent-agent / NACK / long-running-phase /
   stuck-pipeline-rescue detection moves out of
   skills/sdlc/SKILL.md:489-639 into the overseer. The
   in-memory {role: {phase, phase_entered_at, …}} state map
   moves to .egg-state/oversight/agent-timing.json (overseer-
   owned, no orchestrator schema change). Thresholds become
   per-pipeline configurable via PipelineConfig defaulting to
   current values, with stuck-phase-transition bumped from
   ~60 s to 180 s per the human's calibration feedback. A new
   overseer_owns_host_detection: bool = False knob keeps the
   host's detection logic live by default for one release; the
   follow-up PR flips the default and deletes the dormant code.
   /sdlc keeps surfacing alerts, handling HITL decisions, and
   running the Phase-5 final-handoff rescue prompts.

Impact. The host becomes a thin reporter the way the issue
asks for; the overseer escalates more reliably and on a tighter
cost gate; auto-issue filing is gated through HITL while the
policy is calibrated. Existing OVERSEER_ALERT consumers see no
schema change beyond a new optional recommendation field; the
backwards-compat regression test asserts pre-#1962 alert payloads
still parse and render. Cost stays inside the existing
max_llm_cost_per_hour=$5 envelope; a follow-up issue tracks
dedicated advisor budgeting. /sdlc regression behavior stays
identical at first because (a) threshold defaults match today's
hard-coded values and (b) overseer_owns_host_detection=False
keeps the host detectors firing during the calibration window.

Test Plan

Automated:

• Orchestrator tests:
  • test_overseer_issue_filer.py — assertion that the existing
    template literal at lines 86-107 is byte-for-byte preserved
    (the canonical-literal-source contract); no other behavior is
    exercised since the module is marked DEAD.
  • test_overseer_advisor_tool.py (NEW) — orchestrator-side MCP
    tool surface for the advisor (handler returns valid
    AdvisorVerdict JSON; auth-gates to the overseer role; the
    executor → advisor prompt-contract from decision-20 opt-3
    is enforced).
  • Existing tests (test_overseer_monitor.py,
    test_overseer_alert_isolation.py,
    test_overseer_hitl_integration.py,
    test_overseer_decision_maker.py) updated for the new
    action vocabulary, the optional recommendation field on
    OVERSEER_ALERT, and the new agent-timing schema.
  • test_overseer_alert_isolation.py — backwards-compat
    regression assertion: a serialized pre-Improve overseer escalation/issue opening behavior #1962
    OVERSEER_ALERT payload (no recommendation field) round-
    trips through the message store and renders in the /sdlc
    alert-display path verbatim.
• Sandbox tests:
  • test_egg_orch_overseer_file_issue.py (NEW) — CLI verb
    argparse coverage; required-arg validation; happy path
    through gateway-mocked subprocess.run(["gh", "issue", "create", …]); failure path on gh non-zero exit; dedup-hit
    path returns dedup_match without invoking gh.
  • test_overseer_issue_body.py (NEW) — body composition
    helper produces the template body with all five Pipeline
    Links sub-block fields; title format embeds the 8-char
    signature substring; running scrub_secrets over a body
    with embedded ghp_… / AKIA… / hooks.slack.com/… /
    GITHUB_TOKEN=… substitutes them with [REDACTED:<kind>].
  • test_overseer_advisor_invocation.py (NEW) — sandbox-side
    invocation of the orchestrator MCP advisor tool produces a
    populated AdvisorVerdict; trigger gate must require Haiku
    flag AND Tier-1 alert before the tool is called;
    max_uses_per_phase cap respected.
• Shared-package tests:
  • shared/tests/test_overseer_advisor.py (NEW) — pure unit
    coverage of the consult_advisor function with mocked
    run_agent_async for each decision outcome.
  • shared/tests/test_overseer_scrubbing.py (NEW) — secret
    patterns covered with positive + negative cases.
• Gateway tests:
  • test_gh_overseer_file_issue.py (NEW) — gh issue create
    from the overseer role allowed; non-overseer role denied;
    labels auto-injected; title >120 chars rejected; body

    50 KB rejected; cross-repo target rejected; defense-in-
    depth secret-pattern rejection on body.

• Integration tests:
  • integration_tests/test_overseer_auto_issue_filing.py (NEW)
    — end-to-end: synthetic anomaly → Haiku flag + Tier-1 alert
    → advisor recommends file_issue → OVERSEER_ALERT carries
    recommendation → HITL decision created → human approves
    (script-driven provide_input) → sandbox CLI invokes
    gateway-mocked gh issue create with agent:overseer +
    priority labels and the title contains the 8-char
    anomaly-signature substring → re-trigger the same anomaly →
    dedup skips the second gh call.
• Skill regression:
  • skills/sdlc/tests/test_host_migration.py (NEW) — under
    overseer_owns_host_detection=False the host's detectors
    still fire (calibration mode); under =True they no longer
    fire and the overseer's OVERSEER_ALERT arrives at the
    host's surfacing logic with the same content as today.

Manual:

• Run /sdlc submit-task <synthetic issue>; under
  overseer_owns_host_detection=True, inject a long synthetic
  stall (>180 s) and verify the overseer (not /sdlc) emits the
  agent-stall alert.
• Inject a repeated-error anomaly that meets both the
  Haiku-confidence threshold AND a Tier-1 health alert; verify
  OVERSEER_ALERT.recommendation == "file_issue" arrives;
  approve the surfaced HITL decision; verify a GitHub issue
  lands with agent:overseer + the matching priority label,
  the title contains the 8-char anomaly-signature substring,
  and the body has the extended Pipeline Links sub-block.
• Re-trigger the same anomaly; verify the overseer logs a dedup
  hit and does NOT re-recommend filing.
• Smoke-test /sdlc thinness: confirm that during a healthy
  pipeline the host emits no stall / NACK / long-run alerts of
  its own when overseer_owns_host_detection=True (those
  should now come from the overseer if at all).

Manual Steps

Pre-merge:

1. Mark decision-15 (overseer file-boundary policy for
   .egg-state/oversight/) formally resolved with the
   no-op rationale: OVERSEER_PATTERNS at
   shared/egg_restrictions/patterns.py:526-527 already
   permits writes under .egg-state/oversight/. No new
   allowlist entry needed. (Reviewer non-blocking note:
   contract decisions linger as "Defer" forever unless a
   human or planner closes them; this one belongs to the
   planner.)

Post-merge:

1. File the SDK-pin-bump follow-up issue (advisor Option A
   swap) once Anthropic ships advisor_20260301 inside the
   >=0.1.65,<0.2 window — nothing in this PR blocks the
   follow-up, but the SDK pin currently doesn't expose the
   capability.
2. File the advisor-budget follow-up issue that
   decision-19 carved out — track an
   overseer_advisor_max_uses_per_phase (or equivalent
   cost-bound) knob and decide a default after measuring
   advisor-call distribution from production data. This
   follow-up is the only sanctioned reintroduction path for
   the cap.
3. File the metrics-instrumentation follow-up issue for
   feedback-1.Q6.(d) (auto-filed issue acceptance rate —
   closed-as-invalid vs. accepted-and-fixed). TASK-7-5 in
   this PR ships the four log-line shapes (a/b/c) but the
   (d) disposition metric needs GitHub-issue-state polling
   which is out of scope here.
4. Run the production calibration window (≥2 weeks) with
   overseer_owns_host_detection=False (default). Once
   overseer-side detectors are showing parity or improvement
   relative to /sdlc host-side detectors, file the
   host-migration cleanup follow-up PR that flips the default
   to True and deletes the now-dormant /sdlc detection code
   blocks (TASK-6-2 keeps them gated, not deleted, in this PR).

Pipeline Context

Pipeline: issue-1962
Issue: #1962

Per-phase BRC transcripts: refine, plan, implement.

Authored-by: egg

[egg-reviewer]

Re-review at 9efc2a9 — review-nit fixes since 6c081ab

Delta is a single commit, three small nit fixes (49 changed files in the PR overall, but only 3 files in this delta: sandbox/egg_lib/orch_cli.py, sandbox/overseer_monitor.py, sandbox/tests/test_egg_orch_overseer_consult_advisor.py).

Verification of the three fixes

1. Stale comment block at sandbox/overseer_monitor.py:517-527. ✓ Updated. The new comment correctly references egg-orch overseer consult-advisor (the actual CLI verb that exists at sandbox/egg_lib/orch_cli.py:1747) and the actual helper should_consult_advisor (defined at sandbox/overseer_monitor.py:562, a pure predicate as the comment claims — confidence >= HAIKU_CONFIDENCE_THRESHOLD AND tier1, no side effects). The phantom maybe_consult_advisor reference is gone, and the EGG200 boundary citation is correct.

2. Exit-code 3 for advisor SDK runtime failures at sandbox/egg_lib/orch_cli.py:1825-1835. ✓ Implementation is correct:

   • AdvisorParseError (which inherits from RuntimeError, verified at shared/egg_overseer/advisor.py:102) is caught first → exit 1.
   • The new except Exception arm comes second → exit 3, with type(exc).__name__ and the message echoed to stderr so the caller can distinguish e.g. APIConnectionError from RateLimitError from AuthenticationError without parsing the stderr text.
   • BaseException derivatives (KeyboardInterrupt, SystemExit, asyncio.CancelledError) are correctly NOT swallowed.
   • Docstring exit-code table updated and accurate.
   • test_advisor_runtime_error_returns_3 covers the new arm with a synthetic RuntimeError("connection reset by peer"); verifies exit code 3, the "advisor runtime failure" substring, the type name, and the original message all reach stderr.
   • All 17 tests in sandbox/tests/test_egg_orch_overseer_consult_advisor.py pass locally.

3. --json flag help text at sandbox/egg_lib/orch_cli.py:3103-3110. ✓ The help string now spells out the asymmetry: --json tees JSON to stdout when --output-file is set, and is a no-op without --output-file because stdout is already JSON. Inline comment at 1852-1855 mirrors the same explanation. No behavior change — this is documentation only — which matches the author's explicit decision in their feedback-addressing comment to keep the flag rather than drop it.

No new issues introduced

• The bare except Exception is intentionally broad (the docstring explicitly enumerates "or any other unhandled exception from the SDK call"). It will also surface programming bugs (e.g. TypeError from a future consult_advisor signature change) as exit 3 with a diagnostic message — that's a benign outcome since the caller treats exit 3 as "transient/runtime" and stderr names the actual exception type.
• No new pre-existing-issue exposure: the comment update doesn't add new code paths; the exit-code arm only widens what was an unhandled-traceback path; the help-text change doesn't touch behaviour.

Carry-forward non-blocking items (still deferred, none regressed)

The author's feedback-addressing comment explicitly defers these to follow-ups, and none are touched by this delta:

• overseer_advisor_model knob unused (CLI passes config=None so the knob defaults to "opus").
• shared/egg_overseer/state.py:122-126 lock docstring mismatch (claims shared lock with agent-timing.lock, actually uses filed-issues.jsonl.lock).
• shared/egg_overseer/scrubbing.py missing fine-grained GitHub PAT (github_pat_<22>_<59>), sk-ant-api03-…, GCP service-account JSON markers — relevant before flipping overseer_auto_file_issues_mode=live.
• shared/egg_overseer/advisor.py::_default_runner JSON extraction is fence-only; bare-object-with-prose fall-through to AdvisorParseError.
• orchestrator/overseer/issue_filer.py DEAD CODE half-deprecation with byte-equality test.
• sandbox/overseer_monitor.py:334 pipeline_data.get("config", {}) silent default on InvalidPipelineIdError.
• sandbox/agent-config/rules/overseer.md Forbidden Actions table whiplash.
• recent_log_lines byte cap on the prompt-builder side.
• End-to-end test coverage of the subprocess.run / consult_advisor wrapper-vs-CLI handshake without mocks.

These remain appropriate follow-ups, not blockers for this PR.

Verdict

Approve. The three nit fixes are correct, scoped, and well-tested. No new issues introduced; no regressions of previously approved work. The deferred carry-forward items are correctly tracked rather than silently dropped.

— Authored by egg

[james-in-a-box]

egg is addressing review feedback...

[james-in-a-box]

No code changes required.

The two most recent reviews on 9efc2a9 are both approvals:

• egg-reviewer[bot] review 4176390282: APPROVED — verifies the three nit fixes (stale comment block at sandbox/overseer_monitor.py:517-527, new exit code 3 for advisor SDK runtime failures at sandbox/egg_lib/orch_cli.py:1825-1835, and --json flag help-text clarification at sandbox/egg_lib/orch_cli.py:3103-3110). Verdict: "Approve. The three nit fixes are correct, scoped, and well-tested. No new issues introduced; no regressions of previously approved work."
• james-in-a-box[bot] contract-verification review 4176389763: verdict=approve. "No new tasks introduced, no acceptance criteria broken, regression test added for the new exit-code path."

The nine carry-forward items the reviewer enumerated (overseer_advisor_model knob unused; state.py:122-126 lock docstring mismatch; scrubbing.py missing fine-grained-PAT/sk-ant-api03-…/GCP-SA patterns; _default_runner fence-only JSON extraction; issue_filer.py DEAD-CODE half-deprecation; overseer_monitor.py:334 silent default; overseer.md Forbidden Actions whiplash; recent_log_lines byte cap; end-to-end subprocess.run ↔ consult_advisor test) are explicitly tagged: "These remain appropriate follow-ups, not blockers for this PR."

No line-level comments outstanding. All CI checks SUCCESS. Holding.

— Authored by egg

[james-in-a-box]

egg feedback addressed. View run logs

24 previous review(s) hidden.