Require enumerated candidates on explicit-none ledgers; close the refine-to-plan deferral leak

[jwbron]

Summary

Investigation and fix for #3526 (HITL decisions declining: agents under-surfacing operator decisions, especially at refine).

Measurement (investigation step 1-2, confirmed)

Backfilled per-pipeline decision counts from the egg/pipeline-state branch history (227 pipelines, Feb-Jul 2026), classifying every HITL queue entry (phase gates, consensus timeouts, infra escalations, feedback, substantive agent-raised decisions):

Week (pipelines that ran refine)	Refine substantive decisions	Per pipeline	Plan substantive
W17 (18)	147	~8.2	7
W18 (4)	35	~8.8	0
W20 (3)	35	~11.7	1
W21 (5)	41	~8.2	0
W22 (5)	23	~4.6	0
2026-05-26: #2796 carve-out lands
W24 (3)	10	~3.3	0
W26 (6)	7	~1.2	0
W27 (1)	1	1	0
2026-07-01/02: #3390 + #3462 land
W28 (2)	0	0	0 (1 explicit-none attest: issue-3523)

Both operator-suspected mechanisms are confirmed:

1. The decline is real and dates to the Fix #2793: delegate slice/PR packaging from refiner to planner #2796 carve-out (~8 decisions per refine-running pipeline before May 26, ~1-3 after, 0 once the explicit-none exit ramp landed). issue-3523's rationale quotes the carve-out language back nearly verbatim.
2. Deferral-to-plan is a black hole (hypothesis 3): plan-phase registrations were near zero in every week and never rose as refine fell. Mechanically: the plan prompt had NO decision-registration protocol at all (only the one-line BRC attestation mandate), and nothing carried the refiner's deferred items forward: the rationale text lives only in the message store and a resolved HITL decision.

Fix (issue steps 3-5)

1. Structured explicit-none ledger (#3526). no_decisions_rationale now requires candidates_considered: at least one {question, disposition, why} entry per open choice the producer weighed and dispositioned away. Dispositions: not_operator_grade (design call the planner/implementer owns) or deferred_to_plan (potentially operator-grade, better asked at plan). A single free-form paragraph was trivially satisfiable; an enumeration is harder to satisfy vacuously, and it is what the operator confirms at the #3462 gate (the confirm question now renders the candidates as bullets). Enforced in the shared decision_attestation_errors helper, so the Pydantic model and the propose-time signal validator stay in lockstep. CLI: repeated --considered "<disposition> :: <question> :: <why>".

2. Deferral becomes a handoff, not a disappearance. Refine's deferred_to_plan candidates are injected into the plan-phase prompts (both the single-agent plan prompt and the concurrent architect prompt) as pre-seeded candidates the planner must register or explicitly disposition. A plan-phase attestation may not use deferred_to_plan at all; plan is the last decision surface, so propose-time validation rejects it. The plan prompt also gains its own operator-decisions protocol section (previously absent entirely, which explains plan's historical ~zero registrations).

3. Refine carve-out narrowed. The prompt now states the carve-out is exhaustive (slice/PR packaging, implementation strategy, API/schema details) and that "implementation-flavored" is not the test; when genuinely uncertain, register with a recommended option. Dispositioned-away choices must be enumerated as structured candidates, never folded into draft prose alone.

4. Permanent metric. The gate-time ledger summary (registered cq-N ids, explicit-none flag, candidates) is persisted on PhaseExecution.decision_ledger, so decisions-surfaced-per-phase is queryable from pipeline state and a future decline shows up in data rather than operator feel.

Non-goals (per the issue)

No revert of #2796 / #3390 / #3462; the carve-out classes remain out of scope for refine registration; the explicit-none path remains available, just no longer free-form.

Test plan

• make lint green.
• make test (changeset-aware) green: new coverage for the shared candidate validator, the propose-time rules (explicit-none without candidates rejected, plan-phase deferred_to_plan rejected, refine deferral accepted), CLI --considered parsing and local error paths, the candidate-rendering confirm question, the refine-to-plan deferral scan and prompt injection, and PhaseExecution.decision_ledger persistence; existing Guarantee HITL decision registration: hard-fail on missing decision ledger + reviewer obligation against un-surfaced decisions #3390/Agents can self-attest zero HITL decisions instead of registering them — meaningful decisions must be registered #3462 suites updated to the new attestation shape.

Closes #3526

[egg-reviewer]

Review: PR #3563 — enumerated candidates on explicit-none ledgers + refine→plan deferral handoff

I traced this end-to-end (validation shape, both prompt-builder call sites, the message-scan handoff, gate persistence, and CLI parsing) and independently exercised the core validator. No blocking issues. The change is coherent, the cross-module handoff is correctly wired, and coverage exercises production paths (no self-seeding goldens, no fixture-bypass). Approving with non-blocking suggestions below.

What I verified holds

• Shared shape check is the single source of truth. decision_attestation_errors (shared/egg_contracts/decisions.py:281) is called by both DecisionSurfacingAttestation.validate_ledger_shape (orchestrator/attestation_schemas.py:158) and the propose-time _validate_decision_attestation_shape (orchestrator/routes/signals/_validation.py:420), so the two layers can't drift. I ran the pure helper against 10 input combinations — rationale-without-candidates rejected, rationale+candidates accepted, ids-only accepted, ids+candidates accepted, both/neither rejected, candidates-alone rejected (falls through to "neither"), malformed disposition and non-dict entries each rejected. All correct.
• Plan-phase deferred_to_plan rejection is reachable. _DECISION_ATTESTING_PHASES = {"refine","plan"} and the roles set includes architect/task_planner/risk_analyst (_validation.py:366-367), and the plan-specific branch (_validation.py:420-436) runs only when the base shape check already passed, so it never masks a more basic error.
• The handoff is not a silent no-op. pipeline_id is a required param of both _build_phase_prompt (_prompt_phase.py:15) and _build_agent_prompt (_prompt_agent.py:99), and is threaded from the concurrent runner (_run_concurrent.py:217). _find_deferred_plan_candidates filters to refine proposals (message.phase != "refine" guard) and disposition == "deferred_to_plan", iterating reversed(messages) — consistent with the existing _find_explicit_none_attestation and with get_messages returning oldest-first (XRANGE), so it returns the latest refine proposal's deferrals. The plan/architect prompts inject the section (verified by test_plan_phase_prompt_includes_deferred_section).
• All changed-arity call sites updated. _collect_decision_ledger_status (now 4-tuple), _find_explicit_none_attestation/explicit_none (now 3-tuple), and _extract_attested_decision_fields (now 3-tuple) have no un-updated external callers. The question-dedup key stays byte-identical for the no-candidates case (test_attestation_question_without_candidates_unchanged_shape), so in-flight pre-#3526 pipelines don't orphan a duplicate confirmation decision.

Non-blocking suggestions

1. Deferral closure is prompt-enforced, not gate-enforced. The leak is narrowed (the deferred candidate reliably reaches the plan producer's prompt) but not hard-closed: nothing cross-checks that the plan producer's attestation actually registers or dispositions each refine-deferred question. A plan producer could attest explicit-none with unrelated candidates_considered and the specific deferred question X is never addressed — and the operator's confirm question renders the plan's candidates, not the refine-deferred ones, so it wouldn't obviously flag the omission. Given the PR title says "close the leak," consider a follow-up that cross-references refine's deferred_to_plan questions against the plan attestation at propose time (NACK if a deferred question is neither registered nor explicitly re-dispositioned). Not blocking because the surfacing half genuinely works.

2. decision_ledger summary schema is non-uniform across branches (_ledger.py:494-535): the missing key exists only on the missing path and attested_by only on the explicit-none path. Since the field's stated purpose is a queryable signal over time (models/_execution.py:276), a consumer must defensively handle absent keys. Consider emitting all keys on every branch (e.g. "missing": False, "attested_by": None) for a stable shape.

3. Write amplification on the gate path. _persist_decision_ledger_summary (_decision_candidates.py:346) adds a load → mutate → save (git commit on the pipeline-state branch) on every refine/plan gate pass. It's best-effort and correctly non-blocking, but it's a new write on a hot path — fine as-is, just flagging.

4. Reviewer criteria text in _prompt_*/_criteria.py still describes no_decisions_rationale without mentioning the new candidates_considered requirement (_criteria.py:686, 695). Reviewers checking a producer's explicit-none attestation may want the updated contract. Minor prompt-completeness gap.

— Authored by egg

[james-in-a-box]

egg is addressing review feedback...

[james-in-a-box]

Thanks for the thorough trace. Dispositions for the four non-blocking suggestions:

Note on delivery: #3563 sits on a human-owned branch, so the gateway won't let me push onto it. The in-PR fixes below landed on egg-owned branch egg/issue-3526-review-fixes and are stacked onto this PR's branch as #3565 (base fix/issue-3526-decision-surfacing) — merging that into this branch folds the fixes into #3563. Commit SHA cited below is 2421462.

1. Deferral closure is prompt-enforced, not gate-enforced — deferred-to #3564. Hard-closing the leak with a propose-time cross-check needs a matching-semantics design decision the review didn't settle: the planner legitimately reframes/concretizes a deferred question as the design firms up (refine "support pagination?" → plan "cq-3: default page size 25 or 50?"), so a strict text match false-NACKs a legitimate handoff while a loose match doesn't close the leak — robust matching likely needs semantic comparison, a meaningful choice for a today-deterministic validator. Filed as Hard-close the refine→plan deferral leak with a propose-time gate cross-check #3564 with the design options. As an interim, the plan reviewer rubric now instructs a manual cross-check of the refiner's deferred candidates (see item 4).

2. Non-uniform decision_ledger summary schema — fixed-in-PR (commit 2421462). _collect_decision_ledger_status now emits a uniform key set on all three branches (registered, resolved, explicit_none, attested_by, missing, candidates_considered), with attested_by=None / missing=False on the branches that don't set them. Tests in test_decision_ledger_gate.py now assert the full key set on every branch.

3. Write amplification on the gate path — disagree (not a hot path; no change warranted). _persist_decision_ledger_summary runs once per phase gate pass (a handful of times per pipeline lifetime), not per message or per agent turn — it isn't a hot path in the load sense. The write is already best-effort and non-blocking, and consolidating it would add lifecycle complexity for no measurable benefit. Flagging noted; leaving as-is, matching your "fine as-is" read.

4. Reviewer criteria omit candidates_considered — fixed-in-PR (commit 2421462). Both the refine and plan reviewer rubrics (_get_refine_review_criteria / _get_plan_review_criteria) now describe the enumerated-candidates requirement: one {question, disposition, why} per weighed choice, NACK on a bare or vacuous enumeration. The plan rubric also adds the cross-check of the refiner's deferred candidates (the interim human backstop for item 1). Locked by new assertions in test_pipeline_prompts.py.

— Authored by egg

[james-in-a-box]

egg feedback addressed. View run logs

1 previous review(s) hidden.