[issue-3077][slice-3/6] Spec-derived propose-time validation for... - #3142
Conversation
…3077 TASK-3-2) Adds the slice-3 tester suite for #3077: rejection coverage for every registered refine/plan producer (refiner ⇒ analysis-draft, task_planner ⇒ plan-draft, architect ⇒ architect-output + architect-slices, risk_analyst ⇒ risk-analyst-output), with each rejection asserting the exact spec path appears in the error message. Pass-through cases (no_changes_needed, artifact-less roles like coder/documenter in implement) are pinned so the slice-3 generalisation cannot sneak new rejections onto today's accepted paths. The #3081 ``branch_verified`` graceful-degradation posture is pinned for the spec-derived loop on a non-plan role (risk_analyst). The plan-only extensions (#3026 parseability, #2527/#2528 role↔files alignment) are pinned at the integration layer so slice-3's layered design is asserted, not just the underlying ``_validate_plan_proposal`` unit (test_pipeline_prompts.py keeps its unit-level pins). Tests use ``handle_consensus_propose_signal`` as the entry point so the contract is robust to the coder's choice of internal helper name and dispatch shape. A ``_make_subprocess_router`` helper canonicalises the ``git branch --contains`` + per-artifact ``git show`` mocking pattern so each test reads as a one-liner spec-path declaration. Fails today by design (4 of 12): the architect and risk_analyst rejection cases, plus the branch_verified-with-local-commit case for risk_analyst, exercise the architect / risk_analyst gap that TASK-3-1 closes. The other 8 cases (refiner, task_planner, pass-throughs, the no-op skip, branch_verified-commit-absent, and the two plan extension retentions) pass against the current tree and pin behaviour the generalisation must preserve. This is the BRC parallel-mode contract. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Generalises propose-time draft validation in routes/signals.py: every CONSENSUS_PROPOSE with a commit_sha now resolves egg_contracts.artifact_spec.specs_for(phase, agent_role) and runs the existing server-side git show presence check per registered artifact. The plan-draft parseability (#3026) and role↔files alignment (#2527 / #2528) extensions layer on the plan-draft row only via _validate_plan_extensions; the per-role refine/plan dispatch is gone. - _validate_producer_artifacts() is the canonical spec-driven validator. Roles outside the producer set bail out before pipeline-state load; #3081 branch_verified graceful degradation preserved per-spec. - _validate_plan_proposal() is now a thin back-compat wrapper around _validate_producer_artifacts(agent_role="task_planner", phase="plan"). - _validate_producer_draft_present() is DELETED. Direct-call unit tests for the refine path are removed (equivalent coverage moves to test_signals.py via task-3-2); handler-level integration tests stay. - _get_draft_path() in routes/pipelines.py is now a thin spec call for the registered refine/plan phases; pr falls back to _draft_filename so test_pr_phase stays byte-identical, implement still returns None. - Architect / task_planner / risk_analyst prompts derive their agent-output paths via resolve_artifact_path() at the top of _build_role_section. The slice-2 ratchet flips from "literals match the spec" to "literals are gone, spec calls take their place". Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
…t_spec.py
The conflict is in TestConsistencyC: slice-2 asserts agent-output path
literals exist in pipelines.py while slice-3 (this PR) rewrites those
literals into resolve_artifact_path() calls. Slice-3 explicitly retires
the slice-2 tests (per the class docstring) and replaces them with two
new ratchets: ban raw .egg-state/agent-outputs/{_identifier}-… literals
and assert every agent-output spec appears as a resolve_artifact_path
call. Kept the slice-3 (HEAD) version.
Conflict Resolution SummaryResolved merge conflict with
Why HEAD wins on this oneThe conflict is in
The slice-2 tests would fail against the slice-3 implementation by construction (their whole point is that literals exist; slice-3's whole point is that they don't), and slice-3's class docstring narrates the retirement explicitly:
So keeping HEAD is the only resolution that leaves a green suite and honors what slice-3 set out to do. The slice-2 Please review
— Authored by egg |
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Contract Verification — PR #3142 (issue-3077, slice-3/6)
Verdict: Approve. Both slice-3 tasks fully satisfy their acceptance criteria. Spec-derived propose-time validation generalizes cleanly from the old hard-coded refiner/task_planner dispatch to a single registry-driven pass, with no regression to the #3026/#2527/#3081 behaviors layered on top.
Verification method
Orchestrator is UNREACHABLE this session, so egg-contract verify-criterion could not be invoked; the on-disk contract also tracks acceptance per-task (its top-level acceptance_criteria array is empty — no ac-N ids exist to mark). Verification was performed by reading the diff against the contract and running the affected targeted suites at HEAD 47a66f8. CI on the PR HEAD remains the binding gate.
task-3-1 — Generalize propose-time validation
- ✅ Refine producer rejected naming the spec path.
_validate_producer_artifactsraises agit show-presenceValueErrorper registered artifact, namingspec.resolve_path(identifier). (test_refiner_proposal_rejected_when_analysis_draft_absent) - ✅ Plan parseability + role-alignment retained.
_validate_plan_extensionskeepsparse_plan(#3026) andvalidate_task_role_alignment(#2527/#2528), invoked only for theplan-draftrow. Messages unchanged (does not parse into any tasks,role↔files alignment violations). - ✅
branch_verifiedgraceful degradation (#3081) preserved for all producers. Thebranch_verified is None and not _commit_object_resolvable(...)skip is carried into the generalized validator. (test_branch_verified_inconclusive_*— both skip-on-absent and still-validate-on-local cases) - ✅
no_changes_needed+ artifact-less roles not newly rejected. Handler wraps the call inif not no_changes:, and the validator early-bails whenagent_role not in {spec.producer_role for spec in all_specs()}. (coder/documenter/no_changes_needed tests) - ✅
_validate_producer_draft_presentdeleted; no second copy of path knowledge. Function removed (only doc/comment references remain);pipelines.py_get_draft_pathand the architect/risk-analyst prompts now route throughresolve_artifact_path. The rewrittenTestConsistencyC_PromptDerivesFromSpecratchet forbids raw.egg-state/agent-outputs/{_identifier}-…literals from returning.
task-3-2 — Validation tests
- ✅ Rejection coverage for every registered refine/plan producer: refiner→analysis-draft, task_planner→plan-draft, architect→architect-output and architect-slices (both rows), risk_analyst→risk-analyst-output (asserting the underscore disk filename
risk_analyst-output.json). - ✅ Pass-through cases:
no_changes_neededskip (asserts no.egg-state/git show), coder + documenter artifact-less pass-through. - ✅ Existing plan-validation tests unmodified / mechanical-only. The unanchored
match="no plan draft found"/"no analysis draft found"assertions still match the generalized{role} proposal rejected: no {label} found at …message; only docstrings changed.
Targeted test results at HEAD 47a66f8
test_signals.py::TestSpecDerivedProposeValidation— 12 passedtest_artifact_spec.py(slice-2 consistency, incl. rewritten Consistency-C) — 32 passedtest_pipeline_prompts.py::TestPlanProposalValidation+TestProducerDraftPresentValidation— 16 passed
Non-blocking observations
- The task-3-2 acceptance text lists "reviewer messages" as a pass-through case; there is no explicit reviewer-role propose test. This is structurally covered — reviewers issue ACK/NACK (not propose), and the producer-role early-bail-out excludes every reviewer role — so it is not a gap that changes behavior. Worth a one-line test only if you want the acceptance phrasing mirrored literally.
- Cosmetic: in
_validate_plan_extensionsthe role-alignment comment is still numbered(3)though it is now the second check after the parseability(1).
No contract violations found. The diff is confined to files_affected for both tasks (plus the auto-generated brc-history artifacts and a mechanical docstring/rename pass in test_pipeline_prompts.py).
— Authored by egg
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Review: Spec-derived propose-time validation (#3077 slice-3)
I traced the full change end-to-end: the _validate_producer_artifacts generalization in signals.py, the spec-driven _get_draft_path / prompt-path rewrite in pipelines.py, the artifact_spec registry it consumes, and all four test files. This is a clean, correct, and well-tested refactor that also closes a real gap. No blocking issues found.
What I verified (correctness)
- Path equivalence holds. The rewritten
_get_draft_pathroutesrefine/planthroughresolve_artifact_path(...); the resolved templates (.egg-state/drafts/{identifier}-analysis.md,-plan.md) are byte-identical to the oldf".egg-state/drafts/{prefix}-{filename}", and the identifier comes from the same_pipeline_identifier(issue_number, pipeline_id or "unknown")call. Thepr-phase legacy branch is untouched. - Identifier consistency producer↔validator. Prompt generation (
_build_agent_prompt) and validation (_validate_producer_artifacts) both derive the path via_pipeline_identifier(issue_number, pipeline_id)→spec.resolve_path(identifier). An architect is told the exact path the validator later checks, including the re-runissue-<N>-<suffix>case. - No new false rejections. The early membership bail-out (
agent_role not in {spec.producer_role ...}→ return) keeps coder/tester/documenter/all reviewers out of the presence loop. The 409 commit-on-branch short-circuit and #3081branch_verified is None+_commit_object_resolvablegraceful degradation are preserved for every role, not just the legacy plan/refine branches. no_changeshandling is equivalent. The oldelif refiner / elif task_plannerbranches were already unreachable for no-op proposes because the upstream no-op phase gate 400s any refine/plan no-op before validation. The new explicitif not no_changes:guard makes that belt-and-suspenders without changing behavior. The tester branch moving fromelifto a separateifdoes not double-validate (tester has no registered artifact).- Phase resolution is robust. Production passes no
phase; it resolves frompipeline_state.current_phase.value, falling back to the registry's role→phase row whencurrent_phaseis unreadable (e.g. a MagicMock without.value). For every registered producer this resolves to exactly the phase its artifact is registered in, sospecs_foralways returns the right specs — strictly no worse than the old hard-coded phase. - Plan extensions intact.
_validate_plan_extensions(parseability #3026 + role↔files alignment #2527/#2528) is layered onto theplan-draftartifact only, after presence, and retains the same error messages and graceful-degradationtry/exceptposture. - Per-artifact infra-failure isolation is correct. A
git showexception on one artifactcontinues to the next spec rather than masking a second artifact's absence — a reasonable improvement over a single-read.
Tests
Strong. The new TestSpecDerivedProposeValidation suite drives the real handle_consensus_propose_signal production path and mocks only at the subprocess.run boundary via a command-shape router, asserting on the actual rejection messages and handle_propose.assert_not_called(). Coverage spans every registered producer (refiner, task_planner, architect ×2 artifacts, risk_analyst), pass-through for artifact-less roles, the no_changes skip, both directions of #3081 degradation, and plan-extension retention. No self-seeding goldens, no fixtures bypassing the production helper, no name/behavior contradictions. The consistency tests were updated coherently (literal-equality → "resolve call present" + a banned-literal ratchet).
Non-blocking observations
-
Remaining copy of agent-output path knowledge (out of scope).
_synthesize_plan_draft(pipelines.py:19497) still hard-codesarchitect-output.json/architect-slices.yaml/risk_analyst-output.jsonas bare filenames and reconstructs{_synth_id}-{filename}itself, rather than routing throughresolve_artifact_path. This PR does not touch that function, so it's correctly out of scope, but it means the "single source of truth" goal isn't fully complete — a spec template rename would still drift here. Worth a follow-up so the slice-2 ratchet eventually covers this reader too. -
Behavioral tightening worth calling out for the merger. Architect and risk_analyst proposals now get a hard 400 at propose time if their artifacts are absent at the proposed SHA — previously these roles had no propose-time presence check. This is the intended feature (it closes the gap where an architect reached consensus without committing its output and broke task_planner downstream), and it aligns with the unconditional "write both files" instruction + file-restriction list in the architect prompt. Flagging only so it's a conscious behavior change rather than a surprise.
-
Minor test coupling.
test_no_changes_needed_skips_artifact_validationandtest_documenter_implement_propose_passes_throughrely on_contract_completeness_rejectionreturningNonewithout mocking it (works because/tmp/repohas no contract). Fine as-is, just an implicit dependency.
Nice work — the docstrings tracing the #3016/#3026/#3081/#2527 lineage make the invariants easy to audit.
— Authored by egg
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
…numbering, add reviewer pass-through test
Review feedback addressed — commit
|
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Contract Re-Verification — PR #3142 (slice-3, #3077)
Incremental re-review of the delta since my last verification at 47a66f8. The delta is a single commit, 71f3872d ("Address review: route plan-draft synthesis through spec, fix comment numbering, add reviewer pass-through test"), touching three files (+88/−11). Reviewed via git log 47a66f8..HEAD --not origin/egg/issue-3077/slice-2 -p.
Verdict: no contract violations. The delta strengthens compliance with both slice-3 tasks; no previously-verified criterion regresses.
1. orchestrator/routes/pipelines.py — _synthesize_plan_draft now spec-derived ✅
Replaces the hard-coded agent-output filenames (architect-output.json, architect-slices.yaml, risk_analyst-output.json) with spec names resolved through egg_contracts.artifact_spec.resolve_artifact_path. This removes the last copy of agent-output path knowledge in the pipelines.py reader, directly advancing TASK-3-1's acceptance criterion: "no second copy of path knowledge remains in signals.py or pipelines.py" (the prompt-builder at pipelines.py:13701-13703 was already spec-derived).
Verified behaviorally equivalent to the prior literals:
- Prefixed path:
repo_path / resolve_artifact_path(name, id)== oldoutputs_dir / f"{id}-{filename}"(outputs_dirisrepo_path/.egg-state/agent-outputs, line 19484). - Fallback:
Path(prefixed_rel).name.removeprefix(f"{id}-")yields the exact old basename — notablyrisk-analyst-output(hyphen name) →risk_analyst-output.json(underscore file), matching the registry's deliberate name/filename split (artifact_spec.py:117-120). - Log
file=fields updated toglobal_filenameconsistently.
Coverage: the 10 _synthesize_plan_draft tests in test_pipeline_prompts.py (incl. namespaced prefixed + fallback cases) pass. py_compile clean.
2. orchestrator/routes/signals.py — comment renumbering (3) → (2) ✅
Cosmetic only. After the presence-check step was subsumed/deleted in this slice, the surviving steps are (1) parseability and (2) role↔files alignment — the renumber makes the sequence contiguous. No behavior change.
3. orchestrator/tests/test_signals.py — reviewer pass-through test ✅
New test_plan_reviewer_propose_passes_through fills the "reviewer messages" pass-through case named in TASK-3-2's acceptance criteria, which was previously unasserted. It is a meaningful test, not a vacuous pass: it marks all plan-phase artifacts missing (a producer would be rejected) yet asserts the reviewer proposal returns 200 with handle_propose called once. This exercises the genuine producer-membership early bail-out at signals.py:1290 (agent_role not in {spec.producer_role for spec in all_specs()}) — a reviewer produces no registered artifact, so the presence loop is never reached. Helpers/fixtures (_make_subprocess_router, _pipeline_with_phase, _propose_payload, _*_PATH attrs) all pre-exist and match the established pattern.
Coverage: full TestSpecDerivedProposeValidation class (13 tests, incl. the new one) passes.
Regression check
Previously-verified criteria still hold: rejection-per-producer, plan parseability/role-alignment retention, branch_verified degradation, and no_changes_needed/artifact-less pass-through are unchanged by this delta. The pipelines.py change is the reader side and does not touch propose-time validation.
Note for human reviewer
The orchestrator was unreachable during this review (egg-contract show → "Orchestrator unreachable"), so CLI verify-criterion marking could not be performed. Verification was done against the on-disk contract (.egg-state/contracts/issue-3077.json); slice-3 acceptance criteria are stored as free-text within the slice tasks (top-level acceptance_criteria is []), so there are no ac-N ids to mark. Per convention I did not run the full make test suite — only the targeted tests cited above (CI gates the full suite).
— Authored by egg
There was a problem hiding this comment.
Re-review: feedback addressed (#3077 slice-3) — commit 71f3872
Re-reviewed the delta since my last pass (47a66f8..71f3872, a single "Address review" commit touching pipelines.py, signals.py, test_signals.py). Both non-blocking observations from the prior round are properly fixed — not superficially. No blocking issues. Approving.
1. _synthesize_plan_draft routed through the spec — verified correct
The previously-flagged remaining copy of agent-output path knowledge is gone. The function now resolves each section via resolve_artifact_path("<name>", _synth_id) and derives the global fallback as Path(prefixed_rel).name.removeprefix(f"{_synth_id}-").
I verified the resolution is byte-identical to the old hard-coded paths, so this is a pure refactor with no silent-drop risk:
outputs_dir = repo_path / ".egg-state/agent-outputs", andresolve_artifact_path(...)returns.egg-state/agent-outputs/{id}-<file>, sorepo_path / prefixed_rel== oldoutputs_dir / f"{id}-{filename}".- The
removeprefixreconstruction reproduces the old barefilenameexactly ({id}-architect-output.json→architect-output.json). - The one path that mattered most: spec name
risk-analyst-output(hyphen) resolves to…/{id}-risk_analyst-output.json(underscore filename), matching the old literal.artifact_spec.py:117-120documents this intentional hyphen-name/underscore-file split, andTestSynthesizePlanDraftNamespaced::test_reads_prefixed_agent_outputswrites871-risk_analyst-output.jsonand asserts the Risk Assessment section renders — so a future hyphenation slip in the spec would fail that test. Ran the suite locally: 8 passed.
2. Comment renumbering — correct
_validate_plan_extensions now reads (1) Parseability / (2) Role↔files alignment with no orphaned or duplicate marker. Confirmed by reading the full function body.
3. New reviewer pass-through test — valid, exercises the real path
test_plan_reviewer_propose_passes_through drives the production handle_consensus_propose_signal (mocking only at the subprocess.run boundary) with a reviewer role in the plan phase and every plan artifact marked absent — a producer would 400, but the non-producer must bail out before the presence loop. This precisely exercises the agent_role not in {spec.producer_role for spec in all_specs()} early return at signals.py:1290. Non-redundant vs the existing implement-phase pass-throughs (this one uses a phase that does register artifacts). Passes locally.
Minor (non-blocking, no change needed)
- The feedback-addressing comment states the reverse ratchet (
test_every_agent_output_spec_has_resolve_call) "now covers this reader." Strictly,_synthesize_plan_draftcallsresolve_artifact_path(spec_name, …)through a loop variable, so it doesn't add a literalresolve_artifact_path("architect-output"needle — that ratchet was already satisfied by the prompt builder and is unchanged here. The real coverage for this reader isTestSynthesizePlanDraftNamespaced, which is solid. Description nuance only; the code and tests are correct.
Clean refactor that completes the single-source-of-truth goal for the last agent-output reader. Nice work.
— Authored by egg
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
| egg is addressing review feedback... |
Review feedback dispositions — current HEAD
|
|
egg feedback addressed. View run logs 8 previous review(s) hidden. |
…3152) * docs: mark #3077 slices 2-4 shipped in coordination-state doc Slices 2 (#3141), 3 (#3142), and 4 (#3143) have all merged since the coordination-state doc was last updated (it still read "as of slice-1"). Also updates the reviewer-worktree-sync section of concurrent-execution.md to reference egg-artifact as the served channel for spec-registered coordination artifacts (plan-draft, analysis-draft, architect-output) when worktree sync fails, replacing the shared-object-store git show fallback description that predates slice-4. * docs: mark #3077 slices 5-6 shipped in coordination-state All six slices of the #3077 epic have landed on main. Mark the slice-5 (prompt-sync ratchet) and slice-6 (fail-loud memory backend + Redis restart tests) rows Shipped, bump the section header to slice-6, and update the intro/closing prose so the design-target framing no longer contradicts an all-Shipped table. --------- Co-authored-by: jwbron <8340608+jwbron@users.noreply.github.com> Co-authored-by: egg-reviewer[bot] <261018737+egg-reviewer[bot]@users.noreply.github.com>
signals.py propose validation resolves specs_for(phase, producer_role) for every CONSENSUS_PROPOSE with a commit_sha and runs the existing git-show presence check per artifact; _validate_producer_draft_present is subsumed and deleted; path literals asserted in slice 2 are re-derived from the spec.
Base PR: #3139
What's in this PR
Commits (3):
This slice
Spec-derived propose-time validation for all refine/plan producers
Files affected:
orchestrator/routes/signals.pyorchestrator/routes/pipelines.pyorchestrator/tests/test_signals.pyTasks (2) + acceptance criteria
orchestrator/routes/signals.py(≈1076-1139): for every CONSENSUS_PROPOSE carrying acommit_sha, resolvespecs_for(phase, producer_role)and run the existing server-sidegit showpresence check (≈1195) per registered artifact — reusing the single helper and the #3081branch_verifiedgraceful-degradation semantics, no forked code path. Keep the plan-draft extensions (parse_planPlan propose-time validation is existence-only — a draft missing its yaml-tasks appendix passes consensus, then fails the whole pipeline at populate #3026, role↔files alignment Validate task role↔file alignment at plan time, not push time #2527/Per-repo configuration for test/code file patterns #2528) layered on the plan artifact only. Subsume and DELETE_validate_producer_draft_present(≈1115-1117) and re-implement_get_draft_pathas a thin spec call. Re-derive the prompt path literals inorchestrator/routes/pipelines.pyfrom the spec (replacing the literals slice 2 asserted against). Roles with no registered artifact validate nothing;no_changes_neededproposes skip validation.branch_verifiedgraceful degradation behaves as before (#3081) for all producers. -no_changes_neededproposes and artifact-less roles are not newly rejected. -_validate_producer_draft_presentis deleted; no second copy of path knowledge remains in signals.py or pipelines.py.orchestrator/tests/test_signals.py— one rejection case per producer role with a registered artifact (refine analysis-draft, plan plan-draft, architect and risk_analyst outputs); plan extension retention; rejection payload names the expected spec path;branch_verifieddegradation unchanged; pass-through forno_changes_neededand artifact-less roles.Stack
issue-3077egg/issue-3077/slice-2