Skip to content

Multi-repo pipelines: coordinated PRs across repositories (#3393) - #3418

Merged
jwbron merged 36 commits into
mainfrom
egg/issue-3393/work
Jul 2, 2026
Merged

Multi-repo pipelines: coordinated PRs across repositories (#3393)#3418
jwbron merged 36 commits into
mainfrom
egg/issue-3393/work

Conversation

@james-in-a-box

@james-in-a-box james-in-a-box Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Lets a single pipeline operate across an arbitrary number of repositories and open coordinated, cross-referencing PRs — one per slice, each in that slice's repo — with cross-repo ordering expressed through the existing slice-dependency mechanism. Replaces the manual hand-off that today carves the cross-repo half of a logically-single change out of the pipeline and gives it to an engineer.

The change is built on the ratified refine analysis's verified thesis: the single-repo assumption is concentrated, not pervasive. The gateway worktree list, per-repo credentials, create_slice_pr(repo=...), and repo-agnostic push policy already exist; the concentrated gaps are the contract's missing repo dimension and the repos[0] collapse in the agent environment.

Delivered as a single serialized six-slice chain (five slices edit orchestrator/routes/pipelines.py, so per #3046 they are ordered along one dependency chain):

  1. Repo dimension across the two model layers: contract-layer Slice.repo + a pure additive schemaVersion 1.3→1.4 stamp (Slice.repo stays None on legacy load), and orchestrator-layer Pipeline.repos (RepoSpec) + primary_repo + a runtime resolve_slice_repo where absent⇒primary is resolved; N=1 round-trips/behaves unchanged.
  2. List-shaped submission (submit_task / POST /api/v1/pipelines accept a list of {repo, base_branch}) with submission-time validation rejecting mixed-visibility and mixed-auth-mode sets.
  3. Removal of all three enumerated repos[0] collapse sites plus a full repo→worktree map exposed to the agent environment re-keyed by full owner/repo (operator ruling Phase 4: CLI implementation #6), guarded by a ratchet test + grep sweep.
  4. Slice-PR routing to slice.repo and a lazy-per-repo work branch + context PR (every repo with ≥1 slice; PRs cross-reference siblings).
  5. Cross-repo merge-sequencing hold (cq-1 two-tier) with pinned failure semantics: automated draft→ready on the upstream PR's merged boolean/mergedAt (not head-SHA) via a new mark_pr_ready gateway verb; closed-unmerged and never-merging (bounded poll) both escalate to a HITL hold; a separate HITL hold type for beyond-merge-state conditions (release/publish, version pin, genuine development block). Dependencies gate merge-readiness, not development.
  6. Per-repo test-gate + reviewer-diff scoping and per-repo conventions (each slice's gates run in that slice's repo worktree under that repo's CLAUDE.md / linters / check commands).

Out of v1 (deferred): mixed auth modes across repos, richer merge-sequencing machinery, and cross-repo atomic merge (impossible on GitHub — v1 ships an ordering hold, not atomicity).

Test Plan

Automated (per slice; make test narrows to reachable suites, then make test-all before phase exit; make lint green throughout):

  • Slice 1: shared/egg_contracts/tests — schemaVersion 1.3→1.4 migration; absent Slice.repo ⇒ primary repo; legacy singleton Pipeline.repo ⇒ one-element list; JSON round-trip; N=1 back-compat unchanged.
  • Slice 2: orchestrator/tests — list submission accepted; bare single repo still accepted; mixed-visibility set rejected with actionable error; mixed-auth-mode set rejected; primary-repo defaulting.
  • Slice 3: orchestrator/tests — agent env exposes the full repo→path map keyed by owner/repo; all three collapse sites gone; a ratchet test asserts no repos[0] collapse reintroduced (with the len(repos)==1-guarded sdlc_hitl.py:82 explicitly allowed).
  • Slice 4: orchestrator/tests — slice PR created in slice.repo; a repo with ≥1 slice gets a work branch + context PR, a slice-less submitted repo gets neither; sibling-PR cross-reference rendered.
  • Slice 5: orchestrator/tests — dependent slice PR opens draft; upstream-merge poll auto-marks it ready; beyond-merge-state condition routes to a HITL hold, not auto-release; development is not serialized by the dependency.
  • Slice 6: orchestrator/tests — test gate and reviewer diff scope to the slice's repo worktree only; per-repo check commands resolved from the slice's repo.

Regression baseline: single-repo (N=1) pipelines exhibit no behavior change; existing suites stay green.

Manual Steps

Pre-merge: none.

Post-merge: the schema migration (schemaVersion 1.3→1.4) applies lazily on contract load following the four existing precedents — no manual migration of persisted contracts is required; absent Slice.repo resolves to the pipeline's primary repo on read. New multi-repo behavior takes effect for pipelines submitted after the orchestrator image is rebuilt/redeployed on the normal release cadence. File a follow-up issue for the deferred scope: mixed auth modes across repos and any richer first-class merge-sequencing machinery beyond the v1 hold.

Pipeline context

Per-phase BRC transcripts: refine, plan, implement-unattributed.

egg-orchestrator and others added 22 commits July 2, 2026 19:47
Grounded the issue's current-state claims against the live tree
(create_worktree list-shape, per-repo credentials, repo-param PR
creation, Slice schema gap, repos[0] collapse sites). Four grounding
corrections flagged for the planner, acceptance criteria restated,
HITL decision registered for v1 merge-sequencing gate semantics.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Take ownership of 3393-analysis-human.md: verified faithful against the
refiner's 3393-analysis.md (v1, 107c930) and the contract task
description; removed remaining jargon and added the per-repo work-branch/
umbrella-PR point.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e NACK)

Verified live: _spawn.py:452,464, commit_authorship_store.py:932-933, and
routes/pipelines.py:732 (overseer_repo). Added plain-language sweep clause.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Correction #1 un-inverted: client method IS create_worktrees (plural,
  gateway_client/_worktree.py:13); singular create_worktree is the
  gateway-internal helper (worktree_manager/_create.py:115); repo_volumes
  is the live spawner param fed from WorktreeResult.worktrees — rewritten
  as a two-layer naming map.
- repos[0] collapse sites enumerated as THREE (adds
  routes/pipelines.py:732 overseer_repo); sdlc_hitl.py:82 cleared as
  guarded. Verdict row 6, correction #4, AC-4 updated; human summary
  'two spots' -> 'three spots'.
- Per-repo conventions entailment added (design recommendation #5 +
  AC-7): slice agent cwd + CLAUDE.md/linters/check commands of the
  slice's repo.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tions

- Hard-bit #1 now states the operator's decided model: automated draft-hold
  auto-readied on upstream merge; HITL only for beyond-merge-state waits
  (release/publish, version pinning) and genuine development blocks.
- Added per-repo house-rules bullet (refiner v2 design rec #5 / AC-7).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Operator resolved HITL cq-1 (merge-sequencing gate) with a two-tier
model: automated draft->ready on upstream PR merge for plain merge
ordering; HITL-resolved holds for beyond-merge-state conditions
(release/publish waits, version pinning, genuine development blocks).
Added HITL Resolution section with planner-facing consequences; updated
design recommendation #2, hard part #2, AC-6, and the human summary's
hard-bit #1. No new HITL decisions induced.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reverts 3393-analysis-human.md byte-exact to the simplifier's e88c16d
rendering, restoring (1) the per-repo house-rules bullet (design rec #5
/ AC-7 rendering) and (2) the simplifier's hard-bit #1, which keeps the
cq-1 resolution's development-blocks element. Root cause: my v3 rebase
conflict resolution used 'git checkout --theirs', which in a rebase
selects the replayed (stale) commit, overwriting the simplifier's
concurrent v3. 3393-analysis.md is deliberately untouched (reviewers:
ACK-ready). BRC memory updated with the ownership rule.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Hard-bit #3: owner/repo re-key is decided; same-name rejection ruled out;
  prohibitive fan-out returns to the operator, never a silent fallback.
- New 'Where decisions stand' section: all design questions settled and
  binding; only a new operator decision reopens them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ngs as binding

Design questions section retitled to OPERATOR RULINGS (binding, same
standing as cq-1): lazy-per-repo work branches/context PRs, per-slice-repo
test-gate/reviewer-diff scoping, primary-repo naming + per-repo status
surfaces, and per-repo conventions all RATIFIED (substance unchanged,
status upgraded from recommendation). New ruling #6: worktree map MUST be
re-keyed by full owner/repo; reject-same-name-sets is forbidden;
prohibitive fan-out at plan time is a new HITL, never a silent fallback.
Consistency-only knock-ons: correction #2 advice sentence points to the
ruling (facts unchanged), AC-2 drops the same-name-rejection alternative,
AC-4 requires owner/repo keying. cq-1 fold-in and grounding facts
untouched; human summary untouched (simplifier-owned).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…g bullet

Operator ruling (4) is 'first in list unless explicitly flagged'; the bullet
had stated the stricter first-in-list-only rule (reviewer_refine NACK).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…t, repos[0] de-collapse, per-repo PR routing, cq-1 merge gate

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
7 risks; verdict PROCEED_WITH_MITIGATIONS (MEDIUM-HIGH). Load-bearing:
R1 Contract has no repo dimension (Slice.repo absent=>primary unresolvable),
R2 EGG_PIPELINE_REPO is a hard-required overseer consumer (collapse != deletion),
R3 cq-1 merge-poll terminal/failure states unspecified.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Renders task_planner v1 (d067323): fixed-order six-step chain, the two
submission safety checks, the three collapse-site fixes with owner/repo
re-keying, per-repo PR routing, the cq-1 two-tier hold, and per-repo gate
scoping — with the N=1 no-behavior-change guarantee stated plainly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…k_analyst R1 NACK)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
egg and others added 11 commits July 2, 2026 19:47
…r_plan R3 NACK)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator re-materialization of the cq-4 resolution (option 1). The
original materialization (task-4-4) was lost when the phase restart
re-synced the pipeline worktree contract from a stale origin copy;
slice-4 consequently converged structural-only. slice-7 (dep: slice-6)
carries the wiring so coordinated cross-repo PRs work end-to-end.
@jwbron
jwbron merged commit 70d730b into main Jul 2, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant