Add SDLC pipeline architecture guide

[james-in-a-box]

Add comprehensive documentation for the SDLC pipeline architecture

Replaces the original spec and implementation plan documents with a
single architecture guide that documents the actual implemented system:

• Guiding principles: Structural enforcement, contract-as-code,
  worker-reviewer separation, human-in-the-loop at critical points
• Pipeline architecture: Four-phase flow (refine → plan → implement → PR)
  with phase-based operation filtering
• Contract system: JSON schema, role-based field ownership, audit logging
• Implement→review cycle: Cyclic pattern with context window isolation
• Circuit breaker: Thresholds, escalation, and recovery
• HITL decisions: Checkbox-based UI with 30-second debounce
• Failure handling: Rate limits, timeouts, checkpointing
• Implementation reference: Key files, CLI commands, triggering

Issue: #133

Test plan:

• Verify document renders correctly in GitHub
• Review completeness against actual implementation
• Confirm links to related docs work

Authored-by: egg

[jwbron]

@james-in-a-box move forward with phase 1 of the plan. Open a new pr with the implementation.

[james-in-a-box]

egg finished successfully. View run logs

[james-in-a-box]

Review: Specification and Implementation Plan Documents

This PR extracts the specification and implementation plan from the closed PR #184 so they can be reviewed independently before implementation. The documents are well-structured and address a real problem (preventing incidents like #202 where the agent bypassed the planning phase).

Strengths

1. Clear problem framing: The spec correctly identifies that prompt-level instructions aren't reliable constraints and proposes infrastructure-level enforcement via the gateway.

2. Comprehensive architecture: The multi-stage pipeline with separate worker/reviewer contexts and role-based field access is a solid design pattern.

3. Good alignment with existing patterns: The reusable workflow pattern (.github/workflows/sdlc-review.yml) mirrors the existing reusable-review.yml structure.

4. Addresses #202 directly: Phase-based operation blocking (git push blocked during refine/plan phases) would have prevented that incident.

Issues to Address

1. Duplicate "Part 4" headings in the spec

133-structurally-enforced-checkpoints.md has two sections labeled "Part 4":

• Line 110: "Part 4: Task Collection and Role-Based Enforcement"
• Line 199: "Part 4: Human-in-the-Loop Decision System"

One of these should be renumbered (likely the HITL section should be Part 5, and subsequent parts shifted).

2. Contract storage location inconsistency

The spec mentions two different locations for contract storage:

• Part 1 (line 23): "Contract-as-Code: All state is stored in .egg/contracts/{issue-number}.json and committed to the branch"
• Implementation plan (1.2 loader.py): "Load/save contract from .egg/contracts/{issue}.json"

But the .egg/ directory doesn't currently exist in the repo. The implementation plan should note this directory needs to be created, or clarify if this is intentional for the contracts to live only on feature branches.

3. review_cycles field location is inconsistent

In the contract schema (around line 127):

"phases": [{
  "review_cycles": 0,
  "max_cycles": 3,
  ...
}]

But the circuit breaker section (Part 5, around line 292) shows review_cycles on individual tasks:

"tasks": [{
  "review_cycles": 0,
  "max_cycles": 3,
  "escalated": false
}]

The spec should clarify whether cycle tracking is per-phase or per-task (the implementation plan suggests per-task, which makes more sense for granular escalation).

4. Missing review_feedback in contract schema

The role-based field access table (line 174) says reviewers can modify phases[].review_feedback[], but this field isn't present in the contract schema example (line 117-148). Add it to the schema.

5. Implementation plan references missing sandbox/ directory structure

The implementation plan references:

• sandbox/egg_lib/contract_cli.py (3.1)
• sandbox/.claude/rules/contract.md (3.2)
• sandbox/.claude/reviewer-rules.md (Phase 2 in spec)

But the current repo doesn't have a sandbox/ directory. Consider clarifying if this is a new directory to be created or should reference existing paths.

Minor Suggestions

1. The 30-second debounce for HITL checkboxes is reasonable, but consider making it configurable (e.g., HITL_DEBOUNCE_SECONDS env var) for testing purposes.

2. The implementation plan mentions creating templates at docs/templates/analysis.md and docs/templates/plan.md (3.3), but doesn't specify the full template content. The spec has good template examples (lines 43-90) - consider referencing those explicitly.

3. The success criteria (implementation plan lines 228-234) are good but could include a criterion for "Gateway correctly blocks operations per phase" as a separate testable item.

Verdict

The documents are well-thought-out and provide a solid foundation for implementation. The issues above are mostly about internal consistency and should be straightforward to address.

Recommendation: Address the duplicate Part 4 heading and the review_cycles/review_feedback schema inconsistencies before merging. The other items can be addressed during implementation.

— Authored by egg

[james-in-a-box]

egg review completed. View run logs

[jwbron]

@james-in-a-box address the above feedback

[james-in-a-box]

Addressed the review feedback:

1. Duplicate Part 4 headings — Fixed. Renumbered sections sequentially (Part 1–7). The document now goes: Part 1 (Pipeline Architecture), Part 2 (Document Standards), Part 3 (Task Collection), Part 4 (HITL), Part 5 (Circuit Breaker), Part 6 (Resume), Part 7 (Reviewer Workflow).

2. Contract storage location — Added note in implementation plan (1.2) clarifying that .egg/contracts/ is created per-branch during pipeline initialization and contracts are committed to the feature branch, not main.

3. review_cycles field location — Fixed. Moved review_cycles, max_cycles, escalated, and escalation_reason from the phase level to the task level in the contract schema. This aligns with the circuit breaker section and enables granular per-task escalation.

4. Missing review_feedback in schema — Fixed. Added review_feedback: [] field to the phase object in the contract schema.

5. sandbox/ directory references — Added note in implementation plan (Phase 3) clarifying that sandbox/ is the agent's working environment within the container, created during container initialization.

— Authored by egg

[james-in-a-box]

egg finished successfully. View run logs

[james-in-a-box]

Review: Spec and Implementation Plan for Issue #133

Overall this is a well-structured specification and implementation plan that addresses a real architectural need demonstrated by the #202 incident (agent skipping the planning phase). The separation of spec from implementation is a good approach given the scope.

Strengths

1. Clear problem statement: The spec directly addresses the #202 incident where the agent bypassed the planning phase and implemented a full solution prematurely.

2. Layered enforcement model: Role-based access control at the gateway level (not prompt-level) aligns with the design philosophy in agent-mode-design.md: "Prompt-level instructions aren't security controls."

3. Document-first approach: The analysis/plan document templates provide structure for human review and create artifacts that can be evaluated independently of the contract JSON state.

4. Circuit breaker pattern: The implement→review loop with escalation thresholds (3 per-task, 10 total) prevents runaway cycles while still allowing automation to self-correct.

Issues to Address

1. Contract storage location creates branching complexity

The spec says contracts live at .egg/contracts/{issue}.json and are "committed to the feature branch and not to main." However:

• The plan says the .egg/ directory is "created per-branch during pipeline initialization" (Phase 1)
• But Phase 1.2 says to create shared/egg_contracts/ for the library code

This conflates two different things: (1) the contract library (shared code that belongs on main) vs (2) contract instances (per-issue state on feature branches). The spec should clarify this distinction more explicitly. Consider renaming the per-branch directory to .egg-state/ or similar to avoid confusion with the schema directory .egg/schemas/.

2. Gateway role determination is underspecified

The spec says:

"The gateway reads role from workflow context (GitHub Actions job metadata), not environment variables set by the agent"

But the implementation plan doesn't detail how this works. The gateway currently runs as a sidecar (see gateway/gateway.py:1-24). How does it access "GitHub Actions workflow context"? Options include:

• Reading GITHUB_ACTIONS environment variables in the container
• Receiving role as a header in API requests (but then the agent could spoof it)
• A separate authentication flow per-job

This is a critical security boundary and needs explicit design.

3. Plan parser complexity

Phase 3.5 introduces plan_parser.py to "parse plan document to extract phases and tasks." Parsing markdown for structured task extraction is error-prone. Consider:

• Using a more structured format for the tasks section (YAML front matter, or a tasks.json alongside the plan.md)
• Providing explicit task ID format requirements (e.g., [TASK-1] markers)
• Handling parse failures gracefully (don't silently drop tasks)

4. The 30-second debounce may cause confusion

The debounce period for HITL checkboxes means:

• Human checks a box → waits up to 30 seconds → pipeline resumes
• If they change their mind and uncheck within 30s, the action is cancelled

This UX should be communicated clearly in the bot comment, e.g., "Your selection will be confirmed in 30 seconds" with a countdown or status indicator.

5. Missing error handling for external failures

The spec covers circuit breaker for task cycles but doesn't address:

• GitHub API rate limits during pipeline execution
• Network failures between gateway and GitHub
• Workflow timeout limits (GitHub Actions has a 6-hour max)

Minor Observations

• The reusable-review.yml workflow already exists (line 1-80 of the current file). The plan proposes sdlc-review.yml as a new workflow. Consider whether these should be unified or if the naming could cause confusion.

• The spec references sandbox/.claude/rules/contract.md but the current structure is sandbox/.claude/rules/ with multiple rule files. This is fine, just noting consistency.

• The "Test Strategy" table at the end (Unit/Integration/E2E) is helpful but brief. Consider expanding what "E2E: Full pipeline" actually tests — a full issue-to-merged-PR cycle would be expensive to run on every change.

Verdict

The spec and plan are well-thought-out and ready for implementation with the caveats above. The role determination mechanism (issue #2) is the most important gap to clarify before implementation begins.

— Authored by egg

[james-in-a-box]

egg review completed. View run logs

[jwbron]

@james-in-a-box address this feedback #215 (review)

[james-in-a-box]

Addressed the review feedback:

1. Contract storage location creates branching complexity — Fixed. Renamed .egg/contracts/ to .egg-state/contracts/ and added explicit "Directory Structure" section in the spec and a "Directory distinction" note in the implementation plan. The .egg/ directory now exclusively holds the schema library (committed to main), while .egg-state/ holds per-issue contract instances (committed only to feature branches).

2. Gateway role determination is underspecified — Fixed. Added "Gateway Role Determination Mechanism" section with explicit design:

• Uses GitHub Actions OIDC tokens (signed JWTs) for cryptographic role verification
• Gateway validates token signature against GitHub's OIDC public keys
• Role extracted from job_workflow_ref claim via job name convention (implement* → implementer, review* → reviewer)
• Fallback for local development via EGG_GATEWAY_ADMIN_TOKEN
• Added gateway/oidc_auth.py and gateway/role_resolver.py to implementation plan

3. Plan parser complexity — Fixed. Added explicit requirements:

• Task ID format: [TASK-{phase}-{number}] markers (e.g., [TASK-1-1])
• Parse failure handling: placeholder tasks created for unparseable phases, structured errors pause pipeline
• Optional YAML front matter for unambiguous parsing
• Updated plan template in spec to show task ID format

4. The 30-second debounce may cause confusion — Fixed. Added debounce UX communication:

• Initial notice: "Your selection will be confirmed in 30 seconds"
• Countdown updates every 5 seconds while waiting
• Confirmation message when debounce expires

5. Missing error handling for external failures — Fixed. Added "External Failure Handling" section covering:

• GitHub API rate limits (sleep until reset)
• Network failures (exponential backoff, 3 retries)
• GitHub Actions timeout (checkpoint at T-10 minutes)
• Gateway unavailable (retry with backoff, escalate if down >5 minutes)
• Added shared/egg_contracts/resilience.py to implementation plan

— Authored by egg

[james-in-a-box]

egg finished successfully. View run logs

[james-in-a-box]

Summary

This PR extracts the specification and implementation plan from the closed PR #184, allowing the documents to be reviewed and approved before implementation begins. The scope is appropriate: documentation only, no code changes.

Both documents are well-structured and technically sound. The specification addresses a real problem (issue #202 - agent skipping the planning phase) with infrastructure-level enforcement rather than prompt-level instructions, which aligns with the project's design philosophy in docs/guides/agent-mode-design.md.

Feedback

Minor Clarifications (Non-blocking)

1. Plan parser task ID format (implementation-plan.md:159-162) — The regex pattern for parsing task IDs is documented, but consider documenting the behavior when a plan uses inconsistent numbering (e.g., [TASK-1-1] followed by [TASK-1-5]). The parse failure handling (lines 163-166) mentions warnings but not whether gaps in numbering are intentional or should trigger warnings.

2. Phase permissions schema location — The spec mentions .egg/schemas/phase-permissions.schema.json but the implementation plan shows this created in Phase 2.1. Meanwhile Phase 1.1 creates contract.schema.json. Consider noting in Phase 2.1 that this is the second schema file in the same directory.

3. OIDC token audience (spec:57) — The implementation plan references audience=egg-gateway but the spec mentions validating audience in role_resolver.py. These should be consistent — recommend documenting the expected audience value in the spec under the gateway role determination section.

4. Resilience.py placement (implementation-plan:262-265) — The rate limit and timeout handling is described in Phase 5 but depends on gateway modifications described in the same phase. Consider noting the dependency order (gateway changes first, then client-side resilience).

Architectural Observations (for future implementation)

The deep review findings from PR #184 (quoted in comments) identified several issues that should be addressed during implementation:

• Role trust model: The spec correctly states the gateway should read role from workflow context, not agent env vars. Implementation must avoid the privilege escalation path identified in PR #184.
• Path format consistency: Contract mutations use dot-notation paths (phases.*.tasks.task-1.commit). The spec should clarify the path format expected by the API.
• Recursive workflow depth: GitHub caps reusable workflow nesting at 4 levels. The implement→review loop design should account for this constraint.

These are implementation concerns, not spec issues — mentioning them here for visibility during future implementation work.

Conclusion

The documents provide a clear, executable specification for implementing phase-based operation restrictions and role enforcement. The design addresses the root cause of incident #202 with infrastructure-level controls rather than prompt-level instructions.

No blocking issues found. Ready for human review and merge.

— Authored by egg

[james-in-a-box]

egg review completed. View run logs

[jwbron]

@james-in-a-box phase 1 has been completed. Make a new branch off main and implement phase 2. Open a new pr with the implementation.

[james-in-a-box]

egg finished successfully. View run logs

[jwbron]

@james-in-a-box phase 2 is complete. Create a new branch from main and implement phase 3

[james-in-a-box]

egg finished successfully. View run logs

[jwbron]

@james-in-a-box phase 3 is complete. Create a new branch from main and implement phase 4

[james-in-a-box]

egg finished successfully. View run logs

[jwbron]

@james-in-a-box phase 4 is complete. Create a new branch from main and implement phase 5

[james-in-a-box]

egg finished successfully. View run logs

[jwbron]

@james-in-a-box phase 5 is complete. Create a new branch from main and implement phase 6

[james-in-a-box]

egg finished successfully. View run logs

[jwbron]

@james-in-a-box let's convert this to documentation for the SDLC lifecycle. Create a single document based on 133-structurally-enforced-checkpoints.md that outlines the architecture and workflow, as well as the guiding principles. Get rid of the old documents.

Pull in main before you start. Ensure the actual implementation details are reflected in the doc. Push your changes to this branch and update the title and description.

[james-in-a-box]

Converted the spec documents into a single SDLC pipeline architecture guide.

Changes:

• Removed docs/issues/133-implementation-plan.md
• Removed docs/issues/133-structurally-enforced-checkpoints.md
• Created docs/guides/sdlc-pipeline.md

The new document covers:

1. Guiding principles (structural enforcement, contract-as-code, worker-reviewer separation, HITL)
2. Pipeline architecture diagram and phase descriptions
3. Phase-based operation filtering
4. Contract system (schema, directories, role-based field ownership)
5. Implement→review cycle workflow with context window isolation
6. Circuit breaker thresholds and escalation handling
7. HITL checkbox-based decisions with debounce mechanism
8. External failure handling (rate limits, timeouts, checkpointing)
9. Document standards (analysis and plan templates)
10. Implementation reference (key files, CLI commands)

The documentation reflects the actual implementation in main, including references to specific files like shared/egg_contracts/, .github/workflows/sdlc-pipeline.yml, and the JSON schema at .egg/schemas/contract.schema.json.

— Authored by egg

[james-in-a-box]

egg finished successfully. View run logs

[james-in-a-box]

Review: SDLC Pipeline Architecture Guide

This documentation PR consolidates the spec and implementation plan into a single architecture guide. After reviewing the diff against the actual implementation in shared/egg_contracts/, .egg/, and .github/workflows/, I've verified the documentation is accurate and comprehensive.

Verification Summary

Accuracy checks performed:

1. Contract schema — The documented schema structure matches .egg/schemas/contract.schema.json. Task and phase fields, review_cycles, max_cycles, escalated, review_feedback are all present.

2. Role-based field ownership — The FIELD_OWNERSHIP table in the doc matches shared/egg_contracts/roles.py. Field paths use the correct dot-notation format.

3. Phase permissions — The phase-based operation filtering matches .egg/phase-permissions.json. The doc correctly describes refine/plan blocking git push and gh pr create, implement blocking gh pr create, etc.

4. Circuit breaker — The thresholds (3 per-task, 10 total pipeline) and escalation logic match shared/egg_contracts/circuit_breaker.py.

5. Directory structure — The doc correctly distinguishes:

   • .egg/schemas/ — Schema definitions (on main)
   • .egg/phase-permissions.json — Phase restrictions (on main)
   • .egg-state/contracts/ — Per-issue contracts (feature branches only)

6. Workflow structure — The implement → review → loop pattern matches .github/workflows/sdlc-pipeline.yml. The job structure, EGG_AGENT_ROLE env var, and 6-hour timeout are documented correctly.

7. HITL checkboxes — The debounce mechanism (30-second default) and checkbox categories match shared/egg_contracts/hitl.py.

8. Cross-references — Verified the "See also" links exist:

   • docs/templates/analysis.md ✓
   • docs/templates/plan.md ✓
   • docs/guides/github-automation.md ✓

Minor Observations (Non-blocking)

1. get_escalation_summary signature (line ~295) — The doc shows a simplified return structure. The actual implementation in circuit_breaker.py returns additional fields (incomplete_tasks, stuck_phases, requires_intervention). This is fine for documentation purposes—showing the essential fields is clearer.

2. Task ID pattern — The doc mentions [TASK-{phase}-{number}] format for plan documents, but the JSON schema uses task-N pattern (^task-[0-9]+(-[0-9]+)?$). These are different contexts (markdown parsing vs contract IDs), but worth noting the distinction.

3. Contract CLI commands — The doc lists egg-contract CLI commands but the implementation files show this is shared/egg_contracts/ Python code. The CLI wrapper that exposes these as shell commands isn't shown. Presumed to exist as part of the container setup.

Conclusion

The documentation accurately reflects the implemented system. It provides a clear, comprehensive guide to the SDLC pipeline architecture with correct references to actual code paths and configurations.

No blocking issues found. Ready for human review and merge.

— Authored by egg

[james-in-a-box]

egg review completed. View run logs

[jwbron]

@james-in-a-box Actually, move the document from here to #265

[james-in-a-box]

Moved docs/guides/sdlc-pipeline.md to PR #265 as requested.

The document has been added to the egg/issue-133-phase-6 branch and is now part of the Phase 6 PR.

— Authored by egg

[james-in-a-box]

egg finished successfully. View run logs