From 4c257510a483c96d915ac58b6dd1c9c729976089 Mon Sep 17 00:00:00 2001 From: jwbron <8340608+jwbron@users.noreply.github.com> Date: Sat, 14 Feb 2026 03:04:31 +0000 Subject: [PATCH 1/3] docs: Document deployment validation check system Update documentation to reflect deployment validation feature added in #653: - Add deployment_check.py, deployment.py, and devserver.py to STRUCTURE.md - Document deployment validation in architecture README and SDLC guide - Add comprehensive devserver management section to orchestrator architecture - Update phase default checks to include deployment validation (optional) Authored-by: egg --- docs/architecture/README.md | 3 +- docs/architecture/orchestrator.md | 82 +++++++++++++++++++++++++++++++ docs/development/STRUCTURE.md | 13 ++++- docs/guides/sdlc-pipeline.md | 53 ++++++++++++++++++++ 4 files changed, 148 insertions(+), 3 deletions(-) diff --git a/docs/architecture/README.md b/docs/architecture/README.md index 40e92f8825..72002c8403 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -145,12 +145,13 @@ Each SDLC phase can have configurable automated checks that run before completio - `merge-conflict`: Detects merge conflicts with base branch - `draft-validation`: Validates refine phase draft documents - `plan-yaml`: Validates plan phase YAML appendix +- `deployment`: Validates changes against locally running devserver (DinD) - `fixer`: Auto-fixes certain check failures when possible **Phase defaults:** - Refine: draft validation - Plan: YAML validation -- Implement: merge conflict, lint (auto-retry), tests, auto-fixer +- Implement: merge conflict, lint (auto-retry), tests, deployment (optional, auto-retry), auto-fixer - PR: none Contracts can override phase defaults via the `phase_configs` field, allowing per-issue check customization. diff --git a/docs/architecture/orchestrator.md b/docs/architecture/orchestrator.md index 33ac349c5d..075fa5c2ca 100644 --- a/docs/architecture/orchestrator.md +++ b/docs/architecture/orchestrator.md @@ -209,6 +209,9 @@ Fixed IPs: - `GET /pipelines/stream` - Unified SSE stream for all active pipelines (supports `?ascii=true`, `?active_only=false`, `?full_dag=true`) - `POST /pipelines/{id}/signal` - Sandbox signals (complete, progress, error) - `GET /pipelines/{id}/decisions` - HITL decision queue +- `POST /pipelines/{id}/checks/devserver/start` - Start devserver for deployment validation +- `GET /pipelines/{id}/checks/devserver/status` - Poll devserver status +- `POST /pipelines/{id}/checks/devserver/teardown` - Tear down devserver **CLI Access:** The `egg-orch` CLI (`sandbox/bin/egg-orch`) provides command-line access to all orchestrator API endpoints. Available in sandbox containers for agent use, or can be run from the host with appropriate environment variables. See the [README CLI Reference](../../README.md#egg-orch-cli) for command details. @@ -220,6 +223,85 @@ The `egg-orch` CLI (`sandbox/bin/egg-orch`) provides command-line access to all 3. **Gateway → Orchestrator**: Health check (optional) 4. **Orchestrator → GitHub**: Webhook responses, PR updates +## Devserver Management (Deployment Validation) + +The orchestrator manages Docker-in-Docker (DinD) devserver stacks during deployment validation checks. This enables testing agent-modified code against locally running services before merge. + +### Architecture + +**Orchestrator responsibilities:** +- Extract `docker-compose.yml` from committed state (before agent changes) +- Generate override mounts for agent-modified services +- Create isolated Docker network (`egg-check-{pipeline_id}`) +- Start devserver stack with resource limits +- Provide status polling endpoints for sandbox check runner +- Tear down stack after validation completes + +**Sandbox check runner responsibilities:** +- Signal orchestrator to start devserver via REST API +- Poll status until healthy or timeout +- Run health checks against service endpoints +- Run validation tests from `.egg/deployment.yml` +- Signal teardown + +### Security Properties + +**Network isolation:** +- Devserver containers run in dedicated `egg-check-{pipeline_id}` bridge network +- No internet access (internal-only, no gateway, no DNS) +- Services can only communicate within the isolated network +- Sandbox checker makes HTTP requests from outside the devserver network + +**Resource limits (per container):** +- CPU: 1.0 core +- Memory: 512 MB +- PIDs: 256 (prevents fork bombs) +- Hard timeout: 5 minutes for entire lifecycle + +**Credential safety:** +- No cloud credentials or production secrets injected +- Environment variables scanned for suspicious patterns (AWS_*, *_SECRET_KEY, *_TOKEN, etc.) +- Only target repo code is mounted (no access to egg internals) + +### Configuration + +Target repositories opt in by providing `.egg/deployment.yml`: + +```yaml +version: "1" +compose_file: "docker-compose.yml" +services: + - source_dir: "services/api" + service_name: "api" + container_mount_path: "/app" + health_endpoint: "http://api:8000/health" +tests: + - name: "API smoke test" + url: "http://api:8000/users" + method: "GET" + expected_status: 200 +``` + +See `shared/egg_contracts/deployment.py` for full schema. + +### API Flow + +1. **Start**: Sandbox calls `POST /api/v1/pipelines/{id}/checks/devserver/start` + - Orchestrator extracts compose config, generates overrides, starts stack + - Returns immediately with `{"status": "starting"}` + +2. **Poll**: Sandbox polls `GET /api/v1/pipelines/{id}/checks/devserver/status` + - Returns `{"status": "starting" | "healthy" | "unhealthy" | "error"}` + - Includes service IPs and ports when healthy + +3. **Validate**: Sandbox runs health checks and tests against service endpoints + +4. **Teardown**: Sandbox calls `POST /api/v1/pipelines/{id}/checks/devserver/teardown` + - Orchestrator stops containers, removes network + - Returns `{"status": "stopped"}` + +See `orchestrator/devserver.py` and `orchestrator/routes/checks.py` for implementation. + ## Sandbox Lifecycle ### Orchestrator Mode Detection diff --git a/docs/development/STRUCTURE.md b/docs/development/STRUCTURE.md index 68c47b324e..966328b689 100644 --- a/docs/development/STRUCTURE.md +++ b/docs/development/STRUCTURE.md @@ -83,6 +83,7 @@ orchestrator/ ├── dag_visualizer.py # ASCII DAG visualization for pipeline status ├── decision_queue.py # HITL decision queue ├── decision_timeout.py # Decision timeout handling +├── devserver.py # Devserver lifecycle manager for deployment validation (DinD) ├── dispatch.py # Agent dispatch logic ├── docker_client.py # Docker API client ├── events.py # Event bus for pipeline events @@ -99,6 +100,7 @@ orchestrator/ ├── unified_sse.py # Unified SSE stream for all pipelines ├── webhooks.py # GitHub webhook handlers ├── routes/ # API route handlers +│ ├── checks.py # Deployment validation check endpoints │ ├── containers.py # Container management endpoints │ ├── decisions.py # HITL decision endpoints │ ├── health.py # Health check endpoints @@ -149,13 +151,14 @@ sandbox/ ``` shared/ ├── egg_config/ # Configuration utilities -│ ├── constants.py # Centralized constants (ports, networks, container names) +│ ├── constants.py # Centralized constants (ports, networks, container names, devserver resource limits) │ └── validators.py # Validation functions (URLs, emails, tokens, check commands) ├── egg_container/ # Shared container-launch config builder │ └── __init__.py # build_sandbox_config(), build_sandbox_docker_cmd(), git_shadow_mounts(), to_dockerpy_kwargs() ├── egg_contracts/ # SDLC contract models, plan parser, role-based validation, HITL, feedback, phase checks, multi-agent orchestration, checkpoints │ ├── models.py # Pydantic models including CheckDefinition, CheckResult, PhaseConfig, AgentExecutionModel │ ├── phase_defaults.py # Default check configurations per SDLC phase +│ ├── deployment.py # Deployment validation configuration models (.egg/deployment.yml) │ ├── agent_roles.py # Multi-agent role definitions (Coder, Tester, Documenter, Integrator) │ ├── orchestrator.py # Multi-agent orchestration dispatch logic │ ├── orchestration.py # Agent execution state management @@ -197,6 +200,9 @@ integration_tests/ ├── test_policy_enforcement.py # Policy enforcement tests ├── test_rate_limiting.py # Rate limiting tests ├── test_stack_lifecycle.py # Container lifecycle tests +├── deployment_validation/ # Deployment validation integration tests +│ ├── __init__.py +│ └── test_deployment_check_e2e.py # End-to-end devserver lifecycle tests ├── local_pipeline/ # Local orchestrator integration tests │ ├── conftest.py # Local pipeline test fixtures │ ├── docker-compose.yml # Orchestrator test environment @@ -227,11 +233,13 @@ tests/ │ ├── test_contract_cli.py # Contract CLI tests │ └── ... ├── scripts/ -│ └── test_checks.py # Check script framework tests +│ ├── test_checks.py # Check script framework tests +│ └── test_deployment_check.py # Deployment check unit tests ├── shared/ │ └── egg_contracts/ │ ├── test_models.py # Contract model tests including check models │ ├── test_phase_defaults.py # Phase default configuration tests +│ ├── test_deployment_config.py # Deployment configuration tests │ ├── test_agent_recovery.py # Agent recovery and circuit breaker tests │ ├── test_checkpoints.py # Checkpoint model tests │ ├── test_redactor.py # Redactor tests for sensitive data masking @@ -295,6 +303,7 @@ Key workflows for the SDLC pipeline (see `.github/workflows/` for complete list) │ ├── base.py # CheckRunner base class │ ├── run_check.py # Check execution entry point │ ├── check_fixer.py # Auto-fix check runner + │ ├── deployment_check.py # Deployment validation (DinD devserver) │ ├── draft_validation_check.py # Draft document validation │ ├── lint_check.py # Lint check runner │ ├── merge_conflict_check.py # Merge conflict detection diff --git a/docs/guides/sdlc-pipeline.md b/docs/guides/sdlc-pipeline.md index ae6ca34b99..8461615f6b 100644 --- a/docs/guides/sdlc-pipeline.md +++ b/docs/guides/sdlc-pipeline.md @@ -797,6 +797,7 @@ If not configured, the checker falls back to auto-discovery (scanning for Makefi | **Merge Conflict** | `check-merge-conflict` | Detects conflicts with base branch | No | | **Lint** | `check-lint` | Runs `make lint` if available | Yes | | **Test** | `check-test` | Runs `make test` or pytest | No | +| **Deployment Validation** | `check-deployment` | Validates changes against locally running devserver (opt-in via `.egg/deployment.yml`) | No | | **Auto-Fixer** | `check-fixer` | Attempts to auto-fix failed checks | N/A | ### Phase Default Configurations @@ -813,11 +814,63 @@ Default checks for each phase are defined in `shared/egg_contracts/phase_default - Merge conflict check (required) - Lint check (required, 1 retry) - Test check (required) +- Deployment validation (optional, 1 retry, requires `.egg/deployment.yml`) - Auto-fixer (optional) **PR phase:** - No checks +### Deployment Validation + +The deployment validation check (`check-deployment`) runs agent-modified code against a locally running devserver to catch integration issues before merge. This check is **opt-in** and requires target repositories to provide a `.egg/deployment.yml` configuration file. + +**How it works:** + +1. The orchestrator extracts the `docker-compose.yml` from the committed state (before agent changes) +2. Generates override mounts for agent-modified services based on service-to-source mappings +3. Starts the devserver stack in an isolated Docker network with resource limits +4. The sandbox check runner polls health endpoints and runs validation tests +5. The orchestrator tears down the stack after validation completes + +**Configuration (`.egg/deployment.yml`):** + +```yaml +version: "1" +compose_file: "docker-compose.yml" # Path relative to repo root +services: + - source_dir: "services/api" # Source directory (agent changes) + service_name: "api" # docker-compose service name + container_mount_path: "/app" # Mount path inside container + health_endpoint: "http://api:8000/health" # Health check URL +tests: + - name: "API smoke test" + url: "http://api:8000/users" + method: "GET" + expected_status: 200 +``` + +**Security guarantees:** + +- Devserver containers run in an isolated Docker network (no internet, no access to other containers) +- Resource limits prevent exhaustion attacks (CPU, memory, PIDs) +- Hard timeout of 5 minutes for the entire devserver lifecycle +- No cloud credentials or production secrets are injected +- Suspicious environment variables (AWS_*, *_SECRET_KEY, etc.) are rejected + +**When to use:** + +- Microservices with docker-compose devserver setups +- Integration testing that requires multiple services running +- Validating API contracts between services + +**When not to use:** + +- Projects without docker-compose devserver infrastructure +- Simple single-service applications (use `make test` instead) +- Projects where devserver setup is complex or requires external dependencies + +The check is optional by default and will skip if `.egg/deployment.yml` is not present. When enabled, it runs with 1 retry on failure. + ### Customizing Phase Checks Contracts can override phase defaults via the `phase_configs` field: From 67644c154a58fde2761b7f456fafa0ac9340d144 Mon Sep 17 00:00:00 2001 From: "egg-reviewer[bot]" <261018737+egg-reviewer[bot]@users.noreply.github.com> Date: Sat, 14 Feb 2026 03:12:15 +0000 Subject: [PATCH 2/3] Fix docs: correct API paths, YAML schema, and credential patterns MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Fix API endpoint paths: checks/devserver → deployment-check to match actual routes in orchestrator/routes/checks.py - Fix YAML config examples to match DeploymentConfig schema: remove version field, rename tests→validation_tests, rename name→description, url→path, add service field, add health_endpoints top-level dict, remove health_endpoint from services - Expand credential safety pattern list to include all patterns from the actual regex: *_API_KEY, *_ACCESS_KEY, *_PASSWORD, *_CREDENTIALS, plus GCP_*, AZURE_*, GOOGLE_CLOUD_* prefixes --- docs/architecture/orchestrator.md | 25 +++++++++++++------------ docs/guides/sdlc-pipeline.md | 13 +++++++------ 2 files changed, 20 insertions(+), 18 deletions(-) diff --git a/docs/architecture/orchestrator.md b/docs/architecture/orchestrator.md index 075fa5c2ca..9b0eafbf68 100644 --- a/docs/architecture/orchestrator.md +++ b/docs/architecture/orchestrator.md @@ -209,9 +209,9 @@ Fixed IPs: - `GET /pipelines/stream` - Unified SSE stream for all active pipelines (supports `?ascii=true`, `?active_only=false`, `?full_dag=true`) - `POST /pipelines/{id}/signal` - Sandbox signals (complete, progress, error) - `GET /pipelines/{id}/decisions` - HITL decision queue -- `POST /pipelines/{id}/checks/devserver/start` - Start devserver for deployment validation -- `GET /pipelines/{id}/checks/devserver/status` - Poll devserver status -- `POST /pipelines/{id}/checks/devserver/teardown` - Tear down devserver +- `POST /pipelines/{id}/deployment-check/start` - Start devserver for deployment validation +- `GET /pipelines/{id}/deployment-check/status` - Poll devserver status +- `POST /pipelines/{id}/deployment-check/teardown` - Tear down devserver **CLI Access:** The `egg-orch` CLI (`sandbox/bin/egg-orch`) provides command-line access to all orchestrator API endpoints. Available in sandbox containers for agent use, or can be run from the host with appropriate environment variables. See the [README CLI Reference](../../README.md#egg-orch-cli) for command details. @@ -260,7 +260,7 @@ The orchestrator manages Docker-in-Docker (DinD) devserver stacks during deploym **Credential safety:** - No cloud credentials or production secrets injected -- Environment variables scanned for suspicious patterns (AWS_*, *_SECRET_KEY, *_TOKEN, etc.) +- Environment variables scanned for suspicious patterns (AWS_*, GCP_*, AZURE_*, GOOGLE_CLOUD_*, *_SECRET_KEY, *_API_KEY, *_ACCESS_KEY, *_TOKEN, *_PASSWORD, *_CREDENTIALS) - Only target repo code is mounted (no access to egg internals) ### Configuration @@ -268,35 +268,36 @@ The orchestrator manages Docker-in-Docker (DinD) devserver stacks during deploym Target repositories opt in by providing `.egg/deployment.yml`: ```yaml -version: "1" compose_file: "docker-compose.yml" services: - source_dir: "services/api" service_name: "api" container_mount_path: "/app" - health_endpoint: "http://api:8000/health" -tests: - - name: "API smoke test" - url: "http://api:8000/users" +health_endpoints: + api: "/health" +validation_tests: + - service: "api" + path: "/users" method: "GET" expected_status: 200 + description: "API smoke test" ``` See `shared/egg_contracts/deployment.py` for full schema. ### API Flow -1. **Start**: Sandbox calls `POST /api/v1/pipelines/{id}/checks/devserver/start` +1. **Start**: Sandbox calls `POST /api/v1/pipelines/{id}/deployment-check/start` - Orchestrator extracts compose config, generates overrides, starts stack - Returns immediately with `{"status": "starting"}` -2. **Poll**: Sandbox polls `GET /api/v1/pipelines/{id}/checks/devserver/status` +2. **Poll**: Sandbox polls `GET /api/v1/pipelines/{id}/deployment-check/status` - Returns `{"status": "starting" | "healthy" | "unhealthy" | "error"}` - Includes service IPs and ports when healthy 3. **Validate**: Sandbox runs health checks and tests against service endpoints -4. **Teardown**: Sandbox calls `POST /api/v1/pipelines/{id}/checks/devserver/teardown` +4. **Teardown**: Sandbox calls `POST /api/v1/pipelines/{id}/deployment-check/teardown` - Orchestrator stops containers, removes network - Returns `{"status": "stopped"}` diff --git a/docs/guides/sdlc-pipeline.md b/docs/guides/sdlc-pipeline.md index 8461615f6b..877be00a95 100644 --- a/docs/guides/sdlc-pipeline.md +++ b/docs/guides/sdlc-pipeline.md @@ -835,18 +835,19 @@ The deployment validation check (`check-deployment`) runs agent-modified code ag **Configuration (`.egg/deployment.yml`):** ```yaml -version: "1" compose_file: "docker-compose.yml" # Path relative to repo root services: - source_dir: "services/api" # Source directory (agent changes) service_name: "api" # docker-compose service name container_mount_path: "/app" # Mount path inside container - health_endpoint: "http://api:8000/health" # Health check URL -tests: - - name: "API smoke test" - url: "http://api:8000/users" +health_endpoints: + api: "/health" # Service name → health check path +validation_tests: + - service: "api" # Target service name + path: "/users" # Request path method: "GET" expected_status: 200 + description: "API smoke test" ``` **Security guarantees:** @@ -855,7 +856,7 @@ tests: - Resource limits prevent exhaustion attacks (CPU, memory, PIDs) - Hard timeout of 5 minutes for the entire devserver lifecycle - No cloud credentials or production secrets are injected -- Suspicious environment variables (AWS_*, *_SECRET_KEY, etc.) are rejected +- Suspicious environment variables (AWS_*, GCP_*, AZURE_*, GOOGLE_CLOUD_*, *_SECRET_KEY, *_API_KEY, *_ACCESS_KEY, *_TOKEN, *_PASSWORD, *_CREDENTIALS) are rejected **When to use:** From 1712bfa5a3d9bd1a1184e07181d0f359cbe5487c Mon Sep 17 00:00:00 2001 From: "egg-reviewer[bot]" <261018737+egg-reviewer[bot]@users.noreply.github.com> Date: Sat, 14 Feb 2026 03:22:00 +0000 Subject: [PATCH 3/3] Fix check ordering in docs to match phase_defaults.py The implement phase check list in sdlc-pipeline.md and README.md listed deployment validation before auto-fixer, but phase_defaults.py defines check-fixer before check-deployment. Reorder docs to match the code. Authored-by: egg --- docs/architecture/README.md | 2 +- docs/guides/sdlc-pipeline.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/architecture/README.md b/docs/architecture/README.md index 72002c8403..97471c02f3 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -151,7 +151,7 @@ Each SDLC phase can have configurable automated checks that run before completio **Phase defaults:** - Refine: draft validation - Plan: YAML validation -- Implement: merge conflict, lint (auto-retry), tests, deployment (optional, auto-retry), auto-fixer +- Implement: merge conflict, lint (auto-retry), tests, auto-fixer, deployment (optional, auto-retry) - PR: none Contracts can override phase defaults via the `phase_configs` field, allowing per-issue check customization. diff --git a/docs/guides/sdlc-pipeline.md b/docs/guides/sdlc-pipeline.md index 877be00a95..c78a0070aa 100644 --- a/docs/guides/sdlc-pipeline.md +++ b/docs/guides/sdlc-pipeline.md @@ -814,8 +814,8 @@ Default checks for each phase are defined in `shared/egg_contracts/phase_default - Merge conflict check (required) - Lint check (required, 1 retry) - Test check (required) -- Deployment validation (optional, 1 retry, requires `.egg/deployment.yml`) - Auto-fixer (optional) +- Deployment validation (optional, 1 retry, requires `.egg/deployment.yml`) **PR phase:** - No checks