Phase 2b: Anthropic API proxy with credential injection - #20
Merged
Conversation
Implement Phase 2b endpoints for Anthropic API proxying: - POST /v1/messages with streaming SSE support - POST /v1/messages/count_tokens for token counting Key features: - Credential injection (API key and OAuth token support) - Mtime-based credential cache invalidation - Blocked tools filtering in private mode (web_search, WebFetch) - IP-based session lookup for mode detection - Connection pooling via singleton httpx client Implements #15 Authored-by: jib
![james-in-a-box[bot]](https://avatars.githubusercontent.com/u/8340608?s=60&v=4){kind=small}
Contributor
Author
There was a problem hiding this comment.
PR #20 Review: Phase 2b - Anthropic API Proxy with Credential Injection
Summary
This PR implements the Anthropic API proxy for the egg gateway, enabling zero-credential exposure in the sandbox. The implementation closely follows the reference implementation from james-in-a-box and addresses most of the requirements from issue #15.
Alignment with Issue #15 Acceptance Criteria
| Criteria | Status | Notes |
|---|---|---|
anthropic_credentials.py extracted and adapted |
✅ | Properly parameterized for egg (paths, logging) |
/v1/messages endpoint with streaming support |
✅ | Implemented with SSE streaming via httpx |
/v1/messages/count_tokens endpoint |
✅ | Implemented |
| Credential injection (API key and OAuth) | ✅ | Both supported, OAuth takes precedence |
| Blocked tools filtering in private mode | ✅ | Filters web_search, WebSearch, web_fetch, WebFetch |
| IP-based session mode detection | ✅ | Uses get_session_by_ip() from Phase 2a |
| Integration test with mock Anthropic API | Unit tests with mocks, no true integration test |
Alignment with james-in-a-box Reference
The implementation is well-aligned with the reference:
-
anthropic_credentials.py: Nearly identical with proper adaptations:
- Path: ~/.config/jib/secrets.env → ~/.config/egg/secrets.env ✅
- Logging: jib_logging → egg_logging ✅
- Environment variable: JIB_SECRETS_PATH → EGG_SECRETS_PATH ✅
-
Gateway proxy code: Matches reference architecture:
- Same httpx client configuration (timeout, connection pooling)
- Same header filtering approach (blocklist)
- Same streaming implementation pattern
- Same error response format
-
Security features: All present:
- Thread-safe credential caching with mtime invalidation
- Blocked tools filtering for private mode
- Proper 401 response when no credentials available
Test Coverage Analysis
Total: 32 unit tests across 8 test classes - good coverage of core functionality.
Issues Found
1. Missing Streaming Response Test (Medium Priority)
The TestProxyAnthropicMessages class has test_non_streaming_request but lacks a corresponding test_streaming_request. The streaming code path in proxy_anthropic_messages() is not directly tested through the endpoint.
2. Integration Test Missing (Medium Priority)
Issue #15 explicitly requires "Integration test with mock Anthropic API" but the tests are unit tests with unittest.mock. A true integration test would start the gateway and verify end-to-end request/response flow.
3. No Test for WebSearch Variant (Low Priority)
The blocked tools set includes both web_search and WebSearch (case variants), but tests only verify filtering of web_search explicitly.
Security Review
✅ Credential injection: Credentials never exposed to sandbox container
✅ Header stripping: Auth headers stripped before forwarding
✅ Tool filtering: Web tools blocked in private mode
✅ Error responses: Generic messages, no internal details leaked
✅ Credential validation: Short credentials rejected
Recommendations
I'll push a commit addressing these:
- Add streaming response test for
/v1/messagesendpoint - Add integration test as specified in issue #15
- Test both
web_searchandWebSearchvariants
Verdict
The PR is well-implemented and closely follows the reference implementation. Pushing fixes for the test gaps.
— Authored by jib
7 tasks
Addresses review feedback on PR #20: - Add test_streaming_request for /v1/messages SSE streaming - Add test_streaming_filters_tools_in_private_mode for streaming + tool filtering - Add test_filters_all_blocked_tool_variants to verify all 4 tool name variants - Add comprehensive integration tests with mock Anthropic API server: - End-to-end non-streaming and streaming flows - Token counting endpoint - Tool filtering in private mode - Credential injection verification - Header forwarding - Connection error handling Total tests: 42 (35 unit + 7 integration) Authored-by: jib
Contributor
Author
Review Feedback AddressedPushed commit 4356add with the following additions: Unit Tests Added (3 new tests)
Integration Tests Added (7 new tests)Created
Test Summary
— Authored by jib |
- Fix ruff format issue in integration test file - Fix mypy error: handle None case for request.remote_addr Authored-by: jib
Add mypy override to ignore missing type stubs for waitress module. This is a pre-existing issue not specific to this PR. Authored-by: jib
james-in-a-box Bot
pushed a commit
that referenced
this pull request
Apr 20, 2026
- Add lock protection for _pod_states dict access in KubernetesMonitor to prevent data corruption from concurrent thread access (#7) - Fix exit_code=None incorrectly treated as clean exit — only exit_code==0 is a clean exit now (#18) - Prune _clean_exit_skipped when pods are removed to prevent unbounded memory growth (#21) - Remove pods/create from RBAC — orchestrator creates Jobs, not bare pods (#17) - Add sandbox image build and import to test-integration.yml to prevent ImagePullBackOff on agent pod spawns (#20) - Set KUBECONFIG default in Makefile deploy target so it works independently of k3s-setup subshell (#22)
jwbron
added a commit
that referenced
this pull request
Apr 21, 2026
* Initialize SDLC contract for issue #1553 * Add Kubernetes migration documentation and update existing docs Create docs/architecture/kubernetes-migration.md covering the Docker to k8s migration architecture, design decisions, component mapping, network isolation model, storage model, RBAC, developer workflow, and CI/CD changes. Update existing docs to reflect the k8s migration: - docs/guides/deployment.md: Replace Docker Compose with k3s deployment - docs/architecture/orchestrator.md: Update network architecture for k8s - docs/architecture/network-isolation.md: Add Kubernetes NetworkPolicy section - orchestrator/README.md: Update file listing for new k8s modules - docs/development/STRUCTURE.md: Add k8s/ directory, update orchestrator - docs/index.md: Add kubernetes-migration.md to doc index - CONTRIBUTING.md: Update integration test prereq from Docker to k3s * Update remaining docs for Docker-to-Kubernetes terminology - docs/architecture/README.md: Update system overview for k8s components - docs/architecture/git-isolation.md: Update storage/network comparison table - docs/guides/deploy-migration.md: Add deprecation note pointing to k8s - docs/guides/pipeline-health-monitoring.md: Update log reference terminology - docs/guides/concurrent-execution.md: Update worktree isolation for pod/Job naming * Add ContainerBackend protocol, KubernetesClient, and k8s manifests Phase 1: Define ContainerBackend Protocol with runtime_checkable interface that both DockerClient and KubernetesClient satisfy. Implement KubernetesClient wrapping the kubernetes Python client with Job/Pod lifecycle management, custom exception hierarchy, and singleton accessor. Add k8s-native fields (pod_name, namespace, job_name) to ContainerInfo. Phase 2: Create Kustomize manifests with base + local overlay structure. Base includes orchestrator/gateway Deployments and Services, RBAC for Job management, agent Job template with init container for .git shadow mount, and Calico NetworkPolicies enforcing default-deny with gateway-only egress for agent pods. * Update orchestrator architecture doc for k8s terminology Replace Docker-specific references with Kubernetes equivalents throughout: - ContainerMonitor → KubernetesMonitor - container_monitor.py → kubernetes_monitor.py - container_spawner.py → kubernetes_spawner.py - Docker container set → Kubernetes pod set - Docker queries → Kubernetes API queries - container ID → Job name for worktree keying - bind mounts → hostPath volumes - Docker host → host machine * Update orchestrator README for k8s terminology Replace remaining Docker-specific references: state volume, health checks, PATCH behavior, host path translation. * Update docs with accurate implementation details from coder Align migration docs with actual implementation: - NetworkPolicies: add DNS egress policy, correct label selectors (app.kubernetes.io/component, kubernetes.io/metadata.name) - ContainerBackend protocol: match actual method signatures - RBAC: document both ClusterRole and namespace-scoped Role - KubernetesClient: document label scheme (egg.pipeline.id, etc.) * Add tests for ContainerBackend protocol and KubernetesClient - test_container_backend.py: Protocol conformance (Docker, K8s, minimal, incomplete), exception hierarchy, ContainerInfo k8s fields, runtime checkability. - test_kubernetes_client.py: 101 tests covering create/start/stop/remove container, get_container_info, list_containers, logs, wait, cleanup, k8s-native methods (create_job, delete_job, list_jobs, get_pod_for_job, get_pod_logs, get_pod_status), _resolve_job_name, helper functions, singleton accessor, constants. - conftest.py: Mock kubernetes SDK (V1Container, V1Job, etc.) with attribute-storing data classes so tests work without the kubernetes package installed. * Migrate gateway to token-only auth, add KubernetesSpawner and Monitor Gateway auth: Remove IP-based session validation enforcement. Pod IPs are ephemeral in Kubernetes so sessions now authenticate by token only. IP is still recorded for audit logging. container_ip made optional in session registration. KubernetesSpawner: New spawner that creates k8s Jobs instead of Docker containers. Uses label-based identification, token-only gateway sessions, and the same SpawnedContainer interface. Supports agent and overseer job spawning, concurrent spawn functions, pipeline cleanup, and restart tracking. KubernetesMonitor: Replacement for ContainerMonitor using k8s pod polling. Detects pod state transitions, fires event callbacks, and handles orphan cleanup via label-based job listing. Routes updated to support both Docker and k8s backends via EGG_RUNTIME environment variable, defaulting to Docker for backward compatibility. * Add tests for KubernetesSpawner and KubernetesMonitor * Complete k8s migration: CLI runtime, CI/CD, Docker removal Phase 4 - CLI Runtime Migration: - Add to_k8s_job_kwargs() and build_sandbox_job_spec() to shared/egg_container/ for converting SandboxContainerConfig to k8s Job specs with proper volume, env, and security mapping. - Update sandbox/egg_lib/runtime.py with dual Docker/k8s path selected by EGG_RUNTIME env var. K8s path uses Service DNS for gateway resolution. Phase 5 - CI/CD and Docker Removal: - Add Makefile targets: k3s-setup, deploy, k3s-import, k3s-teardown. - Update CI workflows to set up k3s, import images, and deploy. - Replace Docker SDK code with backward-compat shims that re-export from kubernetes equivalents (DockerClient→KubernetesClient, etc.). - Remove docker-compose.yml files. - Replace docker>=7.0.0 with kubernetes>=31.0.0 in dependencies. - Update integration test fixtures for k3s-based test environment. - Add consensus stall recovery methods to KubernetesMonitor for backward compatibility with existing health check infrastructure. * Fix DockerClient test for k8s migration (DockerClient is now alias) * Fix 5 reviewer NACK issues: RBAC, labels, naming, singleton, list 1. SECURITY: Remove ClusterRole/ClusterRoleBinding from rbac.yaml; namespace-scoped Role+RoleBinding in egg-agents is sufficient. 2. CORRECTNESS: Add app.kubernetes.io/component:agent label in spawn_agent_job() so NetworkPolicies apply to agent pods. 3. CORRECTNESS: Prevent JOB_PREFIX double-prepending in create_container() and use correct prefixed name in spawner pre-cleanup. Add backward-compat method aliases and kwargs (docker_client, timeout, spawn_agent_container, etc.). 4. CORRECTNESS: Validate explicit namespace in singleton accessor get_kubernetes_client() using sentinel pattern. 5. CORRECTNESS: Guard against double-prefix in list_containers() when LABEL_CONTAINER_NAME is missing from pod labels. * Fix ruff violations and add _validate_container_id shim - Remove 7 unused imports (F401) from kubernetes_monitor.py and kubernetes_spawner.py via ruff check --fix. - Apply ruff format to all 3 source files. - Add _validate_container_id to docker_client.py shim so test_docker_client.py can collect without import errors. * Fix checks: apply automated formatting fixes * Fix lint: remove unused imports, fix hardcoded ports - Remove unused KubernetesClient, get_kubernetes_client, KubernetesSpawner imports from orchestrator/routes/pipelines.py (ruff F401) - Import GATEWAY_PORT/GATEWAY_PROXY_PORT from egg_config in kubernetes_spawner.py instead of hardcoding 9848/3129 - Add # noqa: EGG002 to k8s YAML manifests where port constants cannot be imported (infrastructure files require literal values) * Fix lint: sort imports in kubernetes_spawner, add raise-from in runtime * Fix mypy errors in runtime.py for kubernetes migration * Fix container_monitor tests for Kubernetes migration * Rewrite docker_client tests for Kubernetes shim layer * Update container_spawner tests for Kubernetes migration * Fix remaining test failures for Kubernetes migration * Fix kubernetes_spawner test assertions * Fix lint formatting in test files * Fix checks: align tests with Docker-to-Kubernetes migration * Address review feedback: fix all blocking issues in k8s migration Fix all 11 remaining blocking issues from the review: 1. Add resource limits (500m/512Mi req, 2CPU/2Gi limits), activeDeadlineSeconds (4h), and ttlSecondsAfterFinished (10m) to programmatic Job specs 2. Remove dead agent-job-template.yaml ConfigMap (never loaded by Python code) 3. Add allow-agent-to-orchestrator egress NetworkPolicy on port 9849 4. Fix namespace default in sandbox/egg_lib/runtime.py from egg-system to egg-agents 5. Forward timeout parameter to delete_job via grace_period_seconds 6. Add set_health_check_runner() method to KubernetesMonitor for cli.py compat 8. Add securityContext (runAsNonRoot, drop ALL caps, no privilege escalation) to gateway and orchestrator deployments 9. Add input validation on container_id/job names in KubernetesClient (_validate_name for create, _resolve_job_name for all other operations) 10. Add SHA256 checksum verification to install-calico.sh 11. Remove || true from CI Calico install and deploy steps 12. Fix EGG_REPO_PATH to include repo name derived from repos list Contract verification gaps addressed: - Add 23 unit tests for to_k8s_job_kwargs() and build_sandbox_job_spec() - Add k8s-based code paths to integration test fixtures (egg_stack, local_pipeline_stack) with test namespace creation/cleanup * Address re-review feedback: fix remaining blocking issues - B2: Add emptyDir volumes for /home/egg/.egg-state and /tmp to orchestrator deployment so it can write state with readOnlyRootFilesystem - B1: Wire health check runner into KubernetesMonitor._check_pod so RUNTIME_TICK checks fire on pod state transitions - B3: Add denylist for security-critical env vars (EGG_SESSION_TOKEN, GATEWAY_URL, HTTP_PROXY, etc.) that extra_env cannot override - N1: Move _UID_RE regex to module scope to avoid recompilation * Address non-blocking review feedback: fix stale comment, move constant to module scope - Fix stale comment in test_health_check_integration.py that incorrectly stated set_health_check_runner and _run_runtime_tick_checks were not carried over to KubernetesMonitor (they were, in da297cb) - Move _PROTECTED_ENV_KEYS from local variable to module-level constant to avoid re-creating the frozenset on every call * Add missing V1ResourceRequirements mock to fix 12 test failures * Port restart improvements from main to kubernetes_spawner: concurrency locks, pre-spawn count increment, mode validation * Fix restart count lock protection in KubernetesSpawner Match ContainerSpawner's thread-safety pattern: - get_restart_count() now acquires per-key lock before reading - reset_restart_counts() holds _restart_locks_lock while modifying both _restart_counts and _restart_locks atomically, using pop() to safely handle already-held locks * Address re-review feedback: namespace default, restart lock timeout, stale template references * Address review feedback: thread safety, correctness, and CI fixes - Add lock protection for _pod_states dict access in KubernetesMonitor to prevent data corruption from concurrent thread access (#7) - Fix exit_code=None incorrectly treated as clean exit — only exit_code==0 is a clean exit now (#18) - Prune _clean_exit_skipped when pods are removed to prevent unbounded memory growth (#21) - Remove pods/create from RBAC — orchestrator creates Jobs, not bare pods (#17) - Add sandbox image build and import to test-integration.yml to prevent ImagePullBackOff on agent pod spawns (#20) - Set KUBECONFIG default in Makefile deploy target so it works independently of k3s-setup subshell (#22) * Fix k3s deploy gaps: orchestrator image, Calico bump, sandbox context Found while testing #1692 on a fresh Fedora aarch64 machine: - `make build` and `make k3s-import` didn't include the orchestrator image, leaving the orchestrator deployment in ImagePullBackOff. - Calico v3.27.2 arm64 image ships without libpcap.so.0.8, so calico-node CrashLoopBackOffs on arm64 hosts (upstream bug, fixed in later patches). Bumped pin to v3.31.5. - The v3.27.2 SHA256 in install-calico.sh never matched the actual upstream manifest. Recomputed and pinned v3.31.5's hash. - Sandbox build fails without a `repo-deps/` directory in the build context, normally assembled by the egg Python build flow. Added a minimal marker bootstrap so `make build` works standalone. - Updated three doc references from v3.27.0 to v3.31.5. - Added `repo-deps/` to .gitignore. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Address review feedback: fix leaky abstraction and VersionConflictError handling - Add .backend property to KubernetesSpawner as runtime-agnostic accessor for the container backend client (Issue #14). This eliminates scattered `spawner.k8s if _RUNTIME == "kubernetes" else spawner.docker` patterns. - Replace all spawner.docker and if/else runtime checks in routes/pipelines.py with spawner.backend - Remove dead code: overseer stop used identical methods on both branches (stop_agent_job == stop_agent_container) — collapsed to single call - Fix VersionConflictError handling in consensus stall recovery (Issue #13): explicit catch with pipeline reload and state verification instead of generic except Exception - Update test fixtures to set mock.backend alongside mock.docker * Fix k3s deploy: gateway/orchestrator actually start end-to-end Continued validation of #1692 on a fresh machine. Prior commit addressed build and Calico install; this one makes the deployments come up. Gateway: - Rewrite base deployment to match what gateway/entrypoint.sh actually reads: /secrets (Secret mount) with launcher-secret + secrets.env + github-app.pem + repositories.yaml, /shared/certs (emptyDir where the entrypoint writes the CA cert), /home/egg emptyDir, /home/egg/.egg-state emptyDir. The previous base mounted /etc/egg-gateway/certs and /var/lib/egg-gateway — paths nothing in the code touches. - Remove runAsNonRoot: the entrypoint is designed to start as root, chown squid dirs + /home/egg, then gosu-drop to HOST_UID. Running as UID 1000 directly hit /run/squid.pid EACCES plus a dozen other issues. Preserve fsGroup: 1000 so emptyDirs are writable post-gosu. - Fix health probe path: /api/v1/health (port 9851), not /healthz. - Add EGG_CONFIG_DIR, EGG_SECRETS_PATH, EGG_REPO_CONFIG env vars so gateway/repo_config resolve their file paths to the mounted Secret. - enableServiceLinks: false to stop the auto-injected GATEWAY_PORT/etc from colliding with the entrypoint's own vars. - Chown squid dirs to egg:egg in the Dockerfile (was proxy:proxy). - Source /secrets/secrets.env in the entrypoint so GITHUB_USER_TOKEN et al. are available (Compose got them from shell env). - Make the chown-everything block in the entrypoint tolerant of read-only bind mounts (k8s hostPath readOnly returns EROFS). Orchestrator: - enableServiceLinks: false (ORCHESTRATOR_PORT was being overwritten by the auto-injected tcp://<ip>:9849 value, breaking --port parsing). - Add emptyDir at /home/egg so .gitconfig / .egg-worktrees writes don't hit the read-only rootfs. - Source /secrets/secrets.env in its entrypoint too. Local overlay: - Replace the invented .egg-gateway hostPaths with strategic-merge additions for /home/egg/repos and /home/egg/.egg-worktrees hostPaths, on both deployments. Local-dev only; paths hardcoded to /home/jwies since kustomize has no env-var substitution. - New make target `k3s-secrets` that creates gateway-secrets from all files under ~/.config/egg/; `make deploy` depends on it. Fix wait targets to match actual deployment names (orchestrator/gateway, not egg-orchestrator/egg-gateway). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add 'timed out' to RESTARTABLE_PATTERNS for restart detection 'timeout' does not match 'timed out' as a substring, causing error messages like 'Agent timed out waiting for response' to miss the restartable keyword check and escalate to HITL unnecessarily. * Fix restart lock race: retain per-key locks in reset_restart_counts Addresses review feedback B1/B2: reset_restart_counts() was deleting per-key locks from _restart_locks, which races with restart_agent_job holding those locks. If a lock is deleted while held, _get_restart_lock creates a new lock for the same key — breaking mutual exclusion. Fix: only clear counter entries in reset_restart_counts(), retain locks. Locks are lightweight and bounded by (pipeline, role) pairs. * Wire orchestrator → gateway connectivity and auth With the previous commit both deployments started, but the orchestrator still couldn't talk to the gateway: - gateway_client.py reads GATEWAY_HOST/GATEWAY_PORT (not GATEWAY_URL). Its default GATEWAY_HOST is "egg-gateway", the old Compose container name — no such name resolves in k8s. Set it to the Service FQDN. - Gateway rejected requests with "Missing or invalid Authorization header" because the orchestrator had no EGG_LAUNCHER_SECRET. Inject it via secretKeyRef from the same gateway-secrets Secret that the gateway mounts at /secrets/launcher-secret. With these in, the orchestrator registers sessions with the gateway and /api/v1/pipelines returns an empty list cleanly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix pipeline submit + expose MCP + block lowercase proxy overrides Pipeline submission tripped three more issues on top of the stack: - state_store._ensure_worktree called logger.warning with structured kwargs (worktree=..., returncode=...) but logger is a stdlib logging.Logger, not a structlog wrapper. submit_task raised TypeError in the warning path when the state worktree needed recreation. Rewrite as a printf-style format. - Local repo mounts on both deployments were readOnly, but the orchestrator creates per-pipeline worktrees inside each repo's .git/worktrees/ and the gateway runs `git worktree prune` on startup. Drop readOnly on both repos mounts. - Nothing exposed the orchestrator MCP port on the host. Added hostPort: 9850 to the local overlay so Claude Code's MCP config (http://localhost:9850/mcp) connects without a port-forward. Also changed the orchestrator Deployment strategy to Recreate because hostPort is singleton per node — a rolling update gets stuck Pending waiting for the port to free up. Also addresses review N3 (flagged 3x): _PROTECTED_ENV_KEYS in kubernetes_spawner.py now blocks the lowercase http_proxy / https_proxy / no_proxy variants too, since curl/libcurl/requests all honor either case and leaving the lowercase forms unblocked is a defense-in-depth gap. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Wire agent Jobs end-to-end: images, mounts, creds, naming Validation of #1692 kept surfacing infrastructure gaps between pipeline submit and the point where agents actually do work. This fixes the remaining ones needed to get all four phase-0 agents (refiner, reviewer_refine, reviewer_agent_design, overseer) spawning with the right mounts, credentials, and names. Agent image resolution - `kubernetes_spawner.DEFAULT_SANDBOX_IMAGE` defaulted to `egg:latest`; `make build` produces `egg-sandbox:latest`, and there is no public `docker.io/library/egg`, so every agent pod ImagePullBackOff'd. Set `EGG_SANDBOX_IMAGE=egg-sandbox:latest` on the orchestrator. - Agent `V1Container` had no `imagePullPolicy`. Default for `:latest` is `Always`, which fails for locally-imported images that only live in containerd's cache. Force `IfNotPresent`. Also added to `shared/egg_container.to_k8s_job_kwargs` for the other code path. Pod security / credentials - Agent pods had no pod-level securityContext so they ran as root. Claude CLI's `--dangerously-skip-permissions` refuses to run as root. Set `runAsUser/Group/fsGroup=1000` (the `egg` user in the sandbox image) on the pod spec built by `kubernetes_client.create_container`. - Agent env had no Anthropic credentials and no proxy routing, so the CLI hit `Not logged in · Please run /login`. Set the same two env vars that `sandbox/entrypoint.py` sets in the Compose flow: `ANTHROPIC_BASE_URL` pointing at the gateway, plus a deliberately- invalid placeholder `CLAUDE_CODE_OAUTH_TOKEN` that satisfies local validation. The gateway strips the placeholder and injects the real credential server-side — real secrets still never enter the sandbox. Volume mounts - `kubernetes_client.create_container` previously dropped volume specs on the floor ("not currently translated to k8s volume mounts"). Add a `host_path_mounts` parameter and translate each entry to a matched `V1Volume`/`V1VolumeMount` pair (hostPath, DirectoryOrCreate). - `kubernetes_spawner.spawn_agent_job` now builds those mounts from `repo_volumes` (owner/repo → host path, one mount per repo) plus a single `worktrees` mount backed by `EGG_HOST_WORKTREES_PATH`. Without these, agents couldn't see the code they were supposed to edit — they tried `gh repo clone` into an empty `/home/egg/repos`. - Added `EGG_HOST_WORKTREES_PATH=/home/jwies/.egg-worktrees` to the local overlay's orchestrator patch. Naming - Job names longer than 63 chars (k8s RFC-1123 limit) failed validation outright. Long pipeline IDs + long role names like `reviewer_agent_design` overflow deterministically. Truncate to 54 chars of readable prefix and append an 8-char SHA1 suffix so uniqueness is preserved. Surfaced by submitting with qualifier `k3s-retry` which pushed the composed name to 64 chars. Other - Gateway `limits.memory: 256Mi` was OOMKilling the pod under normal load (Squid + waitress + git operations). Bumped to 1Gi/512Mi limits. - Orchestrator deployment strategy set to `Recreate` because the local overlay binds a singleton hostPort (9850 for MCP); the default RollingUpdate deadlocks waiting for the port to free. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix B324: mark SHA1 hash as not used for security * Address PR #1692 review: container hardening, volume-name collision Two blocking items from the re-review of `be617281`: 1. Agent V1Container was missing container-level securityContext. The old ConfigMap-based Job template had allowPrivilegeEscalation: false capabilities: drop: [ALL] These disappeared in the switch to programmatic Job specs. Agents already run as UID 1000 via the pod securityContext so there's no reason for them to gain new privs or hold any Linux caps. Added. 2. `kubernetes_spawner.spawn_agent_job` built volume names from the repo basename alone (`repo-{short}`). Two repos from different orgs with the same basename (e.g. `Khan/webapp` + `other-org/webapp`, plausible as the repo list grows) would collide on the volume name and k8s would reject the Job. Include the owner in the name, normalize to RFC-1123, and hash-truncate if the composed name exceeds 63 chars. Non-blocking: strengthened the comment on the local-dev orchestrator overlay patch explaining that every `/home/jwies/...` path and the EGG_HOST_REPO_MAP entries are this developer's layout and must be edited before anyone else can `make deploy`. Portability is tracked as a follow-up in #1760. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix CI: mark sha1 usedforsecurity=False, sort test_cli imports CI's bandit job flagged the new sha1 hash in kubernetes_spawner (introduced in the previous commit's volume-name collision fix) as B324 — weak hash for security. It isn't a security hash (used to pick a unique-per-name suffix); add `usedforsecurity=False` to match the identical treatment already applied in kubernetes_client. Also auto-sorted the import block in orchestrator/tests/test_cli.py that was tripping ruff's I001 (unrelated to our changes, surfaced because `make lint` runs the full tree). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix checks: apply automated formatting fixes --------- Co-authored-by: egg-orchestrator <egg@localhost> Co-authored-by: james-in-a-box[bot] <246424927+james-in-a-box[bot]@users.noreply.github.com> Co-authored-by: egg-reviewer[bot] <261018737+egg-reviewer[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
james-in-a-box Bot
added a commit
that referenced
this pull request
Apr 22, 2026
* Initialize SDLC contract for issue #1553 * Add Kubernetes migration documentation and update existing docs Create docs/architecture/kubernetes-migration.md covering the Docker to k8s migration architecture, design decisions, component mapping, network isolation model, storage model, RBAC, developer workflow, and CI/CD changes. Update existing docs to reflect the k8s migration: - docs/guides/deployment.md: Replace Docker Compose with k3s deployment - docs/architecture/orchestrator.md: Update network architecture for k8s - docs/architecture/network-isolation.md: Add Kubernetes NetworkPolicy section - orchestrator/README.md: Update file listing for new k8s modules - docs/development/STRUCTURE.md: Add k8s/ directory, update orchestrator - docs/index.md: Add kubernetes-migration.md to doc index - CONTRIBUTING.md: Update integration test prereq from Docker to k3s * Update remaining docs for Docker-to-Kubernetes terminology - docs/architecture/README.md: Update system overview for k8s components - docs/architecture/git-isolation.md: Update storage/network comparison table - docs/guides/deploy-migration.md: Add deprecation note pointing to k8s - docs/guides/pipeline-health-monitoring.md: Update log reference terminology - docs/guides/concurrent-execution.md: Update worktree isolation for pod/Job naming * Add ContainerBackend protocol, KubernetesClient, and k8s manifests Phase 1: Define ContainerBackend Protocol with runtime_checkable interface that both DockerClient and KubernetesClient satisfy. Implement KubernetesClient wrapping the kubernetes Python client with Job/Pod lifecycle management, custom exception hierarchy, and singleton accessor. Add k8s-native fields (pod_name, namespace, job_name) to ContainerInfo. Phase 2: Create Kustomize manifests with base + local overlay structure. Base includes orchestrator/gateway Deployments and Services, RBAC for Job management, agent Job template with init container for .git shadow mount, and Calico NetworkPolicies enforcing default-deny with gateway-only egress for agent pods. * Update orchestrator architecture doc for k8s terminology Replace Docker-specific references with Kubernetes equivalents throughout: - ContainerMonitor → KubernetesMonitor - container_monitor.py → kubernetes_monitor.py - container_spawner.py → kubernetes_spawner.py - Docker container set → Kubernetes pod set - Docker queries → Kubernetes API queries - container ID → Job name for worktree keying - bind mounts → hostPath volumes - Docker host → host machine * Update orchestrator README for k8s terminology Replace remaining Docker-specific references: state volume, health checks, PATCH behavior, host path translation. * Update docs with accurate implementation details from coder Align migration docs with actual implementation: - NetworkPolicies: add DNS egress policy, correct label selectors (app.kubernetes.io/component, kubernetes.io/metadata.name) - ContainerBackend protocol: match actual method signatures - RBAC: document both ClusterRole and namespace-scoped Role - KubernetesClient: document label scheme (egg.pipeline.id, etc.) * Add tests for ContainerBackend protocol and KubernetesClient - test_container_backend.py: Protocol conformance (Docker, K8s, minimal, incomplete), exception hierarchy, ContainerInfo k8s fields, runtime checkability. - test_kubernetes_client.py: 101 tests covering create/start/stop/remove container, get_container_info, list_containers, logs, wait, cleanup, k8s-native methods (create_job, delete_job, list_jobs, get_pod_for_job, get_pod_logs, get_pod_status), _resolve_job_name, helper functions, singleton accessor, constants. - conftest.py: Mock kubernetes SDK (V1Container, V1Job, etc.) with attribute-storing data classes so tests work without the kubernetes package installed. * Migrate gateway to token-only auth, add KubernetesSpawner and Monitor Gateway auth: Remove IP-based session validation enforcement. Pod IPs are ephemeral in Kubernetes so sessions now authenticate by token only. IP is still recorded for audit logging. container_ip made optional in session registration. KubernetesSpawner: New spawner that creates k8s Jobs instead of Docker containers. Uses label-based identification, token-only gateway sessions, and the same SpawnedContainer interface. Supports agent and overseer job spawning, concurrent spawn functions, pipeline cleanup, and restart tracking. KubernetesMonitor: Replacement for ContainerMonitor using k8s pod polling. Detects pod state transitions, fires event callbacks, and handles orphan cleanup via label-based job listing. Routes updated to support both Docker and k8s backends via EGG_RUNTIME environment variable, defaulting to Docker for backward compatibility. * Add tests for KubernetesSpawner and KubernetesMonitor * Complete k8s migration: CLI runtime, CI/CD, Docker removal Phase 4 - CLI Runtime Migration: - Add to_k8s_job_kwargs() and build_sandbox_job_spec() to shared/egg_container/ for converting SandboxContainerConfig to k8s Job specs with proper volume, env, and security mapping. - Update sandbox/egg_lib/runtime.py with dual Docker/k8s path selected by EGG_RUNTIME env var. K8s path uses Service DNS for gateway resolution. Phase 5 - CI/CD and Docker Removal: - Add Makefile targets: k3s-setup, deploy, k3s-import, k3s-teardown. - Update CI workflows to set up k3s, import images, and deploy. - Replace Docker SDK code with backward-compat shims that re-export from kubernetes equivalents (DockerClient→KubernetesClient, etc.). - Remove docker-compose.yml files. - Replace docker>=7.0.0 with kubernetes>=31.0.0 in dependencies. - Update integration test fixtures for k3s-based test environment. - Add consensus stall recovery methods to KubernetesMonitor for backward compatibility with existing health check infrastructure. * Fix DockerClient test for k8s migration (DockerClient is now alias) * Fix 5 reviewer NACK issues: RBAC, labels, naming, singleton, list 1. SECURITY: Remove ClusterRole/ClusterRoleBinding from rbac.yaml; namespace-scoped Role+RoleBinding in egg-agents is sufficient. 2. CORRECTNESS: Add app.kubernetes.io/component:agent label in spawn_agent_job() so NetworkPolicies apply to agent pods. 3. CORRECTNESS: Prevent JOB_PREFIX double-prepending in create_container() and use correct prefixed name in spawner pre-cleanup. Add backward-compat method aliases and kwargs (docker_client, timeout, spawn_agent_container, etc.). 4. CORRECTNESS: Validate explicit namespace in singleton accessor get_kubernetes_client() using sentinel pattern. 5. CORRECTNESS: Guard against double-prefix in list_containers() when LABEL_CONTAINER_NAME is missing from pod labels. * Fix ruff violations and add _validate_container_id shim - Remove 7 unused imports (F401) from kubernetes_monitor.py and kubernetes_spawner.py via ruff check --fix. - Apply ruff format to all 3 source files. - Add _validate_container_id to docker_client.py shim so test_docker_client.py can collect without import errors. * Fix checks: apply automated formatting fixes * Fix lint: remove unused imports, fix hardcoded ports - Remove unused KubernetesClient, get_kubernetes_client, KubernetesSpawner imports from orchestrator/routes/pipelines.py (ruff F401) - Import GATEWAY_PORT/GATEWAY_PROXY_PORT from egg_config in kubernetes_spawner.py instead of hardcoding 9848/3129 - Add # noqa: EGG002 to k8s YAML manifests where port constants cannot be imported (infrastructure files require literal values) * Fix lint: sort imports in kubernetes_spawner, add raise-from in runtime * Fix mypy errors in runtime.py for kubernetes migration * Fix container_monitor tests for Kubernetes migration * Rewrite docker_client tests for Kubernetes shim layer * Update container_spawner tests for Kubernetes migration * Fix remaining test failures for Kubernetes migration * Fix kubernetes_spawner test assertions * Fix lint formatting in test files * Fix checks: align tests with Docker-to-Kubernetes migration * Address review feedback: fix all blocking issues in k8s migration Fix all 11 remaining blocking issues from the review: 1. Add resource limits (500m/512Mi req, 2CPU/2Gi limits), activeDeadlineSeconds (4h), and ttlSecondsAfterFinished (10m) to programmatic Job specs 2. Remove dead agent-job-template.yaml ConfigMap (never loaded by Python code) 3. Add allow-agent-to-orchestrator egress NetworkPolicy on port 9849 4. Fix namespace default in sandbox/egg_lib/runtime.py from egg-system to egg-agents 5. Forward timeout parameter to delete_job via grace_period_seconds 6. Add set_health_check_runner() method to KubernetesMonitor for cli.py compat 8. Add securityContext (runAsNonRoot, drop ALL caps, no privilege escalation) to gateway and orchestrator deployments 9. Add input validation on container_id/job names in KubernetesClient (_validate_name for create, _resolve_job_name for all other operations) 10. Add SHA256 checksum verification to install-calico.sh 11. Remove || true from CI Calico install and deploy steps 12. Fix EGG_REPO_PATH to include repo name derived from repos list Contract verification gaps addressed: - Add 23 unit tests for to_k8s_job_kwargs() and build_sandbox_job_spec() - Add k8s-based code paths to integration test fixtures (egg_stack, local_pipeline_stack) with test namespace creation/cleanup * Address re-review feedback: fix remaining blocking issues - B2: Add emptyDir volumes for /home/egg/.egg-state and /tmp to orchestrator deployment so it can write state with readOnlyRootFilesystem - B1: Wire health check runner into KubernetesMonitor._check_pod so RUNTIME_TICK checks fire on pod state transitions - B3: Add denylist for security-critical env vars (EGG_SESSION_TOKEN, GATEWAY_URL, HTTP_PROXY, etc.) that extra_env cannot override - N1: Move _UID_RE regex to module scope to avoid recompilation * Address non-blocking review feedback: fix stale comment, move constant to module scope - Fix stale comment in test_health_check_integration.py that incorrectly stated set_health_check_runner and _run_runtime_tick_checks were not carried over to KubernetesMonitor (they were, in da297cb) - Move _PROTECTED_ENV_KEYS from local variable to module-level constant to avoid re-creating the frozenset on every call * Add missing V1ResourceRequirements mock to fix 12 test failures * Port restart improvements from main to kubernetes_spawner: concurrency locks, pre-spawn count increment, mode validation * Fix restart count lock protection in KubernetesSpawner Match ContainerSpawner's thread-safety pattern: - get_restart_count() now acquires per-key lock before reading - reset_restart_counts() holds _restart_locks_lock while modifying both _restart_counts and _restart_locks atomically, using pop() to safely handle already-held locks * Address re-review feedback: namespace default, restart lock timeout, stale template references * Address review feedback: thread safety, correctness, and CI fixes - Add lock protection for _pod_states dict access in KubernetesMonitor to prevent data corruption from concurrent thread access (#7) - Fix exit_code=None incorrectly treated as clean exit — only exit_code==0 is a clean exit now (#18) - Prune _clean_exit_skipped when pods are removed to prevent unbounded memory growth (#21) - Remove pods/create from RBAC — orchestrator creates Jobs, not bare pods (#17) - Add sandbox image build and import to test-integration.yml to prevent ImagePullBackOff on agent pod spawns (#20) - Set KUBECONFIG default in Makefile deploy target so it works independently of k3s-setup subshell (#22) * Fix k3s deploy gaps: orchestrator image, Calico bump, sandbox context Found while testing #1692 on a fresh Fedora aarch64 machine: - `make build` and `make k3s-import` didn't include the orchestrator image, leaving the orchestrator deployment in ImagePullBackOff. - Calico v3.27.2 arm64 image ships without libpcap.so.0.8, so calico-node CrashLoopBackOffs on arm64 hosts (upstream bug, fixed in later patches). Bumped pin to v3.31.5. - The v3.27.2 SHA256 in install-calico.sh never matched the actual upstream manifest. Recomputed and pinned v3.31.5's hash. - Sandbox build fails without a `repo-deps/` directory in the build context, normally assembled by the egg Python build flow. Added a minimal marker bootstrap so `make build` works standalone. - Updated three doc references from v3.27.0 to v3.31.5. - Added `repo-deps/` to .gitignore. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Address review feedback: fix leaky abstraction and VersionConflictError handling - Add .backend property to KubernetesSpawner as runtime-agnostic accessor for the container backend client (Issue #14). This eliminates scattered `spawner.k8s if _RUNTIME == "kubernetes" else spawner.docker` patterns. - Replace all spawner.docker and if/else runtime checks in routes/pipelines.py with spawner.backend - Remove dead code: overseer stop used identical methods on both branches (stop_agent_job == stop_agent_container) — collapsed to single call - Fix VersionConflictError handling in consensus stall recovery (Issue #13): explicit catch with pipeline reload and state verification instead of generic except Exception - Update test fixtures to set mock.backend alongside mock.docker * Fix k3s deploy: gateway/orchestrator actually start end-to-end Continued validation of #1692 on a fresh machine. Prior commit addressed build and Calico install; this one makes the deployments come up. Gateway: - Rewrite base deployment to match what gateway/entrypoint.sh actually reads: /secrets (Secret mount) with launcher-secret + secrets.env + github-app.pem + repositories.yaml, /shared/certs (emptyDir where the entrypoint writes the CA cert), /home/egg emptyDir, /home/egg/.egg-state emptyDir. The previous base mounted /etc/egg-gateway/certs and /var/lib/egg-gateway — paths nothing in the code touches. - Remove runAsNonRoot: the entrypoint is designed to start as root, chown squid dirs + /home/egg, then gosu-drop to HOST_UID. Running as UID 1000 directly hit /run/squid.pid EACCES plus a dozen other issues. Preserve fsGroup: 1000 so emptyDirs are writable post-gosu. - Fix health probe path: /api/v1/health (port 9851), not /healthz. - Add EGG_CONFIG_DIR, EGG_SECRETS_PATH, EGG_REPO_CONFIG env vars so gateway/repo_config resolve their file paths to the mounted Secret. - enableServiceLinks: false to stop the auto-injected GATEWAY_PORT/etc from colliding with the entrypoint's own vars. - Chown squid dirs to egg:egg in the Dockerfile (was proxy:proxy). - Source /secrets/secrets.env in the entrypoint so GITHUB_USER_TOKEN et al. are available (Compose got them from shell env). - Make the chown-everything block in the entrypoint tolerant of read-only bind mounts (k8s hostPath readOnly returns EROFS). Orchestrator: - enableServiceLinks: false (ORCHESTRATOR_PORT was being overwritten by the auto-injected tcp://<ip>:9849 value, breaking --port parsing). - Add emptyDir at /home/egg so .gitconfig / .egg-worktrees writes don't hit the read-only rootfs. - Source /secrets/secrets.env in its entrypoint too. Local overlay: - Replace the invented .egg-gateway hostPaths with strategic-merge additions for /home/egg/repos and /home/egg/.egg-worktrees hostPaths, on both deployments. Local-dev only; paths hardcoded to /home/jwies since kustomize has no env-var substitution. - New make target `k3s-secrets` that creates gateway-secrets from all files under ~/.config/egg/; `make deploy` depends on it. Fix wait targets to match actual deployment names (orchestrator/gateway, not egg-orchestrator/egg-gateway). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add 'timed out' to RESTARTABLE_PATTERNS for restart detection 'timeout' does not match 'timed out' as a substring, causing error messages like 'Agent timed out waiting for response' to miss the restartable keyword check and escalate to HITL unnecessarily. * Fix restart lock race: retain per-key locks in reset_restart_counts Addresses review feedback B1/B2: reset_restart_counts() was deleting per-key locks from _restart_locks, which races with restart_agent_job holding those locks. If a lock is deleted while held, _get_restart_lock creates a new lock for the same key — breaking mutual exclusion. Fix: only clear counter entries in reset_restart_counts(), retain locks. Locks are lightweight and bounded by (pipeline, role) pairs. * Wire orchestrator → gateway connectivity and auth With the previous commit both deployments started, but the orchestrator still couldn't talk to the gateway: - gateway_client.py reads GATEWAY_HOST/GATEWAY_PORT (not GATEWAY_URL). Its default GATEWAY_HOST is "egg-gateway", the old Compose container name — no such name resolves in k8s. Set it to the Service FQDN. - Gateway rejected requests with "Missing or invalid Authorization header" because the orchestrator had no EGG_LAUNCHER_SECRET. Inject it via secretKeyRef from the same gateway-secrets Secret that the gateway mounts at /secrets/launcher-secret. With these in, the orchestrator registers sessions with the gateway and /api/v1/pipelines returns an empty list cleanly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix pipeline submit + expose MCP + block lowercase proxy overrides Pipeline submission tripped three more issues on top of the stack: - state_store._ensure_worktree called logger.warning with structured kwargs (worktree=..., returncode=...) but logger is a stdlib logging.Logger, not a structlog wrapper. submit_task raised TypeError in the warning path when the state worktree needed recreation. Rewrite as a printf-style format. - Local repo mounts on both deployments were readOnly, but the orchestrator creates per-pipeline worktrees inside each repo's .git/worktrees/ and the gateway runs `git worktree prune` on startup. Drop readOnly on both repos mounts. - Nothing exposed the orchestrator MCP port on the host. Added hostPort: 9850 to the local overlay so Claude Code's MCP config (http://localhost:9850/mcp) connects without a port-forward. Also changed the orchestrator Deployment strategy to Recreate because hostPort is singleton per node — a rolling update gets stuck Pending waiting for the port to free up. Also addresses review N3 (flagged 3x): _PROTECTED_ENV_KEYS in kubernetes_spawner.py now blocks the lowercase http_proxy / https_proxy / no_proxy variants too, since curl/libcurl/requests all honor either case and leaving the lowercase forms unblocked is a defense-in-depth gap. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Wire agent Jobs end-to-end: images, mounts, creds, naming Validation of #1692 kept surfacing infrastructure gaps between pipeline submit and the point where agents actually do work. This fixes the remaining ones needed to get all four phase-0 agents (refiner, reviewer_refine, reviewer_agent_design, overseer) spawning with the right mounts, credentials, and names. Agent image resolution - `kubernetes_spawner.DEFAULT_SANDBOX_IMAGE` defaulted to `egg:latest`; `make build` produces `egg-sandbox:latest`, and there is no public `docker.io/library/egg`, so every agent pod ImagePullBackOff'd. Set `EGG_SANDBOX_IMAGE=egg-sandbox:latest` on the orchestrator. - Agent `V1Container` had no `imagePullPolicy`. Default for `:latest` is `Always`, which fails for locally-imported images that only live in containerd's cache. Force `IfNotPresent`. Also added to `shared/egg_container.to_k8s_job_kwargs` for the other code path. Pod security / credentials - Agent pods had no pod-level securityContext so they ran as root. Claude CLI's `--dangerously-skip-permissions` refuses to run as root. Set `runAsUser/Group/fsGroup=1000` (the `egg` user in the sandbox image) on the pod spec built by `kubernetes_client.create_container`. - Agent env had no Anthropic credentials and no proxy routing, so the CLI hit `Not logged in · Please run /login`. Set the same two env vars that `sandbox/entrypoint.py` sets in the Compose flow: `ANTHROPIC_BASE_URL` pointing at the gateway, plus a deliberately- invalid placeholder `CLAUDE_CODE_OAUTH_TOKEN` that satisfies local validation. The gateway strips the placeholder and injects the real credential server-side — real secrets still never enter the sandbox. Volume mounts - `kubernetes_client.create_container` previously dropped volume specs on the floor ("not currently translated to k8s volume mounts"). Add a `host_path_mounts` parameter and translate each entry to a matched `V1Volume`/`V1VolumeMount` pair (hostPath, DirectoryOrCreate). - `kubernetes_spawner.spawn_agent_job` now builds those mounts from `repo_volumes` (owner/repo → host path, one mount per repo) plus a single `worktrees` mount backed by `EGG_HOST_WORKTREES_PATH`. Without these, agents couldn't see the code they were supposed to edit — they tried `gh repo clone` into an empty `/home/egg/repos`. - Added `EGG_HOST_WORKTREES_PATH=/home/jwies/.egg-worktrees` to the local overlay's orchestrator patch. Naming - Job names longer than 63 chars (k8s RFC-1123 limit) failed validation outright. Long pipeline IDs + long role names like `reviewer_agent_design` overflow deterministically. Truncate to 54 chars of readable prefix and append an 8-char SHA1 suffix so uniqueness is preserved. Surfaced by submitting with qualifier `k3s-retry` which pushed the composed name to 64 chars. Other - Gateway `limits.memory: 256Mi` was OOMKilling the pod under normal load (Squid + waitress + git operations). Bumped to 1Gi/512Mi limits. - Orchestrator deployment strategy set to `Recreate` because the local overlay binds a singleton hostPort (9850 for MCP); the default RollingUpdate deadlocks waiting for the port to free. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix B324: mark SHA1 hash as not used for security * Address PR #1692 review: container hardening, volume-name collision Two blocking items from the re-review of `be617281`: 1. Agent V1Container was missing container-level securityContext. The old ConfigMap-based Job template had allowPrivilegeEscalation: false capabilities: drop: [ALL] These disappeared in the switch to programmatic Job specs. Agents already run as UID 1000 via the pod securityContext so there's no reason for them to gain new privs or hold any Linux caps. Added. 2. `kubernetes_spawner.spawn_agent_job` built volume names from the repo basename alone (`repo-{short}`). Two repos from different orgs with the same basename (e.g. `Khan/webapp` + `other-org/webapp`, plausible as the repo list grows) would collide on the volume name and k8s would reject the Job. Include the owner in the name, normalize to RFC-1123, and hash-truncate if the composed name exceeds 63 chars. Non-blocking: strengthened the comment on the local-dev orchestrator overlay patch explaining that every `/home/jwies/...` path and the EGG_HOST_REPO_MAP entries are this developer's layout and must be edited before anyone else can `make deploy`. Portability is tracked as a follow-up in #1760. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix CI: mark sha1 usedforsecurity=False, sort test_cli imports CI's bandit job flagged the new sha1 hash in kubernetes_spawner (introduced in the previous commit's volume-name collision fix) as B324 — weak hash for security. It isn't a security hash (used to pick a unique-per-name suffix); add `usedforsecurity=False` to match the identical treatment already applied in kubernetes_client. Also auto-sorted the import block in orchestrator/tests/test_cli.py that was tripping ruff's I001 (unrelated to our changes, surfaced because `make lint` runs the full tree). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix checks: apply automated formatting fixes --------- Co-authored-by: egg-orchestrator <egg@localhost> Co-authored-by: james-in-a-box[bot] <246424927+james-in-a-box[bot]@users.noreply.github.com> Co-authored-by: egg-reviewer[bot] <261018737+egg-reviewer[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
james-in-a-box Bot
added a commit
that referenced
this pull request
Apr 22, 2026
* Initialize SDLC contract for issue #1553 * Add Kubernetes migration documentation and update existing docs Create docs/architecture/kubernetes-migration.md covering the Docker to k8s migration architecture, design decisions, component mapping, network isolation model, storage model, RBAC, developer workflow, and CI/CD changes. Update existing docs to reflect the k8s migration: - docs/guides/deployment.md: Replace Docker Compose with k3s deployment - docs/architecture/orchestrator.md: Update network architecture for k8s - docs/architecture/network-isolation.md: Add Kubernetes NetworkPolicy section - orchestrator/README.md: Update file listing for new k8s modules - docs/development/STRUCTURE.md: Add k8s/ directory, update orchestrator - docs/index.md: Add kubernetes-migration.md to doc index - CONTRIBUTING.md: Update integration test prereq from Docker to k3s * Update remaining docs for Docker-to-Kubernetes terminology - docs/architecture/README.md: Update system overview for k8s components - docs/architecture/git-isolation.md: Update storage/network comparison table - docs/guides/deploy-migration.md: Add deprecation note pointing to k8s - docs/guides/pipeline-health-monitoring.md: Update log reference terminology - docs/guides/concurrent-execution.md: Update worktree isolation for pod/Job naming * Add ContainerBackend protocol, KubernetesClient, and k8s manifests Phase 1: Define ContainerBackend Protocol with runtime_checkable interface that both DockerClient and KubernetesClient satisfy. Implement KubernetesClient wrapping the kubernetes Python client with Job/Pod lifecycle management, custom exception hierarchy, and singleton accessor. Add k8s-native fields (pod_name, namespace, job_name) to ContainerInfo. Phase 2: Create Kustomize manifests with base + local overlay structure. Base includes orchestrator/gateway Deployments and Services, RBAC for Job management, agent Job template with init container for .git shadow mount, and Calico NetworkPolicies enforcing default-deny with gateway-only egress for agent pods. * Update orchestrator architecture doc for k8s terminology Replace Docker-specific references with Kubernetes equivalents throughout: - ContainerMonitor → KubernetesMonitor - container_monitor.py → kubernetes_monitor.py - container_spawner.py → kubernetes_spawner.py - Docker container set → Kubernetes pod set - Docker queries → Kubernetes API queries - container ID → Job name for worktree keying - bind mounts → hostPath volumes - Docker host → host machine * Update orchestrator README for k8s terminology Replace remaining Docker-specific references: state volume, health checks, PATCH behavior, host path translation. * Update docs with accurate implementation details from coder Align migration docs with actual implementation: - NetworkPolicies: add DNS egress policy, correct label selectors (app.kubernetes.io/component, kubernetes.io/metadata.name) - ContainerBackend protocol: match actual method signatures - RBAC: document both ClusterRole and namespace-scoped Role - KubernetesClient: document label scheme (egg.pipeline.id, etc.) * Add tests for ContainerBackend protocol and KubernetesClient - test_container_backend.py: Protocol conformance (Docker, K8s, minimal, incomplete), exception hierarchy, ContainerInfo k8s fields, runtime checkability. - test_kubernetes_client.py: 101 tests covering create/start/stop/remove container, get_container_info, list_containers, logs, wait, cleanup, k8s-native methods (create_job, delete_job, list_jobs, get_pod_for_job, get_pod_logs, get_pod_status), _resolve_job_name, helper functions, singleton accessor, constants. - conftest.py: Mock kubernetes SDK (V1Container, V1Job, etc.) with attribute-storing data classes so tests work without the kubernetes package installed. * Migrate gateway to token-only auth, add KubernetesSpawner and Monitor Gateway auth: Remove IP-based session validation enforcement. Pod IPs are ephemeral in Kubernetes so sessions now authenticate by token only. IP is still recorded for audit logging. container_ip made optional in session registration. KubernetesSpawner: New spawner that creates k8s Jobs instead of Docker containers. Uses label-based identification, token-only gateway sessions, and the same SpawnedContainer interface. Supports agent and overseer job spawning, concurrent spawn functions, pipeline cleanup, and restart tracking. KubernetesMonitor: Replacement for ContainerMonitor using k8s pod polling. Detects pod state transitions, fires event callbacks, and handles orphan cleanup via label-based job listing. Routes updated to support both Docker and k8s backends via EGG_RUNTIME environment variable, defaulting to Docker for backward compatibility. * Add tests for KubernetesSpawner and KubernetesMonitor * Complete k8s migration: CLI runtime, CI/CD, Docker removal Phase 4 - CLI Runtime Migration: - Add to_k8s_job_kwargs() and build_sandbox_job_spec() to shared/egg_container/ for converting SandboxContainerConfig to k8s Job specs with proper volume, env, and security mapping. - Update sandbox/egg_lib/runtime.py with dual Docker/k8s path selected by EGG_RUNTIME env var. K8s path uses Service DNS for gateway resolution. Phase 5 - CI/CD and Docker Removal: - Add Makefile targets: k3s-setup, deploy, k3s-import, k3s-teardown. - Update CI workflows to set up k3s, import images, and deploy. - Replace Docker SDK code with backward-compat shims that re-export from kubernetes equivalents (DockerClient→KubernetesClient, etc.). - Remove docker-compose.yml files. - Replace docker>=7.0.0 with kubernetes>=31.0.0 in dependencies. - Update integration test fixtures for k3s-based test environment. - Add consensus stall recovery methods to KubernetesMonitor for backward compatibility with existing health check infrastructure. * Fix DockerClient test for k8s migration (DockerClient is now alias) * Fix 5 reviewer NACK issues: RBAC, labels, naming, singleton, list 1. SECURITY: Remove ClusterRole/ClusterRoleBinding from rbac.yaml; namespace-scoped Role+RoleBinding in egg-agents is sufficient. 2. CORRECTNESS: Add app.kubernetes.io/component:agent label in spawn_agent_job() so NetworkPolicies apply to agent pods. 3. CORRECTNESS: Prevent JOB_PREFIX double-prepending in create_container() and use correct prefixed name in spawner pre-cleanup. Add backward-compat method aliases and kwargs (docker_client, timeout, spawn_agent_container, etc.). 4. CORRECTNESS: Validate explicit namespace in singleton accessor get_kubernetes_client() using sentinel pattern. 5. CORRECTNESS: Guard against double-prefix in list_containers() when LABEL_CONTAINER_NAME is missing from pod labels. * Fix ruff violations and add _validate_container_id shim - Remove 7 unused imports (F401) from kubernetes_monitor.py and kubernetes_spawner.py via ruff check --fix. - Apply ruff format to all 3 source files. - Add _validate_container_id to docker_client.py shim so test_docker_client.py can collect without import errors. * Fix checks: apply automated formatting fixes * Fix lint: remove unused imports, fix hardcoded ports - Remove unused KubernetesClient, get_kubernetes_client, KubernetesSpawner imports from orchestrator/routes/pipelines.py (ruff F401) - Import GATEWAY_PORT/GATEWAY_PROXY_PORT from egg_config in kubernetes_spawner.py instead of hardcoding 9848/3129 - Add # noqa: EGG002 to k8s YAML manifests where port constants cannot be imported (infrastructure files require literal values) * Fix lint: sort imports in kubernetes_spawner, add raise-from in runtime * Fix mypy errors in runtime.py for kubernetes migration * Fix container_monitor tests for Kubernetes migration * Rewrite docker_client tests for Kubernetes shim layer * Update container_spawner tests for Kubernetes migration * Fix remaining test failures for Kubernetes migration * Fix kubernetes_spawner test assertions * Fix lint formatting in test files * Fix checks: align tests with Docker-to-Kubernetes migration * Address review feedback: fix all blocking issues in k8s migration Fix all 11 remaining blocking issues from the review: 1. Add resource limits (500m/512Mi req, 2CPU/2Gi limits), activeDeadlineSeconds (4h), and ttlSecondsAfterFinished (10m) to programmatic Job specs 2. Remove dead agent-job-template.yaml ConfigMap (never loaded by Python code) 3. Add allow-agent-to-orchestrator egress NetworkPolicy on port 9849 4. Fix namespace default in sandbox/egg_lib/runtime.py from egg-system to egg-agents 5. Forward timeout parameter to delete_job via grace_period_seconds 6. Add set_health_check_runner() method to KubernetesMonitor for cli.py compat 8. Add securityContext (runAsNonRoot, drop ALL caps, no privilege escalation) to gateway and orchestrator deployments 9. Add input validation on container_id/job names in KubernetesClient (_validate_name for create, _resolve_job_name for all other operations) 10. Add SHA256 checksum verification to install-calico.sh 11. Remove || true from CI Calico install and deploy steps 12. Fix EGG_REPO_PATH to include repo name derived from repos list Contract verification gaps addressed: - Add 23 unit tests for to_k8s_job_kwargs() and build_sandbox_job_spec() - Add k8s-based code paths to integration test fixtures (egg_stack, local_pipeline_stack) with test namespace creation/cleanup * Address re-review feedback: fix remaining blocking issues - B2: Add emptyDir volumes for /home/egg/.egg-state and /tmp to orchestrator deployment so it can write state with readOnlyRootFilesystem - B1: Wire health check runner into KubernetesMonitor._check_pod so RUNTIME_TICK checks fire on pod state transitions - B3: Add denylist for security-critical env vars (EGG_SESSION_TOKEN, GATEWAY_URL, HTTP_PROXY, etc.) that extra_env cannot override - N1: Move _UID_RE regex to module scope to avoid recompilation * Address non-blocking review feedback: fix stale comment, move constant to module scope - Fix stale comment in test_health_check_integration.py that incorrectly stated set_health_check_runner and _run_runtime_tick_checks were not carried over to KubernetesMonitor (they were, in da297cb) - Move _PROTECTED_ENV_KEYS from local variable to module-level constant to avoid re-creating the frozenset on every call * Add missing V1ResourceRequirements mock to fix 12 test failures * Port restart improvements from main to kubernetes_spawner: concurrency locks, pre-spawn count increment, mode validation * Fix restart count lock protection in KubernetesSpawner Match ContainerSpawner's thread-safety pattern: - get_restart_count() now acquires per-key lock before reading - reset_restart_counts() holds _restart_locks_lock while modifying both _restart_counts and _restart_locks atomically, using pop() to safely handle already-held locks * Address re-review feedback: namespace default, restart lock timeout, stale template references * Address review feedback: thread safety, correctness, and CI fixes - Add lock protection for _pod_states dict access in KubernetesMonitor to prevent data corruption from concurrent thread access (#7) - Fix exit_code=None incorrectly treated as clean exit — only exit_code==0 is a clean exit now (#18) - Prune _clean_exit_skipped when pods are removed to prevent unbounded memory growth (#21) - Remove pods/create from RBAC — orchestrator creates Jobs, not bare pods (#17) - Add sandbox image build and import to test-integration.yml to prevent ImagePullBackOff on agent pod spawns (#20) - Set KUBECONFIG default in Makefile deploy target so it works independently of k3s-setup subshell (#22) * Fix k3s deploy gaps: orchestrator image, Calico bump, sandbox context Found while testing #1692 on a fresh Fedora aarch64 machine: - `make build` and `make k3s-import` didn't include the orchestrator image, leaving the orchestrator deployment in ImagePullBackOff. - Calico v3.27.2 arm64 image ships without libpcap.so.0.8, so calico-node CrashLoopBackOffs on arm64 hosts (upstream bug, fixed in later patches). Bumped pin to v3.31.5. - The v3.27.2 SHA256 in install-calico.sh never matched the actual upstream manifest. Recomputed and pinned v3.31.5's hash. - Sandbox build fails without a `repo-deps/` directory in the build context, normally assembled by the egg Python build flow. Added a minimal marker bootstrap so `make build` works standalone. - Updated three doc references from v3.27.0 to v3.31.5. - Added `repo-deps/` to .gitignore. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Address review feedback: fix leaky abstraction and VersionConflictError handling - Add .backend property to KubernetesSpawner as runtime-agnostic accessor for the container backend client (Issue #14). This eliminates scattered `spawner.k8s if _RUNTIME == "kubernetes" else spawner.docker` patterns. - Replace all spawner.docker and if/else runtime checks in routes/pipelines.py with spawner.backend - Remove dead code: overseer stop used identical methods on both branches (stop_agent_job == stop_agent_container) — collapsed to single call - Fix VersionConflictError handling in consensus stall recovery (Issue #13): explicit catch with pipeline reload and state verification instead of generic except Exception - Update test fixtures to set mock.backend alongside mock.docker * Fix k3s deploy: gateway/orchestrator actually start end-to-end Continued validation of #1692 on a fresh machine. Prior commit addressed build and Calico install; this one makes the deployments come up. Gateway: - Rewrite base deployment to match what gateway/entrypoint.sh actually reads: /secrets (Secret mount) with launcher-secret + secrets.env + github-app.pem + repositories.yaml, /shared/certs (emptyDir where the entrypoint writes the CA cert), /home/egg emptyDir, /home/egg/.egg-state emptyDir. The previous base mounted /etc/egg-gateway/certs and /var/lib/egg-gateway — paths nothing in the code touches. - Remove runAsNonRoot: the entrypoint is designed to start as root, chown squid dirs + /home/egg, then gosu-drop to HOST_UID. Running as UID 1000 directly hit /run/squid.pid EACCES plus a dozen other issues. Preserve fsGroup: 1000 so emptyDirs are writable post-gosu. - Fix health probe path: /api/v1/health (port 9851), not /healthz. - Add EGG_CONFIG_DIR, EGG_SECRETS_PATH, EGG_REPO_CONFIG env vars so gateway/repo_config resolve their file paths to the mounted Secret. - enableServiceLinks: false to stop the auto-injected GATEWAY_PORT/etc from colliding with the entrypoint's own vars. - Chown squid dirs to egg:egg in the Dockerfile (was proxy:proxy). - Source /secrets/secrets.env in the entrypoint so GITHUB_USER_TOKEN et al. are available (Compose got them from shell env). - Make the chown-everything block in the entrypoint tolerant of read-only bind mounts (k8s hostPath readOnly returns EROFS). Orchestrator: - enableServiceLinks: false (ORCHESTRATOR_PORT was being overwritten by the auto-injected tcp://<ip>:9849 value, breaking --port parsing). - Add emptyDir at /home/egg so .gitconfig / .egg-worktrees writes don't hit the read-only rootfs. - Source /secrets/secrets.env in its entrypoint too. Local overlay: - Replace the invented .egg-gateway hostPaths with strategic-merge additions for /home/egg/repos and /home/egg/.egg-worktrees hostPaths, on both deployments. Local-dev only; paths hardcoded to /home/jwies since kustomize has no env-var substitution. - New make target `k3s-secrets` that creates gateway-secrets from all files under ~/.config/egg/; `make deploy` depends on it. Fix wait targets to match actual deployment names (orchestrator/gateway, not egg-orchestrator/egg-gateway). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add 'timed out' to RESTARTABLE_PATTERNS for restart detection 'timeout' does not match 'timed out' as a substring, causing error messages like 'Agent timed out waiting for response' to miss the restartable keyword check and escalate to HITL unnecessarily. * Fix restart lock race: retain per-key locks in reset_restart_counts Addresses review feedback B1/B2: reset_restart_counts() was deleting per-key locks from _restart_locks, which races with restart_agent_job holding those locks. If a lock is deleted while held, _get_restart_lock creates a new lock for the same key — breaking mutual exclusion. Fix: only clear counter entries in reset_restart_counts(), retain locks. Locks are lightweight and bounded by (pipeline, role) pairs. * Wire orchestrator → gateway connectivity and auth With the previous commit both deployments started, but the orchestrator still couldn't talk to the gateway: - gateway_client.py reads GATEWAY_HOST/GATEWAY_PORT (not GATEWAY_URL). Its default GATEWAY_HOST is "egg-gateway", the old Compose container name — no such name resolves in k8s. Set it to the Service FQDN. - Gateway rejected requests with "Missing or invalid Authorization header" because the orchestrator had no EGG_LAUNCHER_SECRET. Inject it via secretKeyRef from the same gateway-secrets Secret that the gateway mounts at /secrets/launcher-secret. With these in, the orchestrator registers sessions with the gateway and /api/v1/pipelines returns an empty list cleanly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix pipeline submit + expose MCP + block lowercase proxy overrides Pipeline submission tripped three more issues on top of the stack: - state_store._ensure_worktree called logger.warning with structured kwargs (worktree=..., returncode=...) but logger is a stdlib logging.Logger, not a structlog wrapper. submit_task raised TypeError in the warning path when the state worktree needed recreation. Rewrite as a printf-style format. - Local repo mounts on both deployments were readOnly, but the orchestrator creates per-pipeline worktrees inside each repo's .git/worktrees/ and the gateway runs `git worktree prune` on startup. Drop readOnly on both repos mounts. - Nothing exposed the orchestrator MCP port on the host. Added hostPort: 9850 to the local overlay so Claude Code's MCP config (http://localhost:9850/mcp) connects without a port-forward. Also changed the orchestrator Deployment strategy to Recreate because hostPort is singleton per node — a rolling update gets stuck Pending waiting for the port to free up. Also addresses review N3 (flagged 3x): _PROTECTED_ENV_KEYS in kubernetes_spawner.py now blocks the lowercase http_proxy / https_proxy / no_proxy variants too, since curl/libcurl/requests all honor either case and leaving the lowercase forms unblocked is a defense-in-depth gap. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Wire agent Jobs end-to-end: images, mounts, creds, naming Validation of #1692 kept surfacing infrastructure gaps between pipeline submit and the point where agents actually do work. This fixes the remaining ones needed to get all four phase-0 agents (refiner, reviewer_refine, reviewer_agent_design, overseer) spawning with the right mounts, credentials, and names. Agent image resolution - `kubernetes_spawner.DEFAULT_SANDBOX_IMAGE` defaulted to `egg:latest`; `make build` produces `egg-sandbox:latest`, and there is no public `docker.io/library/egg`, so every agent pod ImagePullBackOff'd. Set `EGG_SANDBOX_IMAGE=egg-sandbox:latest` on the orchestrator. - Agent `V1Container` had no `imagePullPolicy`. Default for `:latest` is `Always`, which fails for locally-imported images that only live in containerd's cache. Force `IfNotPresent`. Also added to `shared/egg_container.to_k8s_job_kwargs` for the other code path. Pod security / credentials - Agent pods had no pod-level securityContext so they ran as root. Claude CLI's `--dangerously-skip-permissions` refuses to run as root. Set `runAsUser/Group/fsGroup=1000` (the `egg` user in the sandbox image) on the pod spec built by `kubernetes_client.create_container`. - Agent env had no Anthropic credentials and no proxy routing, so the CLI hit `Not logged in · Please run /login`. Set the same two env vars that `sandbox/entrypoint.py` sets in the Compose flow: `ANTHROPIC_BASE_URL` pointing at the gateway, plus a deliberately- invalid placeholder `CLAUDE_CODE_OAUTH_TOKEN` that satisfies local validation. The gateway strips the placeholder and injects the real credential server-side — real secrets still never enter the sandbox. Volume mounts - `kubernetes_client.create_container` previously dropped volume specs on the floor ("not currently translated to k8s volume mounts"). Add a `host_path_mounts` parameter and translate each entry to a matched `V1Volume`/`V1VolumeMount` pair (hostPath, DirectoryOrCreate). - `kubernetes_spawner.spawn_agent_job` now builds those mounts from `repo_volumes` (owner/repo → host path, one mount per repo) plus a single `worktrees` mount backed by `EGG_HOST_WORKTREES_PATH`. Without these, agents couldn't see the code they were supposed to edit — they tried `gh repo clone` into an empty `/home/egg/repos`. - Added `EGG_HOST_WORKTREES_PATH=/home/jwies/.egg-worktrees` to the local overlay's orchestrator patch. Naming - Job names longer than 63 chars (k8s RFC-1123 limit) failed validation outright. Long pipeline IDs + long role names like `reviewer_agent_design` overflow deterministically. Truncate to 54 chars of readable prefix and append an 8-char SHA1 suffix so uniqueness is preserved. Surfaced by submitting with qualifier `k3s-retry` which pushed the composed name to 64 chars. Other - Gateway `limits.memory: 256Mi` was OOMKilling the pod under normal load (Squid + waitress + git operations). Bumped to 1Gi/512Mi limits. - Orchestrator deployment strategy set to `Recreate` because the local overlay binds a singleton hostPort (9850 for MCP); the default RollingUpdate deadlocks waiting for the port to free. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix B324: mark SHA1 hash as not used for security * Address PR #1692 review: container hardening, volume-name collision Two blocking items from the re-review of `be617281`: 1. Agent V1Container was missing container-level securityContext. The old ConfigMap-based Job template had allowPrivilegeEscalation: false capabilities: drop: [ALL] These disappeared in the switch to programmatic Job specs. Agents already run as UID 1000 via the pod securityContext so there's no reason for them to gain new privs or hold any Linux caps. Added. 2. `kubernetes_spawner.spawn_agent_job` built volume names from the repo basename alone (`repo-{short}`). Two repos from different orgs with the same basename (e.g. `Khan/webapp` + `other-org/webapp`, plausible as the repo list grows) would collide on the volume name and k8s would reject the Job. Include the owner in the name, normalize to RFC-1123, and hash-truncate if the composed name exceeds 63 chars. Non-blocking: strengthened the comment on the local-dev orchestrator overlay patch explaining that every `/home/jwies/...` path and the EGG_HOST_REPO_MAP entries are this developer's layout and must be edited before anyone else can `make deploy`. Portability is tracked as a follow-up in #1760. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix CI: mark sha1 usedforsecurity=False, sort test_cli imports CI's bandit job flagged the new sha1 hash in kubernetes_spawner (introduced in the previous commit's volume-name collision fix) as B324 — weak hash for security. It isn't a security hash (used to pick a unique-per-name suffix); add `usedforsecurity=False` to match the identical treatment already applied in kubernetes_client. Also auto-sorted the import block in orchestrator/tests/test_cli.py that was tripping ruff's I001 (unrelated to our changes, surfaced because `make lint` runs the full tree). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix checks: apply automated formatting fixes --------- Co-authored-by: egg-orchestrator <egg@localhost> Co-authored-by: james-in-a-box[bot] <246424927+james-in-a-box[bot]@users.noreply.github.com> Co-authored-by: egg-reviewer[bot] <261018737+egg-reviewer[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
james-in-a-box Bot
pushed a commit
that referenced
this pull request
May 11, 2026
…lockers Reviewer_plan NACKed the v1 plan with one blocking item (single-OR JQL fails on team-managed Jira projects) plus 20 non-blocking flags ranked by impact. This revision lands the blocker plus the 10 highest-impact non-blockers in one re-propose. Blocker: - TASK-1-3 + TASK-1-12: replace the single-OR JQL `parent = <K> OR "Epic Link" = <K>` with two separate queries (`parent = "<K>"` and `"Epic Link" = "<K>"`) and merge results, tolerating per-query HTTP 400 (architect ad-9 / risk_analyst R4). Single-OR fails on team-managed projects that lack the "Epic Link" custom field; auto-detection silently downgrades to fresh-path and the sweep returns empty. Exports the helper `search_epic_children` so TASK-1-12 reuses it. Top non-blocking (reviewer-flagged as most impactful): - #1 In-flight gate trust-boundary trade-off: add explicit acknowledgement that gateway-side enforcement is deferred and v1 relies on agent-side gating + apply-time re-check by TASK-1-13. - #5 APPLY_EPIC role registration: expand TASK-1-10 to enumerate all FIVE registration steps (AgentRole, AgentRoleDefinition, get_roles_for_phase, file-restrictions patterns, spawner branch). - #6 epic_apply persistence MCP surface: add `mcp__sdlc__update_epic_apply` MCP tool to TASK-1-7 so the sandbox-side agent can persist artifact updates. - #7 Concurrent-edit guard: TASK-1-10 now fetches the current epic Description, sha256s it, and registers a divergence HITL on mismatch; TASK-1-9 records the baseline sha256; TASK-1-7 adds `refine_description_sha256` to the schema. Additional non-blockers folded in: - #2: jira_effective_mode added to primitives table. - #3: TASK-1-5 introduces `shared/egg_jira_credentials.py` shared module to eliminate the orchestrator → gateway coupling. - #8: TASK-1-11 commits to extending `parse_plan` (not pass-through). - #9: TASK-1-5/TASK-1-14 add already-in-state idempotent short-circuit for Won't-Do transitions. - #10: TASK-1-15 introduces `Pipeline.jira_parent_epic_key` so PR phase doesn't need an extra Jira call. - #11: TASK-1-16 adds `PipelinePhase.PLAN_STOPPED` documented terminal phase + updates overseer monitor short-circuit. - #14: TASK-1-11 requires `wont_do_reason` per node + ⚠ warning rendering in the plan draft (R6). - #15: TASK-1-5 gates the orchestrator-direct cred surface behind `EGG_ENABLE_ORCH_JIRA_TRANSITIONS` (default off — R1). - #16: TASK-1-7 schema gains `version`, `idempotency_seed`, per-edit `summary_hash` + `applied_at`, `wont_do_reason`, signal_source as a list (R10). - #19: TASK-1-19 drops orchestrator-cli.md, adds submit-task-mcp.md. - #13: TASK-1-18 adds the lint regression test `test_no_outbound_jira_writes.py` (R7). - #12: TASK-1-12 introduces a reverse-index `.egg-state/jira-child-pipeline-index.json` to bound the sweep to O(K) (R3 performance mitigation). - #20: New "Risk-analyst items addressed" section summarises how R1/R2/R6/R7/R10/R12 are resolved in-plan (no fresh HITLs). Plan still parses cleanly: 1 slice, 19 tasks, 0 warnings, 0 role-alignment errors. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
james-in-a-box Bot
pushed a commit
that referenced
this pull request
Jul 27, 2026
…ouble-fire guard, register Tier 3 exceptions, reorder midturn_messages Reviewer feedback identified four issues in v1: 1. AC-1 was unsatisfiable: claimed all 13 fields populated but only 5 in scope 2. detect_heartbeat_stall/ConsensusStallCheck double-fire not addressed 3. TASK-2-2 (#19) and TASK-3-2 (#20) are Tier 3 without registered decisions 4. midturn_messages sequenced last despite gating the primary deliverable All four corrected. ACK verdict unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the Anthropic API proxy that enables zero-credential exposure in the sandbox. This allows Claude Code to use
ANTHROPIC_BASE_URLto route API traffic through the gateway for automatic credential injection.New endpoints:
POST /v1/messages- Main messages API with streaming SSE supportPOST /v1/messages/count_tokens- Token counting APIKey features:
x-api-key) and OAuth tokens (Authorization: Bearer)web_search,WebSearch,web_fetch,WebFetch)Closes #15
Files Changed
gateway/anthropic_credentials.pygateway/gateway.pypyproject.tomltests/unit/test_anthropic_proxy.pyTest plan
pytest tests/unit/test_anthropic_proxy.py -vpytest tests/unit/ -vANTHROPIC_API_KEYorANTHROPIC_OAUTH_TOKENAuthored-by: jib