diff --git a/README.md b/README.md index 310e7b986d..bf4eecd7f4 100644 --- a/README.md +++ b/README.md @@ -135,7 +135,7 @@ egg can run as a GitHub Action for CI/CD automation: anthropic-oauth-token: ${{ secrets.ANTHROPIC_OAUTH_TOKEN }} ``` -Trigger egg via @mentions in issues and PRs, or run it on any GitHub Actions event. See the [GitHub Action documentation](docs/features/github-action.md) and [@mention trigger setup guide](docs/guides/mention-trigger-setup.md) for details. +Trigger egg via @mentions in issues and PRs, or run it on any GitHub Actions event. See the [GitHub Action documentation](action/README.md) and the [GitHub Actions ADR](docs/adr/in-progress/ADR-GitHub-Actions-Support.md) for details. ## CLI Reference @@ -160,10 +160,8 @@ Trigger egg via @mentions in issues and PRs, or run it on any GitHub Actions eve - [Documentation Index](docs/index.md) - Navigation hub for all docs - [Architecture](docs/architecture/README.md) - System design and component overview -- [Setup Guide](docs/setup/README.md) - First-time setup and configuration -- [GitHub Action](docs/features/github-action.md) - CI/CD integration -- [@mention Trigger Setup](docs/guides/mention-trigger-setup.md) - GitHub @mention automation -- [Troubleshooting](docs/troubleshooting/) - Common issues and solutions +- [Contributing](CONTRIBUTING.md) - Development setup and workflow +- [GitHub Action](action/README.md) - CI/CD integration ### Component Documentation diff --git a/action/README.md b/action/README.md index 8d69d4c5f5..78178fd60a 100644 --- a/action/README.md +++ b/action/README.md @@ -26,6 +26,6 @@ This action runs the egg autonomous coding agent within GitHub Actions. It sets ## Documentation -For full usage documentation including all inputs, outputs, examples, and security considerations, see the [GitHub Action feature documentation](../docs/features/github-action.md). +For design details, inputs, outputs, and implementation notes, see the [GitHub Actions Support ADR](../docs/adr/in-progress/ADR-GitHub-Actions-Support.md). -For setting up @mention triggers, see the [@mention trigger setup guide](../docs/guides/mention-trigger-setup.md). +For the existing @mention trigger workflow, see [`.github/workflows/on-mention.yml`](../.github/workflows/on-mention.yml) and [`build-mention-prompt.sh`](build-mention-prompt.sh). diff --git a/config/README.md b/config/README.md index e6394f68bd..fd438bf5f5 100644 --- a/config/README.md +++ b/config/README.md @@ -97,7 +97,7 @@ Egg supports separate tokens for writable and readable repositories: | `GITHUB_TOKEN` | Token for writable repos (or use GitHub App for auto-refresh) | | `GITHUB_READONLY_TOKEN` | Separate PAT for read-only repos (optional, falls back to `GITHUB_TOKEN`) | -Using a separate read-only token provides security benefits. See [GitHub App Setup](../docs/setup/github-app-setup.md) for details. +Using a separate read-only token provides security benefits. GitHub App setup is covered in `./setup.py` and the ADRs. ## repositories.yaml (Source of Truth for Repo Access) diff --git a/config/secrets.template.env b/config/secrets.template.env index 760cfb947e..e3b014362e 100644 --- a/config/secrets.template.env +++ b/config/secrets.template.env @@ -44,7 +44,7 @@ SLACK_APP_TOKEN="" # 1. GITHUB_TOKEN - For writable repos (full access) # Create a fine-grained token at https://github.com/settings/tokens?type=beta # Required permissions: Contents (R/W), Pull requests (R/W), Workflows (R) -# OR configure a GitHub App (recommended) - see docs/setup/github-app-setup.md +# OR configure a GitHub App (recommended) - run ./setup.py for guided setup # # 2. GITHUB_READONLY_TOKEN - For read-only repos (optional) # Used when monitoring repos where egg doesn't have write access diff --git a/docs/README.md b/docs/README.md index 55e73ce081..75c0b6dec2 100644 --- a/docs/README.md +++ b/docs/README.md @@ -6,34 +6,11 @@ Complete documentation for egg: a hardened sandbox for autonomous LLM code agent ## Documentation Structure -### [Setup](setup/) -Installation and configuration guides. - -- **[Setup Overview](setup/README.md)** - Quick start and prerequisites -- **[GitHub App Setup](setup/github-app-setup.md)** - GitHub App permissions and installation -- **[GitHub Auth Comparison](setup/github-auth-comparison.md)** - Auth method options - ### [Architecture](architecture/) System design and technical details. - **[Overview](architecture/README.md)** - Gateway + sandbox architecture -### [Features](features/) -Feature documentation and capabilities. - -- **[Container Infrastructure](features/container-infrastructure.md)** - Container management and development environment -- **[GitHub Action](features/github-action.md)** - Running egg in GitHub Actions CI/CD - -### [Guides](guides/) -How-to guides for specific tasks. - -- **[@mention Trigger Setup](guides/mention-trigger-setup.md)** - Trigger egg via GitHub @mentions - -### [Reference](reference/) -Quick reference guides. - -- **[Log Persistence](reference/log-persistence.md)** - Container log persistence and correlation - ### [Development](development/) For contributors and developers. @@ -46,16 +23,6 @@ Architecture Decision Records. - **[ADR Index](adr/README.md)** - All decisions and their status - **[Autonomous Software Engineer](adr/in-progress/ADR-Autonomous-Software-Engineer.md)** - Main system architecture -### [Troubleshooting](troubleshooting/) -Common issues and solutions. - -- **[GitHub Auth Issues](troubleshooting/github-auth-in-long-running-containers.md)** - Token expiry and refresh - -### [Plans](plans/) -Implementation plans for future work. - -- **[GitHub Actions Implementation](plans/github-actions-implementation-plan.md)** - GitHub Actions integration plan - ## Component READMEs Each component directory contains its own README with detailed documentation: @@ -71,7 +38,6 @@ Each component directory contains its own README with detailed documentation: **Getting Started:** 1. [Main README](../README.md) - Project overview -2. [Setup Overview](setup/README.md) - Installation and configuration **Understanding the System:** 1. [Architecture Overview](architecture/README.md) - Component design diff --git a/docs/development/README.md b/docs/development/README.md deleted file mode 100644 index 898a8f8884..0000000000 --- a/docs/development/README.md +++ /dev/null @@ -1,67 +0,0 @@ -# Development Documentation - -For contributors and developers working on egg. - -## Available Guides - -- **[Project Structure](STRUCTURE.md)** - Directory conventions and code organization -- **[Contributing](../../CONTRIBUTING.md)** - Development setup, workflow, testing, and PR process - -## Development Setup - -```bash -# Clone the repository -git clone https://github.com/YOUR_USERNAME/egg.git -cd egg - -# Set up development environment (venv + pre-commit hooks) -make setup - -# Verify the setup -make lint -make test -``` - -See [CONTRIBUTING.md](../../CONTRIBUTING.md) for full setup instructions including `act` installation. - -## Key Commands - -| Command | What it does | -|---------|--------------| -| `make setup` | Install dependencies and pre-commit hooks | -| `make lint` | Run all linters (via act) | -| `make test` | Run all tests (via act) | -| `make security` | Run security scan (via act) | -| `make ci` | Run full CI pipeline (via act) | -| `make lint-fix` | Auto-fix lint issues (native) | -| `make build` | Build Docker images (gateway + sandbox) | - -For direct tool access without act: -```bash -.venv/bin/ruff check . # Python lint -.venv/bin/pytest tests/ -v # Run tests -.venv/bin/bandit -r gateway shared sandbox # Security scan -``` - -## Testing - -- **Unit tests**: `tests/` - Fast, isolated tests -- **Gateway tests**: `gateway/tests/` - Gateway-specific tests -- **Integration tests**: `tests/integration/` - Tests requiring Docker - -Coverage targets (aspirational — CI enforces a 20% floor via `--cov-fail-under`): -- Target 80% overall coverage -- Target 95%+ for security-critical code (`policy.py`, `session_manager.py`) - -## Code Standards - -- **Python**: PEP 8, type hints, docstrings, explicit imports -- **Formatting**: ruff (100 char line length) -- **Shell**: shfmt (2-space indent) -- **Config files**: `.yaml` (not `.yml`) - -## See Also - -- [Architecture](../architecture/) - System design -- [Main README](../../README.md) - Project overview -- [Project Structure](STRUCTURE.md) - Directory layout diff --git a/docs/features/container-infrastructure.md b/docs/features/container-infrastructure.md deleted file mode 100644 index 61a31e3aee..0000000000 --- a/docs/features/container-infrastructure.md +++ /dev/null @@ -1,92 +0,0 @@ -# Container Infrastructure Features - -Core egg container management and development environment. - -## Overview - -The egg container provides a sandboxed development environment: -- **Container Management**: Build, run, exec operations via the `egg` CLI -- **Custom Commands**: Slash commands for common agent operations -- **Gateway Integration**: All git/GitHub operations route through the gateway sidecar - -## Features - -### Claude Custom Commands - -**Purpose**: Slash command system for common agent operations including metrics display. - -**Location**: [`sandbox/.claude/commands/`](../../sandbox/.claude/commands/README.md) - -**Available commands:** -- `/show-metrics` - Generate activity report - -### Container Management System - -**Purpose**: The `egg` CLI provides the primary interface for starting, managing, and interacting with the sandboxed Docker environment. Includes container lifecycle management, log viewing, and configuration validation. - -**Location**: -- [`sandbox/egg`](../../sandbox/egg) - Main egg CLI script -- [`sandbox/egg_lib/`](../../sandbox/egg_lib/) - CLI library modules - -**Key modules:** -- `cli.py` - CLI command handling and argument parsing -- `config.py` - Configuration management -- `docker.py` - Docker container operations -- `gateway.py` - Gateway sidecar communication -- `network_mode.py` - Public/private network mode handling - -### Docker Development Environment Setup - -**Purpose**: Automates installation of development tools in the Docker container, including Python, Node.js, Go, Java, PostgreSQL, Redis, and development utilities. - -**Location**: [`sandbox/docker-setup.py`](../../sandbox/docker-setup.py) - -### Container Entrypoint - -**Purpose**: Orchestrates container startup including environment configuration, gateway connectivity, Claude Code launch, and session management. - -**Location**: [`sandbox/entrypoint.py`](../../sandbox/entrypoint.py) - -### Git/GitHub Wrappers - -**Purpose**: Intercept all git and GitHub CLI commands, routing them through the gateway sidecar for credential injection and policy enforcement. - -**Location**: [`sandbox/bin/`](../../sandbox/bin/) -- `git` - Git wrapper (routes push, fetch, clone through gateway) -- `gh` - GitHub CLI wrapper (routes PR/issue operations through gateway) -- `git-credential-github-token` - Credential helper for git operations - -### Container Directory Communication - -**Purpose**: Shared directory structure enabling communication between container and host. - -**Location**: Documented in [`sandbox/README.md`](../../sandbox/README.md) - -``` -~/sharing/ # Shared with host -├── notifications/ # Agent -> Human (notifications) -├── incoming/ # Human -> Agent (tasks) -├── responses/ # Human -> Agent (responses) -└── context/ # Persistent knowledge across rebuilds - -~/context-sync/ # Read-only context (mounted from host) -├── confluence/ # Confluence documentation -└── jira/ # JIRA tickets -``` - -## Related Documentation - -- [Sandbox README](../../sandbox/README.md) - Full sandbox documentation -- [Claude Code Configuration](../../sandbox/.claude/README.md) - Agent rules and commands -- [Architecture Overview](../architecture/README.md) - System design - -## Source Files - -| Component | Path | -|-----------|------| -| Claude Custom Commands | [`sandbox/.claude/commands/`](../../sandbox/.claude/commands/) | -| Container Management | [`sandbox/egg`](../../sandbox/egg), [`sandbox/egg_lib/`](../../sandbox/egg_lib/) | -| Docker Setup | [`sandbox/docker-setup.py`](../../sandbox/docker-setup.py) | -| Container Entrypoint | [`sandbox/entrypoint.py`](../../sandbox/entrypoint.py) | -| Git/GitHub Wrappers | [`sandbox/bin/`](../../sandbox/bin/) | -| LLM Integration | [`sandbox/llm/`](../../sandbox/llm/) | diff --git a/docs/features/github-action.md b/docs/features/github-action.md deleted file mode 100644 index 75cfa0e0b2..0000000000 --- a/docs/features/github-action.md +++ /dev/null @@ -1,104 +0,0 @@ -# GitHub Action - -Run egg as a GitHub Action for CI/CD automation. - -## Overview - -egg provides a composite GitHub Action that runs the autonomous coding agent within GitHub Actions workflows. This enables automated code changes triggered by events like @mentions, issue comments, or scheduled workflows. - -## Usage - -```yaml -- uses: jwbron/egg@main - with: - prompt: "Fix the failing tests in src/utils.ts" - anthropic-oauth-token: ${{ secrets.ANTHROPIC_OAUTH_TOKEN }} - github-token: ${{ secrets.GITHUB_TOKEN }} -``` - -## Inputs - -| Input | Required | Default | Description | -|-------|----------|---------|-------------| -| `prompt` | Yes | - | Task prompt for Claude Code | -| `anthropic-oauth-token` | Yes | - | Anthropic OAuth token for Claude API | -| `github-token` | Yes | `${{ github.token }}` | GitHub token for git operations | -| `bot-app-id` | No | - | GitHub App ID for bot identity | -| `bot-app-private-key` | No | - | GitHub App private key (PEM) | -| `bot-app-installation-id` | No | - | GitHub App installation ID | -| `bot-username` | No | `egg` | Bot GitHub username (for self-comment filtering) | -| `mode` | No | `auto` | Network mode: `public`, `private`, or `auto` | -| `timeout` | No | `30` | Timeout in minutes | -| `model` | No | `opus` | Claude model to use | -| `image-tag` | No | `latest` | Docker image tag | - -## Outputs - -| Output | Description | -|--------|-------------| -| `exit-code` | Sandbox container exit code (0 = success) | -| `pr-url` | URL of created PR, if any | -| `log-file` | Path to full Claude output log | - -## Network Modes - -| Mode | Behavior | -|------|----------| -| `auto` | Detects from repository visibility (public repo = public mode, private = private) | -| `public` | Full internet access, public GitHub repos | -| `private` | Anthropic API only, private GitHub repos only | - -## Bot Mode vs Token Mode - -**Token mode** (default): Uses the provided `github-token` for all operations. Simple but limited to the token's permissions. - -**Bot mode**: Uses GitHub App credentials (`bot-app-id`, `bot-app-private-key`, `bot-app-installation-id`) for identity. The gateway generates short-lived tokens automatically. Recommended for production use. - -## Example: @mention Trigger - -The most common use case is triggering egg when mentioned in GitHub comments. See the [@mention trigger setup guide](../guides/mention-trigger-setup.md) for a complete walkthrough. - -```yaml -name: egg @mention -on: - issue_comment: - types: [created] - pull_request_review_comment: - types: [created] - -jobs: - respond: - if: contains(github.event.comment.body, '@my-bot') - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - uses: jwbron/egg@main - with: - prompt: ${{ steps.build-prompt.outputs.prompt }} - anthropic-oauth-token: ${{ secrets.ANTHROPIC_OAUTH_TOKEN }} - bot-app-id: ${{ secrets.BOT_APP_ID }} - bot-app-private-key: ${{ secrets.BOT_APP_PRIVATE_KEY }} - bot-app-installation-id: ${{ secrets.BOT_APP_INSTALLATION_ID }} -``` - -## Security - -- The gateway sidecar runs alongside the sandbox within the Action, enforcing the same policies as local execution -- Branch ownership rules still apply (agent can only push to `egg/*` branches) -- Merge operations are still blocked -- In private mode, network access is restricted to Anthropic API only - -## Source Files - -| File | Description | -|------|-------------| -| [`action/action.yml`](../../action/action.yml) | Action metadata and input/output definitions | -| [`action/entrypoint.sh`](../../action/entrypoint.sh) | Action entry point script | -| [`action/build-mention-prompt.sh`](../../action/build-mention-prompt.sh) | Prompt builder for @mention events | -| [`action/generate-config.sh`](../../action/generate-config.sh) | Config generator for the action | - -## Related Documentation - -- [@mention Trigger Setup](../guides/mention-trigger-setup.md) - Setting up @mention triggers -- [ADR: GitHub Actions Support](../adr/in-progress/ADR-GitHub-Actions-Support.md) - Design decisions -- [GitHub Actions Implementation Plan](../plans/github-actions-implementation-plan.md) - Implementation details diff --git a/docs/guides/mention-trigger-setup.md b/docs/guides/mention-trigger-setup.md deleted file mode 100644 index 839306f6aa..0000000000 --- a/docs/guides/mention-trigger-setup.md +++ /dev/null @@ -1,149 +0,0 @@ -# Setting Up @mention Triggers for egg - -This guide explains how to configure a GitHub Actions workflow that triggers egg when a bot is mentioned in issues or pull requests. - -## Overview - -The @mention trigger lets authorized users invoke egg by mentioning a bot username (e.g., `@james-in-a-box`) in: - -- Issue comments -- PR conversation comments -- Inline PR review comments (on diffs) -- New issue descriptions - -egg receives a context-rich prompt with the issue/PR details, recent conversation, and the user's request, then works autonomously to complete the task. - -## Prerequisites - -### Required Secrets - -Configure these in **Settings > Secrets and variables > Actions**: - -| Secret | Description | -|--------|-------------| -| `ANTHROPIC_OAUTH_TOKEN` | Anthropic OAuth token for Claude API access | -| `BOT_APP_ID` | GitHub App ID (numeric) for the bot identity | -| `BOT_APP_PRIVATE_KEY` | GitHub App private key (PEM format) | -| `BOT_APP_INSTALLATION_ID` | GitHub App installation ID for this repo/org | - -### GitHub App Setup - -The bot GitHub App needs these permissions: -- **Contents**: Read & Write (push commits) -- **Pull requests**: Read & Write (create/update PRs, post comments) -- **Issues**: Read & Write (post comments, add reactions) - -### Finding the App credentials - -1. **App ID**: Go to the App's settings page at `https://github.com/settings/apps/` — the App ID is displayed near the top. -2. **Private key**: On the same settings page, scroll to "Private keys" and click "Generate a private key". Save the downloaded `.pem` file contents as the `BOT_APP_PRIVATE_KEY` secret. -3. **Installation ID**: Go to `https://github.com/settings/installations`, click the App installation, and note the numeric ID in the URL (e.g., `https://github.com/settings/installations/12345678` → ID is `12345678`). - -## How Authentication Works - -The egg gateway generates **short-lived GitHub tokens** (1-hour TTL) from your App credentials at runtime. It: - -1. Creates a JWT signed with the App's private key -2. Exchanges the JWT for an installation access token via GitHub's API -3. Caches the token in-memory and auto-refreshes 15 minutes before expiry - -This means no long-lived tokens are stored — the gateway generates fresh tokens on demand. - -For workflow steps that run outside the egg action (reactions, checkout, prompt building), the `actions/create-github-app-token` action generates a separate short-lived token from the same App credentials. - -## Files - -The implementation consists of two files: - -### `.github/workflows/on-mention.yml` - -The workflow file that listens for GitHub events and orchestrates the response. It: - -1. Filters events to only those mentioning the bot from authorized users -2. Generates a short-lived bot token via `actions/create-github-app-token` -3. Adds a reaction to acknowledge the mention -4. Checks out the correct branch (PR head branch or main) -5. Builds a context-rich prompt using `build-mention-prompt.sh` -6. Runs the egg action (which generates its own tokens inside the gateway) -7. Posts a summary comment with a link to the workflow run - -### `action/build-mention-prompt.sh` - -A reusable bash script that constructs prompts from GitHub event payloads. It: - -- Parses `$GITHUB_EVENT_PATH` with `jq` to extract event data -- Uses `gh api` to fetch additional context (comment threads, PR files) -- Outputs a structured prompt to `$GITHUB_OUTPUT` -- Truncates long content to stay within prompt limits (~50K chars) - -## Customization - -### Changing the Bot Username - -1. In `on-mention.yml`, replace `@james-in-a-box` in the `if:` conditions -2. Update the `BOT_USERNAME` env var in the "Build prompt" step -3. Update the `bot-username` input in the "Run egg" step - -### Changing Authorized Users - -The workflow uses `github.event.sender.login == 'jwbron'` to restrict who can trigger egg. - -**Single user:** -```yaml -github.event.sender.login == 'your-username' -``` - -**Multiple users:** -```yaml -contains(fromJSON('["user1","user2","user3"]'), github.event.sender.login) -``` - -**Team membership** (requires an API call in a prior step): -```yaml -- name: Check team membership - id: auth - run: | - gh api orgs/YOUR_ORG/teams/YOUR_TEAM/memberships/${{ github.event.sender.login }} \ - && echo "authorized=true" >> "$GITHUB_OUTPUT" \ - || echo "authorized=false" >> "$GITHUB_OUTPUT" - env: - GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} -``` - -### Adjusting Timeout - -The egg action defaults to 30 minutes. Override with: -```yaml -- uses: jwbron/egg@main - with: - timeout: "60" # 60 minutes -``` - -### Concurrency - -The default configuration groups concurrent runs by issue/PR number and does not cancel in-progress runs. To change this: - -```yaml -concurrency: - group: egg-mention-${{ github.event.issue.number || github.event.pull_request.number }} - cancel-in-progress: true # Cancel previous run if new mention arrives -``` - -## Security Considerations - -- **Authorization**: Always restrict who can trigger egg. Without the sender check, anyone who can comment could invoke it. -- **Loop prevention**: The workflow filters out the bot's own username to prevent infinite loops where egg's comments trigger itself. -- **Concurrency**: Runs are serialized per issue/PR to prevent race conditions when egg pushes code. -- **Short-lived tokens**: The gateway generates tokens with 1-hour TTL and auto-refreshes them. No long-lived PATs are stored. -- **Prompt injection**: The prompt includes user-provided content (comment bodies, issue descriptions). egg's system prompt and sandbox provide defense in depth. - -## Troubleshooting - -| Problem | Solution | -|---------|----------| -| Workflow doesn't trigger | Check the `if:` conditions — username, bot mention, and event type must all match | -| Reaction not added | Verify `BOT_APP_ID` and `BOT_APP_PRIVATE_KEY` secrets are set and the app has Issues/PRs write permission | -| "Generate bot token" step fails | Verify all three `BOT_APP_*` secrets are correctly set — check App ID, private key PEM format, and installation ID | -| egg can't push to PR | Ensure the checkout step uses the PR head ref and the bot has Contents write permission | -| Gateway logs "Token refresher not configured" | Check that `BOT_APP_ID`, `BOT_APP_PRIVATE_KEY`, and `BOT_APP_INSTALLATION_ID` are all passed to the action | -| Prompt too large | Adjust `MAX_BODY_CHARS` and `MAX_COMMENT_CHARS` in `build-mention-prompt.sh` | diff --git a/docs/index.md b/docs/index.md index 9b2e4e9e92..7434c468e0 100644 --- a/docs/index.md +++ b/docs/index.md @@ -27,33 +27,6 @@ This index helps both humans and LLMs navigate the documentation efficiently. |----------|-------------| | [Architecture Overview](architecture/README.md) | High-level system design and security model | -### Setup Guides - -| Document | Description | -|----------|-------------| -| [Setup Overview](setup/README.md) | Installation and configuration summary | -| [GitHub App Setup](setup/github-app-setup.md) | GitHub App permissions and installation | -| [GitHub Auth Comparison](setup/github-auth-comparison.md) | Auth method options (GitHub App vs PAT) | - -### Features - -| Document | Description | -|----------|-------------| -| [Container Infrastructure](features/container-infrastructure.md) | Container management and development environment | -| [GitHub Action](features/github-action.md) | Running egg in GitHub Actions CI/CD | - -### Guides - -| Document | Description | -|----------|-------------| -| [@mention Trigger Setup](guides/mention-trigger-setup.md) | Trigger egg via GitHub @mentions | - -### Reference - -| Document | Description | -|----------|-------------| -| [Log Persistence](reference/log-persistence.md) | Container log persistence and correlation | - ### Development | Document | Description | @@ -61,12 +34,6 @@ This index helps both humans and LLMs navigate the documentation efficiently. | [Project Structure](development/STRUCTURE.md) | Directory conventions and organization | | [Contributing](../CONTRIBUTING.md) | Development setup, workflow, and PR process | -### Troubleshooting - -| Document | Description | -|----------|-------------| -| [GitHub Auth Issues](troubleshooting/github-auth-in-long-running-containers.md) | Token expiry and refresh in containers | - ## Component Documentation Each major component has its own README with detailed documentation: @@ -87,30 +54,22 @@ Each major component has its own README with detailed documentation: |-----------|------------|--------------| | **Gateway changes** | [Architecture Overview](architecture/README.md) | [ADR: Git Isolation](adr/implemented/ADR-Git-Isolation-Architecture.md), [Gateway README](../gateway/README.md) | | **Security-related changes** | [ADR: Autonomous SE](adr/in-progress/ADR-Autonomous-Software-Engineer.md) | [Git Isolation](adr/implemented/ADR-Git-Isolation-Architecture.md) | -| **Sandbox changes** | [Sandbox README](../sandbox/README.md) | [Container Infrastructure](features/container-infrastructure.md) | +| **Sandbox changes** | [Sandbox README](../sandbox/README.md) | [Architecture Overview](architecture/README.md) | | **Configuration changes** | [Config README](../config/README.md) | [egg_config README](../shared/egg_config/README.md) | -| **@mention trigger setup** | [Mention Trigger Setup](guides/mention-trigger-setup.md) | [Architecture Overview](architecture/README.md) | -| **GitHub Action setup** | [GitHub Action](features/github-action.md) | [ADR: GitHub Actions](adr/in-progress/ADR-GitHub-Actions-Support.md) | +| **GitHub Action setup** | [ADR: GitHub Actions](adr/in-progress/ADR-GitHub-Actions-Support.md) | [Architecture Overview](architecture/README.md) | | **Adding tests** | [Contributing](../CONTRIBUTING.md) | [Project Structure](development/STRUCTURE.md) | ## Quick Navigation **Getting Started:** 1. [Main README](../README.md) - Project overview and quick start -2. [Setup Overview](setup/README.md) - Installation and configuration -3. [Contributing](../CONTRIBUTING.md) - Development setup +2. [Contributing](../CONTRIBUTING.md) - Development setup **Understanding the System:** 1. [Architecture Overview](architecture/README.md) - Component design 2. [ADR: Autonomous SE](adr/in-progress/ADR-Autonomous-Software-Engineer.md) - Full architecture 3. [Project Structure](development/STRUCTURE.md) - Code organization -**Operating egg:** -1. [GitHub Action](features/github-action.md) - CI/CD integration -2. [@mention Trigger Setup](guides/mention-trigger-setup.md) - Trigger via comments -3. [Log Persistence](reference/log-persistence.md) - Log management -4. [Troubleshooting](troubleshooting/github-auth-in-long-running-containers.md) - Common issues - --- *Last updated: 2026-02-05* diff --git a/docs/plans/github-actions-implementation-plan.md b/docs/plans/github-actions-implementation-plan.md deleted file mode 100644 index d521003691..0000000000 --- a/docs/plans/github-actions-implementation-plan.md +++ /dev/null @@ -1,291 +0,0 @@ -# Implementation Plan: GitHub Actions Support - -**ADR:** [GitHub Actions Support](../adr/in-progress/ADR-GitHub-Actions-Support.md) -**Issue:** #78 -**ADR PR:** #98 - -> **Dependency note:** The ADR (PR #98) is still under review and may -> change before merging. This plan should be re-validated against the -> final ADR before implementation begins. If the ADR's design changes -> materially, sections of this plan may need revision. - -## Overview - -This plan covers the Phase 1 (MVP) implementation of running egg as a -GitHub Action. The ADR proposes a composite GitHub Action that -orchestrates the full egg stack (gateway + sandbox) within a GHA runner, -accepting the checked-out repo as the working repository. - -## Codebase Observations - -Findings from reviewing the existing codebase that affect implementation -decisions. - -### 1. Port Discrepancy - -The gateway `Dockerfile` EXPOSE says 9847/3128, but the actual ports are -**9848** (API, per `sandbox/egg_lib/config.py:53` and -`gateway/entrypoint.sh:133`) and **3129** (proxy, per -`sandbox/egg_lib/config.py:54`). The ADR references 9847 in one place — -the implementation must use the correct ports (9848/3129). The Dockerfile -EXPOSE should be fixed as part of this work. - -### 2. Session Creation Is Atomic - -`POST /api/v1/sessions/create` takes -`{container_id, container_ip, mode, repos, uid, gid}` and atomically -creates session + worktrees + filters by visibility -(`sandbox/egg_lib/gateway.py:449-508`). The GHA entrypoint must replicate -this flow via `curl` rather than the existing Python client. - -### 3. Gateway Config Requirements - -The gateway's config layer expects a full `repositories.yaml` including -`writable_repos`, `bot_username`, `repo_settings`, `user_mode` (parsed by -`config/repo_config.py`), and `local_repos.paths` (parsed by -`shared/egg_config/config.py`). See `config/repositories.yaml.example` -for the complete schema. The config generator must produce all required -fields — a minimal config will cause parse failures. - -Note: `gateway/repo_parser.py` is a URL/path parsing utility for -extracting owner/repo from GitHub URLs — it does not parse the config -file. - -### 4. Credential File Layout - -The gateway reads Anthropic credentials from a `secrets.env` file mounted -at `~/.config/egg/secrets.env` (`gateway/anthropic_credentials.py:31-33`). -Launcher auth uses a separate `launcher-secret` file -(`gateway/entrypoint.sh:47-53`). The GHA entrypoint must create both files -and mount them into the gateway container. - -### 5. Build Context - -Both Dockerfiles use the **repo root** as build context: -- Gateway: `docker build -f gateway/Dockerfile .` -- Sandbox: `docker build -f sandbox/Dockerfile .` - (sandbox Dockerfile copies the entire repo into `/opt/egg-runtime/` and - sets `PYTHONPATH` to reference `sandbox/` and `shared/` subdirectories) - -### 6. Dual-Network Architecture - -Locally, the gateway is dual-homed on `egg-isolated` (172.32.0.0/24) and -`egg-external` (172.33.0.0/24). The GHA entrypoint must replicate this -for the full security model — private sandbox containers route through the -proxy on the isolated network, public containers use the external network -with direct internet access. - -### 7. No Gateway/Sandbox Code Changes Required - -The ADR was designed to reuse existing infrastructure. After reviewing the -codebase, this holds: -- Session management API already supports external callers via - `launcher_secret` auth -- Worktree creation works with any repo mounted at - `/home/egg/repos/` -- Anthropic credential injection reads from `secrets.env`, no changes - needed -- Policy enforcement (branch ownership, merge blocking) works as-is -- Config parsing (`config/repo_config.py`, `shared/egg_config/config.py`) - accepts the YAML format we generate - -The only change to existing code is the cosmetic EXPOSE fix in -`gateway/Dockerfile`. - -## Deliverables - -### 1. `action/action.yml` — Composite Action Definition - -Defines the GitHub Action interface with inputs, outputs, and a composite -`runs` block that invokes the entrypoint script. - -**Inputs:** - -| Input | Required | Default | Description | -|-------|----------|---------|-------------| -| `prompt` | Yes | — | Task prompt for Claude Code | -| `anthropic-oauth-token` | Yes | — | Anthropic OAuth token | -| `github-token` | Yes | `${{ github.token }}` | GitHub token for git ops | -| `bot-github-token` | No | — | Bot GitHub App token | -| `bot-username` | No | `egg` | Bot GitHub username | -| `mode` | No | `auto` | `public`, `private`, or `auto` | -| `timeout` | No | `30` | Timeout in minutes | -| `model` | No | `opus` | Claude model to use | - -**Outputs:** - -| Output | Description | -|--------|-------------| -| `exit-code` | Sandbox container exit code | -| `pr-url` | URL of created PR, if any | -| `log-file` | Path to full Claude output log | - -Inputs are passed to the entrypoint as environment variables following -GitHub's composite action convention (`INPUT_PROMPT`, etc.). - -### 2. `action/entrypoint.sh` — Container Orchestration (~300 lines) - -The most complex deliverable. Replicates the orchestration flow from -`sandbox/egg_lib/runtime.py` and `sandbox/egg_lib/gateway.py` in bash. - -**Step-by-step flow:** - -| Step | What | How | Existing reference | -|------|------|-----|-------------------| -| 1 | Pull images | `docker pull ghcr.io/jwbron/egg-gateway:` + sandbox | New | -| 2 | Create networks | Inspect existing Docker networks, allocate unused 172.x.0.0/24 subnets, create `egg-gha-isolated-$RUN_ID` and `egg-gha-external-$RUN_ID`. **Note:** The local dev setup uses hardcoded subnets (`172.32.0.0/24`, `172.33.0.0/24` per `config.py:59-65`), but GHA intentionally uses dynamic allocation to avoid collisions when multiple concurrent runs share a self-hosted runner (or when a runner also runs egg locally). | `config.py:59-65` | -| 3 | Detect mode | If `auto`: read `$GITHUB_EVENT_REPOSITORY_VISIBILITY` (set from `${{ github.event.repository.visibility }}` in `action.yml`) — `private`/`internal`→private, `public`→public. This avoids an extra API call since the visibility is already available in the workflow event context | ADR spec | -| 4 | Generate config | Call `generate-config.sh` to produce repositories.yaml, secrets.env, launcher-secret in temp dir | New (see deliverable #3) | -| 5 | Start gateway | `docker run -d` with mounts for config dir, workspace `.git`, worktrees dir, state dir, certs dir. Env vars: `GITHUB_USER_TOKEN`, `CLAUDE_CODE_OAUTH_TOKEN`, `HOST_UID`, `HOST_GID`, `BOT_GITHUB_TOKEN` (if provided), gateway policy vars | `sandbox/egg_lib/gateway.py:808-831` | -| 6 | Health check | Poll `http://:9848/api/v1/health` via `curl` on Docker network (no port publishing needed) | `sandbox/egg_lib/gateway.py:681-754` | -| 7 | Allocate IP | `docker network inspect` to find assigned IPs, select next available in subnet | `sandbox/egg_lib/runtime.py:189-247` | -| 8 | Create session | `curl -X POST http://:9848/api/v1/sessions/create` with `Authorization: Bearer `. Body: `{container_id, container_ip, mode, repos, uid, gid}`. Parse response for `session_token` and `worktrees` | `sandbox/egg_lib/gateway.py:449-508` | -| 9 | Start sandbox | `docker run` with: worktree path from session response mounted at `/home/egg/repos/`, `.git` shadowed via bind-mount of `/dev/null` (prevents the sandbox from accessing the full repo history or manipulating the worktree's parent repo directly — the sandbox should only interact with its isolated worktree copy), `EGG_SESSION_TOKEN`, `ANTHROPIC_BASE_URL=http://egg-gateway:9848`, Claude Code in `--exec` mode with `--print --output-format=stream-json` | `sandbox/egg_lib/runtime.py:646-740` | -| 10 | Capture output | Tee sandbox container logs to file and `$GITHUB_STEP_SUMMARY`, extract PR URLs via regex for step outputs | New | -| 11 | Cleanup | `trap EXIT` handler: delete session via gateway API, stop+rm containers, remove networks | `sandbox/egg_lib/runtime.py:351-388` | - -**Implementation details:** - -- **Container naming:** Include `$GITHUB_RUN_ID` to avoid conflicts on - self-hosted runners: `egg-gha-gateway-${GITHUB_RUN_ID}`, - `egg-gha-sandbox-${GITHUB_RUN_ID}` -- **Network naming:** `egg-gha-isolated-${GITHUB_RUN_ID}` and - `egg-gha-external-${GITHUB_RUN_ID}` -- **No port publishing:** Containers communicate over Docker network. - Health checks use container IP on the network or `docker exec` -- **Timeout:** Run the sandbox container with - `docker run --stop-timeout 30 ...` and use a background timer that - calls `docker stop` after `$TIMEOUT_MINUTES`. This ensures Docker - sends SIGTERM to PID 1 inside the container and waits for graceful - shutdown, rather than `timeout(1)` sending SIGTERM to the `docker` - client process which may not propagate cleanly to the workload -- **Gateway dual-homing:** Start on isolated network, then - `docker network connect` to external network (matching - `sandbox/egg_lib/gateway.py:839-853`) -- **Exit code propagation:** Capture sandbox exit code and write to - `$GITHUB_OUTPUT` - -### 3. `action/generate-config.sh` — Config Generation (~100 lines) - -Creates a temp directory (`$RUNNER_TEMP/egg-config-$RUN_ID/`) containing -the three files the gateway needs. - -**`repositories.yaml`** (full format for `config/repo_config.py`): -```yaml -github_username: <$GITHUB_ACTOR> -bot_username: -writable_repos: - - <$GITHUB_REPOSITORY> -repo_settings: - <$GITHUB_REPOSITORY>: - auth_mode: <"bot" if bot-github-token provided, else "user"> -user_mode: - github_user: <$GITHUB_ACTOR> - git_name: <$GITHUB_ACTOR> - git_email: <$GITHUB_ACTOR_ID>+<$GITHUB_ACTOR>@users.noreply.github.com -local_repos: - paths: - - /home/egg/repos/ -``` - -**`secrets.env`:** -``` -CLAUDE_CODE_OAUTH_TOKEN= -``` - -**`launcher-secret`:** Generated via `openssl rand -base64 32`. - -**Workspace path mapping:** `$GITHUB_WORKSPACE` is bind-mounted into the -gateway container at `/home/egg/repos/` where `` is -derived from `$GITHUB_REPOSITORY` (strips owner prefix). The gateway's -worktree manager uses the `.git` directory at that path to create -worktrees at `~/.egg-worktrees//`. - -### 4. `.github/workflows/release-images.yml` — GHCR Publishing - -Builds and pushes gateway and sandbox Docker images to GHCR on release -events. - -**Triggers:** `release: [published]` + `workflow_dispatch` for manual -testing. - -**Images:** -- `ghcr.io/jwbron/egg-gateway:latest` and `:$TAG` -- `ghcr.io/jwbron/egg-sandbox:latest` and `:$TAG` - -**Build details:** -- Uses `docker/build-push-action@v6` with repo root as context -- Gateway: `-f gateway/Dockerfile` -- Sandbox: `-f sandbox/Dockerfile` -- amd64 only for Phase 1 (matches `ubuntu-latest` runners) -- Requires `packages: write` permission - -### 5. `.github/workflows/test-action.yml` — Integration Tests - -Runs the action on PRs that modify `action/**` files. - -**Test strategy:** -- Minimal test: verify config generation, container startup, gateway - health (no API key required) -- Full test: run with a simple prompt and verify exit code (requires - `ANTHROPIC_OAUTH_TOKEN` secret) -- Validate outputs: exit-code is set, log-file exists - -### 6. Gateway Dockerfile EXPOSE Fix - -Change `EXPOSE 9847 3128` to `EXPOSE 9848 3129` in `gateway/Dockerfile` -to match actual ports. - -## Implementation Sequence - -| Order | Deliverable | Depends on | Notes | -|-------|------------|------------|-------| -| 1 | Gateway Dockerfile EXPOSE fix | — | One-line change | -| 2 | `release-images.yml` | — | Must publish images before action can pull them | -| 3 | `action/generate-config.sh` | — | Self-contained, testable in isolation | -| 4 | `action/entrypoint.sh` | #2, #3 | Core logic, largest piece | -| 5 | `action/action.yml` | #4 | Thin wrapper around entrypoint | -| 6 | `test-action.yml` | #5 | Integration verification | - -## Risks and Mitigations - -| Risk | Impact | Mitigation | -|------|--------|------------| -| **Sandbox image size (2-3GB)** causes slow pulls | Cold start adds 30-60s | GitHub CDN + `actions/cache` for Docker layers (Phase 2) | -| **Gateway health check timing** | Action may timeout waiting for startup | Aggressive polling (0.5s interval), 60s max wait before failing | -| **`GITHUB_TOKEN` permission scope** insufficient | Push/PR operations fail | Document required permissions: `contents: write`, `pull-requests: write`. Recommend PAT for full functionality | -| **Self-hosted runner conflicts** | Container/network name collisions with concurrent runs | Include `$GITHUB_RUN_ID` in all Docker resource names | -| **Workspace `.git` ownership mismatch** | Gateway can't read `.git` directory | Use existing `HOST_UID`/`HOST_GID` mechanism in gateway entrypoint to match runner user | -| **Session API calls via curl** harder to debug than Python | Opaque failures | Verbose logging with `set -x` in debug mode, clear error messages on each API call failure | -| **ARM64 runners** not supported | Action fails on ARM self-hosted runners | Phase 1: amd64 only (matches `ubuntu-latest`). Document limitation. | -| **Network subnet collisions** | Docker network creation fails | Dynamic subnet allocation: inspect existing networks and pick unused range | - -## Open Items to Resolve During Implementation - -1. **Claude Code `--exec` invocation:** Verify exact CLI flags for - non-interactive mode. ADR specifies `--print --output-format=stream-json` - — validate against installed Claude Code version in sandbox image. - -2. **PR URL extraction from output:** Determine reliable pattern to - extract PR URLs from Claude Code's stream-json output for the `pr-url` - action output. May need to parse JSON lines for specific event types. - -3. **`$GITHUB_WORKSPACE` ownership:** The checkout action creates the repo - as the runner user (typically uid 1001). Verify the gateway's - `HOST_UID`/`HOST_GID` + `gosu` mechanism handles this correctly when - creating worktrees from the mounted `.git`. - -4. **Sandbox Dockerfile build context:** Confirm that - `sandbox/Dockerfile` builds correctly with repo root as context - (the `COPY . /opt/egg-runtime/` line copies the entire repo). - A `.dockerignore` to exclude unnecessary files (docs, tests, - `.github/`) would reduce image size but is a change to the existing - build — this should be handled in a separate PR. - -5. **Image tagging strategy:** `action/action.yml` should reference a - pinned version tag (e.g., `:v1.0.0`) rather than `:latest`. Using - `latest` in a GitHub Action is unreliable — users who pin `@v1` of - the action would still get unpredictable image versions. The release - workflow should tag images with the release version, and the - `action.yml` should be updated as part of each release to reference - the corresponding image tag. diff --git a/docs/reference/README.md b/docs/reference/README.md deleted file mode 100644 index aafb400dcf..0000000000 --- a/docs/reference/README.md +++ /dev/null @@ -1,29 +0,0 @@ -# Reference Documentation - -Quick reference guides for egg. - -## Available References - -### [Log Persistence](log-persistence.md) -Container log persistence and correlation. - -## Common Issues - -**Slack not receiving notifications:** -1. Check service: `systemctl --user status slack-notifier` -2. Verify token: Check logs for authentication errors -3. Test manually: Write file to `~/.egg-sharing/notifications/` - -**Claude Code not available:** -1. Authenticate: `claude auth login` -2. Check version: `claude --version` -3. Verify PATH: `which claude` - -**Container issues:** -1. Rebuild: `bin/egg --rebuild` -2. Check Docker: `docker ps` -3. View logs: `docker logs egg-claude -f` - -## See Also -- [Setup Guides](../setup/) -- [Architecture](../architecture/) diff --git a/docs/reference/log-persistence.md b/docs/reference/log-persistence.md deleted file mode 100644 index 809ee85ad6..0000000000 --- a/docs/reference/log-persistence.md +++ /dev/null @@ -1,218 +0,0 @@ -# Egg Container Log Persistence - -This document describes the log persistence and correlation system for egg containers. - -## Overview - -Egg containers are ephemeral - they're removed after execution completes. To preserve logs for debugging and auditing, the system now: - -1. **Persists container logs** to `~/.egg-sharing/container-logs/` -2. **Creates correlation links** between task IDs, thread timestamps, and container IDs -3. **Maintains a searchable log index** for quick lookups - -## Log Locations - -| Location | Contents | Persisted? | -|----------|----------|------------| -| `~/.egg-sharing/container-logs/` | Docker container stdout/stderr | Yes | -| `~/.egg-sharing/logs/` | Claude output streams (real-time) | Yes | -| Docker daemon logs | Internal Docker logs | Via json-file driver | - -## Log Correlation - -Every container execution is tagged with correlation IDs: - -### Environment Variables (Inside Container) -- `CONTAINER_ID` - Unique container identifier (e.g., `egg-exec-20251129-222239-12345`) -- `EGG_TASK_ID` - Task identifier (e.g., `task-20251129-222239`) -- `EGG_THREAD_TS` - Thread timestamp (e.g., `1764483758.159619`) - -### Docker Labels -- `egg.container_id` - Container ID label -- `egg.task_id` - Task ID label (if available) - -### Log Index File -`~/.egg-sharing/container-logs/log-index.json` contains: -```json -{ - "task_to_container": { - "task-20251129-222239": "egg-exec-20251129-222239-12345" - }, - "thread_to_task": { - "1764483758.159619": "task-20251129-222239" - }, - "entries": [ - { - "container_id": "egg-exec-20251129-222239-12345", - "task_id": "task-20251129-222239", - "thread_ts": "1764483758.159619", - "log_file": "/home/user/.egg-sharing/container-logs/egg-exec-20251129-222239-12345.log", - "timestamp": "2025-11-29T22:22:39.123456" - } - ] -} -``` - -## Using the egg-logs Utility - -The `egg-logs` utility provides easy access to persisted logs: - -### List Recent Logs -```bash -egg-logs # List last 20 logs -egg-logs --list 50 # List last 50 logs -``` - -### View Logs for a Task -```bash -egg-logs task-20251129-222239 -``` - -### Search Logs -```bash -egg-logs --search "error" # Search all logs -egg-logs --search "authentication" # Case-insensitive regex -``` - -### Follow Logs (Tail) -```bash -egg-logs --tail task-20251129-222239 -``` - -### Clean Up Old Logs -```bash -egg-logs --cleanup --days 7 # Remove logs older than 7 days -egg-logs --cleanup --days 30 --dry-run # Preview what would be removed -``` - -## Log File Structure - -Each container log file contains: - -``` -=== Container: egg-exec-20251129-222239-12345 === -=== Saved: 2025-11-29T22:25:00.123456 === -=== Task ID: task-20251129-222239 === -=== Thread TS: 1764483758.159619 === -================================================== - -=== STDOUT === -[Container stdout output...] - -=== STDERR === -[Container stderr output...] -``` - -## Correlation Flow - -When a Slack message triggers a container: - -1. **slack-receiver** writes message to `~/.egg-sharing/incoming/task-{timestamp}.md` -2. **egg --exec** extracts task_id and thread_ts from the task file -3. Container is started with correlation environment variables and Docker labels -4. On container exit, logs are captured via `docker logs` and saved -5. Log index is updated with correlation mappings -6. Symlink is created: `task-{id}.log -> egg-exec-{container-id}.log` - -## Debugging a Slack Thread - -To find all logs related to a Slack thread: - -```bash -# If you have the thread timestamp -egg-logs --search "1764483758.159619" - -# If you have the task ID -egg-logs task-20251129-222239 - -# Search by content -egg-logs --search "your error message" -``` - -## Storage Management - -Logs are stored with rotation: -- Container logs: No automatic rotation (use `egg-logs --cleanup`) -- Claude output logs: Also use manual cleanup - -Recommended cleanup policy: -```bash -# Add to crontab for weekly cleanup -0 0 * * 0 /path/to/egg-logs --cleanup --days 14 -``` - -## Security Considerations - -### Sensitive Data in Logs - -**IMPORTANT**: Container logs may contain sensitive information: - -- **API Keys and Tokens**: Environment variables, authentication headers -- **Credentials**: Database passwords, SSH keys, OAuth secrets -- **Personal Data**: User information, email addresses, API responses -- **Internal Details**: System paths, configuration details - -### Security Recommendations - -1. **Storage Encryption**: Logs are stored **unencrypted** in `~/.egg-sharing/container-logs/` - - Ensure filesystem encryption if handling sensitive data - - Consider encrypting the entire `.egg-sharing` directory - -2. **Access Control**: - - Logs are readable by the user running egg - - Ensure proper file permissions on the `.egg-sharing` directory - - Avoid sharing logs without sanitization - -3. **Regular Cleanup**: - ```bash - # Clean up old logs to minimize exposure window - egg-logs --cleanup --days 7 - ``` - -4. **Shared Environments**: - - **DO NOT** use egg on shared systems without considering log exposure - - Implement log sanitization if logs will be shared (e.g., for debugging) - - Consider using separate egg instances for sensitive vs. non-sensitive work - -5. **Log Retention Policy**: - - Default: Manual cleanup only - - Recommended: Automated weekly/monthly cleanup via cron - - For compliance: Align retention with your organization's data retention policies - -### Sanitizing Logs for Sharing - -Before sharing logs with others: - -```bash -# Create a sanitized copy -egg-logs task-20251129-222239 > /tmp/log.txt - -# Manually review and redact: -# - API keys (look for "Bearer", "token", "key") -# - Passwords (look for "password", "secret") -# - Personal data (emails, names, IDs) -``` - -Consider creating a sanitization script if you frequently share logs. - -## Troubleshooting - -### Logs Not Being Saved -1. Check if `~/.egg-sharing/container-logs/` exists and is writable -2. Verify container completed (not killed mid-execution) -3. Check egg launcher output for errors - -### Can't Find Logs for a Task -1. Try searching: `egg-logs --search "task-20251129"` -2. Check the log index: `cat ~/.egg-sharing/container-logs/log-index.json` -3. List all logs: `egg-logs --list 100` - -### Old Logs Taking Up Space -```bash -# Check disk usage -du -sh ~/.egg-sharing/container-logs/ -du -sh ~/.egg-sharing/logs/ - -# Clean up -egg-logs --cleanup --days 7 -``` diff --git a/docs/setup/README.md b/docs/setup/README.md deleted file mode 100644 index f0640bfb55..0000000000 --- a/docs/setup/README.md +++ /dev/null @@ -1,66 +0,0 @@ -# Setup Guides - -Installation and configuration documentation. - -## Quick Start - -```bash -# Clone and set up -git clone https://github.com/YOUR_USERNAME/egg.git -cd egg -./dev setup - -# Configure credentials -cp secrets.yaml.example ~/.config/egg/secrets.yaml -# Edit with your GitHub App / PAT and Anthropic credentials - -# Start the sandbox (public mode) -egg start --config egg.yaml - -# Start with network lockdown (private mode) -egg start --config egg.yaml --private -``` - -## Setup Requirements - -Before running setup, you'll need: - -1. **GitHub App** (required for PR creation) - - See [GitHub App Setup](github-app-setup.md) for instructions - - Alternative: Personal Access Token (PAT) - - See [GitHub Auth Comparison](github-auth-comparison.md) for trade-offs - -2. **Anthropic API Key** (required for Claude) - -3. **Prerequisites** - - Docker installed and running - - Python 3 installed - -## Available Guides - -### [GitHub App Setup](github-app-setup.md) -GitHub App configuration for automated PR creation: -- Required permissions (read-only and read-write) -- Installation steps -- Token configuration -- Troubleshooting common permission errors - -### [GitHub Auth Comparison](github-auth-comparison.md) -Comparison of authentication methods (GitHub App vs PAT). - -## Configuration Location - -All configuration is stored in `~/.config/egg/`: - -| File | Purpose | -|------|---------| -| `secrets.yaml` | Anthropic API key, GitHub credentials | -| `github-app-id` | GitHub App ID | -| `github-app-installation-id` | GitHub App Installation ID | -| `github-app.pem` | GitHub App private key | - -## See Also - -- [Main README](../../README.md) - Project overview and quick start -- [Architecture](../architecture/) - System design -- [Reference](../reference/) - Quick reference guides diff --git a/docs/setup/github-app-setup.md b/docs/setup/github-app-setup.md deleted file mode 100644 index b2908ddc70..0000000000 --- a/docs/setup/github-app-setup.md +++ /dev/null @@ -1,117 +0,0 @@ -# GitHub App Setup - -This guide covers setting up the `egg` GitHub App for automated PR creation and workflow management. - -## Required Permissions - -The GitHub App requires the following permissions to function properly: - -### Read-Only Permissions - -| Permission | Purpose | -|------------|--------| -| **Actions** | Read workflow run status and logs | -| **Checks** | Read check run status for CI/CD monitoring | -| **Commit statuses** | Read commit status indicators | -| **Dependabot alerts** | Read security vulnerability alerts | -| **Discussions** | Read repository discussions | -| **Merge queues** | Read merge queue status | - -### Read-Write Permissions - -| Permission | Purpose | -|------------|--------| -| **Contents** | Push commits, create/update files | -| **Pull requests** | Create PRs, add comments, request reviews | -| **Workflows** | Trigger and manage GitHub Actions workflows | - -## Installation Steps - -### 1. Create GitHub App - -1. Go to GitHub Settings > Developer settings > GitHub Apps -2. Click "New GitHub App" -3. Configure: - - **Name**: `egg` - - **Homepage URL**: Your documentation URL - - **Webhook**: Disable unless needed - -### 2. Set Permissions - -Under "Permissions & events": - -**Repository permissions:** -- Actions: Read-only -- Checks: Read-only -- Commit statuses: Read-only -- Contents: Read and write -- Dependabot alerts: Read-only -- Discussions: Read-only -- Merge queues: Read-only -- Pull requests: Read and write -- Workflows: Read and write - -### 3. Generate Private Key - -1. Scroll to "Private keys" -2. Click "Generate a private key" -3. Save the `.pem` file securely - -### 4. Install App - -1. Go to the "Install App" tab -2. Select the repositories to grant access to -3. Confirm installation - -### 5. Configure Token - -The GitHub token is configured in your environment: - -```bash -# In your shell config or .env -export GITHUB_TOKEN="your-token-here" -``` - -The `gh` CLI and `git push` automatically use this token for authentication. - -## Verifying Permissions - -To verify the app has correct permissions: - -```bash -# Check token scopes -gh auth status - -# Test PR creation (dry-run) -gh pr create --dry-run --title "Test" --body "Test" - -# Test workflow access -gh workflow list -``` - -## Troubleshooting - -### "Resource not accessible by integration" Error - -This typically means a permission is missing. Check: -1. App permissions in GitHub Settings -2. Installation scope (which repos have access) -3. Token validity - -### Workflow Permission Errors - -If you see errors like "refusing to allow a GitHub App to create or update workflow": -1. Ensure "Workflows" permission is set to "Read and write" -2. Re-install the app to pick up new permissions - -### Contents Permission Errors - -If pushes fail: -1. Verify "Contents" permission is "Read and write" -2. Check branch protection rules -3. Ensure the app is installed on the target repository - -## Related Documentation - -- [GitHub Auth Comparison](github-auth-comparison.md) - Auth method options -- [Architecture Overview](../architecture/) - System design diff --git a/docs/setup/github-auth-comparison.md b/docs/setup/github-auth-comparison.md deleted file mode 100644 index 01187c8574..0000000000 --- a/docs/setup/github-auth-comparison.md +++ /dev/null @@ -1,145 +0,0 @@ -# GitHub Authentication: App vs Personal Access Token - -This document explains the differences between GitHub App authentication and Personal Access Tokens (PATs) for egg. - -## Summary - -**Both GitHub App and PAT authentication are supported.** However, GitHub Apps are required for certain operations that PATs cannot perform. - -## Authentication Methods - -### Personal Access Token (PAT) - -**What it is:** A user-level token that grants permissions based on your personal GitHub account. - -**Supported operations:** -- ✅ Clone repositories -- ✅ Push commits -- ✅ Create pull requests -- ✅ Comment on PRs -- ✅ Trigger workflows -- ✅ Read repository contents -- ❌ **Cannot read PR check runs** (requires GitHub App) - -**When to use:** -- Personal projects -- Simple setups -- You don't need PR check status monitoring - -**Setup:** -1. Go to https://github.com/settings/tokens -2. Create a token with: - - **Read/write repos:** `repo` (full scope), `workflow` - - **Read-only repos:** `repo` (read-only) -3. Use token in setup: `GITHUB_TOKEN=ghp_...` - -### GitHub App - -**What it is:** An application-level token that grants permissions to specific repositories independent of user accounts. - -**Supported operations:** -- ✅ All PAT operations (clone, push, PRs, comments, workflows) -- ✅ **Read PR check runs** (via GitHub Checks API) -- ✅ Team collaboration (not tied to a single user) -- ✅ Fine-grained repository permissions - -**When to use:** -- Team projects -- You need automatic token refresh (handled by gateway sidecar) -- Fine-grained permission control -- Production deployments - -**Setup:** -See [github-app-setup.md](github-app-setup.md) for detailed instructions. - -## Authentication Usage Audit - -### Where PATs are used: - -1. **Container authentication** (`egg` script, `entrypoint.py`) - - Sets `GITHUB_TOKEN` environment variable - - Configures `gh` CLI - - Configures git credential helper - -2. **Read-only monitoring** (`github_readonly_token` in `host_config.py`) - - For repositories where egg only watches but doesn't write - - Falls back to `GITHUB_TOKEN` if not set - -3. **All write operations** (PRs, commits, comments) - - Uses `GITHUB_TOKEN` from environment or config - -### Where GitHub Apps are required: - -1. **Token auto-refresh** (gateway sidecar `token_refresher.py`) - - GitHub App tokens expire after 1 hour - - Gateway sidecar auto-refreshes tokens 15 minutes before expiry - - Not needed for PATs (they don't expire automatically) - -2. **Fine-grained repository permissions** - - GitHub Apps can be installed on specific repositories - - Better security model for team environments - -## Recommendations - -### For Personal Use -```bash -# Use PAT if you don't need PR check monitoring -GITHUB_TOKEN=ghp_... (read/write) -GITHUB_READONLY_TOKEN=ghp_... (optional, for external repos) -``` - -### For Team Use -```bash -# Use GitHub App for full functionality -# Configure via setup.py option 1 -# Tokens auto-refresh via gateway sidecar -``` - -### Hybrid Setup (Current Implementation) -```bash -# You can mix both: -# 1. GitHub App for repositories where you want check monitoring -# 2. PAT for repositories where you only need basic operations - -# Both authentication methods work side-by-side -# The system will use GitHub App tokens when available, -# fall back to PAT otherwise -``` - -## Technical Details - -### Token Precedence - -For bot mode, tokens are managed by the gateway sidecar's in-memory token refresher. - -For user mode, tokens are loaded from: -1. `GITHUB_USER_TOKEN` environment variable -2. `~/.config/egg/secrets.env` - `GITHUB_USER_TOKEN` - -### Why PATs Can't Access Check Runs - -The GitHub Checks API requires: -- `checks:read` permission (only available to GitHub Apps) -- PATs use OAuth scopes which don't include check runs access - -From GitHub's documentation: -> "The Checks API is only available to GitHub Apps. OAuth Apps and authenticated users cannot access this endpoint." - -## Migration Path - -If you're currently using PATs and want PR check monitoring: - -1. Set up a GitHub App (see [github-app-setup.md](github-app-setup.md)) -2. Configure App credentials in `~/.config/egg/`: - - `github-app-id` - - `github-app-installation-id` - - `github-app.pem` -3. Restart the gateway sidecar: `systemctl --user restart gateway` -4. Keep your PAT as `GITHUB_READONLY_TOKEN` for external repos (optional) - -The gateway sidecar will automatically manage token refresh for the GitHub App. - -## Related Documentation - -- [GitHub App Setup](github-app-setup.md) -- [Host Configuration](../../config/README.md) diff --git a/docs/troubleshooting/github-auth-in-long-running-containers.md b/docs/troubleshooting/github-auth-in-long-running-containers.md deleted file mode 100644 index fe351eee45..0000000000 --- a/docs/troubleshooting/github-auth-in-long-running-containers.md +++ /dev/null @@ -1,102 +0,0 @@ -# GitHub Auth in Long-Running Containers - -## Problem Statement - -Long-running containers may lose GitHub access after approximately 1 hour. Users see errors like: -- `HTTP 401: Bad credentials` -- `403 Resource not accessible by integration` - -## Root Cause Analysis - -### Token Lifecycle - -1. **Token Generation**: GitHub App installation tokens expire after **1 hour** -2. **Token Refresh**: The gateway sidecar refreshes tokens automatically 15 minutes before expiry -3. **GitHub Access**: The `gh` CLI and `git` commands are routed through the gateway sidecar - -### Architecture - -``` -+-----------------------------------------------------------------+ -| GATEWAY SIDECAR | -+-----------------------------------------------------------------+ -| TokenRefresher (in-memory) | -| +- Refreshes tokens 15 minutes before expiry | -| +- Caches tokens in memory (thread-safe) | -| +- Falls back to cached token on refresh failure (up to 3x) | -| +- Clears cache after 3 consecutive failures (fail closed) | -+-----------------------------------------------------------------+ - | - | git/gh commands routed via gateway - v -+-----------------------------------------------------------------+ -| CONTAINER | -+-----------------------------------------------------------------+ -| git/gh wrappers -> gateway sidecar API -> GitHub | -+-----------------------------------------------------------------+ -``` - -## Diagnosis Steps - -### 1. Check Gateway Sidecar Health - -```bash -# Check if gateway is running -curl http://egg-gateway:9847/api/v1/health -``` - -### 2. Check Gateway Logs - -```bash -# On host -journalctl --user -u gateway -f - -# Or view recent logs -journalctl --user -u gateway --since "1 hour ago" -``` - -### 3. Test GitHub Token - -```bash -# In container - test via gateway -gh auth status -``` - -## Common Issues - -### Gateway Sidecar Not Running - -```bash -# On host - restart gateway -systemctl --user restart gateway -``` - -### Token Refresh Failures - -If you see "Max refresh failures reached" in gateway logs: - -1. Check GitHub App credentials in `~/.config/egg/`: - - `github-app-id` - - `github-app-installation-id` - - `github-app.pem` - -2. Verify the GitHub App is still installed on the target repositories - -3. Restart the gateway sidecar to reset the failure counter: - ```bash - systemctl --user restart gateway - ``` - -## Known Limitations - -### GitHub App Token Limitations - -GitHub App installation tokens have limited scope: -- `/user` endpoint returns `403 Resource not accessible by integration` (this is expected) -- Token only has access to repositories where the App is installed -- Use `/installation/repositories` to verify token validity - -## See Also - -- `docs/setup/github-app-setup.md` - GitHub App configuration -- `gateway/token_refresher.py` - In-memory token refresh implementation diff --git a/gateway/README.md b/gateway/README.md index 22a5c7bba8..730353abdb 100644 --- a/gateway/README.md +++ b/gateway/README.md @@ -158,4 +158,3 @@ make test - [ADR: Git Isolation](../docs/adr/implemented/ADR-Git-Isolation-Architecture.md) - Worktree isolation design - [ADR: Credential Injection](../docs/adr/implemented/ADR-Gateway-Credential-Injection.md) - Zero-credential sandbox - [ADR: Internet Lockdown](../docs/adr/in-progress/ADR-Internet-Tool-Access-Lockdown.md) - Network modes -- [Troubleshooting: GitHub Auth](../docs/troubleshooting/github-auth-in-long-running-containers.md) - Token refresh issues diff --git a/gateway/setup.sh b/gateway/setup.sh index 2164ff5261..50a0822c2b 100755 --- a/gateway/setup.sh +++ b/gateway/setup.sh @@ -152,7 +152,7 @@ check_prerequisites() { echo " - secrets.env with GITHUB_APP_ID and GITHUB_APP_INSTALLATION_ID" echo " - github-app.pem (private key)" echo "" - echo "Run 'egg --setup' to configure credentials, or see docs/setup/github-app-setup.md" + echo "Run 'egg --setup' to configure credentials" echo "" read -p "Continue anyway? (y/N) " -n 1 -r echo diff --git a/sandbox/README.md b/sandbox/README.md index c6b25574ce..1b965bbbf8 100644 --- a/sandbox/README.md +++ b/sandbox/README.md @@ -136,7 +136,6 @@ Container setup is automated via `docker-setup.py`, which runs on container star ## Related Documentation -- [Container Infrastructure Features](../docs/features/container-infrastructure.md) - Feature overview - [Claude Code Configuration](.claude/README.md) - Agent rules and commands - [Gateway Sidecar](../gateway/README.md) - Policy enforcement gateway - [Architecture Overview](../docs/architecture/README.md) - System design diff --git a/sandbox/egg_lib/setup_flow.py b/sandbox/egg_lib/setup_flow.py index 78bea0c030..56b7bb61f8 100644 --- a/sandbox/egg_lib/setup_flow.py +++ b/sandbox/egg_lib/setup_flow.py @@ -217,7 +217,7 @@ def _create_secrets_config() -> bool: if not current_app_id or not current_install_id: print() print("Enter GitHub App credentials:") - print(" (See docs/setup/github-app-setup.md for instructions)") + print(" (See https://docs.github.com/en/apps/creating-github-apps for GitHub App setup)") print() app_id = input("GitHub App ID: ").strip()