Skip to content

feat(e2e): implement working admin e2e tests with Playwright - #158

Closed
ralphbean wants to merge 1 commit into
mainfrom
agent-e2e-tests-working
Closed

feat(e2e): implement working admin e2e tests with Playwright#158
ralphbean wants to merge 1 commit into
mainfrom
agent-e2e-tests-working

Conversation

@ralphbean

@ralphbean ralphbean commented Apr 3, 2026

Copy link
Copy Markdown
Member

Summary

- Merges the design spec and implementation plan from PR #156

  • Implements the full admin e2e test suite using Playwright for browser automation
  • Fixes numerous reliability issues discovered during implementation
  • Tests pass reliably on consecutive runs (~84 seconds each)

Key Changes

E2E Test Infrastructure

  • E2E_GITHUB_PASSWORD_FILE support: Reads password from a file path when E2E_GITHUB_PASSWORD env var is not set
  • Cross-origin manifest fix: Fetches local manifest form via HTTP and submits from GitHub's origin to avoid SameSite=Lax cookie loss
  • workflow PAT scope: Required for creating files in .github/workflows/ directories
  • Retry logic for 404s: CreateOrUpdateFile retries with exponential backoff for newly created repos (GitHub eventual consistency)

Browser Automation

  • Replaced WaitUntilStateNetworkidle with WaitUntilStateDomcontentloaded (GitHub keeps persistent connections)
  • Added hook_attributes and redirect_url to manifest JSON for GitHub's manifest flow
  • Fixed app deletion: org-scoped URL, type app name to confirm, correct button selectors
  • Added debug screenshot helper for troubleshooting failures

Lock and Cleanup

  • Reduced lock timeout to 2 minutes, poll interval to 30 seconds
  • Added stale lock detection for non-UUID content
  • Uses CreateOrUpdateFile instead of CreateFile for lock (handles auto_init README)
  • Cleans up stale enrollment branches via GitHub API
  • Creates test-repo if it doesn't exist

Test Flow (all 6 phases pass)

  1. Cleanup: Delete stale apps, repos, branches from previous runs
  2. Install: Create 4 GitHub Apps, install them, write config/workflows/secrets
  3. Verify: Check all resources exist (config, workflows, secrets, variables, enrollment PR)
  4. Analyze: Verify layer stack reports correct status
  5. Uninstall: Delete config repo
  6. Verify uninstall: Confirm cleanup

Bug Fix

  • Fixed compilation error in deletePAT (wrong number of return values)

@ralphbean
ralphbean marked this pull request as ready for review April 3, 2026 13:19
@ralphbean
ralphbean force-pushed the agent-e2e-tests-working branch from b0dd44a to f5174de Compare April 3, 2026 14:00
Full lifecycle test: lock acquisition, teardown-first cleanup,
app creation via Playwright browser automation, layer stack
install/verify/analyze/uninstall/verify.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@ralphbean

Copy link
Copy Markdown
Member Author

Closing in favor of #167

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant