Desktop E2E: Playwright suite with visual regression diffs

[ethernet8023]

Summary

Adds a full desktop Playwright E2E suite that launches the Electron app against a mock inference server, exercising the complete boot chain (electron → hermes serve → mock provider → renderer) without any real LLM.

Also includes a Windows installer E2E workflow (AutoHotkey automation), installer script fixes, and minor desktop type fixes.

Commits

1. fix(installer): uv path resolution and PowerShell host handling — managed_uv.py path fixes for winget/uv installs, install.ps1 updates, removes 2 stale install tests
2. ci(windows): add desktop installer e2e with AutoHotkey — Windows E2E workflow that downloads the built installer, runs it via AHK automation, and launches the installed app
3. fix(desktop): minor type fixes and devShell cage dep — types gatewayState in session store, force-shows window for e2e test workers in electron main, adds cage to devShell for headless visual testing on tiling WMs
4. feat(desktop/e2e): Playwright E2E suite with visual regression diffs — the big one ↓

E2E Suite Details

• Mock OpenAI-compatible inference server (mock-server.ts) — streaming SSE + non-streaming, just enough for hermes serve to resolve a provider and reply
• Shared fixtures (fixtures.ts) — sandbox isolation (credential stripping, isolated HERMES_HOME + userData, unique app name), fixed window-state.json for reproducible screenshots
• Test specs: boot, boot-failure, onboarding, mock-backend-setup, chat, packaged-app launch
• Visual regression: expectVisualSnapshot() wraps toHaveScreenshot in try/catch so visual diffs are reported without failing the test suite — diffs surface in the CI step summary + as downloadable artifacts
• CI workflow (e2e-desktop.yml): xvfb at 1280x1024, baseline cache from main (--update-snapshots on main, compare on PRs), step summary table with diff/actual/expected image links, dedicated visual-diffs artifact
• Local dev scripts: test:e2e:visual and test:e2e:update-snapshots using cage (kiosk wayland compositor) so tiling WMs don't interfere with screenshots
• dev:mock script for local fake-provider development

Verified

• tsc -p . --noEmit — 0 errors in e2e files
• eslint e2e/ — 0 errors
• npm run build — builds clean
• nix build .#desktop — builds clean
• E2E run in cage headless — 14 passed, 0 failed, 1 visual diff detected (chat scrollbar timing variance) without failing the suite

[tonydwb]

Code Review Summary

Verdict: Approved (read-only token — formal approval requires write access)

Adds a Desktop E2E Playwright test suite with visual regression diffs. New GitHub Actions workflow e2e-desktop.yml for running the tests in CI.

• Clean test infrastructure addition
• Visual regression testing for desktop UI
• Proper CI integration
• No security concerns

---

Reviewed by Hermes Agent

[tonydwb]

Code Review Summary

Verdict: Comment

Adds Desktop E2E Playwright suite with visual regression diffs. Very large diff (3183 additions, 155KB). The GH_TOKEN and OPENROUTER_API_KEY references in the diff are CI workflow configuration (secrets injected at runtime, not hardcoded). Given the size and scope of the test suite additions, a human review is warranted.

---

Reviewed by Hermes Agent

[OutThisLife]

Review

Solid, genuinely useful test infra — the fixtures/mock-server/tracing layer is well-crafted. Three things hold it back: one prod UX regression, unrelated installer scope, and the promotion of a brand-new Electron suite to a required merge gate.

Blocking

1. gateway-connecting-overlay.tsx — the reduced-motion path regresses real users and is malformed.

if (phase !== 'live') return

if(reduce) {
  setPhase('gone')      // runs for ANY reduced-motion user, no return
}

  if (previewing) {      // 6-space misindent; brace closes at 4
    if(reduce) { setPhase('gone'); return }
    const id = window.setTimeout(() => setPhase('text-out'), PREVIEW_CONNECT_MS)
    return () => window.clearTimeout(id)
}

if (gatewayState === 'open' && shownRef.current) {
  setPhase(reduce ? 'gone' : 'text-out')
}

• The top if(reduce) setPhase('gone') fires unconditionally on mount whenever reduce is true — not gated on connecting / gatewayState / shownRef. So every user with OS "reduce motion" enabled loses the CONNECTING overlay during cold boot entirely (jumps to gone before the gateway is even open). The intent was to skip the exit choreography, not to skip showing the overlay. This ships an accessibility regression to serve screenshots.
• if(reduce) (missing space), the 6-space indent, and the same 3-line comment pasted three times won't survive prettier.
• Fix: only short-circuit the exit on connect (gatewayState === 'open' && shownRef.current), leave the appearance/connecting path intact. The third branch already does reduce ? 'gone' : 'text-out' correctly — the top block and the nested preview block are redundant with it.

The other prod tweaks are good and should stay: the boot.ts error-latch (don't let a late error:null clobber a set failure), onboarding Preparing returning null on error so BootFailureOverlay owns the screen, the blanket styles.css reduced-motion rule, and the session.ts ConnectionState typing.

Should split / reconsider

2. Installer rewrite is unrelated scope with its validation disabled. scripts/install.ps1 swaps the astral install-script for a direct GitHub-zip download (a real Windows-installer behavior change) and deletes two install tests, yet e2e-windows.yml is if: false on both jobs and the final revert(installer) commit only reverted managed_uv.py — the install.ps1 rewrite remains. So a production installer change lands in a "Desktop E2E" PR with its E2E path off and no other coverage (can't run on Linux CI). Please pull it into its own installer PR. Deleting the two tests is itself fine — they're the banned "read source with regex" pattern — but do it alongside the installer change.

3. e2e-desktop added to the required all-checks-pass gate. A new Electron + Playwright + real hermes serve suite under xvfb now blocks every PR touching python or frontend. The mitigations are real (skip→success for unrelated PRs, retries: 1, visual diffs advisory via expectVisualSnapshot), but the functional specs (boot chain, chat send/receive through the real backend) are classic flake sources and it adds a full npm ci + uv sync --extra all --extra dev + desktop build to a large share of PRs. Suggest running it non-required for a few weeks to gather a flake baseline before making it a hard gate.

Nits

• tsconfig.json now excludes e2e, so the specs aren't covered by the desktop typecheck job (the "0 tsc errors" claim was a manual run). Add an e2e tsconfig to CI typecheck or they'll rot.
• fix-electron-tracing.ts monkeypatches Playwright internals (_playwright, _allContexts, tracing.start). Acceptable only because @playwright/test is pinned exact (=1.58.2) — call that out in the file comment so a future bump knows to re-verify.
• main.ts TEST_WORKER_INDEX show-window block has stray 6-space indentation.
• check dropping the trailing npm run build is a fair perf fix but is unrelated scope riding along.
• e2e-windows.yml lands fully if: false — dead CI in-tree. Fine as a parked scaffold, just be explicit it exercises nothing yet.

[ethernet8023]

1. fixed.
2. partially fixed a couple hrs ago in 6ddbe8e, fixed fr now

3. but the functional specs (boot chain, chat send/receive through the real backend) are classic flake sources and it adds a full npm ci + uv sync --extra all --extra dev + desktop build to a large share of PRs.

these are not flaking in any of my runs, and i think it's important to actually excercise these pathways!! catching and fixing any flakes will fix real bugs IMO. the runtime is short - 1:41 on the last run - and it runs in parallel to other tests, so no extra time.

nits:

1. added e2e tsconfig!
2. noted in a comment.
3. fixed
4. dropping the trailing build is technically unrelated but feels related enough :p
5. removed from tree.

[OutThisLife]

Approving — all review items addressed:

• Overlay a11y (was blocking): exit short-circuit now lives only in the gatewayState === 'open' && shownRef.current branch (setPhase(reduce ? 'gone' : 'text-out')). Connecting/appearance path is intact, redundant blocks + duplicated comments gone. Reduced-motion users keep the boot overlay.
• Installer scope: install.ps1 rewrite + test deletions dropped; e2e-windows.yml pulled to its own branch (33→26 files).
• e2e typecheck: tsconfig.e2e.json added so specs are covered.

Remaining prod changes are sound (boot.ts error-latch, reduced-motion decode-text/styles, session typing, env-gated main.ts hooks). Desktop E2E green (<2min).

Only conscious call left is e2e-desktop in the required all-checks-pass gate — acceptable given visual diffs are advisory, skip→success covers unrelated PRs, and retries: 1. Fine to own it; revisit if flake shows up.