Skip to content

Wake E2E display before strict activation test - #4942

Closed
lawrencecchen wants to merge 2 commits into
mainfrom
feat-e2e-remove-activation-masks
Closed

lawrencecchen wants to merge 2 commits into
mainfrom
feat-e2e-remove-activation-masks

Conversation

@lawrencecchen

@lawrencecchen lawrencecchen commented May 28, 2026 •

Copy link
Copy Markdown
Contributor

Adds a display wake/prevent-sleep guard before hosted E2E xcodebuild, then removes the CommandPalette activation expected-failure wrappers as the strict activation proof case.

Why this scope: after #4940, the depot-macos-latest label can still land on cmux-aws-m4pro-* runners whose display is asleep. Strict launch runs on those machines still failed with Running Background, so this PR wakes and keeps the display awake during xcodebuild. Other masks stay in place until their behavior-level class failures or runner-specific activation failures are handled separately.

Validation:

Summary by CodeRabbit

  • Tests

    • Improved UI test stability by refactoring app launch/activation and clearer failure reporting.
  • Chores

    • Updated CI runner defaults and validation to use the new macfleet runner.
    • Added the new runner label to internal CI validation rules and test guards.

Review Change Stack


Note

Low Risk
Changes only affect CI workflow and UI test expectations, not production app logic or user data.

Overview
Hosted E2E on depot-macos-latest can run on self-hosted runners whose display is asleep, so strict UI tests still saw the app stuck in Running Background after launch. This PR adds a display wake / keep-awake step around xcodebuild in .github/workflows/test-e2e.yml so the GUI session stays active for the test run.

Command palette UI tests drop XCTExpectFailure wrappers around app activation, so launch must reach the foreground on those runners instead of being tolerated as a known CI flake.

Reviewed by Cursor Bugbot for commit 068d872. Bugbot is set up for automated code reviews on this repo. Configure here.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@vercel

vercel Bot commented May 28, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
cmux Ready Ready Preview, Comment May 28, 2026 11:27am
cmux-staging Building Building Preview, Comment May 28, 2026 11:27am

@coderabbitai

coderabbitai Bot commented May 28, 2026 •

Copy link
Copy Markdown

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Removes XCTExpectFailure masking from a UI test helper and rewrites app launch/activation to unconditionally launch and poll for .runningForeground with a 4s timeout. Separately, the E2E workflow, actionlint allowlist, and guard script were updated to prefer and validate the macfleet self-hosted runner label.

Changes

UI Test App Launch and Activation Failure Handling

Layer / File(s) Summary
App-foreground activation polling with timeout
cmuxUITests/CommandPaletteIdentifierClipboardUITests.swift
launchAndActivate(_:) now calls app.launch() unconditionally, returns if already .runningForeground, otherwise polls via app.activate() up to 4s and calls XCTFail with the app state on timeout; removes XCTExpectFailure wrapping and .runningBackground special-case.

CI Runner and Guard Updates

Layer / File(s) Summary
Workflow runner defaults and runs-on
.github/workflows/test-e2e.yml
run-name and workflow_dispatch runner input default changed to use `inputs.runner
Actionlint allowlist
.github/actionlint.yaml
Added macfleet to the self-hosted-runner label allowlist.
Guard script updates
tests/test_ci_self_hosted_guard.sh
Awk-based dynamic run-name detection accepts inputs.runner/macfleet; required runner-label checks now include macfleet; PASS message and a comment were reworded to reference GUI runner choices.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related issues

  • manaflow-ai/cmux-dev-artifacts#1930: The rewritten launchAndActivate addresses activation/flakiness reported for CommandPaletteIdentifierClipboardUITests.
  • manaflow-ai/cmux-dev-artifacts#1889: Implements the "feat-ui-test-activation-deflake" behavior related to the issue.
  • manaflow-ai/cmux-dev-artifacts#1915: Targets the same UI test activation deflake for CommandPaletteIdentifierClipboardUITests.
  • manaflow-ai/cmux-dev-artifacts#1958: Addresses app-activation/foregrounding behavior that caused related UI test failures.

Possibly related PRs

  • manaflow-ai/cmux#4922: Overlaps with workflow runner-fallback and E2E guard updates (runner -> macfleet).
  • manaflow-ai/cmux#4928: Also modifies launchAndActivate(_:) removal of XCTExpectFailure and activation polling.
  • manaflow-ai/cmux#4940: Related changes to test app-start behavior and removal of XCTExpectFailure masking.

Poem

🐰 I hopped through tests both night and day,

Launched the app straight, no masks in the way.
Polling steady for the foreground light,
CI now prefers macfleet's flight.
A cheerful hop — tests pass in sight!

🚥 Pre-merge checks | ✅ 16 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check ❓ Inconclusive The PR description is detailed and includes context, but it does not follow the required template structure with all mandatory sections clearly addressed. Reorganize the description to match the template: add a clear 'Summary' section with 'What changed?' and 'Why?', a 'Testing' section documenting how the change was tested, specify if a demo video is needed, ensure the review trigger block is included, and complete the checklist items.
✅ Passed checks (16 passed)
Check name Status Explanation
Title check ✅ Passed The title 'Wake E2E display before strict activation test' accurately summarizes the main change: adding a display wake guard for E2E tests and removing activation expected-failure wrappers.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Cmux Swift Actor Isolation ✅ Passed PR contains only test files and CI/workflow configuration changes; no production Swift code was modified, per check instruction to pass for tests.
Cmux Swift Blocking Runtime ✅ Passed PR contains only test-only changes in cmuxUITests/ and CI files; no production Swift code introduces blocking synchronization. Polling uses existing RunLoop mechanism in test scaffolding.
Cmux No Hacky Sleeps ✅ Passed All changes are in test-only Swift files, GitHub Actions workflows (out of scope per rule), or test/validation shell scripts. No production non-Swift runtime code introducing hacky sleeps.
Cmux Algorithmic Complexity ✅ Passed All 4 modified files are test-only or CI configuration; no production code modified. Rule explicitly passes test-only scaffolding. Test code has no algorithmic complexity violations.
Cmux Swift Concurrency ✅ Passed Test-only changes; no legacy async patterns introduced. RunLoop.current.run(until:) for XCTest polling is an allowed AppKit/XCTest boundary.
Cmux Swift @Concurrent ✅ Passed The Swift changes in CommandPaletteIdentifierClipboardUITests.swift rewritten launchAndActivate method are synchronous operations with no async/concurrent violations per the rules.
Cmux Swift File And Package Boundaries ✅ Passed Modified file is a test file (cmuxUITests/CommandPaletteIdentifierClipboardUITests.swift). Rule explicitly allows test fixtures. Focused helper improvement only.
Cmux Swift Logging ✅ Passed Only test code and CI changes; Swift file has no logging violations—no print, debugPrint, dump, NSLog, or ad hoc logging added.
Cmux User-Facing Error Privacy ✅ Passed Test-only CI/workflow PR. All error messages are in test assertions and CI logs (developer-only), excluded from the check. No sensitive information exposed.
Cmux Full Internationalization ✅ Passed All changes are test files and CI configuration files with no production code or user-facing strings. Full-internationalization rule explicitly exempts tests and operational docs.
Cmux Swiftui State Layout ✅ Passed PR contains only test code, CI configuration, and shell scripts—no SwiftUI implementation changes, so the SwiftUI state layout check is not applicable.
Cmux Architecture Rethink ✅ Passed PR contains only test-only modifications (UI test polling helper, CI config changes). No production Swift code changes. Test-only synchronization is explicitly allowed by architectural rules.
Cmux Swift Auxiliary Window Close Shortcuts ✅ Passed PR contains test-only and CI/workflow changes. No production Swift window declarations (NSWindow, NSPanel, NSWindowController, Window, WindowGroup) added or modified.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat-e2e-remove-activation-masks

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@greptile-apps

greptile-apps Bot commented May 28, 2026 •

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR removes XCTExpectFailure tolerance wrappers from the launchAndActivate helper in CommandPaletteIdentifierClipboardUITests, making display/activation failures strict test failures rather than expected (swallowed) ones. The change is motivated by a display-sleep issue on cmux-aws-m4pro-* runners: a preceding caffeinate/wake step now keeps the display awake, so CI can reliably bring the app to the foreground.

  • Drops XCTExpectFailure around app.launch() — launch failures now fail the test immediately.
  • Replaces the old reachedForeground || app.state == .runningBackground silent-return with a guard reachedForeground else { XCTFail(...); return }, so a background-only app no longer silently passes through to UI interactions.

Confidence Score: 5/5

Safe to merge — the change is confined to test scaffolding and intentionally tightens CI signal by removing activation leniency.

The diff touches only one UI test helper. The logic is straightforward: launch strictly, poll for foreground, fail hard if not reached. The old silent path through .runningBackground is correctly removed, and the guard plus continueAfterFailure=false ensures no further UI interactions run on a mis-activated app.

No files require special attention.

Important Files Changed

Filename Overview
cmuxUITests/CommandPaletteIdentifierClipboardUITests.swift Removes XCTExpectFailure leniency around app.launch() and the foreground-activation poll; replaces the runningBackground silent-return path with a strict XCTFail guard, making activation a hard requirement on hosted runners.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[launchAndActivate called] --> B[app.launch]
    B --> C{runningForeground?}
    C -- yes --> D[return early]
    C -- no --> E[pollUntil 4s loop calling app.activate]
    E --> F{reachedForeground?}
    F -- yes --> G[return normally]
    F -- no --> H[XCTFail with state value]
    H --> I[return - test stops via continueAfterFailure=false]
Loading

Reviews (6): Last reviewed commit: "Remove CommandPalette activation expecte..." | Re-trigger Greptile

Comment thread cmuxUITests/CommandPaletteIdentifierClipboardUITests.swift
@lawrencecchen
lawrencecchen force-pushed the feat-e2e-remove-activation-masks branch from 6b7da97 to 81f0233 Compare May 28, 2026 10:07
@lawrencecchen lawrencecchen changed the title Remove UI test activation expected failures Remove proven UI test activation expected failures May 28, 2026
@lawrencecchen
lawrencecchen force-pushed the feat-e2e-remove-activation-masks branch from 81f0233 to 748755a Compare May 28, 2026 10:24
@lawrencecchen lawrencecchen changed the title Remove proven UI test activation expected failures Wake E2E display before strict activation tests May 28, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 748755a. Configure here.

Comment thread .github/workflows/test-e2e.yml Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/test-e2e.yml:
- Around line 286-291: The new trap line is overriding the earlier EXIT trap and
preventing MANIFEST_PATH cleanup; remove the trap 'kill "$CAFFEINATE_PID" …'
invocation (or instead append to the existing EXIT trap rather than replacing
it) so the original EXIT handler from earlier in the script is preserved; locate
the caffeinate usage and CAFFEINATE_PID variable and either delete the trap
command or implement appending to the existing EXIT trap for cleanup rather than
setting a second trap.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: bcd93be2-45f5-400b-8053-ce8014e39050

📥 Commits

Reviewing files that changed from the base of the PR and between 81f0233 and 748755a.

📒 Files selected for processing (3)
  • .github/workflows/test-e2e.yml
  • cmuxUITests/CommandPaletteIdentifierClipboardUITests.swift
  • cmuxUITests/UpdatePillUITests.swift

Comment thread .github/workflows/test-e2e.yml Outdated
Comment on lines +286 to +291
if command -v caffeinate >/dev/null 2>&1; then
echo "Keeping display awake during xcodebuild"
caffeinate -dim -w $$ >/tmp/cmux-e2e-caffeinate.log 2>&1 &
CAFFEINATE_PID=$!
trap 'kill "$CAFFEINATE_PID" 2>/dev/null || true' EXIT
caffeinate -u -t 5 >/dev/null 2>&1 || true

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Preserve existing EXIT trap when enabling caffeinate.

At Line 290, this trap overrides the earlier EXIT trap from Line 249, so MANIFEST_PATH cleanup can be skipped in the display-regression branch. Keep existing trap behavior or avoid setting a second EXIT trap here.

Suggested fix
           if command -v caffeinate >/dev/null 2>&1; then
             echo "Keeping display awake during xcodebuild"
             caffeinate -dim -w $$ >/tmp/cmux-e2e-caffeinate.log 2>&1 &
             CAFFEINATE_PID=$!
-            trap 'kill "$CAFFEINATE_PID" 2>/dev/null || true' EXIT
             caffeinate -u -t 5 >/dev/null 2>&1 || true
           fi

caffeinate -w $$ already exits when the current shell exits, so this avoids clobbering prior EXIT cleanup.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/test-e2e.yml around lines 286 - 291, The new trap line is
overriding the earlier EXIT trap and preventing MANIFEST_PATH cleanup; remove
the trap 'kill "$CAFFEINATE_PID" …' invocation (or instead append to the
existing EXIT trap rather than replacing it) so the original EXIT handler from
earlier in the script is preserved; locate the caffeinate usage and
CAFFEINATE_PID variable and either delete the trap command or implement
appending to the existing EXIT trap for cleanup rather than setting a second
trap.

@lawrencecchen
lawrencecchen force-pushed the feat-e2e-remove-activation-masks branch from 748755a to 517e325 Compare May 28, 2026 10:47
@lawrencecchen lawrencecchen changed the title Wake E2E display before strict activation tests Wake E2E display before strict activation test May 28, 2026
Comment thread .github/workflows/test-e2e.yml Outdated
echo "Keeping display awake during xcodebuild"
caffeinate -dim -w $$ >/tmp/cmux-e2e-caffeinate.log 2>&1 &
CAFFEINATE_PID=$!
trap 'kill "$CAFFEINATE_PID" 2>/dev/null || true' EXIT

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 EXIT trap silently replaces the earlier manifest cleanup trap

When TEST_FILTER=DisplayResolutionRegressionUITests, line 249 installs trap 'rm -f "$MANIFEST_PATH"' EXIT. On macOS, caffeinate is always present, so the new trap 'kill "$CAFFEINATE_PID" 2>/dev/null || true' EXIT unconditionally overwrites that earlier trap. The manifest JSON in /tmp is then never removed on exit for that test filter.

Note that the caffeinate trap is actually redundant: caffeinate -dim -w $$ already causes caffeinate to exit as soon as the watched shell PID dies, so no explicit kill is needed. Removing the trap line entirely avoids the clobber while preserving the caffeinate wake behaviour.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

♻️ Duplicate comments (1)
.github/workflows/test-e2e.yml (1)

290-290: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Do not overwrite the existing EXIT trap.

At Line 290, this trap replaces the earlier EXIT handler from Line 249, so MANIFEST_PATH cleanup can be skipped in display-regression runs. Prefer relying on caffeinate -w $$ lifecycle (or append safely), not replacing the trap.

Suggested fix
           if command -v caffeinate >/dev/null 2>&1; then
             echo "Keeping display awake during xcodebuild"
             caffeinate -dim -w $$ >/tmp/cmux-e2e-caffeinate.log 2>&1 &
-            CAFFEINATE_PID=$!
-            trap 'kill "$CAFFEINATE_PID" 2>/dev/null || true' EXIT
             caffeinate -u -t 5 >/dev/null 2>&1 || true
           fi
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/test-e2e.yml at line 290, The new trap for CAFFEINATE_PID
overwrites the earlier EXIT handler (which performs MANIFEST_PATH cleanup), so
change it to append instead of replace: capture the existing trap into a
variable (e.g. PREV_TRAP="$(trap -p EXIT | sed -E "s/trap -- '(.+)' EXIT/\1/")"
or similar), then install a combined EXIT handler that first kills
"$CAFFEINATE_PID" (kill "$CAFFEINATE_PID" 2>/dev/null || true) and then invokes
the previous handler; alternatively rely on caffeinate -w $$ lifecycle and
remove the replacing trap so you do not discard the original cleanup logic for
MANIFEST_PATH.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Duplicate comments:
In @.github/workflows/test-e2e.yml:
- Line 290: The new trap for CAFFEINATE_PID overwrites the earlier EXIT handler
(which performs MANIFEST_PATH cleanup), so change it to append instead of
replace: capture the existing trap into a variable (e.g. PREV_TRAP="$(trap -p
EXIT | sed -E "s/trap -- '(.+)' EXIT/\1/")" or similar), then install a combined
EXIT handler that first kills "$CAFFEINATE_PID" (kill "$CAFFEINATE_PID"
2>/dev/null || true) and then invokes the previous handler; alternatively rely
on caffeinate -w $$ lifecycle and remove the replacing trap so you do not
discard the original cleanup logic for MANIFEST_PATH.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 08e0edd9-e2bf-43d3-b545-d5b9dc0e238d

📥 Commits

Reviewing files that changed from the base of the PR and between 748755a and 517e325.

📒 Files selected for processing (2)
  • .github/workflows/test-e2e.yml
  • cmuxUITests/CommandPaletteIdentifierClipboardUITests.swift

@coderabbitai

coderabbitai Bot commented May 28, 2026

Copy link
Copy Markdown

Actionable comments posted: 0

@lawrencecchen

Copy link
Copy Markdown
Contributor Author

Closing this PR because the strict CommandPalette activation change failed its exact hosted proof run. After the runner-label cleanup, the focused run still landed on a true Depot runner (depot-w8l2gw3nfv, console GUI user runner, Xcode 16.4) and both CommandPaletteIdentifierClipboardUITests cases failed with Failed to activate application ... Running Background: https://github.com/manaflow-ai/cmux/actions/runs/26571172281. I pushed a revert commit so the PR is net-empty against main. The separate external cleanup removed the accidental depot-macos-latest label from AWS self-hosted runners, but this activation mask is not safe to remove yet.

This branch was successfully deployed

1 active deployment
Preview – cmux — 068d8723 Deployed May 28, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant