Repository navigation
ci: run tests-job xcodebuild in the console GUI session (fix testmanagerd on self-hosted minis) - #6401
Conversation
The `tests` job is the only macOS test job that runs xcodebuild test with no
session setup, unlike `ui-regressions` (enables automation mode) and
`perf-activation` (launchctl asuser into the console user's Aqua session). On a
self-hosted runner whose agent is not itself in a logged-in GUI session,
testmanagerd's control service is not in the runner's bootstrap namespace, so
xcodebuild times out initiating the control session and 0 tests run (the
austins-mac-mini failure).
Add scripts/ci/run-in-console-session.sh: it elevates a command into the
logged-in console user's Aqua session via `launchctl asuser`, guarded so it
falls back to the current bootstrap when no console user is logged in or
passwordless sudo is unavailable (never worse than today; a no-op on runners
already in a session). It also forwards only the env vars that are actually set,
so it can't blank out a downstream `${VAR:-default}`. Wire it around the three
app-host xcodebuild invocations in the `tests` job, and add the same
automation-mode enable step `ui-regressions` already uses.
The wrapper logs whether it found a logged-in console user, so the job output
now also reports each runner's session state.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
📝 WalkthroughWalkthroughIntroduces ChangesConsole Session Wrapper and CI Wiring
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 21 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (21 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Greptile SummaryThis PR fixes
Confidence Score: 5/5Safe to merge; the fallback path ensures this change can never make a runner worse than it is today. The elevation logic is guarded: no console user or no passwordless sudo means the command runs exactly as before. The three previously-flagged issues (warning annotation on stderr, HOME fallback on dscl failure, sudo -E SETENV requirement) are all in the fallback/diagnostic path and do not break the happy path. The only new gap found is two lock-tuning env vars omitted from the forward list, which only matters if those vars are explicitly set in CI — they are not currently set in the workflow. scripts/ci/run-in-console-session.sh — the env forward list and the diagnostic warning path are the areas most likely to need follow-up. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A["CI tests job step\nruns run-in-console-session.sh CMD"] --> B{console user\nlogged in AND\npasswordless sudo?}
B -- No --> C["warn to stderr\nexec CMD directly\n(current bootstrap)"]
B -- Yes --> D["resolve console_uid\nbuild env_pairs"]
D --> E["sudo -n launchctl asuser console_uid\n sudo -n -u console_user -E\n env HOME=console_home …\n bash -c 'cd GITHUB_WORKSPACE && exec CMD'"]
E --> F["CMD runs in\nconsole user's\nAqua GUI bootstrap"]
F --> G{run-app-host-xcodebuild.sh\nor xcodebuild_noninteractive.py}
G --> H["flock per-machine lock\n(app_host_test_lock.py)"]
H --> I["xcodebuild test\ntestmanagerd control session\nreachable ✓"]
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
A["CI tests job step\nruns run-in-console-session.sh CMD"] --> B{console user\nlogged in AND\npasswordless sudo?}
B -- No --> C["warn to stderr\nexec CMD directly\n(current bootstrap)"]
B -- Yes --> D["resolve console_uid\nbuild env_pairs"]
D --> E["sudo -n launchctl asuser console_uid\n sudo -n -u console_user -E\n env HOME=console_home …\n bash -c 'cd GITHUB_WORKSPACE && exec CMD'"]
E --> F["CMD runs in\nconsole user's\nAqua GUI bootstrap"]
F --> G{run-app-host-xcodebuild.sh\nor xcodebuild_noninteractive.py}
G --> H["flock per-machine lock\n(app_host_test_lock.py)"]
H --> I["xcodebuild test\ntestmanagerd control session\nreachable ✓"]
Reviews (2): Last reviewed commit: "Merge origin/main into feat-ci-tests-con..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| bash -c 'cd "$GITHUB_WORKSPACE" && exec "$@"' bash "$@" | ||
| fi | ||
|
|
||
| echo "::warning::No logged-in console user (or no passwordless sudo) on this runner; running in the current bootstrap. XCTest will fail here if this runner has no GUI session." >&2 |
There was a problem hiding this comment.
::warning:: workflow command is sent to stderr (>&2), but GitHub Actions processes ::workflow-command:: annotations from stdout only. This is the most important diagnostic message in the script — it tells operators their runner has no GUI session — but it will appear only in the raw log and not as a visible job annotation. The success-path log on line 53 is correctly stderr (it's not a workflow command), but this one should go to stdout.
| echo "::warning::No logged-in console user (or no passwordless sudo) on this runner; running in the current bootstrap. XCTest will fail here if this runner has no GUI session." >&2 | |
| echo "::warning::No logged-in console user (or no passwordless sudo) on this runner; running in the current bootstrap. XCTest will fail here if this runner has no GUI session." |
| if [ -n "$console_user" ] && [ "$console_user" != "root" ] \ | ||
| && console_uid="$(id -u "$console_user" 2>/dev/null)" && sudo -n true 2>/dev/null; then | ||
| console_home="$( (dscl . -read "/Users/$console_user" NFSHomeDirectory 2>/dev/null || true) | awk '{print $2}')" | ||
| [ -n "$console_home" ] || console_home="$HOME" |
There was a problem hiding this comment.
dscl fails to read NFSHomeDirectory, console_home falls back to $HOME, which is the runner agent's home directory — not the console user's home. The inner sudo -n -u "$console_user" then runs as the console user with HOME pointing to a directory that user may not own or be able to write to, which can cause Xcode tooling (~/Library/Developer, etc.) to fail. A safer default is /Users/$console_user, which is the canonical macOS home path for the user and is very unlikely to be wrong when the user does exist.
| [ -n "$console_home" ] || console_home="$HOME" | |
| [ -n "$console_home" ] || console_home="/Users/$console_user" |
| exec sudo -n launchctl asuser "$console_uid" sudo -n -u "$console_user" -E \ | ||
| env HOME="$console_home" "${env_pairs[@]}" \ | ||
| bash -c 'cd "$GITHUB_WORKSPACE" && exec "$@"' bash "$@" |
There was a problem hiding this comment.
sudo -E may fail if SETENV is not granted in sudoers
sudo -n -u "$console_user" -E requires the SETENV tag (or !env_reset / an env_keep allowlist) for the runner account in /etc/sudoers. On hardened CI images that have Defaults env_reset without an explicit SETENV rule, this flag causes sudo to print an error and exit non-zero, failing the step entirely. Since the important variables are already forwarded explicitly via env HOME=... "${env_pairs[@]}", dropping -E would make the command resilient to stricter sudoers configs. As-is, if a new austin-mini runner image has tighter sudoers, the fallback path is never reached — the step just errors out.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
.github/workflows/ci.yml (1)
414-418:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winMissing error handling for
automationmodetoolcommand failure.If
automationmodetool enable-automationmode-without-authenticationreturns non-zero (e.g., tool error, unsupported configuration), the step fails the job due toset -euo pipefail. The analogous code intest-e2e.ymlhandles this gracefully with|| echo "::warning::...".🛡️ Proposed fix to match test-e2e.yml behavior
if sudo -n true 2>/dev/null; then - sudo -n automationmodetool enable-automationmode-without-authentication + sudo -n automationmodetool enable-automationmode-without-authentication \ + || echo "::warning::Could not enable Automation Mode" else echo "::warning::Passwordless sudo unavailable; XCTest will use its default automation-mode setup" fi🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/ci.yml around lines 414 - 418, The automationmodetool enable-automationmode-without-authentication command lacks error handling and will cause the job to fail if the command returns a non-zero exit code due to the set -euo pipefail setting. Add error handling to the sudo command that runs automationmodetool by appending a fallback pattern similar to what is used in test-e2e.yml, so that when the automationmodetool command fails, it logs a warning message instead of failing the entire step.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In @.github/workflows/ci.yml:
- Around line 414-418: The automationmodetool
enable-automationmode-without-authentication command lacks error handling and
will cause the job to fail if the command returns a non-zero exit code due to
the set -euo pipefail setting. Add error handling to the sudo command that runs
automationmodetool by appending a fallback pattern similar to what is used in
test-e2e.yml, so that when the automationmodetool command fails, it logs a
warning message instead of failing the entire step.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 5f159ed0-58d2-4d8f-8248-b020f23d86fd
📒 Files selected for processing (1)
.github/workflows/ci.yml
This branch was successfully deployed
1 participant
Why
Diagnosis of the austin-mini testmanagerd failures:
xcodebuild testneedstestmanagerd's control service, which only exists in the console user's GUI (Aqua) login session bootstrap namespace. Thetestsjob is the only macOS test job that runs xcodebuild with no session setup (unlikeui-regressions, which enables automation mode, andperf-activation, whichlaunchctl asusers into the console session). So on a runner whose agent isn't itself in a logged-in GUI session, the control session never initiates (No such process,Timed out … initiating control session,Executed 0 tests).What
scripts/ci/run-in-console-session.sh: elevates a command into the logged-in console user's Aqua session vialaunchctl asuser. Guarded — if no real console user is logged in (/dev/consoleowned byroot) or passwordless sudo is unavailable, it falls back to the current bootstrap, i.e. exactly today's behavior. It forwards only env vars that are actually set (so it can't blank a downstream${VAR:-default}), and is a no-op on runners already in a session.testsjob, and add the automation-mode enable stepui-regressionsalready uses.Experiment value
The wrapper logs whether it found a logged-in console user, so the job output reports each runner's session state. If this job lands on an austin mini and the
BrowserSystemProxyMirrorTestsstep (the one that was failing) now passes, that both fixes it and confirms a user is logged in. If it logs "No logged-in console user," that runner needs the machine-side fix (run the agent as the GUI user / log in and stay logged in).Caveat
This rescues a runner where a console user is logged in but the agent runs outside that session. It cannot rescue a runner with no GUI login at all; that still needs the machine fix.
🤖 Generated with Claude Code
Need help on this PR? Tag
/codesmithwith what you need. Autofix is disabled.Summary by cubic
Run macOS
testsjobxcodebuildsteps in the console GUI session to restoretestmanagerdcontrol and stop 0-test runs on self-hosted minis. Also enable XCTest automation mode to matchui-regressions.scripts/ci/run-in-console-session.shto elevate commands into the logged-in console user vialaunchctl asuser; guarded fallback if no GUI login or no passwordless sudo; forwards only set env vars; logs session status.xcodebuildcalls intestswith the new script.automationmodetool; warn when missing or sudo not available.Written for commit c1daa22. Summary will update on new commits.
Summary by CodeRabbit