Skip to content

fix(ci): avoid app-host rsync metadata failures - #18616

Open
teamleaderleo wants to merge 1 commit into
mainfrom
fix/app-host-rsync-metadata
Open

teamleaderleo wants to merge 1 commit into
mainfrom
fix/app-host-rsync-metadata

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

App-host framework staging can fail on reused macOS runners when rsync tries to restore source mtimes with utimensat, and enumeration can leave an app-host child without the receipt identity cleanup requires.

This change makes framework staging metadata-neutral with rsync --no-times across the app-host paths and forwards the isolated app-host receipt identity through enumeration's Xcode invocation.

Changelog

Fixed app-host CI staging and cleanup on reused macOS runners.

Validation

  • python3 tests/test_ci_console_session_env_forwarding.py
  • python3 tests/test_ci_app_host_home_isolation.py
  • bash tests/test_ci_app_host_xcodebuild_retry.sh
  • python3 tests/test_ci_change_areas.py -k macos_compile_admission_precedes_expensive_shards
  • Bash syntax checks and git diff --check

The source runner investigation found no root-owned files, immutable flags, or ACLs in the failed isolation/package paths. The failure was in mtime preservation plus missing enumeration receipt identity.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Summary by cubic

Fixes app-host CI staging and cleanup failures on reused macOS runners.

  • Adds --no-times to all app-host rsync calls so mtime restoration no longer fails with EPERM on framework files left by a previous runner account.
  • Forwards the isolated app-host receipt identity through Xcode's TEST_RUNNER_ environment during enumeration so cleanup can prove ownership of an orphaned app-host child.

Written for commit 6390049. Summary will update on new commits.

View guided diff Turn on auto-fix


Note

Low Risk
CI-only staging and test-enumeration env forwarding; no production app or auth paths.

Overview
App-host CI now stages CmuxAgentJournal package frameworks with rsync --no-times everywhere those copies are made (workflows, restore-app-host-test-product.sh, run-app-host-xcodebuild.sh). That skips mtime restoration via utimensat, which was raising EPERM on reused macOS runners when framework files were readable but owned by another account.

When CMUX_CI_APP_HOST_ISOLATION_REQUIRED=1, enumerate-app-host-tests.sh injects the same isolated app-host receipt and home identity through TEST_RUNNER_* env vars before xcodebuild -enumerate-tests, so post-run cleanup can attribute stray app-host children the same way as full test batches.

Workflow contract tests were updated to expect the new rsync flags.

Reviewed by Cursor Bugbot for commit 6390049. Bugbot is set up for automated code reviews on this repo. Configure here.

Summary by CodeRabbit

  • Bug Fixes
    • Improved the consistency of macOS app packaging by preventing copied frameworks from retaining source timestamps.
    • Improved app-host test setup in isolated CI environments, helping test enumeration run with the required environment settings.

@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@coderabbitai

coderabbitai Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: fa0e0d11-5d80-4389-83ea-2faa9d3b3217
📥 Commits

Reviewing files that changed from the base of the PR and between f0a2bad and 6390049.

📒 Files selected for processing (7)
  • .github/workflows/app-host-test-rerun.yml
  • .github/workflows/ci-macos.yml
  • .github/workflows/seed-derived-data.yml
  • scripts/ci/enumerate-app-host-tests.sh
  • scripts/ci/restore-app-host-test-product.sh
  • scripts/ci/run-app-host-xcodebuild.sh
  • tests/test_ci_change_areas.py

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 4 remain after this review.


📝 Walkthrough

Walkthrough

CI workflows and scripts now prevent package-framework copies from preserving source timestamps. App-host test enumeration also prepares and passes a runner environment when isolation is required.

Changes

App-host CI updates

Layer / File(s) Summary
Package-framework copy timestamps
.github/workflows/app-host-test-rerun.yml, .github/workflows/ci-macos.yml, .github/workflows/seed-derived-data.yml, scripts/ci/restore-app-host-test-product.sh, scripts/ci/run-app-host-xcodebuild.sh, tests/test_ci_change_areas.py
Package-framework rsync commands add --no-times. The related test assertion expects the option.
Isolated enumeration environment
scripts/ci/enumerate-app-host-tests.sh
When isolation is required, the script prepares the configured home, XDG config, receipt directory, app-host keys, and available CI variables, then runs enumeration with that environment.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: austinywang

Merge Risk: ⚪ Minimal · up to 63900

Framework staging avoids restoring source timestamps, and app-host enumeration receives its prepared isolation identity. No merge-blocking regression is established.

🚥 Pre-merge checks | ✅ 24 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 4 files. (3 skipped: 3 … Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (24 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise, specific, and accurately summarizes the primary CI fix for app-host rsync metadata failures.
Description check ✅ Passed The description explains the problem, implementation, affected behavior, changelog, and validation commands. It is mostly complete, but it omits the required Proof and Checklist sections, uses Validat…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Cmux Cloud Persistent Session And Early Input ✅ Passed PASS: The pull request changes only app-host CI workflows/scripts and a CI assertion. The diff adds rsync --no-times and forwards app-host test-runner environment variables. It does not change Cloud…
Cmux Swift Actor Isolation ✅ Passed The pull request changes only YAML workflows, shell scripts, and one Python test. The diff contains no Swift files or Swift actor-isolation declarations. The production changes only add `rsync --no-ti…
Cmux Swift Blocking Runtime ✅ Passed The pull request changes only GitHub Actions YAML, CI shell scripts, and a Python test. It changes no Swift files and adds no Swift blocking-runtime synchronization. The custom check is therefore not …
Cmux Browser Automation Off-Main ✅ Passed PASS. The pull request changes only CI workflows, app-host shell scripts, and a CI test. The rule-scoped browser automation files, Sources/TerminalController.swift and `Packages/macOS/CmuxControlSoc…
Cmux Expensive Synchronous Load ✅ Passed PASS. The reviewed diff changes only GitHub workflows, CI shell scripts, and a Python test. It adds no production Swift files or Swift code, so it cannot introduce an expensive synchronous agent-histo…
Cmux Cache Substitution Correctness ✅ Passed PASS: The pull request changes only GitHub Actions YAML, Bash CI scripts, and a Python test. It does not change production Swift, TypeScript, or JavaScript, and it does not substitute a cached value f…
Cmux No Hacky Sleeps ✅ Passed PASS. The diff introduces no fixed sleep, timer, polling loop, delayed dispatch, or wall-clock wait in covered runtime scripts. It adds app-host environment forwarding and changes rsync to use `--no…
Cmux Algorithmic Complexity ✅ Passed The production diff does not introduce a scalable collection scan or a slower algorithm. It adds --no-times to existing rsync calls in the CI workflows and shell scripts, plus a fixed-size environ…
Cmux Swift Concurrency ✅ Passed PASS: The pull request changes only CI YAML, shell scripts, and one Python test. The authoritative diff contains no Swift files or Swift concurrency changes, so it does not introduce or expand the pro…
Cmux Swift @Concurrent ✅ Passed The pull request changes only GitHub Actions YAML, shell scripts, and one Python test. The authoritative diff contains no Swift files or Swift declarations, so the Swift @concurrent check is not appli…
Cmux Swift Package Boundaries ✅ Passed PASS: The pull request changes only CI workflows, shell scripts, and a Python test. The authoritative diff contains no Swift files or production Swift changes, so the Swift package-boundary rule does …
Cmux Swiftpm Lockfiles ✅ Passed PASS. The PR changes only CI workflows, CI shell scripts, and one test assertion. The authoritative diff contains no Package.swift, Package.resolved, .gitignore, or Xcode project changes, and no…
Cmux Swift Logging ✅ Passed PASS. The pull request changes only GitHub workflows, Bash scripts, and a Python test; it changes no Swift or other app/runtime source. The added lines only configure rsync --no-times and environmen…
Cmux User-Facing Error Privacy ✅ Passed PASS — The diff changes only internal macOS CI workflows and CI helper scripts. It adds rsync --no-times, forwards TEST_RUNNER_* environment values to Xcode test enumeration, and updates a CI asse…
Cmux Full Internationalization ✅ Passed PASS. The PR changes only CI workflows, CI shell scripts, and a CI test assertion. The diff adds rsync --no-times and forwards test-runner environment variables; it introduces no Swift UI text, stri…
Cmux Swiftui State Layout ✅ Passed The pull request changes only CI YAML, Bash scripts, and one Python test. It introduces no Swift or SwiftUI views, state, GeometryReader, lazy/list row, or render-time state changes. The `run-app-host…
Cmux Architecture Rethink ✅ Passed PASS: The pull request changes only CI workflow YAML, shell scripts, and a Python test expectation. It does not change Swift architecture or introduce any of the forbidden Swift timing, state-ownershi…
Cmux Swift Auxiliary Window Close Shortcuts ✅ Passed PASS: The pull request changes only workflow YAML, CI shell scripts, and a Python test. The authoritative diff contains no Swift files and no standalone cmux-owned window code. The auxiliary-window cl…
Cmux Source Artifacts ✅ Passed All seven changed paths are hand-written CI workflows, shell scripts, or a test. The diff adds only rsync options, app-host environment handling, and a matching test assertion. It adds no logs, screen…
Cmux No Test Or Debug Seam In Production Source ✅ Passed PASS. The pull request changes only CI workflow files, shell scripts, and a Python test. It changes no Swift file under a production Sources/ path, so it cannot introduce the specified test or debug…
Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 4 files. (3 skipped: 3 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 2
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
🛠️ Fix failing CI checks 💡
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Passes: CI passes on 6390049d01.

CI passes on 6390049d01 (run 38349564193 attempt 3).

Written by scripts/ci/classify_failures.py (ci-failure-attribution.yml); signatures are its SIGNATURES table. A machine verdict is the runner's fault, not this PR's; yours means the failing file is one this PR changes, also red on main that main's latest full suite fails the same way, seen on other PRs that it failed on another pull request's run lately.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Host receipt for the original cmux13s failure:

  • Run 37828161625 attempt 1, job 113486867165, failed on cmux13s-mac-mini-glaeda during package-framework restaging with rsync ... utimensat: Operation not permitted.
  • Read-only inspection found the isolation directory, receipt directory, and confirmation file owned by the build user with expected restrictive modes. No root-owned files, immutable flags, or ACLs were present in the inspected paths; the failed process had exited.
  • The repair is in head 6390049d01e: framework staging uses rsync --no-times, and enumeration forwards the isolated app-host receipt identity through TEST_RUNNER_* so cleanup can verify a surviving child.

The PR workflow's first macOS admission attempt was canceled before runner assignment; Linux and guard checks passed. A full rerun is currently queued at the same exact head.

@teamleaderleo teamleaderleo added the needs a call Finished and held for a team design or product decision (see #13742 and the gallery in #15427) label Oct 13, 2026
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

CI fast guards passes on 6390049d01 (https://github.com/manaflow-ai/cmux/actions/runs/38349564193). This covers only the fast guards, not CI: CI's result is the ci-status check, and the CI failure attribution comment names any failing test.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs a call Finished and held for a team design or product decision (see #13742 and the gallery in #15427)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant