Skip to content

ci: point restored products at their own package frameworks - #14930

Merged
teamleaderleo merged 2 commits into
mainfrom
ci/relocate-package-framework-rpaths
Sep 27, 2026
Merged

teamleaderleo merged 2 commits into
mainfrom
ci/relocate-package-framework-rpaths

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Sep 27, 2026 •

Copy link
Copy Markdown
Collaborator

Main's app-host shards and tests-build-and-lag load a different commit's package frameworks on the owned Macs.

Every Mach-O in a Debug build has an absolute rpath to the DerivedData it was compiled in, ahead of the bundle-relative ones:

/tmp/cmux-ci-2/derived-data-compile-admission/Build/Products/Debug/PackageFrameworks
@executable_path
@executable_path/../Frameworks
...
/private/tmp/cmux-app-host-package-frameworks

A consumer restores the product into its own DerivedData. On an owned Mac the producer's path usually exists anyway: it is that canonical root's kept build, from whatever commit last compiled there. dyld loads package frameworks from it first. When their API has moved, the load fails:

dyld: Symbol not found: _$s15CMUXAgentLaunch05AgentB18CaptureArgvVerdictO11processName9arguments4kindACSSSg_SaySSGSgSStcfC
  Referenced from: .../cmux-derived-data-tests-36292396824-1-shard-1-layers/.../cmux DEV.app/Contents/Resources/bin/cmux
  Expected in:     /private/tmp/cmux-ci-2/derived-data-compile-admission/Build/Products/Debug/PackageFrameworks/CMUXAgentLaunch_...

On the last three main runs this is most of the red:

  • 36292396824 shard 1: 63 of these, every CLI test that runs the bundled cmux exits 6
  • 36290042989 shard 5: 77, the hook and session-start suites
  • 36295033926 tests-build-and-lag: the app itself aborts at startup on the missing symbol

restore-app-host-test-product.sh, which every consumer uses, now runs scripts/ci/relocate_package_framework_rpaths.py right after the product lands:

  • It rewrites that one rpath, in every Mach-O under the products directory, to the products' own PackageFrameworks.
  • It re-signs ad hoc what it changed, innermost first, keeping identifier, entitlements (get-task-allow), flags and runtime.
  • It skips the plain-file copies rsync -aL leaves at X.framework/X and Versions/Current/X: install names always say Versions/A, so dyld never loads them.
  • A product restored where it was compiled (compile admission) has nothing to rewrite, and neither does a second run.

Verification

  • On cmux8s, against an APFS clone of a real product from its root 1:
    • 157 Mach-O files rewritten, then re-signed, in about 6 s.
    • codesign --verify --deep --strict passes on the app, and get-task-allow is still there.
    • The bundled CLI runs, and DYLD_PRINT_LIBRARIES shows CMUXAgentLaunch loading from the clone's own PackageFrameworks.
    • A second run prints nothing to rewrite.
  • tests/test_relocate_package_framework_rpaths.py: path matching, framework-copy skipping and signing targets, plus a macOS round trip. The round trip links a small binary with a producer rpath, relocates it, verifies the signature, runs it, and checks that a second pass is a no-op.
  • Not yet run by a CI consumer. This PR's own app-host shards restore through the changed script, so they are the first real run.

Not covered here: cmux7s has one canonical root, so a shard holding a root-2 product that lands there still fails at take-root (shard 3 of 36295033926). That is fleet config and goes in a separate change.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Summary by cubic

Fixes app-host test products on owned Macs loading package frameworks from another commit's DerivedData, which caused Symbol not found failures across CLI, hook, and session-start suites on main. restore-app-host-test-product.sh now runs a new script that rewrites each restored product's compile-time package-framework rpath to an @loader_path-relative path into the product's own PackageFrameworks — an absolute replacement overflowed the linker's load-command padding on deep runner paths — then re-signs ad hoc what it changed and its enclosing bundles. Adds unit tests and a macOS round-trip test.

Verification

  • On a real product, 157 Mach-O files were rewritten and re-signed in about 6 seconds; codesign --verify passes, the CLI runs, and a second run is a no-op.
  • Not covered: on cmux7s, the single canonical root still trips shards holding root-2 products; that needs a separate fleet-config change.

Written for commit 606dd70. Summary will update on new commits.

Review in cubic

Debug builds link every Mach-O with an absolute rpath to the DerivedData
they were compiled in, ahead of the bundle-relative ones. A consumer that
restores a product into its own DerivedData on an owned Mac often still
has that producer path: the canonical root's kept build of another
commit. dyld then loads that commit's package frameworks first. On main
the bundled CLI died with "Symbol not found" in CMUXAgentLaunch, failing
dozens of CLI and hook tests with status 6, and tests-build-and-lag's app
crashed at startup.

restore-app-host-test-product.sh now rewrites that rpath, in every Mach-O
under the restored products, to the products' own PackageFrameworks and
re-signs ad hoc what changed, innermost first. A product restored where it
was compiled has nothing to rewrite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 2 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 619a579f-c625-440c-ae99-f0505bee70f2

📥 Commits

Reviewing files that changed from the base of the PR and between 9a5f926 and 606dd70.

📒 Files selected for processing (11)
  • .github/workflows/ci-artifact-transport.yml
  • .github/workflows/ci.yml
  • scripts/ci/choose_ci_suite.py
  • scripts/ci/detect_ci_change_areas.py
  • scripts/ci/relocate_package_framework_rpaths.py
  • scripts/ci/restore-app-host-test-product.sh
  • scripts/ci/workflow_guard_groups.py
  • tests/test-execution.toml
  • tests/test_ci_change_areas.py
  • tests/test_ci_linux_guard_routing.py
  • tests/test_relocate_package_framework_rpaths.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

The absolute replacement did not fit: on cmux11s the CLI product's
restore failed because the runner's _work/_temp products path is longer
than the linker's load-command padding allows. An @loader_path-relative
path is always shorter than the 81-byte producer path and still holds
when the whole products directory is copied.

Also from review: print the failing tool's stderr, delete a second
producer spelling instead of duplicating the rpath, re-sign every
bundle around a rewritten file, and list the script with its
neighbours in ci.yml's router and the routing tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cursor

cursor Bot commented Sep 27, 2026

Copy link
Copy Markdown

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

@teamleaderleo
teamleaderleo merged commit 28d1eaf into main Sep 27, 2026
71 of 72 checks passed
@teamleaderleo
teamleaderleo deleted the ci/relocate-package-framework-rpaths branch September 27, 2026 10:29
@github-actions

Copy link
Copy Markdown
Contributor

Merge receipt for 606dd7097e: every check was green at merge (20 verified; 21 skipped by policy). Full suite runs on main after merge.

teamleaderleo added a commit that referenced this pull request Sep 27, 2026
…4947)

The shortest producer spelling is 82 bytes, not 81 (review note on #14930).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
rustybret pushed a commit to rustybret/bmux that referenced this pull request Sep 27, 2026
5090403 UI test frames: sample XCTest screen recordings; SIGKILL stuck prompts (manaflow-ai#14956)
9943115 Canvas: keep agent panes from moving the viewport; honor Reduce Motion (manaflow-ai#14939)
f873b5a Fix duplicate-instance handler terminating unrelated helpers (manaflow-ai#13845)
0c151d1 Open Settings panes at their natural top (manaflow-ai#14950)
cfdde0b cmux-tui: inject Claude hooks through a PATH shim, including under sr (manaflow-ai#14908)
320a966 ci: correct the producer rpath length in the relocation docstring (manaflow-ai#14947)
8f79066 Hover never outshouts selection; focus, badge, and feed pill edges (manaflow-ai#14941)
5617ac3 cmux-tui: publish the agent's session id on the agents roster (manaflow-ai#14904)
533a5b9 fix: stop WindowAccessor storing a deallocating window (manaflow-ai#14946)
9546e06 reloadp.sh: exclude only this build's own bundle from the stable check (manaflow-ai#14889)
e27f361 docs: say full-ci runs only selected cmuxUITests targets (manaflow-ai#14945)
28d1eaf ci: point restored products at their own package frameworks (manaflow-ai#14930)
75caaa5 Land hot-path sidebar, feed, palette and notification state changes in the next frame (manaflow-ai#14927)
6b58884 docs: tighten CLAUDE.md and CONTRIBUTING.md; move procedures to skills (manaflow-ai#14920)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant