ci(e2e): give DerivedData adoption a budget, and log where its time goes

[teamleaderleo]

Problem

Since #14016, every macOS e2e run's "Adopt main's DerivedData" step has run until the step timeout: 612 s under the old 10-minute limit, 912 s under the 15-minute one. None has ever hit. Every focused run pays those minutes before it compiles cold anyway. The step prints nothing until it finishes, so the logs can't say which phase is slow.

Sample of timed-out runs: 35937847074, 35938349258, 35938871900, 35938878301, 35939305209, 35939623231, 35940458918.

What is and is not slow

Phase	Off the fleet (Linux, same 1.9 GB artifact)	On the macOS fleet
Parallel-range download	34 s	35 s for the test product, in another step of the same runs
Unzip	1 s	—
List + extract 65,028 tar members (6.3 GB)	35 s	unmeasured
Hash the workspace (Record step, same walk as replay)	—	5–8 s

The likeliest cost is extraction onto the VM's disk. That's a hypothesis: this PR's phase timings will confirm or refute it on the first run.

Change

The restore step now:

• Logs as it goes: each phase and its duration print to stderr immediately, not at the end.
• Gives up on its own: it stops after CMUX_WARM_BUDGET_SECONDS (default 420 s), using SIGALRM inside the process. Its own cleanup then deletes the partial DerivedData and reports which phase it was in. A step timeout kills the process before any of that can run.
• Can't cut off a finished restore: the alarm is cancelled in a finally.

The step timeout drops to 10 minutes as a backstop. #14084's discard step still covers a process that dies anyway.

Testing

• New tests in tests/test_e2e_warm_derived_data.py:
  • An adoption over budget returns hit=false, leaves DerivedData empty, and names the phase it was in.
  • An adoption within budget is not interrupted afterwards.
  • Negative control: removing the alarm makes the first test fail.
• Guard lane: linux-guard exits 0 with 434 passing.
• Other checks: the self-hosted guard exits 0, and actionlint is clean.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

---

Summary by cubic

Stops macOS e2e DerivedData adoption from burning the full step timeout (612 s, then 912 s since #14016) with no hit. The restore step now gives up on its own after a time budget and logs where its time goes as it runs.

• Stops after CMUX_WARM_BUDGET_SECONDS (default 420 s) via SIGALRM inside the process, so cleanup deletes the partial DerivedData and names the phase it was in; a zero or malformed budget falls back to the default instead of disabling or crashing.
• Each phase (find archive, download, verify digest, unzip, extract, replay) logs its duration to stderr as it goes, and a timeout just after a phase ends no longer blames that phase.
• The alarm is cancelled in a finally, so a finished restore can't be interrupted afterwards.
• The step timeout drops to 10 minutes as a backstop.
• Adds tests covering over-budget and in-budget adoptions; they drive the budget by delivering the alarm signal itself instead of sleeping, and restore the signal handler they change.

^{Written for commit fc63447. Summary will update on new commits.}

[coderabbitai]

Warning

Review limit reached

Next included review available in 18 seconds.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: e16029cc-b039-4acc-8ac7-a201f91f9f19

📥 Commits

Reviewing files that changed from the base of the PR and between 19144b6 and fc63447.

📒 Files selected for processing (3)

• .github/workflows/test-e2e.yml
• scripts/ci/e2e_warm_derived_data.py
• tests/test_e2e_warm_derived_data.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[teamleaderleo]

Independent subagent review at 68a3821: no blocking defects. Follow-ups are in 855c2bff1b; the budget tests pass (11/11).

• Timeout message: a timeout right after a phase no longer names that phase.
• Budget setting: a zero or malformed CMUX_WARM_BUDGET_SECONDS now falls back to a real budget. Before, zero disabled the budget and a malformed value crashed outside the try.
• Test cleanup: the budget tests restore the SIGALRM handler and phase they change.

Accepted as-is:

• The alarm can arrive late during the download. Leaving the ThreadPoolExecutor block waits for in-flight range requests, so there the effective limit is the transport's own deadline (360 s plus at most one 60 s read). That still fits inside the 10-minute step. The alarm interrupts the verify, unzip, extract and replay phases promptly.
• A staging directory can be left behind if the alarm fires during tempdir cleanup on the success path. It only matters on non-ephemeral runners.

[teamleaderleo]

Closing as mostly superseded by #14098 (with #14084). Adoption now happens only when the app build is unchanged, which removes most of the 612–912 s timeouts that motivated this PR. Reopen if adopt steps still run to the timeout.