glaeda-cmux-runner-hook: take-root --switch and take-gui for cmux's E2E build

[teamleaderleo]

Stacked on #1259 (which gives test-e2e's build the compile-gui class).

Why

On an owned mini, test-e2e's build holds the gui token for its whole job, about 15 min, but uses the console session only for its tests, about 1 min. It also can't adopt a product compiled at another canonical root. The job holds one root from admission, and take-root refuses a second, different root. In the E2E reuse misses I checked, one product compiled at /private/tmp/cmux-ci-2 was never adopted by a build holding /private/tmp/cmux-ci. This adds the two step-time helpers the cmux side needs. Nothing changes for any job until cmux calls them.

What

Both go through the existing glaeda-canonical-root shim.

• take ROOT --switch.
  • The job waits for ROOT while still holding its current root, and lets the old one go only once ROOT is held. So it never holds two, and a timeout (exit 1) leaves it where it was. On success it writes the new CMUX_CI_CANONICAL_ROOT to $GITHUB_ENV.
  • It only works when every held root has a live holder of its own: one taken with take, or the admission root of a ROOT_SWITCHERS class. compile-gui is the only such class, and admission now hands its root to a separate <holder>-root-k.pid holder instead of the main one, which the main holder doesn't inherit. A root held by the main holder can't be let go (exit 2).
• take-gui [--wait S].
  • Holds the gui token from that step to the end of the job, in a <holder>-gui.pid holder that job-completed releases. It does nothing when admission already gave the job gui.
  • Deadlock guard. A job in take-gui holds a root, and a gui job waiting in take-root holds gui: the opposite lock order. So take-root waiters write capacity/root-k.want-<pid>, which says gui when their job holds gui and is rewritten every second. take-gui exits 3 at once when a gui holder is waiting for a root this job holds. The caller then hands its tests to another job. Markers older than 10 s, or with a dead pid, are ignored and removed.
• The shim gains take-gui. It is regenerated on fleet re-apply, while the hook file runs fresh for every job.

Unchanged: compile-gui still takes gui at admission. Dropping that is a follow-up once cmux's build calls take-gui before its tests; otherwise a build would test without the token.

Compatibility

• An old shim answers take-gui with a usage error, exit 2. An old hook rejects --switch and take-gui through argparse, also exit 2. cmux has to treat 2 as "unsupported, fall back".
• Rollout order: this PR and a fleet re-apply, then the cmux call sites, then the gui drop.

Review

An independent read-only review of 87f5561 found nothing blocking. It checked the fd inheritance across the two forks, the error paths, and the deadlock argument: every wait is bounded, --switch never holds two roots, and the gui-for-root two-party cycle is broken by the marker and exit 3. It found four smaller issues, all fixed in 19c3012:

• a switch timeout left the job with no root while its environment still named the old one
• a SIGKILLed holder's pid file passed the switch check
• a recycled or foreign pid on a stale marker could force a false give-way
• an unrecorded gui hold broke a second take-gui call
  The fixes are being re-reviewed on 19c3012.

Tests

• New tests:
  • test_switch_swaps_a_producers_root_without_holding_two: timeout keeps the old root, env export, dead holder, admission-held root
  • test_take_gui_holds_the_token_for_the_rest_of_the_job
  • test_take_gui_gives_way_to_a_gui_job_waiting_for_its_root: real concurrent waiter, stale marker swept
• The shim test covers the usage errors.
• Results on 19c3012:
  • scripts/test-glaeda-cmux-runner.py: 155 OK
  • test-glaeda-fleet-status.py and test-glaeda-cmux-runner-fleet.py: OK

🤖 Generated with Claude Code