Skip to content

runner-hook: a signing job waits out an earlier job's leftover processes in any runner's _work - #1738

Merged
teamleaderleo merged 2 commits into
mainfrom
leo/exclusive-host-lock-offline
Oct 29, 2026
Merged

teamleaderleo merged 2 commits into
mainfrom
leo/exclusive-host-lock-offline

Conversation

@teamleaderleo

Copy link
Copy Markdown
Owner

A signing job (the exclusive class) is now admitted only when no leftover job process is still working on the mini, not just when every capacity unit is free.

Leo, 2026-10-28, on the release signing jobs running on owned Macs: "ok. idk what the "risk" was before but like.. we can just read the code and stuff." The rule: the job claims the Mac exclusively, so nothing else runs beside it.

Why: cmux's signing picker is about to offer a mini whose offline runners GitHub calls idle. glaeda parks most of a mini's runners offline, and counting those as busy left no mini ever drained. A runner whose listener died mid-job looks the same: offline, not busy, and its units released with its Runner.Worker. But the steps it started can still be running.

Change:

  • After the exclusive class takes every unit, it lists this user's processes that work in (lsof cwd) or name (ps) any runner's _work (actions-runner-glaeda*/_work, glaeda-runner/_work).
  • Excluded from that list: this job's own processes (under its Runner.Worker), its ancestors, and the hook's own processes.
  • If any are left, it lets go of what it took and keeps waiting like a capacity wait, at most EXCLUSIVE_CAPACITY_WAIT_S (300 s). Then it is refused, and the job's re-run takes Blacksmith. When ps or lsof can't tell, it is refused as well.
  • A held host.lock still refuses it (now covered by a test).
  • The hook's new sha256 is pinned in glaeda-disk.

Test: test_a_signing_job_waits_out_an_offline_runners_leftover_job_and_the_host_lock.

  • An orphaned process working in a fake runner's _work gets the job refused, with the leftover's pid named, no GLAEDA_JOB_CLAIM written, and everything it took released.
  • A held host.lock gets the job refused.
  • With neither, the job is admitted with the claim.
  • test_a_signing_job_takes_the_whole_mini still passes.

Agent session: agent session

🤖 Generated with Claude Code

…ses in any runner's _work

A runner whose listener died mid-job shows offline and not busy, its units go
with its Runner.Worker, yet its steps can still run. cmux's signing picker now
offers a mini with such runners, so the exclusive class also requires that no
process of this user outside this job works in or names a runner's _work; it
waits for them like for units (300 s at most), then is refused. Pins the hook's
new sha256 in glaeda-disk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
@teamleaderleo

Copy link
Copy Markdown
Owner Author

Owned by an agent session · session

@teamleaderleo
teamleaderleo merged commit cb2186b into main Oct 29, 2026
20 of 21 checks passed
@teamleaderleo
teamleaderleo deleted the leo/exclusive-host-lock-offline branch October 29, 2026 07:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant